` where *url* is the url of the server and +Other :command:`upload` options include ``--repository=url`` or +``--repository=section`` where *url* is the url of the server and *section* the name of the section in :file:`$HOME/.pypirc`, and -:option:`--show-response` (which displays the full response text from the PyPI +``--show-response`` (which displays the full response text from the PyPI server for help in debugging upload problems). PyPI package display -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 08:13:11 2013 From: python-checkins at python.org (ezio.melotti) Date: Fri, 11 Jan 2013 08:13:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogIzE2MTU0OiBmaXgg?= =?utf-8?q?some_doctests_in_Doc/library=2E__Patch_by_Ravi_Sinha=2E?= Message-ID: <3YjFfH17zzzQpP@mail.python.org> http://hg.python.org/cpython/rev/9b3d0bdb9256 changeset: 81395:9b3d0bdb9256 branch: 2.7 parent: 81387:2bc04449fd8c user: Ezio Melotti date: Fri Jan 11 08:53:10 2013 +0200 summary: #16154: fix some doctests in Doc/library. Patch by Ravi Sinha. files: Doc/library/colorsys.rst | 8 ++++---- Doc/library/filecmp.rst | 8 ++++---- Doc/library/math.rst | 2 ++ Doc/library/string.rst | 2 +- Misc/ACKS | 1 + 5 files changed, 12 insertions(+), 9 deletions(-) diff --git a/Doc/library/colorsys.rst b/Doc/library/colorsys.rst --- a/Doc/library/colorsys.rst +++ b/Doc/library/colorsys.rst @@ -58,7 +58,7 @@ Example:: >>> import colorsys - >>> colorsys.rgb_to_hsv(.3, .4, .2) - (0.25, 0.5, 0.4) - >>> colorsys.hsv_to_rgb(0.25, 0.5, 0.4) - (0.3, 0.4, 0.2) + >>> colorsys.rgb_to_hsv(0.2, 0.4, 0.4) + (0.5, 0.5, 0.4) + >>> colorsys.hsv_to_rgb(0.5, 0.5, 0.4) + (0.2, 0.4, 0.4) diff --git a/Doc/library/filecmp.rst b/Doc/library/filecmp.rst --- a/Doc/library/filecmp.rst +++ b/Doc/library/filecmp.rst @@ -54,9 +54,9 @@ Example:: >>> import filecmp - >>> filecmp.cmp('undoc.rst', 'undoc.rst') + >>> filecmp.cmp('undoc.rst', 'undoc.rst') # doctest: +SKIP True - >>> filecmp.cmp('undoc.rst', 'index.rst') + >>> filecmp.cmp('undoc.rst', 'index.rst') # doctest: +SKIP False @@ -190,6 +190,6 @@ ... for sub_dcmp in dcmp.subdirs.values(): ... print_diff_files(sub_dcmp) ... - >>> dcmp = dircmp('dir1', 'dir2') - >>> print_diff_files(dcmp) + >>> dcmp = dircmp('dir1', 'dir2') # doctest: +SKIP + >>> print_diff_files(dcmp) # doctest: +SKIP diff --git a/Doc/library/math.rst b/Doc/library/math.rst --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -82,6 +82,8 @@ .. function:: fsum(iterable) +.. testsetup:: + >>> from math import fsum Return an accurate floating point sum of values in the iterable. Avoids loss of precision by tracking multiple intermediate partial sums:: diff --git a/Doc/library/string.rst b/Doc/library/string.rst --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -717,7 +717,7 @@ >>> Template('Give $who $100').substitute(d) Traceback (most recent call last): [...] - ValueError: Invalid placeholder in string: line 1, col 10 + ValueError: Invalid placeholder in string: line 1, col 11 >>> Template('$who likes $what').substitute(d) Traceback (most recent call last): [...] diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -923,6 +923,7 @@ Ionel Simionescu Kirill Simonov Nathan Paul Simons +Ravi Sinha Janne Sinkkonen Ng Pheng Siong George Sipe -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 08:13:12 2013 From: python-checkins at python.org (ezio.melotti) Date: Fri, 11 Jan 2013 08:13:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf-8?q?_Merge_heads=2E?= Message-ID: <3YjFfJ4nPPzRxl@mail.python.org> http://hg.python.org/cpython/rev/66e2040c1756 changeset: 81396:66e2040c1756 branch: 2.7 parent: 81391:a30c36fbefcf parent: 81395:9b3d0bdb9256 user: Ezio Melotti date: Fri Jan 11 09:06:12 2013 +0200 summary: Merge heads. files: Doc/library/colorsys.rst | 8 ++++---- Doc/library/filecmp.rst | 8 ++++---- Doc/library/math.rst | 2 ++ Doc/library/string.rst | 2 +- Misc/ACKS | 1 + 5 files changed, 12 insertions(+), 9 deletions(-) diff --git a/Doc/library/colorsys.rst b/Doc/library/colorsys.rst --- a/Doc/library/colorsys.rst +++ b/Doc/library/colorsys.rst @@ -58,7 +58,7 @@ Example:: >>> import colorsys - >>> colorsys.rgb_to_hsv(.3, .4, .2) - (0.25, 0.5, 0.4) - >>> colorsys.hsv_to_rgb(0.25, 0.5, 0.4) - (0.3, 0.4, 0.2) + >>> colorsys.rgb_to_hsv(0.2, 0.4, 0.4) + (0.5, 0.5, 0.4) + >>> colorsys.hsv_to_rgb(0.5, 0.5, 0.4) + (0.2, 0.4, 0.4) diff --git a/Doc/library/filecmp.rst b/Doc/library/filecmp.rst --- a/Doc/library/filecmp.rst +++ b/Doc/library/filecmp.rst @@ -54,9 +54,9 @@ Example:: >>> import filecmp - >>> filecmp.cmp('undoc.rst', 'undoc.rst') + >>> filecmp.cmp('undoc.rst', 'undoc.rst') # doctest: +SKIP True - >>> filecmp.cmp('undoc.rst', 'index.rst') + >>> filecmp.cmp('undoc.rst', 'index.rst') # doctest: +SKIP False @@ -190,6 +190,6 @@ ... for sub_dcmp in dcmp.subdirs.values(): ... print_diff_files(sub_dcmp) ... - >>> dcmp = dircmp('dir1', 'dir2') - >>> print_diff_files(dcmp) + >>> dcmp = dircmp('dir1', 'dir2') # doctest: +SKIP + >>> print_diff_files(dcmp) # doctest: +SKIP diff --git a/Doc/library/math.rst b/Doc/library/math.rst --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -82,6 +82,8 @@ .. function:: fsum(iterable) +.. testsetup:: + >>> from math import fsum Return an accurate floating point sum of values in the iterable. Avoids loss of precision by tracking multiple intermediate partial sums:: diff --git a/Doc/library/string.rst b/Doc/library/string.rst --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -717,7 +717,7 @@ >>> Template('Give $who $100').substitute(d) Traceback (most recent call last): [...] - ValueError: Invalid placeholder in string: line 1, col 10 + ValueError: Invalid placeholder in string: line 1, col 11 >>> Template('$who likes $what').substitute(d) Traceback (most recent call last): [...] diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -923,6 +923,7 @@ Ionel Simionescu Kirill Simonov Nathan Paul Simons +Ravi Sinha Janne Sinkkonen Ng Pheng Siong George Sipe -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 08:13:14 2013 From: python-checkins at python.org (ezio.melotti) Date: Fri, 11 Jan 2013 08:13:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogIzE2MTU0OiBmaXgg?= =?utf-8?q?some_doctests_in_Doc/library=2E__Patch_by_Ravi_Sinha=2E?= Message-ID: <3YjFfL11C5zS4w@mail.python.org> http://hg.python.org/cpython/rev/5b405df8518d changeset: 81397:5b405df8518d branch: 3.2 parent: 81392:dcb645b9de4c user: Ezio Melotti date: Fri Jan 11 09:09:07 2013 +0200 summary: #16154: fix some doctests in Doc/library. Patch by Ravi Sinha. files: Doc/library/colorsys.rst | 8 ++++---- Doc/library/filecmp.rst | 8 ++++---- Doc/library/math.rst | 2 ++ Doc/library/string.rst | 4 ++-- Misc/ACKS | 1 + 5 files changed, 13 insertions(+), 10 deletions(-) diff --git a/Doc/library/colorsys.rst b/Doc/library/colorsys.rst --- a/Doc/library/colorsys.rst +++ b/Doc/library/colorsys.rst @@ -58,7 +58,7 @@ Example:: >>> import colorsys - >>> colorsys.rgb_to_hsv(.3, .4, .2) - (0.25, 0.5, 0.4) - >>> colorsys.hsv_to_rgb(0.25, 0.5, 0.4) - (0.3, 0.4, 0.2) + >>> colorsys.rgb_to_hsv(0.2, 0.4, 0.4) + (0.5, 0.5, 0.4) + >>> colorsys.hsv_to_rgb(0.5, 0.5, 0.4) + (0.2, 0.4, 0.4) diff --git a/Doc/library/filecmp.rst b/Doc/library/filecmp.rst --- a/Doc/library/filecmp.rst +++ b/Doc/library/filecmp.rst @@ -54,9 +54,9 @@ Example:: >>> import filecmp - >>> filecmp.cmp('undoc.rst', 'undoc.rst') + >>> filecmp.cmp('undoc.rst', 'undoc.rst') # doctest: +SKIP True - >>> filecmp.cmp('undoc.rst', 'index.rst') + >>> filecmp.cmp('undoc.rst', 'index.rst') # doctest: +SKIP False @@ -191,6 +191,6 @@ ... for sub_dcmp in dcmp.subdirs.values(): ... print_diff_files(sub_dcmp) ... - >>> dcmp = dircmp('dir1', 'dir2') - >>> print_diff_files(dcmp) + >>> dcmp = dircmp('dir1', 'dir2') # doctest: +SKIP + >>> print_diff_files(dcmp) # doctest: +SKIP diff --git a/Doc/library/math.rst b/Doc/library/math.rst --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -77,6 +77,8 @@ .. function:: fsum(iterable) +.. testsetup:: + >>> from math import fsum Return an accurate floating point sum of values in the iterable. Avoids loss of precision by tracking multiple intermediate partial sums:: diff --git a/Doc/library/string.rst b/Doc/library/string.rst --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -610,7 +610,7 @@ 3232235521 >>> >>> width = 5 - >>> for num in range(5,12): + >>> for num in range(5,12): #doctest: +NORMALIZE_WHITESPACE ... for base in 'dXob': ... print('{0:{width}{base}}'.format(num, base=base, width=width), end=' ') ... print() @@ -698,7 +698,7 @@ >>> Template('Give $who $100').substitute(d) Traceback (most recent call last): [...] - ValueError: Invalid placeholder in string: line 1, col 10 + ValueError: Invalid placeholder in string: line 1, col 11 >>> Template('$who likes $what').substitute(d) Traceback (most recent call last): [...] diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -998,6 +998,7 @@ Ionel Simionescu Kirill Simonov Nathan Paul Simons +Ravi Sinha Janne Sinkkonen Ng Pheng Siong George Sipe -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 08:13:15 2013 From: python-checkins at python.org (ezio.melotti) Date: Fri, 11 Jan 2013 08:13:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_=2316154=3A_merge_with_3=2E2=2E?= Message-ID: <3YjFfM4HL4zS47@mail.python.org> http://hg.python.org/cpython/rev/3e884b3804b3 changeset: 81398:3e884b3804b3 branch: 3.3 parent: 81393:63fff55a167f parent: 81397:5b405df8518d user: Ezio Melotti date: Fri Jan 11 09:12:28 2013 +0200 summary: #16154: merge with 3.2. files: Doc/library/colorsys.rst | 8 ++++---- Doc/library/filecmp.rst | 4 ++-- Doc/library/math.rst | 2 ++ Doc/library/string.rst | 4 ++-- Misc/ACKS | 1 + 5 files changed, 11 insertions(+), 8 deletions(-) diff --git a/Doc/library/colorsys.rst b/Doc/library/colorsys.rst --- a/Doc/library/colorsys.rst +++ b/Doc/library/colorsys.rst @@ -58,7 +58,7 @@ Example:: >>> import colorsys - >>> colorsys.rgb_to_hsv(.3, .4, .2) - (0.25, 0.5, 0.4) - >>> colorsys.hsv_to_rgb(0.25, 0.5, 0.4) - (0.3, 0.4, 0.2) + >>> colorsys.rgb_to_hsv(0.2, 0.4, 0.4) + (0.5, 0.5, 0.4) + >>> colorsys.hsv_to_rgb(0.5, 0.5, 0.4) + (0.2, 0.4, 0.4) diff --git a/Doc/library/filecmp.rst b/Doc/library/filecmp.rst --- a/Doc/library/filecmp.rst +++ b/Doc/library/filecmp.rst @@ -176,6 +176,6 @@ ... for sub_dcmp in dcmp.subdirs.values(): ... print_diff_files(sub_dcmp) ... - >>> dcmp = dircmp('dir1', 'dir2') - >>> print_diff_files(dcmp) + >>> dcmp = dircmp('dir1', 'dir2') # doctest: +SKIP + >>> print_diff_files(dcmp) # doctest: +SKIP diff --git a/Doc/library/math.rst b/Doc/library/math.rst --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -77,6 +77,8 @@ .. function:: fsum(iterable) +.. testsetup:: + >>> from math import fsum Return an accurate floating point sum of values in the iterable. Avoids loss of precision by tracking multiple intermediate partial sums:: diff --git a/Doc/library/string.rst b/Doc/library/string.rst --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -610,7 +610,7 @@ 3232235521 >>> >>> width = 5 - >>> for num in range(5,12): + >>> for num in range(5,12): #doctest: +NORMALIZE_WHITESPACE ... for base in 'dXob': ... print('{0:{width}{base}}'.format(num, base=base, width=width), end=' ') ... print() @@ -698,7 +698,7 @@ >>> Template('Give $who $100').substitute(d) Traceback (most recent call last): [...] - ValueError: Invalid placeholder in string: line 1, col 10 + ValueError: Invalid placeholder in string: line 1, col 11 >>> Template('$who likes $what').substitute(d) Traceback (most recent call last): [...] diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1103,6 +1103,7 @@ Kirill Simonov Nathan Paul Simons Adam Simpkins +Ravi Sinha Janne Sinkkonen Ng Pheng Siong George Sipe -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 08:13:17 2013 From: python-checkins at python.org (ezio.melotti) Date: Fri, 11 Jan 2013 08:13:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogIzE2MTU0OiBtZXJnZSB3aXRoIDMuMy4=?= Message-ID: <3YjFfP0QjGzS5g@mail.python.org> http://hg.python.org/cpython/rev/5ec8daab477a changeset: 81399:5ec8daab477a parent: 81394:860a2f6282c4 parent: 81398:3e884b3804b3 user: Ezio Melotti date: Fri Jan 11 09:12:49 2013 +0200 summary: #16154: merge with 3.3. files: Doc/library/colorsys.rst | 8 ++++---- Doc/library/filecmp.rst | 4 ++-- Doc/library/math.rst | 2 ++ Doc/library/string.rst | 4 ++-- Misc/ACKS | 1 + 5 files changed, 11 insertions(+), 8 deletions(-) diff --git a/Doc/library/colorsys.rst b/Doc/library/colorsys.rst --- a/Doc/library/colorsys.rst +++ b/Doc/library/colorsys.rst @@ -58,7 +58,7 @@ Example:: >>> import colorsys - >>> colorsys.rgb_to_hsv(.3, .4, .2) - (0.25, 0.5, 0.4) - >>> colorsys.hsv_to_rgb(0.25, 0.5, 0.4) - (0.3, 0.4, 0.2) + >>> colorsys.rgb_to_hsv(0.2, 0.4, 0.4) + (0.5, 0.5, 0.4) + >>> colorsys.hsv_to_rgb(0.5, 0.5, 0.4) + (0.2, 0.4, 0.4) diff --git a/Doc/library/filecmp.rst b/Doc/library/filecmp.rst --- a/Doc/library/filecmp.rst +++ b/Doc/library/filecmp.rst @@ -176,6 +176,6 @@ ... for sub_dcmp in dcmp.subdirs.values(): ... print_diff_files(sub_dcmp) ... - >>> dcmp = dircmp('dir1', 'dir2') - >>> print_diff_files(dcmp) + >>> dcmp = dircmp('dir1', 'dir2') # doctest: +SKIP + >>> print_diff_files(dcmp) # doctest: +SKIP diff --git a/Doc/library/math.rst b/Doc/library/math.rst --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -77,6 +77,8 @@ .. function:: fsum(iterable) +.. testsetup:: + >>> from math import fsum Return an accurate floating point sum of values in the iterable. Avoids loss of precision by tracking multiple intermediate partial sums:: diff --git a/Doc/library/string.rst b/Doc/library/string.rst --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -610,7 +610,7 @@ 3232235521 >>> >>> width = 5 - >>> for num in range(5,12): + >>> for num in range(5,12): #doctest: +NORMALIZE_WHITESPACE ... for base in 'dXob': ... print('{0:{width}{base}}'.format(num, base=base, width=width), end=' ') ... print() @@ -698,7 +698,7 @@ >>> Template('Give $who $100').substitute(d) Traceback (most recent call last): [...] - ValueError: Invalid placeholder in string: line 1, col 10 + ValueError: Invalid placeholder in string: line 1, col 11 >>> Template('$who likes $what').substitute(d) Traceback (most recent call last): [...] diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1113,6 +1113,7 @@ Kirill Simonov Nathan Paul Simons Adam Simpkins +Ravi Sinha Janne Sinkkonen Ng Pheng Siong George Sipe -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 08:30:50 2013 From: python-checkins at python.org (chris.jerdonek) Date: Fri, 11 Jan 2013 08:30:50 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_Improve_and_make_more_con?= =?utf-8?q?cise_the_=22Building_the_documentation=22_section=2E?= Message-ID: <3YjG2f74RhzS69@mail.python.org> http://hg.python.org/devguide/rev/157066a204ab changeset: 587:157066a204ab user: Chris Jerdonek date: Thu Jan 10 23:29:47 2013 -0800 summary: Improve and make more concise the "Building the documentation" section. A couple of the changes include adding hyperlinks for each toolset component and moving the viewing instructions outside of the "Using make" section. files: documenting.rst | 34 +++++++++++++++++++--------------- 1 files changed, 19 insertions(+), 15 deletions(-) diff --git a/documenting.rst b/documenting.rst --- a/documenting.rst +++ b/documenting.rst @@ -18,8 +18,6 @@ :ref:`CPython Mercurial repository `. .. _reStructuredText: http://docutils.sf.net/rst.html -.. _docutils: http://docutils.sf.net/ -.. _Sphinx: http://sphinx-doc.org/ .. note:: @@ -1629,29 +1627,30 @@ Building the documentation ========================== -You need to have Python 2.4 or higher installed; the toolset used to build the -docs is written in Python. It is called *Sphinx*, it is not included in this -tree, but maintained separately. Also needed are the docutils, supplying the -base markup that Sphinx uses, Jinja, a templating engine, and optionally -Pygments, a code highlighter. +You need to have Python 2.4 or higher installed. The toolset used to build +the docs is written in Python and is called Sphinx_. Sphinx is maintained +separately and is not included in this tree. Also needed are docutils_, +supplying the base markup that Sphinx uses; Jinja_, a templating engine; and +optionally Pygments_, a code highlighter. + +To build the documentation, follow the instructions from one of the sections +below. You can view the documentation after building the HTML by pointing +a browser at the file :file:`Doc/build/html/index.html`. Using make ---------- -Luckily, a Makefile has been prepared so that on Unix, provided you have -installed Python and Subversion, you can just go to your :ref:`clone of the -CPython Mercurial repository ` and run :: +On Unix, if you have Subversion installed, run the following from the root of +your :ref:`repository clone `:: cd Doc make html -to check out the necessary toolset in the :file:`Doc/tools/` subdirectory and -build the HTML output files. To view the generated HTML, point your favorite -browser at the top-level index :file:`Doc/build/html/index.html` after running -:command:`make`. +This checks out the needed toolset in the :file:`Doc/tools/` directory and +builds the output as HTML. -Available make targets are: +Available :command:`make` targets are: * "html", which builds standalone HTML files for offline viewing. @@ -1721,3 +1720,8 @@ where `` is one of html, text, latex, or htmlhelp (for explanations see the make targets above). + +.. _docutils: http://docutils.sourceforge.net/ +.. _Jinja: http://jinja.pocoo.org/ +.. _Pygments: http://pygments.org/ +.. _Sphinx: http://sphinx-doc.org/ -- Repository URL: http://hg.python.org/devguide From root at python.org Fri Jan 11 08:35:08 2013 From: root at python.org (Cron Daemon) Date: Fri, 11 Jan 2013 08:35:08 +0100 Subject: [Python-checkins] Cron /home/docs/build-devguide Message-ID: /home/docs/devguide/documenting.rst:766: WARNING: term not in glossary: bytecode From python-checkins at python.org Fri Jan 11 09:16:57 2013 From: python-checkins at python.org (chris.jerdonek) Date: Fri, 11 Jan 2013 09:16:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_Issue_=2316814=3A_add_=22?= =?utf-8?q?make_-C_Doc_html=22_short-cut_to_documentation_instructions=2E?= Message-ID: <3YjH3s2HhvzS3j@mail.python.org> http://hg.python.org/devguide/rev/5f24a77e7beb changeset: 588:5f24a77e7beb user: Chris Jerdonek date: Fri Jan 11 00:10:09 2013 -0800 summary: Issue #16814: add "make -C Doc html" short-cut to documentation instructions. files: documenting.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/documenting.rst b/documenting.rst --- a/documenting.rst +++ b/documenting.rst @@ -1647,8 +1647,8 @@ cd Doc make html -This checks out the needed toolset in the :file:`Doc/tools/` directory and -builds the output as HTML. +or alternatively ``make -C Doc html``. This checks out the needed toolset +in the :file:`Doc/tools/` directory and builds the output as HTML. Available :command:`make` targets are: -- Repository URL: http://hg.python.org/devguide From root at python.org Fri Jan 11 09:20:07 2013 From: root at python.org (Cron Daemon) Date: Fri, 11 Jan 2013 09:20:07 +0100 Subject: [Python-checkins] Cron /home/docs/build-devguide Message-ID: /home/docs/devguide/documenting.rst:766: WARNING: term not in glossary: bytecode From python-checkins at python.org Fri Jan 11 09:20:52 2013 From: python-checkins at python.org (georg.brandl) Date: Fri, 11 Jan 2013 09:20:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_Update_Py3k_doc_URL=2E?= Message-ID: <3YjH8N4f1xzS34@mail.python.org> http://hg.python.org/devguide/rev/5c8851aaf386 changeset: 589:5c8851aaf386 user: Georg Brandl date: Fri Jan 11 09:21:52 2013 +0100 summary: Update Py3k doc URL. files: conf.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/conf.py b/conf.py --- a/conf.py +++ b/conf.py @@ -26,7 +26,7 @@ # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = ['sphinx.ext.intersphinx', 'sphinx.ext.todo'] -intersphinx_mapping = {'python': ('http://docs.python.org/py3k', None)} +intersphinx_mapping = {'python': ('http://docs.python.org/3', None)} todo_include_todos = True # Add any paths that contain templates here, relative to this directory. -- Repository URL: http://hg.python.org/devguide From root at python.org Fri Jan 11 09:25:12 2013 From: root at python.org (Cron Daemon) Date: Fri, 11 Jan 2013 09:25:12 +0100 Subject: [Python-checkins] Cron /home/docs/build-devguide Message-ID: /home/docs/devguide/documenting.rst:766: WARNING: term not in glossary: bytecode From python-checkins at python.org Fri Jan 11 10:50:13 2013 From: python-checkins at python.org (sandro.tosi) Date: Fri, 11 Jan 2013 10:50:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_grammatical_fi?= =?utf-8?q?x=3B_thanks_to_Alex_Rudnick_from_docs=40?= Message-ID: <3YjK7T2YKtzS5v@mail.python.org> http://hg.python.org/cpython/rev/7555399134af changeset: 81400:7555399134af branch: 2.7 parent: 81396:66e2040c1756 user: Sandro Tosi date: Fri Jan 11 10:48:34 2013 +0100 summary: grammatical fix; thanks to Alex Rudnick from docs@ files: Doc/library/argparse.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1116,7 +1116,7 @@ metavar ^^^^^^^ -When :class:`ArgumentParser` generates help messages, it need some way to refer +When :class:`ArgumentParser` generates help messages, it needs some way to refer to each expected argument. By default, ArgumentParser objects use the dest_ value as the "name" of each object. By default, for positional argument actions, the dest_ value is used directly, and for optional argument actions, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 10:50:14 2013 From: python-checkins at python.org (sandro.tosi) Date: Fri, 11 Jan 2013 10:50:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E2=29=3A_grammatical_fi?= =?utf-8?q?x=3B_thanks_to_Alex_Rudnick_from_docs=40?= Message-ID: <3YjK7V58Z3zS56@mail.python.org> http://hg.python.org/cpython/rev/64d1cbc0f813 changeset: 81401:64d1cbc0f813 branch: 3.2 parent: 81397:5b405df8518d user: Sandro Tosi date: Fri Jan 11 10:49:00 2013 +0100 summary: grammatical fix; thanks to Alex Rudnick from docs@ files: Doc/library/argparse.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1117,7 +1117,7 @@ metavar ^^^^^^^ -When :class:`ArgumentParser` generates help messages, it need some way to refer +When :class:`ArgumentParser` generates help messages, it needs some way to refer to each expected argument. By default, ArgumentParser objects use the dest_ value as the "name" of each object. By default, for positional argument actions, the dest_ value is used directly, and for optional argument actions, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 10:50:16 2013 From: python-checkins at python.org (sandro.tosi) Date: Fri, 11 Jan 2013 10:50:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_merge_with_3=2E2?= Message-ID: <3YjK7X0dcZzS4y@mail.python.org> http://hg.python.org/cpython/rev/ad806dac3ff3 changeset: 81402:ad806dac3ff3 branch: 3.3 parent: 81398:3e884b3804b3 parent: 81401:64d1cbc0f813 user: Sandro Tosi date: Fri Jan 11 10:49:28 2013 +0100 summary: merge with 3.2 files: Doc/library/argparse.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1136,7 +1136,7 @@ metavar ^^^^^^^ -When :class:`ArgumentParser` generates help messages, it need some way to refer +When :class:`ArgumentParser` generates help messages, it needs some way to refer to each expected argument. By default, ArgumentParser objects use the dest_ value as the "name" of each object. By default, for positional argument actions, the dest_ value is used directly, and for optional argument actions, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 10:50:17 2013 From: python-checkins at python.org (sandro.tosi) Date: Fri, 11 Jan 2013 10:50:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_merge_with_3=2E3?= Message-ID: <3YjK7Y3BvmzS5r@mail.python.org> http://hg.python.org/cpython/rev/6eae84170536 changeset: 81403:6eae84170536 parent: 81399:5ec8daab477a parent: 81402:ad806dac3ff3 user: Sandro Tosi date: Fri Jan 11 10:49:43 2013 +0100 summary: merge with 3.3 files: Doc/library/argparse.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1136,7 +1136,7 @@ metavar ^^^^^^^ -When :class:`ArgumentParser` generates help messages, it need some way to refer +When :class:`ArgumentParser` generates help messages, it needs some way to refer to each expected argument. By default, ArgumentParser objects use the dest_ value as the "name" of each object. By default, for positional argument actions, the dest_ value is used directly, and for optional argument actions, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 11:40:07 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 11 Jan 2013 11:40:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE1NTM5?= =?utf-8?q?=3A_Fix_a_number_of_bugs_in_Tools/scripts/pindent=2Epy=2E?= Message-ID: <3YjLF33x8jzP94@mail.python.org> http://hg.python.org/cpython/rev/f783db4a58ba changeset: 81404:f783db4a58ba branch: 2.7 parent: 81396:66e2040c1756 user: Serhiy Storchaka date: Fri Jan 11 11:59:59 2013 +0200 summary: Issue #15539: Fix a number of bugs in Tools/scripts/pindent.py. Now pindent.py works with a "with" statement. pindent.py no longer produces improper indentation. pindent.py now works with continued lines broken after "class" or "def" keywords and with continuations at the start of line. Added regression tests for pindent.py. Modernized pindent.py. files: Lib/test/test_tools.py | 334 ++++++++++++++++++++++++++- Misc/NEWS | 7 + Tools/scripts/pindent.py | 166 ++++-------- 3 files changed, 399 insertions(+), 108 deletions(-) diff --git a/Lib/test/test_tools.py b/Lib/test/test_tools.py --- a/Lib/test/test_tools.py +++ b/Lib/test/test_tools.py @@ -5,10 +5,15 @@ """ import os +import sys import unittest +import shutil +import subprocess import sysconfig +import tempfile +import textwrap from test import test_support -from test.script_helper import assert_python_ok +from test.script_helper import assert_python_ok, temp_dir if not sysconfig.is_python_build(): # XXX some installers do contain the tools, should we detect that @@ -17,10 +22,11 @@ basepath = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'Tools') +scriptsdir = os.path.join(basepath, 'scripts') class ReindentTests(unittest.TestCase): - script = os.path.join(basepath, 'scripts', 'reindent.py') + script = os.path.join(scriptsdir, 'reindent.py') def test_noargs(self): assert_python_ok(self.script) @@ -31,8 +37,330 @@ self.assertGreater(err, b'') +class PindentTests(unittest.TestCase): + script = os.path.join(scriptsdir, 'pindent.py') + + def assertFileEqual(self, fn1, fn2): + with open(fn1) as f1, open(fn2) as f2: + self.assertEqual(f1.readlines(), f2.readlines()) + + def pindent(self, source, *args): + proc = subprocess.Popen( + (sys.executable, self.script) + args, + stdin=subprocess.PIPE, stdout=subprocess.PIPE, + universal_newlines=True) + out, err = proc.communicate(source) + self.assertIsNone(err) + return out + + def lstriplines(self, data): + return '\n'.join(line.lstrip() for line in data.splitlines()) + '\n' + + def test_selftest(self): + with temp_dir() as directory: + data_path = os.path.join(directory, '_test.py') + with open(self.script) as f: + closed = f.read() + with open(data_path, 'w') as f: + f.write(closed) + + rc, out, err = assert_python_ok(self.script, '-d', data_path) + self.assertEqual(out, b'') + self.assertEqual(err, b'') + backup = data_path + '~' + self.assertTrue(os.path.exists(backup)) + with open(backup) as f: + self.assertEqual(f.read(), closed) + with open(data_path) as f: + clean = f.read() + compile(clean, '_test.py', 'exec') + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + + rc, out, err = assert_python_ok(self.script, '-c', data_path) + self.assertEqual(out, b'') + self.assertEqual(err, b'') + with open(backup) as f: + self.assertEqual(f.read(), clean) + with open(data_path) as f: + self.assertEqual(f.read(), closed) + + broken = self.lstriplines(closed) + with open(data_path, 'w') as f: + f.write(broken) + rc, out, err = assert_python_ok(self.script, '-r', data_path) + self.assertEqual(out, b'') + self.assertEqual(err, b'') + with open(backup) as f: + self.assertEqual(f.read(), broken) + with open(data_path) as f: + indented = f.read() + compile(indented, '_test.py', 'exec') + self.assertEqual(self.pindent(broken, '-r'), indented) + + def pindent_test(self, clean, closed): + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + broken = self.lstriplines(closed) + self.assertEqual(self.pindent(broken, '-r', '-e', '-s', '4'), closed) + + def test_statements(self): + clean = textwrap.dedent("""\ + if a: + pass + + if a: + pass + else: + pass + + if a: + pass + elif: + pass + else: + pass + + while a: + break + + while a: + break + else: + pass + + for i in a: + break + + for i in a: + break + else: + pass + + try: + pass + finally: + pass + + try: + pass + except TypeError: + pass + except ValueError: + pass + else: + pass + + try: + pass + except TypeError: + pass + except ValueError: + pass + finally: + pass + + with a: + pass + + class A: + pass + + def f(): + pass + """) + + closed = textwrap.dedent("""\ + if a: + pass + # end if + + if a: + pass + else: + pass + # end if + + if a: + pass + elif: + pass + else: + pass + # end if + + while a: + break + # end while + + while a: + break + else: + pass + # end while + + for i in a: + break + # end for + + for i in a: + break + else: + pass + # end for + + try: + pass + finally: + pass + # end try + + try: + pass + except TypeError: + pass + except ValueError: + pass + else: + pass + # end try + + try: + pass + except TypeError: + pass + except ValueError: + pass + finally: + pass + # end try + + with a: + pass + # end with + + class A: + pass + # end class A + + def f(): + pass + # end def f + """) + self.pindent_test(clean, closed) + + def test_multilevel(self): + clean = textwrap.dedent("""\ + def foobar(a, b): + if a == b: + a = a+1 + elif a < b: + b = b-1 + if b > a: a = a-1 + else: + print 'oops!' + """) + closed = textwrap.dedent("""\ + def foobar(a, b): + if a == b: + a = a+1 + elif a < b: + b = b-1 + if b > a: a = a-1 + # end if + else: + print 'oops!' + # end if + # end def foobar + """) + self.pindent_test(clean, closed) + + def test_preserve_indents(self): + clean = textwrap.dedent("""\ + if a: + if b: + pass + """) + closed = textwrap.dedent("""\ + if a: + if b: + pass + # end if + # end if + """) + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + broken = self.lstriplines(closed) + self.assertEqual(self.pindent(broken, '-r', '-e', '-s', '9'), closed) + clean = textwrap.dedent("""\ + if a: + \tif b: + \t\tpass + """) + closed = textwrap.dedent("""\ + if a: + \tif b: + \t\tpass + \t# end if + # end if + """) + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + broken = self.lstriplines(closed) + self.assertEqual(self.pindent(broken, '-r'), closed) + + def test_escaped_newline(self): + clean = textwrap.dedent("""\ + class\\ + \\ + A: + def\ + \\ + f: + pass + """) + closed = textwrap.dedent("""\ + class\\ + \\ + A: + def\ + \\ + f: + pass + # end def f + # end class A + """) + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + + def test_empty_line(self): + clean = textwrap.dedent("""\ + if a: + + pass + """) + closed = textwrap.dedent("""\ + if a: + + pass + # end if + """) + self.pindent_test(clean, closed) + + def test_oneline(self): + clean = textwrap.dedent("""\ + if a: pass + """) + closed = textwrap.dedent("""\ + if a: pass + # end if + """) + self.pindent_test(clean, closed) + + def test_main(): - test_support.run_unittest(ReindentTests) + test_support.run_unittest(*[obj for obj in globals().values() + if isinstance(obj, type)]) if __name__ == '__main__': diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -634,6 +634,8 @@ Tests ----- +- Issue #15539: Added regression tests for Tools/scripts/pindent.py. + - Issue #15324: Fix regrtest parsing of --fromfile and --randomize options. - Issue #16618: Add more regression tests for glob. @@ -709,6 +711,11 @@ Tools/Demos ----------- +- Issue #15539: Fix a number of bugs in Tools/scripts/pindent.py. Now + pindent.py works with a "with" statement. pindent.py no longer produces + improper indentation. pindent.py now works with continued lines broken after + "class" or "def" keywords and with continuations at the start of line. + - Issue #13301: use ast.literal_eval() instead of eval() in Tools/i18n/msgfmt.py Patch by Serhiy Storchaka. diff --git a/Tools/scripts/pindent.py b/Tools/scripts/pindent.py --- a/Tools/scripts/pindent.py +++ b/Tools/scripts/pindent.py @@ -79,8 +79,9 @@ # Defaults STEPSIZE = 8 TABSIZE = 8 -EXPANDTABS = 0 +EXPANDTABS = False +import io import re import sys @@ -89,7 +90,8 @@ next['while'] = next['for'] = 'else', 'end' next['try'] = 'except', 'finally' next['except'] = 'except', 'else', 'finally', 'end' -next['else'] = next['finally'] = next['def'] = next['class'] = 'end' +next['else'] = next['finally'] = next['with'] = \ + next['def'] = next['class'] = 'end' next['end'] = () start = 'if', 'while', 'for', 'try', 'with', 'def', 'class' @@ -105,11 +107,11 @@ self.expandtabs = expandtabs self._write = fpo.write self.kwprog = re.compile( - r'^\s*(?P[a-z]+)' - r'(\s+(?P[a-zA-Z_]\w*))?' + r'^(?:\s|\\\n)*(?P[a-z]+)' + r'((?:\s|\\\n)+(?P[a-zA-Z_]\w*))?' r'[^\w]') self.endprog = re.compile( - r'^\s*#?\s*end\s+(?P[a-z]+)' + r'^(?:\s|\\\n)*#?\s*end\s+(?P[a-z]+)' r'(\s+(?P[a-zA-Z_]\w*))?' r'[^\w]') self.wsprog = re.compile(r'^[ \t]*') @@ -125,7 +127,7 @@ def readline(self): line = self.fpi.readline() - if line: self.lineno = self.lineno + 1 + if line: self.lineno += 1 # end if return line # end def readline @@ -143,27 +145,24 @@ line2 = self.readline() if not line2: break # end if - line = line + line2 + line += line2 # end while return line # end def getline - def putline(self, line, indent = None): - if indent is None: - self.write(line) - return + def putline(self, line, indent): + tabs, spaces = divmod(indent*self.indentsize, self.tabsize) + i = self.wsprog.match(line).end() + line = line[i:] + if line[:1] not in ('\n', '\r', ''): + line = '\t'*tabs + ' '*spaces + line # end if - tabs, spaces = divmod(indent*self.indentsize, self.tabsize) - i = 0 - m = self.wsprog.match(line) - if m: i = m.end() - # end if - self.write('\t'*tabs + ' '*spaces + line[i:]) + self.write(line) # end def putline def reformat(self): stack = [] - while 1: + while True: line = self.getline() if not line: break # EOF # end if @@ -173,10 +172,9 @@ kw2 = m.group('kw') if not stack: self.error('unexpected end') - elif stack[-1][0] != kw2: + elif stack.pop()[0] != kw2: self.error('unmatched end') # end if - del stack[-1:] self.putline(line, len(stack)) continue # end if @@ -208,23 +206,23 @@ def delete(self): begin_counter = 0 end_counter = 0 - while 1: + while True: line = self.getline() if not line: break # EOF # end if m = self.endprog.match(line) if m: - end_counter = end_counter + 1 + end_counter += 1 continue # end if m = self.kwprog.match(line) if m: kw = m.group('kw') if kw in start: - begin_counter = begin_counter + 1 + begin_counter += 1 # end if # end if - self.putline(line) + self.write(line) # end while if begin_counter - end_counter < 0: sys.stderr.write('Warning: input contained more end tags than expected\n') @@ -234,17 +232,12 @@ # end def delete def complete(self): - self.indentsize = 1 stack = [] todo = [] - thisid = '' - current, firstkw, lastkw, topid = 0, '', '', '' - while 1: + currentws = thisid = firstkw = lastkw = topid = '' + while True: line = self.getline() - i = 0 - m = self.wsprog.match(line) - if m: i = m.end() - # end if + i = self.wsprog.match(line).end() m = self.endprog.match(line) if m: thiskw = 'end' @@ -269,7 +262,9 @@ thiskw = '' # end if # end if - indent = len(line[:i].expandtabs(self.tabsize)) + indentws = line[:i] + indent = len(indentws.expandtabs(self.tabsize)) + current = len(currentws.expandtabs(self.tabsize)) while indent < current: if firstkw: if topid: @@ -278,11 +273,11 @@ else: s = '# end %s\n' % firstkw # end if - self.putline(s, current) + self.write(currentws + s) firstkw = lastkw = '' # end if - current, firstkw, lastkw, topid = stack[-1] - del stack[-1] + currentws, firstkw, lastkw, topid = stack.pop() + current = len(currentws.expandtabs(self.tabsize)) # end while if indent == current and firstkw: if thiskw == 'end': @@ -297,18 +292,18 @@ else: s = '# end %s\n' % firstkw # end if - self.putline(s, current) + self.write(currentws + s) firstkw = lastkw = topid = '' # end if # end if if indent > current: - stack.append((current, firstkw, lastkw, topid)) + stack.append((currentws, firstkw, lastkw, topid)) if thiskw and thiskw not in start: # error thiskw = '' # end if - current, firstkw, lastkw, topid = \ - indent, thiskw, thiskw, thisid + currentws, firstkw, lastkw, topid = \ + indentws, thiskw, thiskw, thisid # end if if thiskw: if thiskw in start: @@ -326,7 +321,6 @@ self.write(line) # end while # end def complete - # end class PythonIndenter # Simplified user interface @@ -352,76 +346,34 @@ pi.reformat() # end def reformat_filter -class StringReader: - def __init__(self, buf): - self.buf = buf - self.pos = 0 - self.len = len(self.buf) - # end def __init__ - def read(self, n = 0): - if n <= 0: - n = self.len - self.pos - else: - n = min(n, self.len - self.pos) - # end if - r = self.buf[self.pos : self.pos + n] - self.pos = self.pos + n - return r - # end def read - def readline(self): - i = self.buf.find('\n', self.pos) - return self.read(i + 1 - self.pos) - # end def readline - def readlines(self): - lines = [] - line = self.readline() - while line: - lines.append(line) - line = self.readline() - # end while - return lines - # end def readlines - # seek/tell etc. are left as an exercise for the reader -# end class StringReader - -class StringWriter: - def __init__(self): - self.buf = '' - # end def __init__ - def write(self, s): - self.buf = self.buf + s - # end def write - def getvalue(self): - return self.buf - # end def getvalue -# end class StringWriter - def complete_string(source, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - input = StringReader(source) - output = StringWriter() + input = io.BytesIO(source) + output = io.BytesIO() pi = PythonIndenter(input, output, stepsize, tabsize, expandtabs) pi.complete() return output.getvalue() # end def complete_string def delete_string(source, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - input = StringReader(source) - output = StringWriter() + input = io.BytesIO(source) + output = io.BytesIO() pi = PythonIndenter(input, output, stepsize, tabsize, expandtabs) pi.delete() return output.getvalue() # end def delete_string def reformat_string(source, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - input = StringReader(source) - output = StringWriter() + input = io.BytesIO(source) + output = io.BytesIO() pi = PythonIndenter(input, output, stepsize, tabsize, expandtabs) pi.reformat() return output.getvalue() # end def reformat_string def complete_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - source = open(filename, 'r').read() + with open(filename, 'r') as f: + source = f.read() + # end with result = complete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if @@ -429,14 +381,16 @@ try: os.rename(filename, filename + '~') except os.error: pass # end try - f = open(filename, 'w') - f.write(result) - f.close() + with open(filename, 'w') as f: + f.write(result) + # end with return 1 # end def complete_file def delete_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - source = open(filename, 'r').read() + with open(filename, 'r') as f: + source = f.read() + # end with result = delete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if @@ -444,14 +398,16 @@ try: os.rename(filename, filename + '~') except os.error: pass # end try - f = open(filename, 'w') - f.write(result) - f.close() + with open(filename, 'w') as f: + f.write(result) + # end with return 1 # end def delete_file def reformat_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - source = open(filename, 'r').read() + with open(filename, 'r') as f: + source = f.read() + # end with result = reformat_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if @@ -459,9 +415,9 @@ try: os.rename(filename, filename + '~') except os.error: pass # end try - f = open(filename, 'w') - f.write(result) - f.close() + with open(filename, 'w') as f: + f.write(result) + # end with return 1 # end def reformat_file @@ -474,7 +430,7 @@ -r : reformat a completed program (use #end directives) -s stepsize: indentation step (default %(STEPSIZE)d) -t tabsize : the worth in spaces of a tab (default %(TABSIZE)d) --e : expand TABs into spaces (defailt OFF) +-e : expand TABs into spaces (default OFF) [file] ... : files are changed in place, with backups in file~ If no files are specified or a single - is given, the program acts as a filter (reads stdin, writes stdout). @@ -517,7 +473,7 @@ elif o == '-t': tabsize = int(a) elif o == '-e': - expandtabs = 1 + expandtabs = True # end if # end for if not action: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 11:40:09 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 11 Jan 2013 11:40:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE1NTM5?= =?utf-8?q?=3A_Fix_a_number_of_bugs_in_Tools/scripts/pindent=2Epy=2E?= Message-ID: <3YjLF52FWdzR2Y@mail.python.org> http://hg.python.org/cpython/rev/9df6b707aef9 changeset: 81405:9df6b707aef9 branch: 3.2 parent: 81397:5b405df8518d user: Serhiy Storchaka date: Fri Jan 11 12:04:23 2013 +0200 summary: Issue #15539: Fix a number of bugs in Tools/scripts/pindent.py. Now pindent.py works with a "with" statement. pindent.py no longer produces improper indentation. pindent.py now works with continued lines broken after "class" or "def" keywords and with continuations at the start of line. Added regression tests for pindent.py. Modernized pindent.py. files: Lib/test/test_tools.py | 326 ++++++++++++++++++++++++++- Misc/NEWS | 7 + Tools/scripts/pindent.py | 166 +++++-------- 3 files changed, 393 insertions(+), 106 deletions(-) diff --git a/Lib/test/test_tools.py b/Lib/test/test_tools.py --- a/Lib/test/test_tools.py +++ b/Lib/test/test_tools.py @@ -8,10 +8,13 @@ import sys import imp import unittest +import shutil +import subprocess import sysconfig import tempfile +import textwrap from test import support -from test.script_helper import assert_python_ok +from test.script_helper import assert_python_ok, temp_dir if not sysconfig.is_python_build(): # XXX some installers do contain the tools, should we detect that @@ -35,6 +38,327 @@ self.assertGreater(err, b'') +class PindentTests(unittest.TestCase): + script = os.path.join(scriptsdir, 'pindent.py') + + def assertFileEqual(self, fn1, fn2): + with open(fn1) as f1, open(fn2) as f2: + self.assertEqual(f1.readlines(), f2.readlines()) + + def pindent(self, source, *args): + with subprocess.Popen( + (sys.executable, self.script) + args, + stdin=subprocess.PIPE, stdout=subprocess.PIPE, + universal_newlines=True) as proc: + out, err = proc.communicate(source.encode()) + self.assertIsNone(err) + return out + + def lstriplines(self, data): + return '\n'.join(line.lstrip() for line in data.splitlines()) + '\n' + + def test_selftest(self): + with temp_dir() as directory: + data_path = os.path.join(directory, '_test.py') + with open(self.script) as f: + closed = f.read() + with open(data_path, 'w') as f: + f.write(closed) + + rc, out, err = assert_python_ok(self.script, '-d', data_path) + self.assertEqual(out, b'') + self.assertEqual(err, b'') + backup = data_path + '~' + self.assertTrue(os.path.exists(backup)) + with open(backup) as f: + self.assertEqual(f.read(), closed) + with open(data_path) as f: + clean = f.read() + compile(clean, '_test.py', 'exec') + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + + rc, out, err = assert_python_ok(self.script, '-c', data_path) + self.assertEqual(out, b'') + self.assertEqual(err, b'') + with open(backup) as f: + self.assertEqual(f.read(), clean) + with open(data_path) as f: + self.assertEqual(f.read(), closed) + + broken = self.lstriplines(closed) + with open(data_path, 'w') as f: + f.write(broken) + rc, out, err = assert_python_ok(self.script, '-r', data_path) + self.assertEqual(out, b'') + self.assertEqual(err, b'') + with open(backup) as f: + self.assertEqual(f.read(), broken) + with open(data_path) as f: + indented = f.read() + compile(indented, '_test.py', 'exec') + self.assertEqual(self.pindent(broken, '-r'), indented) + + def pindent_test(self, clean, closed): + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + broken = self.lstriplines(closed) + self.assertEqual(self.pindent(broken, '-r', '-e', '-s', '4'), closed) + + def test_statements(self): + clean = textwrap.dedent("""\ + if a: + pass + + if a: + pass + else: + pass + + if a: + pass + elif: + pass + else: + pass + + while a: + break + + while a: + break + else: + pass + + for i in a: + break + + for i in a: + break + else: + pass + + try: + pass + finally: + pass + + try: + pass + except TypeError: + pass + except ValueError: + pass + else: + pass + + try: + pass + except TypeError: + pass + except ValueError: + pass + finally: + pass + + with a: + pass + + class A: + pass + + def f(): + pass + """) + + closed = textwrap.dedent("""\ + if a: + pass + # end if + + if a: + pass + else: + pass + # end if + + if a: + pass + elif: + pass + else: + pass + # end if + + while a: + break + # end while + + while a: + break + else: + pass + # end while + + for i in a: + break + # end for + + for i in a: + break + else: + pass + # end for + + try: + pass + finally: + pass + # end try + + try: + pass + except TypeError: + pass + except ValueError: + pass + else: + pass + # end try + + try: + pass + except TypeError: + pass + except ValueError: + pass + finally: + pass + # end try + + with a: + pass + # end with + + class A: + pass + # end class A + + def f(): + pass + # end def f + """) + self.pindent_test(clean, closed) + + def test_multilevel(self): + clean = textwrap.dedent("""\ + def foobar(a, b): + if a == b: + a = a+1 + elif a < b: + b = b-1 + if b > a: a = a-1 + else: + print 'oops!' + """) + closed = textwrap.dedent("""\ + def foobar(a, b): + if a == b: + a = a+1 + elif a < b: + b = b-1 + if b > a: a = a-1 + # end if + else: + print 'oops!' + # end if + # end def foobar + """) + self.pindent_test(clean, closed) + + def test_preserve_indents(self): + clean = textwrap.dedent("""\ + if a: + if b: + pass + """) + closed = textwrap.dedent("""\ + if a: + if b: + pass + # end if + # end if + """) + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + broken = self.lstriplines(closed) + self.assertEqual(self.pindent(broken, '-r', '-e', '-s', '9'), closed) + clean = textwrap.dedent("""\ + if a: + \tif b: + \t\tpass + """) + closed = textwrap.dedent("""\ + if a: + \tif b: + \t\tpass + \t# end if + # end if + """) + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + broken = self.lstriplines(closed) + self.assertEqual(self.pindent(broken, '-r'), closed) + + def test_escaped_newline(self): + clean = textwrap.dedent("""\ + class\\ + \\ + A: + def\ + \\ + f: + pass + """) + closed = textwrap.dedent("""\ + class\\ + \\ + A: + def\ + \\ + f: + pass + # end def f + # end class A + """) + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + + def test_empty_line(self): + clean = textwrap.dedent("""\ + if a: + + pass + """) + closed = textwrap.dedent("""\ + if a: + + pass + # end if + """) + self.pindent_test(clean, closed) + + def test_oneline(self): + clean = textwrap.dedent("""\ + if a: pass + """) + closed = textwrap.dedent("""\ + if a: pass + # end if + """) + self.pindent_test(clean, closed) + + class TestSundryScripts(unittest.TestCase): # At least make sure the rest don't have syntax errors. When tests are # added for a script it should be added to the whitelist below. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -770,6 +770,8 @@ Tests ----- +- Issue #15539: Added regression tests for Tools/scripts/pindent.py. + - Issue #15324: Fix regrtest parsing of --fromfile, --match, and --randomize options. @@ -951,6 +953,11 @@ Tools/Demos ----------- +- Issue #15539: Fix a number of bugs in Tools/scripts/pindent.py. Now + pindent.py works with a "with" statement. pindent.py no longer produces + improper indentation. pindent.py now works with continued lines broken after + "class" or "def" keywords and with continuations at the start of line. + - Issue #15378: Fix Tools/unicode/comparecodecs.py. Patch by Serhiy Storchaka. - Issue #14695: Fix missing support for starred assignments in diff --git a/Tools/scripts/pindent.py b/Tools/scripts/pindent.py --- a/Tools/scripts/pindent.py +++ b/Tools/scripts/pindent.py @@ -79,8 +79,9 @@ # Defaults STEPSIZE = 8 TABSIZE = 8 -EXPANDTABS = 0 +EXPANDTABS = False +import io import re import sys @@ -89,7 +90,8 @@ next['while'] = next['for'] = 'else', 'end' next['try'] = 'except', 'finally' next['except'] = 'except', 'else', 'finally', 'end' -next['else'] = next['finally'] = next['def'] = next['class'] = 'end' +next['else'] = next['finally'] = next['with'] = \ + next['def'] = next['class'] = 'end' next['end'] = () start = 'if', 'while', 'for', 'try', 'with', 'def', 'class' @@ -105,11 +107,11 @@ self.expandtabs = expandtabs self._write = fpo.write self.kwprog = re.compile( - r'^\s*(?P[a-z]+)' - r'(\s+(?P[a-zA-Z_]\w*))?' + r'^(?:\s|\\\n)*(?P[a-z]+)' + r'((?:\s|\\\n)+(?P[a-zA-Z_]\w*))?' r'[^\w]') self.endprog = re.compile( - r'^\s*#?\s*end\s+(?P[a-z]+)' + r'^(?:\s|\\\n)*#?\s*end\s+(?P[a-z]+)' r'(\s+(?P[a-zA-Z_]\w*))?' r'[^\w]') self.wsprog = re.compile(r'^[ \t]*') @@ -125,7 +127,7 @@ def readline(self): line = self.fpi.readline() - if line: self.lineno = self.lineno + 1 + if line: self.lineno += 1 # end if return line # end def readline @@ -143,27 +145,24 @@ line2 = self.readline() if not line2: break # end if - line = line + line2 + line += line2 # end while return line # end def getline - def putline(self, line, indent = None): - if indent is None: - self.write(line) - return + def putline(self, line, indent): + tabs, spaces = divmod(indent*self.indentsize, self.tabsize) + i = self.wsprog.match(line).end() + line = line[i:] + if line[:1] not in ('\n', '\r', ''): + line = '\t'*tabs + ' '*spaces + line # end if - tabs, spaces = divmod(indent*self.indentsize, self.tabsize) - i = 0 - m = self.wsprog.match(line) - if m: i = m.end() - # end if - self.write('\t'*tabs + ' '*spaces + line[i:]) + self.write(line) # end def putline def reformat(self): stack = [] - while 1: + while True: line = self.getline() if not line: break # EOF # end if @@ -173,10 +172,9 @@ kw2 = m.group('kw') if not stack: self.error('unexpected end') - elif stack[-1][0] != kw2: + elif stack.pop()[0] != kw2: self.error('unmatched end') # end if - del stack[-1:] self.putline(line, len(stack)) continue # end if @@ -208,23 +206,23 @@ def delete(self): begin_counter = 0 end_counter = 0 - while 1: + while True: line = self.getline() if not line: break # EOF # end if m = self.endprog.match(line) if m: - end_counter = end_counter + 1 + end_counter += 1 continue # end if m = self.kwprog.match(line) if m: kw = m.group('kw') if kw in start: - begin_counter = begin_counter + 1 + begin_counter += 1 # end if # end if - self.putline(line) + self.write(line) # end while if begin_counter - end_counter < 0: sys.stderr.write('Warning: input contained more end tags than expected\n') @@ -234,17 +232,12 @@ # end def delete def complete(self): - self.indentsize = 1 stack = [] todo = [] - thisid = '' - current, firstkw, lastkw, topid = 0, '', '', '' - while 1: + currentws = thisid = firstkw = lastkw = topid = '' + while True: line = self.getline() - i = 0 - m = self.wsprog.match(line) - if m: i = m.end() - # end if + i = self.wsprog.match(line).end() m = self.endprog.match(line) if m: thiskw = 'end' @@ -269,7 +262,9 @@ thiskw = '' # end if # end if - indent = len(line[:i].expandtabs(self.tabsize)) + indentws = line[:i] + indent = len(indentws.expandtabs(self.tabsize)) + current = len(currentws.expandtabs(self.tabsize)) while indent < current: if firstkw: if topid: @@ -278,11 +273,11 @@ else: s = '# end %s\n' % firstkw # end if - self.putline(s, current) + self.write(currentws + s) firstkw = lastkw = '' # end if - current, firstkw, lastkw, topid = stack[-1] - del stack[-1] + currentws, firstkw, lastkw, topid = stack.pop() + current = len(currentws.expandtabs(self.tabsize)) # end while if indent == current and firstkw: if thiskw == 'end': @@ -297,18 +292,18 @@ else: s = '# end %s\n' % firstkw # end if - self.putline(s, current) + self.write(currentws + s) firstkw = lastkw = topid = '' # end if # end if if indent > current: - stack.append((current, firstkw, lastkw, topid)) + stack.append((currentws, firstkw, lastkw, topid)) if thiskw and thiskw not in start: # error thiskw = '' # end if - current, firstkw, lastkw, topid = \ - indent, thiskw, thiskw, thisid + currentws, firstkw, lastkw, topid = \ + indentws, thiskw, thiskw, thisid # end if if thiskw: if thiskw in start: @@ -326,7 +321,6 @@ self.write(line) # end while # end def complete - # end class PythonIndenter # Simplified user interface @@ -352,76 +346,34 @@ pi.reformat() # end def reformat_filter -class StringReader: - def __init__(self, buf): - self.buf = buf - self.pos = 0 - self.len = len(self.buf) - # end def __init__ - def read(self, n = 0): - if n <= 0: - n = self.len - self.pos - else: - n = min(n, self.len - self.pos) - # end if - r = self.buf[self.pos : self.pos + n] - self.pos = self.pos + n - return r - # end def read - def readline(self): - i = self.buf.find('\n', self.pos) - return self.read(i + 1 - self.pos) - # end def readline - def readlines(self): - lines = [] - line = self.readline() - while line: - lines.append(line) - line = self.readline() - # end while - return lines - # end def readlines - # seek/tell etc. are left as an exercise for the reader -# end class StringReader - -class StringWriter: - def __init__(self): - self.buf = '' - # end def __init__ - def write(self, s): - self.buf = self.buf + s - # end def write - def getvalue(self): - return self.buf - # end def getvalue -# end class StringWriter - def complete_string(source, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - input = StringReader(source) - output = StringWriter() + input = io.StringIO(source) + output = io.StringIO() pi = PythonIndenter(input, output, stepsize, tabsize, expandtabs) pi.complete() return output.getvalue() # end def complete_string def delete_string(source, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - input = StringReader(source) - output = StringWriter() + input = io.StringIO(source) + output = io.StringIO() pi = PythonIndenter(input, output, stepsize, tabsize, expandtabs) pi.delete() return output.getvalue() # end def delete_string def reformat_string(source, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - input = StringReader(source) - output = StringWriter() + input = io.StringIO(source) + output = io.StringIO() pi = PythonIndenter(input, output, stepsize, tabsize, expandtabs) pi.reformat() return output.getvalue() # end def reformat_string def complete_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - source = open(filename, 'r').read() + with open(filename, 'r') as f: + source = f.read() + # end with result = complete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if @@ -429,14 +381,16 @@ try: os.rename(filename, filename + '~') except os.error: pass # end try - f = open(filename, 'w') - f.write(result) - f.close() + with open(filename, 'w') as f: + f.write(result) + # end with return 1 # end def complete_file def delete_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - source = open(filename, 'r').read() + with open(filename, 'r') as f: + source = f.read() + # end with result = delete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if @@ -444,14 +398,16 @@ try: os.rename(filename, filename + '~') except os.error: pass # end try - f = open(filename, 'w') - f.write(result) - f.close() + with open(filename, 'w') as f: + f.write(result) + # end with return 1 # end def delete_file def reformat_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - source = open(filename, 'r').read() + with open(filename, 'r') as f: + source = f.read() + # end with result = reformat_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if @@ -459,9 +415,9 @@ try: os.rename(filename, filename + '~') except os.error: pass # end try - f = open(filename, 'w') - f.write(result) - f.close() + with open(filename, 'w') as f: + f.write(result) + # end with return 1 # end def reformat_file @@ -474,7 +430,7 @@ -r : reformat a completed program (use #end directives) -s stepsize: indentation step (default %(STEPSIZE)d) -t tabsize : the worth in spaces of a tab (default %(TABSIZE)d) --e : expand TABs into spaces (defailt OFF) +-e : expand TABs into spaces (default OFF) [file] ... : files are changed in place, with backups in file~ If no files are specified or a single - is given, the program acts as a filter (reads stdin, writes stdout). @@ -517,7 +473,7 @@ elif o == '-t': tabsize = int(a) elif o == '-e': - expandtabs = 1 + expandtabs = True # end if # end for if not action: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 11:40:11 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 11 Jan 2013 11:40:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=2315539=3A_Fix_a_number_of_bugs_in_Tools/scripts/pinden?= =?utf-8?b?dC5weS4=?= Message-ID: <3YjLF70XwczRxl@mail.python.org> http://hg.python.org/cpython/rev/01df1f7841b2 changeset: 81406:01df1f7841b2 branch: 3.3 parent: 81398:3e884b3804b3 parent: 81405:9df6b707aef9 user: Serhiy Storchaka date: Fri Jan 11 12:10:57 2013 +0200 summary: Issue #15539: Fix a number of bugs in Tools/scripts/pindent.py. Now pindent.py works with a "with" statement. pindent.py no longer produces improper indentation. pindent.py now works with continued lines broken after "class" or "def" keywords and with continuations at the start of line. Added regression tests for pindent.py. Modernized pindent.py. files: Lib/test/test_tools.py | 326 ++++++++++++++++++++++++++- Misc/NEWS | 7 + Tools/scripts/pindent.py | 166 +++++-------- 3 files changed, 393 insertions(+), 106 deletions(-) diff --git a/Lib/test/test_tools.py b/Lib/test/test_tools.py --- a/Lib/test/test_tools.py +++ b/Lib/test/test_tools.py @@ -9,10 +9,13 @@ import importlib.machinery import unittest from unittest import mock +import shutil +import subprocess import sysconfig import tempfile +import textwrap from test import support -from test.script_helper import assert_python_ok +from test.script_helper import assert_python_ok, temp_dir if not sysconfig.is_python_build(): # XXX some installers do contain the tools, should we detect that @@ -36,6 +39,327 @@ self.assertGreater(err, b'') +class PindentTests(unittest.TestCase): + script = os.path.join(scriptsdir, 'pindent.py') + + def assertFileEqual(self, fn1, fn2): + with open(fn1) as f1, open(fn2) as f2: + self.assertEqual(f1.readlines(), f2.readlines()) + + def pindent(self, source, *args): + with subprocess.Popen( + (sys.executable, self.script) + args, + stdin=subprocess.PIPE, stdout=subprocess.PIPE, + universal_newlines=True) as proc: + out, err = proc.communicate(source) + self.assertIsNone(err) + return out + + def lstriplines(self, data): + return '\n'.join(line.lstrip() for line in data.splitlines()) + '\n' + + def test_selftest(self): + with temp_dir() as directory: + data_path = os.path.join(directory, '_test.py') + with open(self.script) as f: + closed = f.read() + with open(data_path, 'w') as f: + f.write(closed) + + rc, out, err = assert_python_ok(self.script, '-d', data_path) + self.assertEqual(out, b'') + self.assertEqual(err, b'') + backup = data_path + '~' + self.assertTrue(os.path.exists(backup)) + with open(backup) as f: + self.assertEqual(f.read(), closed) + with open(data_path) as f: + clean = f.read() + compile(clean, '_test.py', 'exec') + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + + rc, out, err = assert_python_ok(self.script, '-c', data_path) + self.assertEqual(out, b'') + self.assertEqual(err, b'') + with open(backup) as f: + self.assertEqual(f.read(), clean) + with open(data_path) as f: + self.assertEqual(f.read(), closed) + + broken = self.lstriplines(closed) + with open(data_path, 'w') as f: + f.write(broken) + rc, out, err = assert_python_ok(self.script, '-r', data_path) + self.assertEqual(out, b'') + self.assertEqual(err, b'') + with open(backup) as f: + self.assertEqual(f.read(), broken) + with open(data_path) as f: + indented = f.read() + compile(indented, '_test.py', 'exec') + self.assertEqual(self.pindent(broken, '-r'), indented) + + def pindent_test(self, clean, closed): + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + broken = self.lstriplines(closed) + self.assertEqual(self.pindent(broken, '-r', '-e', '-s', '4'), closed) + + def test_statements(self): + clean = textwrap.dedent("""\ + if a: + pass + + if a: + pass + else: + pass + + if a: + pass + elif: + pass + else: + pass + + while a: + break + + while a: + break + else: + pass + + for i in a: + break + + for i in a: + break + else: + pass + + try: + pass + finally: + pass + + try: + pass + except TypeError: + pass + except ValueError: + pass + else: + pass + + try: + pass + except TypeError: + pass + except ValueError: + pass + finally: + pass + + with a: + pass + + class A: + pass + + def f(): + pass + """) + + closed = textwrap.dedent("""\ + if a: + pass + # end if + + if a: + pass + else: + pass + # end if + + if a: + pass + elif: + pass + else: + pass + # end if + + while a: + break + # end while + + while a: + break + else: + pass + # end while + + for i in a: + break + # end for + + for i in a: + break + else: + pass + # end for + + try: + pass + finally: + pass + # end try + + try: + pass + except TypeError: + pass + except ValueError: + pass + else: + pass + # end try + + try: + pass + except TypeError: + pass + except ValueError: + pass + finally: + pass + # end try + + with a: + pass + # end with + + class A: + pass + # end class A + + def f(): + pass + # end def f + """) + self.pindent_test(clean, closed) + + def test_multilevel(self): + clean = textwrap.dedent("""\ + def foobar(a, b): + if a == b: + a = a+1 + elif a < b: + b = b-1 + if b > a: a = a-1 + else: + print 'oops!' + """) + closed = textwrap.dedent("""\ + def foobar(a, b): + if a == b: + a = a+1 + elif a < b: + b = b-1 + if b > a: a = a-1 + # end if + else: + print 'oops!' + # end if + # end def foobar + """) + self.pindent_test(clean, closed) + + def test_preserve_indents(self): + clean = textwrap.dedent("""\ + if a: + if b: + pass + """) + closed = textwrap.dedent("""\ + if a: + if b: + pass + # end if + # end if + """) + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + broken = self.lstriplines(closed) + self.assertEqual(self.pindent(broken, '-r', '-e', '-s', '9'), closed) + clean = textwrap.dedent("""\ + if a: + \tif b: + \t\tpass + """) + closed = textwrap.dedent("""\ + if a: + \tif b: + \t\tpass + \t# end if + # end if + """) + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + broken = self.lstriplines(closed) + self.assertEqual(self.pindent(broken, '-r'), closed) + + def test_escaped_newline(self): + clean = textwrap.dedent("""\ + class\\ + \\ + A: + def\ + \\ + f: + pass + """) + closed = textwrap.dedent("""\ + class\\ + \\ + A: + def\ + \\ + f: + pass + # end def f + # end class A + """) + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + + def test_empty_line(self): + clean = textwrap.dedent("""\ + if a: + + pass + """) + closed = textwrap.dedent("""\ + if a: + + pass + # end if + """) + self.pindent_test(clean, closed) + + def test_oneline(self): + clean = textwrap.dedent("""\ + if a: pass + """) + closed = textwrap.dedent("""\ + if a: pass + # end if + """) + self.pindent_test(clean, closed) + + class TestSundryScripts(unittest.TestCase): # At least make sure the rest don't have syntax errors. When tests are # added for a script it should be added to the whitelist below. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -424,6 +424,8 @@ Tests ----- +- Issue #15539: Added regression tests for Tools/scripts/pindent.py. + - Issue #16925: test_configparser now works with unittest test discovery. Patch by Zachary Ware. @@ -571,6 +573,11 @@ Tools/Demos ----------- +- Issue #15539: Fix a number of bugs in Tools/scripts/pindent.py. Now + pindent.py works with a "with" statement. pindent.py no longer produces + improper indentation. pindent.py now works with continued lines broken after + "class" or "def" keywords and with continuations at the start of line. + - Issue #15378: Fix Tools/unicode/comparecodecs.py. Patch by Serhiy Storchaka. diff --git a/Tools/scripts/pindent.py b/Tools/scripts/pindent.py --- a/Tools/scripts/pindent.py +++ b/Tools/scripts/pindent.py @@ -79,8 +79,9 @@ # Defaults STEPSIZE = 8 TABSIZE = 8 -EXPANDTABS = 0 +EXPANDTABS = False +import io import re import sys @@ -89,7 +90,8 @@ next['while'] = next['for'] = 'else', 'end' next['try'] = 'except', 'finally' next['except'] = 'except', 'else', 'finally', 'end' -next['else'] = next['finally'] = next['def'] = next['class'] = 'end' +next['else'] = next['finally'] = next['with'] = \ + next['def'] = next['class'] = 'end' next['end'] = () start = 'if', 'while', 'for', 'try', 'with', 'def', 'class' @@ -105,11 +107,11 @@ self.expandtabs = expandtabs self._write = fpo.write self.kwprog = re.compile( - r'^\s*(?P[a-z]+)' - r'(\s+(?P[a-zA-Z_]\w*))?' + r'^(?:\s|\\\n)*(?P[a-z]+)' + r'((?:\s|\\\n)+(?P[a-zA-Z_]\w*))?' r'[^\w]') self.endprog = re.compile( - r'^\s*#?\s*end\s+(?P[a-z]+)' + r'^(?:\s|\\\n)*#?\s*end\s+(?P[a-z]+)' r'(\s+(?P[a-zA-Z_]\w*))?' r'[^\w]') self.wsprog = re.compile(r'^[ \t]*') @@ -125,7 +127,7 @@ def readline(self): line = self.fpi.readline() - if line: self.lineno = self.lineno + 1 + if line: self.lineno += 1 # end if return line # end def readline @@ -143,27 +145,24 @@ line2 = self.readline() if not line2: break # end if - line = line + line2 + line += line2 # end while return line # end def getline - def putline(self, line, indent = None): - if indent is None: - self.write(line) - return + def putline(self, line, indent): + tabs, spaces = divmod(indent*self.indentsize, self.tabsize) + i = self.wsprog.match(line).end() + line = line[i:] + if line[:1] not in ('\n', '\r', ''): + line = '\t'*tabs + ' '*spaces + line # end if - tabs, spaces = divmod(indent*self.indentsize, self.tabsize) - i = 0 - m = self.wsprog.match(line) - if m: i = m.end() - # end if - self.write('\t'*tabs + ' '*spaces + line[i:]) + self.write(line) # end def putline def reformat(self): stack = [] - while 1: + while True: line = self.getline() if not line: break # EOF # end if @@ -173,10 +172,9 @@ kw2 = m.group('kw') if not stack: self.error('unexpected end') - elif stack[-1][0] != kw2: + elif stack.pop()[0] != kw2: self.error('unmatched end') # end if - del stack[-1:] self.putline(line, len(stack)) continue # end if @@ -208,23 +206,23 @@ def delete(self): begin_counter = 0 end_counter = 0 - while 1: + while True: line = self.getline() if not line: break # EOF # end if m = self.endprog.match(line) if m: - end_counter = end_counter + 1 + end_counter += 1 continue # end if m = self.kwprog.match(line) if m: kw = m.group('kw') if kw in start: - begin_counter = begin_counter + 1 + begin_counter += 1 # end if # end if - self.putline(line) + self.write(line) # end while if begin_counter - end_counter < 0: sys.stderr.write('Warning: input contained more end tags than expected\n') @@ -234,17 +232,12 @@ # end def delete def complete(self): - self.indentsize = 1 stack = [] todo = [] - thisid = '' - current, firstkw, lastkw, topid = 0, '', '', '' - while 1: + currentws = thisid = firstkw = lastkw = topid = '' + while True: line = self.getline() - i = 0 - m = self.wsprog.match(line) - if m: i = m.end() - # end if + i = self.wsprog.match(line).end() m = self.endprog.match(line) if m: thiskw = 'end' @@ -269,7 +262,9 @@ thiskw = '' # end if # end if - indent = len(line[:i].expandtabs(self.tabsize)) + indentws = line[:i] + indent = len(indentws.expandtabs(self.tabsize)) + current = len(currentws.expandtabs(self.tabsize)) while indent < current: if firstkw: if topid: @@ -278,11 +273,11 @@ else: s = '# end %s\n' % firstkw # end if - self.putline(s, current) + self.write(currentws + s) firstkw = lastkw = '' # end if - current, firstkw, lastkw, topid = stack[-1] - del stack[-1] + currentws, firstkw, lastkw, topid = stack.pop() + current = len(currentws.expandtabs(self.tabsize)) # end while if indent == current and firstkw: if thiskw == 'end': @@ -297,18 +292,18 @@ else: s = '# end %s\n' % firstkw # end if - self.putline(s, current) + self.write(currentws + s) firstkw = lastkw = topid = '' # end if # end if if indent > current: - stack.append((current, firstkw, lastkw, topid)) + stack.append((currentws, firstkw, lastkw, topid)) if thiskw and thiskw not in start: # error thiskw = '' # end if - current, firstkw, lastkw, topid = \ - indent, thiskw, thiskw, thisid + currentws, firstkw, lastkw, topid = \ + indentws, thiskw, thiskw, thisid # end if if thiskw: if thiskw in start: @@ -326,7 +321,6 @@ self.write(line) # end while # end def complete - # end class PythonIndenter # Simplified user interface @@ -352,76 +346,34 @@ pi.reformat() # end def reformat_filter -class StringReader: - def __init__(self, buf): - self.buf = buf - self.pos = 0 - self.len = len(self.buf) - # end def __init__ - def read(self, n = 0): - if n <= 0: - n = self.len - self.pos - else: - n = min(n, self.len - self.pos) - # end if - r = self.buf[self.pos : self.pos + n] - self.pos = self.pos + n - return r - # end def read - def readline(self): - i = self.buf.find('\n', self.pos) - return self.read(i + 1 - self.pos) - # end def readline - def readlines(self): - lines = [] - line = self.readline() - while line: - lines.append(line) - line = self.readline() - # end while - return lines - # end def readlines - # seek/tell etc. are left as an exercise for the reader -# end class StringReader - -class StringWriter: - def __init__(self): - self.buf = '' - # end def __init__ - def write(self, s): - self.buf = self.buf + s - # end def write - def getvalue(self): - return self.buf - # end def getvalue -# end class StringWriter - def complete_string(source, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - input = StringReader(source) - output = StringWriter() + input = io.StringIO(source) + output = io.StringIO() pi = PythonIndenter(input, output, stepsize, tabsize, expandtabs) pi.complete() return output.getvalue() # end def complete_string def delete_string(source, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - input = StringReader(source) - output = StringWriter() + input = io.StringIO(source) + output = io.StringIO() pi = PythonIndenter(input, output, stepsize, tabsize, expandtabs) pi.delete() return output.getvalue() # end def delete_string def reformat_string(source, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - input = StringReader(source) - output = StringWriter() + input = io.StringIO(source) + output = io.StringIO() pi = PythonIndenter(input, output, stepsize, tabsize, expandtabs) pi.reformat() return output.getvalue() # end def reformat_string def complete_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - source = open(filename, 'r').read() + with open(filename, 'r') as f: + source = f.read() + # end with result = complete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if @@ -429,14 +381,16 @@ try: os.rename(filename, filename + '~') except os.error: pass # end try - f = open(filename, 'w') - f.write(result) - f.close() + with open(filename, 'w') as f: + f.write(result) + # end with return 1 # end def complete_file def delete_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - source = open(filename, 'r').read() + with open(filename, 'r') as f: + source = f.read() + # end with result = delete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if @@ -444,14 +398,16 @@ try: os.rename(filename, filename + '~') except os.error: pass # end try - f = open(filename, 'w') - f.write(result) - f.close() + with open(filename, 'w') as f: + f.write(result) + # end with return 1 # end def delete_file def reformat_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - source = open(filename, 'r').read() + with open(filename, 'r') as f: + source = f.read() + # end with result = reformat_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if @@ -459,9 +415,9 @@ try: os.rename(filename, filename + '~') except os.error: pass # end try - f = open(filename, 'w') - f.write(result) - f.close() + with open(filename, 'w') as f: + f.write(result) + # end with return 1 # end def reformat_file @@ -474,7 +430,7 @@ -r : reformat a completed program (use #end directives) -s stepsize: indentation step (default %(STEPSIZE)d) -t tabsize : the worth in spaces of a tab (default %(TABSIZE)d) --e : expand TABs into spaces (defailt OFF) +-e : expand TABs into spaces (default OFF) [file] ... : files are changed in place, with backups in file~ If no files are specified or a single - is given, the program acts as a filter (reads stdin, writes stdout). @@ -517,7 +473,7 @@ elif o == '-t': tabsize = int(a) elif o == '-e': - expandtabs = 1 + expandtabs = True # end if # end for if not action: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 11:40:12 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 11 Jan 2013 11:40:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2315539=3A_Fix_a_number_of_bugs_in_Tools/scripts/?= =?utf-8?q?pindent=2Epy=2E?= Message-ID: <3YjLF866ftzS6g@mail.python.org> http://hg.python.org/cpython/rev/8452c23139c6 changeset: 81407:8452c23139c6 parent: 81399:5ec8daab477a parent: 81406:01df1f7841b2 user: Serhiy Storchaka date: Fri Jan 11 12:12:32 2013 +0200 summary: Issue #15539: Fix a number of bugs in Tools/scripts/pindent.py. Now pindent.py works with a "with" statement. pindent.py no longer produces improper indentation. pindent.py now works with continued lines broken after "class" or "def" keywords and with continuations at the start of line. Added regression tests for pindent.py. Modernized pindent.py. files: Lib/test/test_tools.py | 326 ++++++++++++++++++++++++++- Misc/NEWS | 7 + Tools/scripts/pindent.py | 166 +++++-------- 3 files changed, 393 insertions(+), 106 deletions(-) diff --git a/Lib/test/test_tools.py b/Lib/test/test_tools.py --- a/Lib/test/test_tools.py +++ b/Lib/test/test_tools.py @@ -9,10 +9,13 @@ import importlib.machinery import unittest from unittest import mock +import shutil +import subprocess import sysconfig import tempfile +import textwrap from test import support -from test.script_helper import assert_python_ok +from test.script_helper import assert_python_ok, temp_dir if not sysconfig.is_python_build(): # XXX some installers do contain the tools, should we detect that @@ -36,6 +39,327 @@ self.assertGreater(err, b'') +class PindentTests(unittest.TestCase): + script = os.path.join(scriptsdir, 'pindent.py') + + def assertFileEqual(self, fn1, fn2): + with open(fn1) as f1, open(fn2) as f2: + self.assertEqual(f1.readlines(), f2.readlines()) + + def pindent(self, source, *args): + with subprocess.Popen( + (sys.executable, self.script) + args, + stdin=subprocess.PIPE, stdout=subprocess.PIPE, + universal_newlines=True) as proc: + out, err = proc.communicate(source) + self.assertIsNone(err) + return out + + def lstriplines(self, data): + return '\n'.join(line.lstrip() for line in data.splitlines()) + '\n' + + def test_selftest(self): + with temp_dir() as directory: + data_path = os.path.join(directory, '_test.py') + with open(self.script) as f: + closed = f.read() + with open(data_path, 'w') as f: + f.write(closed) + + rc, out, err = assert_python_ok(self.script, '-d', data_path) + self.assertEqual(out, b'') + self.assertEqual(err, b'') + backup = data_path + '~' + self.assertTrue(os.path.exists(backup)) + with open(backup) as f: + self.assertEqual(f.read(), closed) + with open(data_path) as f: + clean = f.read() + compile(clean, '_test.py', 'exec') + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + + rc, out, err = assert_python_ok(self.script, '-c', data_path) + self.assertEqual(out, b'') + self.assertEqual(err, b'') + with open(backup) as f: + self.assertEqual(f.read(), clean) + with open(data_path) as f: + self.assertEqual(f.read(), closed) + + broken = self.lstriplines(closed) + with open(data_path, 'w') as f: + f.write(broken) + rc, out, err = assert_python_ok(self.script, '-r', data_path) + self.assertEqual(out, b'') + self.assertEqual(err, b'') + with open(backup) as f: + self.assertEqual(f.read(), broken) + with open(data_path) as f: + indented = f.read() + compile(indented, '_test.py', 'exec') + self.assertEqual(self.pindent(broken, '-r'), indented) + + def pindent_test(self, clean, closed): + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + broken = self.lstriplines(closed) + self.assertEqual(self.pindent(broken, '-r', '-e', '-s', '4'), closed) + + def test_statements(self): + clean = textwrap.dedent("""\ + if a: + pass + + if a: + pass + else: + pass + + if a: + pass + elif: + pass + else: + pass + + while a: + break + + while a: + break + else: + pass + + for i in a: + break + + for i in a: + break + else: + pass + + try: + pass + finally: + pass + + try: + pass + except TypeError: + pass + except ValueError: + pass + else: + pass + + try: + pass + except TypeError: + pass + except ValueError: + pass + finally: + pass + + with a: + pass + + class A: + pass + + def f(): + pass + """) + + closed = textwrap.dedent("""\ + if a: + pass + # end if + + if a: + pass + else: + pass + # end if + + if a: + pass + elif: + pass + else: + pass + # end if + + while a: + break + # end while + + while a: + break + else: + pass + # end while + + for i in a: + break + # end for + + for i in a: + break + else: + pass + # end for + + try: + pass + finally: + pass + # end try + + try: + pass + except TypeError: + pass + except ValueError: + pass + else: + pass + # end try + + try: + pass + except TypeError: + pass + except ValueError: + pass + finally: + pass + # end try + + with a: + pass + # end with + + class A: + pass + # end class A + + def f(): + pass + # end def f + """) + self.pindent_test(clean, closed) + + def test_multilevel(self): + clean = textwrap.dedent("""\ + def foobar(a, b): + if a == b: + a = a+1 + elif a < b: + b = b-1 + if b > a: a = a-1 + else: + print 'oops!' + """) + closed = textwrap.dedent("""\ + def foobar(a, b): + if a == b: + a = a+1 + elif a < b: + b = b-1 + if b > a: a = a-1 + # end if + else: + print 'oops!' + # end if + # end def foobar + """) + self.pindent_test(clean, closed) + + def test_preserve_indents(self): + clean = textwrap.dedent("""\ + if a: + if b: + pass + """) + closed = textwrap.dedent("""\ + if a: + if b: + pass + # end if + # end if + """) + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + broken = self.lstriplines(closed) + self.assertEqual(self.pindent(broken, '-r', '-e', '-s', '9'), closed) + clean = textwrap.dedent("""\ + if a: + \tif b: + \t\tpass + """) + closed = textwrap.dedent("""\ + if a: + \tif b: + \t\tpass + \t# end if + # end if + """) + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + broken = self.lstriplines(closed) + self.assertEqual(self.pindent(broken, '-r'), closed) + + def test_escaped_newline(self): + clean = textwrap.dedent("""\ + class\\ + \\ + A: + def\ + \\ + f: + pass + """) + closed = textwrap.dedent("""\ + class\\ + \\ + A: + def\ + \\ + f: + pass + # end def f + # end class A + """) + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + + def test_empty_line(self): + clean = textwrap.dedent("""\ + if a: + + pass + """) + closed = textwrap.dedent("""\ + if a: + + pass + # end if + """) + self.pindent_test(clean, closed) + + def test_oneline(self): + clean = textwrap.dedent("""\ + if a: pass + """) + closed = textwrap.dedent("""\ + if a: pass + # end if + """) + self.pindent_test(clean, closed) + + class TestSundryScripts(unittest.TestCase): # At least make sure the rest don't have syntax errors. When tests are # added for a script it should be added to the whitelist below. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -621,6 +621,8 @@ Tests ----- +- Issue #15539: Added regression tests for Tools/scripts/pindent.py. + - Issue #16836: Enable IPv6 support even if IPv6 is disabled on the build host. - Issue #16925: test_configparser now works with unittest test discovery. @@ -777,6 +779,11 @@ Tools/Demos ----------- +- Issue #15539: Fix a number of bugs in Tools/scripts/pindent.py. Now + pindent.py works with a "with" statement. pindent.py no longer produces + improper indentation. pindent.py now works with continued lines broken after + "class" or "def" keywords and with continuations at the start of line. + - Issue #11797: Add a 2to3 fixer that maps reload() to imp.reload(). - Issue #10966: Remove the concept of unexpected skipped tests. diff --git a/Tools/scripts/pindent.py b/Tools/scripts/pindent.py --- a/Tools/scripts/pindent.py +++ b/Tools/scripts/pindent.py @@ -79,8 +79,9 @@ # Defaults STEPSIZE = 8 TABSIZE = 8 -EXPANDTABS = 0 +EXPANDTABS = False +import io import re import sys @@ -89,7 +90,8 @@ next['while'] = next['for'] = 'else', 'end' next['try'] = 'except', 'finally' next['except'] = 'except', 'else', 'finally', 'end' -next['else'] = next['finally'] = next['def'] = next['class'] = 'end' +next['else'] = next['finally'] = next['with'] = \ + next['def'] = next['class'] = 'end' next['end'] = () start = 'if', 'while', 'for', 'try', 'with', 'def', 'class' @@ -105,11 +107,11 @@ self.expandtabs = expandtabs self._write = fpo.write self.kwprog = re.compile( - r'^\s*(?P[a-z]+)' - r'(\s+(?P[a-zA-Z_]\w*))?' + r'^(?:\s|\\\n)*(?P[a-z]+)' + r'((?:\s|\\\n)+(?P[a-zA-Z_]\w*))?' r'[^\w]') self.endprog = re.compile( - r'^\s*#?\s*end\s+(?P[a-z]+)' + r'^(?:\s|\\\n)*#?\s*end\s+(?P[a-z]+)' r'(\s+(?P[a-zA-Z_]\w*))?' r'[^\w]') self.wsprog = re.compile(r'^[ \t]*') @@ -125,7 +127,7 @@ def readline(self): line = self.fpi.readline() - if line: self.lineno = self.lineno + 1 + if line: self.lineno += 1 # end if return line # end def readline @@ -143,27 +145,24 @@ line2 = self.readline() if not line2: break # end if - line = line + line2 + line += line2 # end while return line # end def getline - def putline(self, line, indent = None): - if indent is None: - self.write(line) - return + def putline(self, line, indent): + tabs, spaces = divmod(indent*self.indentsize, self.tabsize) + i = self.wsprog.match(line).end() + line = line[i:] + if line[:1] not in ('\n', '\r', ''): + line = '\t'*tabs + ' '*spaces + line # end if - tabs, spaces = divmod(indent*self.indentsize, self.tabsize) - i = 0 - m = self.wsprog.match(line) - if m: i = m.end() - # end if - self.write('\t'*tabs + ' '*spaces + line[i:]) + self.write(line) # end def putline def reformat(self): stack = [] - while 1: + while True: line = self.getline() if not line: break # EOF # end if @@ -173,10 +172,9 @@ kw2 = m.group('kw') if not stack: self.error('unexpected end') - elif stack[-1][0] != kw2: + elif stack.pop()[0] != kw2: self.error('unmatched end') # end if - del stack[-1:] self.putline(line, len(stack)) continue # end if @@ -208,23 +206,23 @@ def delete(self): begin_counter = 0 end_counter = 0 - while 1: + while True: line = self.getline() if not line: break # EOF # end if m = self.endprog.match(line) if m: - end_counter = end_counter + 1 + end_counter += 1 continue # end if m = self.kwprog.match(line) if m: kw = m.group('kw') if kw in start: - begin_counter = begin_counter + 1 + begin_counter += 1 # end if # end if - self.putline(line) + self.write(line) # end while if begin_counter - end_counter < 0: sys.stderr.write('Warning: input contained more end tags than expected\n') @@ -234,17 +232,12 @@ # end def delete def complete(self): - self.indentsize = 1 stack = [] todo = [] - thisid = '' - current, firstkw, lastkw, topid = 0, '', '', '' - while 1: + currentws = thisid = firstkw = lastkw = topid = '' + while True: line = self.getline() - i = 0 - m = self.wsprog.match(line) - if m: i = m.end() - # end if + i = self.wsprog.match(line).end() m = self.endprog.match(line) if m: thiskw = 'end' @@ -269,7 +262,9 @@ thiskw = '' # end if # end if - indent = len(line[:i].expandtabs(self.tabsize)) + indentws = line[:i] + indent = len(indentws.expandtabs(self.tabsize)) + current = len(currentws.expandtabs(self.tabsize)) while indent < current: if firstkw: if topid: @@ -278,11 +273,11 @@ else: s = '# end %s\n' % firstkw # end if - self.putline(s, current) + self.write(currentws + s) firstkw = lastkw = '' # end if - current, firstkw, lastkw, topid = stack[-1] - del stack[-1] + currentws, firstkw, lastkw, topid = stack.pop() + current = len(currentws.expandtabs(self.tabsize)) # end while if indent == current and firstkw: if thiskw == 'end': @@ -297,18 +292,18 @@ else: s = '# end %s\n' % firstkw # end if - self.putline(s, current) + self.write(currentws + s) firstkw = lastkw = topid = '' # end if # end if if indent > current: - stack.append((current, firstkw, lastkw, topid)) + stack.append((currentws, firstkw, lastkw, topid)) if thiskw and thiskw not in start: # error thiskw = '' # end if - current, firstkw, lastkw, topid = \ - indent, thiskw, thiskw, thisid + currentws, firstkw, lastkw, topid = \ + indentws, thiskw, thiskw, thisid # end if if thiskw: if thiskw in start: @@ -326,7 +321,6 @@ self.write(line) # end while # end def complete - # end class PythonIndenter # Simplified user interface @@ -352,76 +346,34 @@ pi.reformat() # end def reformat_filter -class StringReader: - def __init__(self, buf): - self.buf = buf - self.pos = 0 - self.len = len(self.buf) - # end def __init__ - def read(self, n = 0): - if n <= 0: - n = self.len - self.pos - else: - n = min(n, self.len - self.pos) - # end if - r = self.buf[self.pos : self.pos + n] - self.pos = self.pos + n - return r - # end def read - def readline(self): - i = self.buf.find('\n', self.pos) - return self.read(i + 1 - self.pos) - # end def readline - def readlines(self): - lines = [] - line = self.readline() - while line: - lines.append(line) - line = self.readline() - # end while - return lines - # end def readlines - # seek/tell etc. are left as an exercise for the reader -# end class StringReader - -class StringWriter: - def __init__(self): - self.buf = '' - # end def __init__ - def write(self, s): - self.buf = self.buf + s - # end def write - def getvalue(self): - return self.buf - # end def getvalue -# end class StringWriter - def complete_string(source, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - input = StringReader(source) - output = StringWriter() + input = io.StringIO(source) + output = io.StringIO() pi = PythonIndenter(input, output, stepsize, tabsize, expandtabs) pi.complete() return output.getvalue() # end def complete_string def delete_string(source, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - input = StringReader(source) - output = StringWriter() + input = io.StringIO(source) + output = io.StringIO() pi = PythonIndenter(input, output, stepsize, tabsize, expandtabs) pi.delete() return output.getvalue() # end def delete_string def reformat_string(source, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - input = StringReader(source) - output = StringWriter() + input = io.StringIO(source) + output = io.StringIO() pi = PythonIndenter(input, output, stepsize, tabsize, expandtabs) pi.reformat() return output.getvalue() # end def reformat_string def complete_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - source = open(filename, 'r').read() + with open(filename, 'r') as f: + source = f.read() + # end with result = complete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if @@ -429,14 +381,16 @@ try: os.rename(filename, filename + '~') except OSError: pass # end try - f = open(filename, 'w') - f.write(result) - f.close() + with open(filename, 'w') as f: + f.write(result) + # end with return 1 # end def complete_file def delete_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - source = open(filename, 'r').read() + with open(filename, 'r') as f: + source = f.read() + # end with result = delete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if @@ -444,14 +398,16 @@ try: os.rename(filename, filename + '~') except OSError: pass # end try - f = open(filename, 'w') - f.write(result) - f.close() + with open(filename, 'w') as f: + f.write(result) + # end with return 1 # end def delete_file def reformat_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - source = open(filename, 'r').read() + with open(filename, 'r') as f: + source = f.read() + # end with result = reformat_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if @@ -459,9 +415,9 @@ try: os.rename(filename, filename + '~') except OSError: pass # end try - f = open(filename, 'w') - f.write(result) - f.close() + with open(filename, 'w') as f: + f.write(result) + # end with return 1 # end def reformat_file @@ -474,7 +430,7 @@ -r : reformat a completed program (use #end directives) -s stepsize: indentation step (default %(STEPSIZE)d) -t tabsize : the worth in spaces of a tab (default %(TABSIZE)d) --e : expand TABs into spaces (defailt OFF) +-e : expand TABs into spaces (default OFF) [file] ... : files are changed in place, with backups in file~ If no files are specified or a single - is given, the program acts as a filter (reads stdin, writes stdout). @@ -517,7 +473,7 @@ elif o == '-t': tabsize = int(a) elif o == '-e': - expandtabs = 1 + expandtabs = True # end if # end for if not action: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 11:40:14 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 11 Jan 2013 11:40:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf-8?q?_Merge_heads?= Message-ID: <3YjLFB1hW7zR5K@mail.python.org> http://hg.python.org/cpython/rev/4a86efa5a97f changeset: 81408:4a86efa5a97f branch: 2.7 parent: 81404:f783db4a58ba parent: 81400:7555399134af user: Serhiy Storchaka date: Fri Jan 11 12:27:43 2013 +0200 summary: Merge heads files: Doc/library/argparse.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1116,7 +1116,7 @@ metavar ^^^^^^^ -When :class:`ArgumentParser` generates help messages, it need some way to refer +When :class:`ArgumentParser` generates help messages, it needs some way to refer to each expected argument. By default, ArgumentParser objects use the dest_ value as the "name" of each object. By default, for positional argument actions, the dest_ value is used directly, and for optional argument actions, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 11:40:15 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 11 Jan 2013 11:40:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMik6?= =?utf-8?q?_Merge_heads?= Message-ID: <3YjLFC4MGtzRxl@mail.python.org> http://hg.python.org/cpython/rev/973f3a253ab0 changeset: 81409:973f3a253ab0 branch: 3.2 parent: 81405:9df6b707aef9 parent: 81401:64d1cbc0f813 user: Serhiy Storchaka date: Fri Jan 11 12:28:53 2013 +0200 summary: Merge heads files: Doc/library/argparse.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1117,7 +1117,7 @@ metavar ^^^^^^^ -When :class:`ArgumentParser` generates help messages, it need some way to refer +When :class:`ArgumentParser` generates help messages, it needs some way to refer to each expected argument. By default, ArgumentParser objects use the dest_ value as the "name" of each object. By default, for positional argument actions, the dest_ value is used directly, and for optional argument actions, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 11:40:17 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 11 Jan 2013 11:40:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuMyk6?= =?utf-8?q?_Merge_heads?= Message-ID: <3YjLFF2TjvzS6J@mail.python.org> http://hg.python.org/cpython/rev/be350b08fc9c changeset: 81410:be350b08fc9c branch: 3.3 parent: 81402:ad806dac3ff3 parent: 81406:01df1f7841b2 user: Serhiy Storchaka date: Fri Jan 11 12:32:37 2013 +0200 summary: Merge heads files: Lib/test/test_tools.py | 326 ++++++++++++++++++++++++++- Misc/NEWS | 7 + Tools/scripts/pindent.py | 166 +++++-------- 3 files changed, 393 insertions(+), 106 deletions(-) diff --git a/Lib/test/test_tools.py b/Lib/test/test_tools.py --- a/Lib/test/test_tools.py +++ b/Lib/test/test_tools.py @@ -9,10 +9,13 @@ import importlib.machinery import unittest from unittest import mock +import shutil +import subprocess import sysconfig import tempfile +import textwrap from test import support -from test.script_helper import assert_python_ok +from test.script_helper import assert_python_ok, temp_dir if not sysconfig.is_python_build(): # XXX some installers do contain the tools, should we detect that @@ -36,6 +39,327 @@ self.assertGreater(err, b'') +class PindentTests(unittest.TestCase): + script = os.path.join(scriptsdir, 'pindent.py') + + def assertFileEqual(self, fn1, fn2): + with open(fn1) as f1, open(fn2) as f2: + self.assertEqual(f1.readlines(), f2.readlines()) + + def pindent(self, source, *args): + with subprocess.Popen( + (sys.executable, self.script) + args, + stdin=subprocess.PIPE, stdout=subprocess.PIPE, + universal_newlines=True) as proc: + out, err = proc.communicate(source) + self.assertIsNone(err) + return out + + def lstriplines(self, data): + return '\n'.join(line.lstrip() for line in data.splitlines()) + '\n' + + def test_selftest(self): + with temp_dir() as directory: + data_path = os.path.join(directory, '_test.py') + with open(self.script) as f: + closed = f.read() + with open(data_path, 'w') as f: + f.write(closed) + + rc, out, err = assert_python_ok(self.script, '-d', data_path) + self.assertEqual(out, b'') + self.assertEqual(err, b'') + backup = data_path + '~' + self.assertTrue(os.path.exists(backup)) + with open(backup) as f: + self.assertEqual(f.read(), closed) + with open(data_path) as f: + clean = f.read() + compile(clean, '_test.py', 'exec') + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + + rc, out, err = assert_python_ok(self.script, '-c', data_path) + self.assertEqual(out, b'') + self.assertEqual(err, b'') + with open(backup) as f: + self.assertEqual(f.read(), clean) + with open(data_path) as f: + self.assertEqual(f.read(), closed) + + broken = self.lstriplines(closed) + with open(data_path, 'w') as f: + f.write(broken) + rc, out, err = assert_python_ok(self.script, '-r', data_path) + self.assertEqual(out, b'') + self.assertEqual(err, b'') + with open(backup) as f: + self.assertEqual(f.read(), broken) + with open(data_path) as f: + indented = f.read() + compile(indented, '_test.py', 'exec') + self.assertEqual(self.pindent(broken, '-r'), indented) + + def pindent_test(self, clean, closed): + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + broken = self.lstriplines(closed) + self.assertEqual(self.pindent(broken, '-r', '-e', '-s', '4'), closed) + + def test_statements(self): + clean = textwrap.dedent("""\ + if a: + pass + + if a: + pass + else: + pass + + if a: + pass + elif: + pass + else: + pass + + while a: + break + + while a: + break + else: + pass + + for i in a: + break + + for i in a: + break + else: + pass + + try: + pass + finally: + pass + + try: + pass + except TypeError: + pass + except ValueError: + pass + else: + pass + + try: + pass + except TypeError: + pass + except ValueError: + pass + finally: + pass + + with a: + pass + + class A: + pass + + def f(): + pass + """) + + closed = textwrap.dedent("""\ + if a: + pass + # end if + + if a: + pass + else: + pass + # end if + + if a: + pass + elif: + pass + else: + pass + # end if + + while a: + break + # end while + + while a: + break + else: + pass + # end while + + for i in a: + break + # end for + + for i in a: + break + else: + pass + # end for + + try: + pass + finally: + pass + # end try + + try: + pass + except TypeError: + pass + except ValueError: + pass + else: + pass + # end try + + try: + pass + except TypeError: + pass + except ValueError: + pass + finally: + pass + # end try + + with a: + pass + # end with + + class A: + pass + # end class A + + def f(): + pass + # end def f + """) + self.pindent_test(clean, closed) + + def test_multilevel(self): + clean = textwrap.dedent("""\ + def foobar(a, b): + if a == b: + a = a+1 + elif a < b: + b = b-1 + if b > a: a = a-1 + else: + print 'oops!' + """) + closed = textwrap.dedent("""\ + def foobar(a, b): + if a == b: + a = a+1 + elif a < b: + b = b-1 + if b > a: a = a-1 + # end if + else: + print 'oops!' + # end if + # end def foobar + """) + self.pindent_test(clean, closed) + + def test_preserve_indents(self): + clean = textwrap.dedent("""\ + if a: + if b: + pass + """) + closed = textwrap.dedent("""\ + if a: + if b: + pass + # end if + # end if + """) + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + broken = self.lstriplines(closed) + self.assertEqual(self.pindent(broken, '-r', '-e', '-s', '9'), closed) + clean = textwrap.dedent("""\ + if a: + \tif b: + \t\tpass + """) + closed = textwrap.dedent("""\ + if a: + \tif b: + \t\tpass + \t# end if + # end if + """) + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + broken = self.lstriplines(closed) + self.assertEqual(self.pindent(broken, '-r'), closed) + + def test_escaped_newline(self): + clean = textwrap.dedent("""\ + class\\ + \\ + A: + def\ + \\ + f: + pass + """) + closed = textwrap.dedent("""\ + class\\ + \\ + A: + def\ + \\ + f: + pass + # end def f + # end class A + """) + self.assertEqual(self.pindent(clean, '-c'), closed) + self.assertEqual(self.pindent(closed, '-d'), clean) + + def test_empty_line(self): + clean = textwrap.dedent("""\ + if a: + + pass + """) + closed = textwrap.dedent("""\ + if a: + + pass + # end if + """) + self.pindent_test(clean, closed) + + def test_oneline(self): + clean = textwrap.dedent("""\ + if a: pass + """) + closed = textwrap.dedent("""\ + if a: pass + # end if + """) + self.pindent_test(clean, closed) + + class TestSundryScripts(unittest.TestCase): # At least make sure the rest don't have syntax errors. When tests are # added for a script it should be added to the whitelist below. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -424,6 +424,8 @@ Tests ----- +- Issue #15539: Added regression tests for Tools/scripts/pindent.py. + - Issue #16925: test_configparser now works with unittest test discovery. Patch by Zachary Ware. @@ -571,6 +573,11 @@ Tools/Demos ----------- +- Issue #15539: Fix a number of bugs in Tools/scripts/pindent.py. Now + pindent.py works with a "with" statement. pindent.py no longer produces + improper indentation. pindent.py now works with continued lines broken after + "class" or "def" keywords and with continuations at the start of line. + - Issue #15378: Fix Tools/unicode/comparecodecs.py. Patch by Serhiy Storchaka. diff --git a/Tools/scripts/pindent.py b/Tools/scripts/pindent.py --- a/Tools/scripts/pindent.py +++ b/Tools/scripts/pindent.py @@ -79,8 +79,9 @@ # Defaults STEPSIZE = 8 TABSIZE = 8 -EXPANDTABS = 0 +EXPANDTABS = False +import io import re import sys @@ -89,7 +90,8 @@ next['while'] = next['for'] = 'else', 'end' next['try'] = 'except', 'finally' next['except'] = 'except', 'else', 'finally', 'end' -next['else'] = next['finally'] = next['def'] = next['class'] = 'end' +next['else'] = next['finally'] = next['with'] = \ + next['def'] = next['class'] = 'end' next['end'] = () start = 'if', 'while', 'for', 'try', 'with', 'def', 'class' @@ -105,11 +107,11 @@ self.expandtabs = expandtabs self._write = fpo.write self.kwprog = re.compile( - r'^\s*(?P[a-z]+)' - r'(\s+(?P[a-zA-Z_]\w*))?' + r'^(?:\s|\\\n)*(?P[a-z]+)' + r'((?:\s|\\\n)+(?P[a-zA-Z_]\w*))?' r'[^\w]') self.endprog = re.compile( - r'^\s*#?\s*end\s+(?P[a-z]+)' + r'^(?:\s|\\\n)*#?\s*end\s+(?P[a-z]+)' r'(\s+(?P[a-zA-Z_]\w*))?' r'[^\w]') self.wsprog = re.compile(r'^[ \t]*') @@ -125,7 +127,7 @@ def readline(self): line = self.fpi.readline() - if line: self.lineno = self.lineno + 1 + if line: self.lineno += 1 # end if return line # end def readline @@ -143,27 +145,24 @@ line2 = self.readline() if not line2: break # end if - line = line + line2 + line += line2 # end while return line # end def getline - def putline(self, line, indent = None): - if indent is None: - self.write(line) - return + def putline(self, line, indent): + tabs, spaces = divmod(indent*self.indentsize, self.tabsize) + i = self.wsprog.match(line).end() + line = line[i:] + if line[:1] not in ('\n', '\r', ''): + line = '\t'*tabs + ' '*spaces + line # end if - tabs, spaces = divmod(indent*self.indentsize, self.tabsize) - i = 0 - m = self.wsprog.match(line) - if m: i = m.end() - # end if - self.write('\t'*tabs + ' '*spaces + line[i:]) + self.write(line) # end def putline def reformat(self): stack = [] - while 1: + while True: line = self.getline() if not line: break # EOF # end if @@ -173,10 +172,9 @@ kw2 = m.group('kw') if not stack: self.error('unexpected end') - elif stack[-1][0] != kw2: + elif stack.pop()[0] != kw2: self.error('unmatched end') # end if - del stack[-1:] self.putline(line, len(stack)) continue # end if @@ -208,23 +206,23 @@ def delete(self): begin_counter = 0 end_counter = 0 - while 1: + while True: line = self.getline() if not line: break # EOF # end if m = self.endprog.match(line) if m: - end_counter = end_counter + 1 + end_counter += 1 continue # end if m = self.kwprog.match(line) if m: kw = m.group('kw') if kw in start: - begin_counter = begin_counter + 1 + begin_counter += 1 # end if # end if - self.putline(line) + self.write(line) # end while if begin_counter - end_counter < 0: sys.stderr.write('Warning: input contained more end tags than expected\n') @@ -234,17 +232,12 @@ # end def delete def complete(self): - self.indentsize = 1 stack = [] todo = [] - thisid = '' - current, firstkw, lastkw, topid = 0, '', '', '' - while 1: + currentws = thisid = firstkw = lastkw = topid = '' + while True: line = self.getline() - i = 0 - m = self.wsprog.match(line) - if m: i = m.end() - # end if + i = self.wsprog.match(line).end() m = self.endprog.match(line) if m: thiskw = 'end' @@ -269,7 +262,9 @@ thiskw = '' # end if # end if - indent = len(line[:i].expandtabs(self.tabsize)) + indentws = line[:i] + indent = len(indentws.expandtabs(self.tabsize)) + current = len(currentws.expandtabs(self.tabsize)) while indent < current: if firstkw: if topid: @@ -278,11 +273,11 @@ else: s = '# end %s\n' % firstkw # end if - self.putline(s, current) + self.write(currentws + s) firstkw = lastkw = '' # end if - current, firstkw, lastkw, topid = stack[-1] - del stack[-1] + currentws, firstkw, lastkw, topid = stack.pop() + current = len(currentws.expandtabs(self.tabsize)) # end while if indent == current and firstkw: if thiskw == 'end': @@ -297,18 +292,18 @@ else: s = '# end %s\n' % firstkw # end if - self.putline(s, current) + self.write(currentws + s) firstkw = lastkw = topid = '' # end if # end if if indent > current: - stack.append((current, firstkw, lastkw, topid)) + stack.append((currentws, firstkw, lastkw, topid)) if thiskw and thiskw not in start: # error thiskw = '' # end if - current, firstkw, lastkw, topid = \ - indent, thiskw, thiskw, thisid + currentws, firstkw, lastkw, topid = \ + indentws, thiskw, thiskw, thisid # end if if thiskw: if thiskw in start: @@ -326,7 +321,6 @@ self.write(line) # end while # end def complete - # end class PythonIndenter # Simplified user interface @@ -352,76 +346,34 @@ pi.reformat() # end def reformat_filter -class StringReader: - def __init__(self, buf): - self.buf = buf - self.pos = 0 - self.len = len(self.buf) - # end def __init__ - def read(self, n = 0): - if n <= 0: - n = self.len - self.pos - else: - n = min(n, self.len - self.pos) - # end if - r = self.buf[self.pos : self.pos + n] - self.pos = self.pos + n - return r - # end def read - def readline(self): - i = self.buf.find('\n', self.pos) - return self.read(i + 1 - self.pos) - # end def readline - def readlines(self): - lines = [] - line = self.readline() - while line: - lines.append(line) - line = self.readline() - # end while - return lines - # end def readlines - # seek/tell etc. are left as an exercise for the reader -# end class StringReader - -class StringWriter: - def __init__(self): - self.buf = '' - # end def __init__ - def write(self, s): - self.buf = self.buf + s - # end def write - def getvalue(self): - return self.buf - # end def getvalue -# end class StringWriter - def complete_string(source, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - input = StringReader(source) - output = StringWriter() + input = io.StringIO(source) + output = io.StringIO() pi = PythonIndenter(input, output, stepsize, tabsize, expandtabs) pi.complete() return output.getvalue() # end def complete_string def delete_string(source, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - input = StringReader(source) - output = StringWriter() + input = io.StringIO(source) + output = io.StringIO() pi = PythonIndenter(input, output, stepsize, tabsize, expandtabs) pi.delete() return output.getvalue() # end def delete_string def reformat_string(source, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - input = StringReader(source) - output = StringWriter() + input = io.StringIO(source) + output = io.StringIO() pi = PythonIndenter(input, output, stepsize, tabsize, expandtabs) pi.reformat() return output.getvalue() # end def reformat_string def complete_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - source = open(filename, 'r').read() + with open(filename, 'r') as f: + source = f.read() + # end with result = complete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if @@ -429,14 +381,16 @@ try: os.rename(filename, filename + '~') except os.error: pass # end try - f = open(filename, 'w') - f.write(result) - f.close() + with open(filename, 'w') as f: + f.write(result) + # end with return 1 # end def complete_file def delete_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - source = open(filename, 'r').read() + with open(filename, 'r') as f: + source = f.read() + # end with result = delete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if @@ -444,14 +398,16 @@ try: os.rename(filename, filename + '~') except os.error: pass # end try - f = open(filename, 'w') - f.write(result) - f.close() + with open(filename, 'w') as f: + f.write(result) + # end with return 1 # end def delete_file def reformat_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): - source = open(filename, 'r').read() + with open(filename, 'r') as f: + source = f.read() + # end with result = reformat_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if @@ -459,9 +415,9 @@ try: os.rename(filename, filename + '~') except os.error: pass # end try - f = open(filename, 'w') - f.write(result) - f.close() + with open(filename, 'w') as f: + f.write(result) + # end with return 1 # end def reformat_file @@ -474,7 +430,7 @@ -r : reformat a completed program (use #end directives) -s stepsize: indentation step (default %(STEPSIZE)d) -t tabsize : the worth in spaces of a tab (default %(TABSIZE)d) --e : expand TABs into spaces (defailt OFF) +-e : expand TABs into spaces (default OFF) [file] ... : files are changed in place, with backups in file~ If no files are specified or a single - is given, the program acts as a filter (reads stdin, writes stdout). @@ -517,7 +473,7 @@ elif o == '-t': tabsize = int(a) elif o == '-e': - expandtabs = 1 + expandtabs = True # end if # end for if not action: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 11:40:18 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 11 Jan 2013 11:40:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?q?=29=3A_Merge_heads?= Message-ID: <3YjLFG6CldzS6d@mail.python.org> http://hg.python.org/cpython/rev/479da8b0e36d changeset: 81411:479da8b0e36d parent: 81407:8452c23139c6 parent: 81403:6eae84170536 user: Serhiy Storchaka date: Fri Jan 11 12:33:10 2013 +0200 summary: Merge heads files: Doc/library/argparse.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1136,7 +1136,7 @@ metavar ^^^^^^^ -When :class:`ArgumentParser` generates help messages, it need some way to refer +When :class:`ArgumentParser` generates help messages, it needs some way to refer to each expected argument. By default, ArgumentParser objects use the dest_ value as the "name" of each object. By default, for positional argument actions, the dest_ value is used directly, and for optional argument actions, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 11:40:20 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 11 Jan 2013 11:40:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Null_merge?= Message-ID: <3YjLFJ3wgpzS6T@mail.python.org> http://hg.python.org/cpython/rev/086c6e8d0a24 changeset: 81412:086c6e8d0a24 branch: 3.3 parent: 81410:be350b08fc9c parent: 81409:973f3a253ab0 user: Serhiy Storchaka date: Fri Jan 11 12:34:21 2013 +0200 summary: Null merge files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 11:40:21 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 11 Jan 2013 11:40:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Null_merge?= Message-ID: <3YjLFK6QbGzS21@mail.python.org> http://hg.python.org/cpython/rev/59d9d970b2d0 changeset: 81413:59d9d970b2d0 parent: 81411:479da8b0e36d parent: 81412:086c6e8d0a24 user: Serhiy Storchaka date: Fri Jan 11 12:34:48 2013 +0200 summary: Null merge files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 12:41:02 2013 From: python-checkins at python.org (ezio.melotti) Date: Fri, 11 Jan 2013 12:41:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_Add_a_paragraph_about_che?= =?utf-8?q?cking_for_refleaks=2E__Initial_patch_by_Daniel_Shahaf=2E?= Message-ID: <3YjMbL0mThzS41@mail.python.org> http://hg.python.org/devguide/rev/de93193d7a33 changeset: 590:de93193d7a33 user: Ezio Melotti date: Fri Jan 11 13:40:53 2013 +0200 summary: Add a paragraph about checking for refleaks. Initial patch by Daniel Shahaf. files: committing.rst | 1 + runtests.rst | 5 +++++ 2 files changed, 6 insertions(+), 0 deletions(-) diff --git a/committing.rst b/committing.rst --- a/committing.rst +++ b/committing.rst @@ -35,6 +35,7 @@ * Has ``configure`` been regenerated, if necessary? * Has ``pyconfig.h.in`` been regenerated, if necessary? * Has the test suite been run? +* Are there any reference leaks? Note that the automated patch check can't actually *answer* all of these questions, and even if it could, it still wouldn't know whether or not diff --git a/runtests.rst b/runtests.rst --- a/runtests.rst +++ b/runtests.rst @@ -71,6 +71,11 @@ The ``-uall`` flag allows the use of all available resources so as to not skip tests requiring, e.g., Internet access. +To check for reference leaks (only needed if you modified C code), use the +``-R`` flag. For example, ``-R 3:2`` will first run the test 3 times to settle +down the reference count, and then run it 2 more times to verify if there are +any leaks. + You can also execute the ``Tools/scripts/run_tests.py`` script as found in a CPython checkout. The script tries to balance speed with thoroughness. But if you want the most thorough tests you should use the strenuous approach shown -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Fri Jan 11 13:12:55 2013 From: python-checkins at python.org (ezio.melotti) Date: Fri, 11 Jan 2013 13:12:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Add_a_note_abo?= =?utf-8?q?ut_checking_refleaks_to_patchcheck=2E?= Message-ID: <3YjNJ71v55zP57@mail.python.org> http://hg.python.org/cpython/rev/1cfe0f50fd0c changeset: 81414:1cfe0f50fd0c branch: 2.7 parent: 81408:4a86efa5a97f user: Ezio Melotti date: Fri Jan 11 14:07:47 2013 +0200 summary: Add a note about checking refleaks to patchcheck. files: Tools/scripts/patchcheck.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Tools/scripts/patchcheck.py b/Tools/scripts/patchcheck.py --- a/Tools/scripts/patchcheck.py +++ b/Tools/scripts/patchcheck.py @@ -174,8 +174,9 @@ # Test suite run and passed. if python_files or c_files: + end = " and check for refleaks?" if c_files else "?" print - print "Did you run the test suite?" + print "Did you run the test suite" + end if __name__ == '__main__': -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 13:12:56 2013 From: python-checkins at python.org (ezio.melotti) Date: Fri, 11 Jan 2013 13:12:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E2=29=3A_Add_a_note_abo?= =?utf-8?q?ut_checking_refleaks_to_patchcheck=2E?= Message-ID: <3YjNJ84QyfzS6v@mail.python.org> http://hg.python.org/cpython/rev/0ed68fb13b3a changeset: 81415:0ed68fb13b3a branch: 3.2 parent: 81409:973f3a253ab0 user: Ezio Melotti date: Fri Jan 11 14:07:47 2013 +0200 summary: Add a note about checking refleaks to patchcheck. files: Tools/scripts/patchcheck.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Tools/scripts/patchcheck.py b/Tools/scripts/patchcheck.py --- a/Tools/scripts/patchcheck.py +++ b/Tools/scripts/patchcheck.py @@ -170,8 +170,9 @@ # Test suite run and passed. if python_files or c_files: + end = " and check for refleaks?" if c_files else "?" print() - print("Did you run the test suite?") + print("Did you run the test suite" + end) if __name__ == '__main__': -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 13:12:58 2013 From: python-checkins at python.org (ezio.melotti) Date: Fri, 11 Jan 2013 13:12:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Merge_patchcheck_changes_with_3=2E2?= Message-ID: <3YjNJB0Bq8zS6x@mail.python.org> http://hg.python.org/cpython/rev/d3374d393975 changeset: 81416:d3374d393975 branch: 3.3 parent: 81412:086c6e8d0a24 parent: 81415:0ed68fb13b3a user: Ezio Melotti date: Fri Jan 11 14:12:20 2013 +0200 summary: Merge patchcheck changes with 3.2 files: Tools/scripts/patchcheck.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Tools/scripts/patchcheck.py b/Tools/scripts/patchcheck.py --- a/Tools/scripts/patchcheck.py +++ b/Tools/scripts/patchcheck.py @@ -173,8 +173,9 @@ # Test suite run and passed. if python_files or c_files: + end = " and check for refleaks?" if c_files else "?" print() - print("Did you run the test suite?") + print("Did you run the test suite" + end) if __name__ == '__main__': -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 13:12:59 2013 From: python-checkins at python.org (ezio.melotti) Date: Fri, 11 Jan 2013 13:12:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge_patchcheck_changes_with_3=2E3?= Message-ID: <3YjNJC3LwFzS6x@mail.python.org> http://hg.python.org/cpython/rev/f9feb11130e7 changeset: 81417:f9feb11130e7 parent: 81413:59d9d970b2d0 parent: 81416:d3374d393975 user: Ezio Melotti date: Fri Jan 11 14:12:42 2013 +0200 summary: Merge patchcheck changes with 3.3 files: Tools/scripts/patchcheck.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Tools/scripts/patchcheck.py b/Tools/scripts/patchcheck.py --- a/Tools/scripts/patchcheck.py +++ b/Tools/scripts/patchcheck.py @@ -173,8 +173,9 @@ # Test suite run and passed. if python_files or c_files: + end = " and check for refleaks?" if c_files else "?" print() - print("Did you run the test suite?") + print("Did you run the test suite" + end) if __name__ == '__main__': -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 14:31:30 2013 From: python-checkins at python.org (tim.golden) Date: Fri, 11 Jan 2013 14:31:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_=2316921_Since_Win9x_is_no?= =?utf-8?q?_longer_supported=2C_CREATE=5FNEW=5FCONSOLE_is?= Message-ID: <3YjQ2p0KW5zS2B@mail.python.org> http://hg.python.org/cpython/rev/1aac333f320a changeset: 81418:1aac333f320a user: Tim Golden date: Fri Jan 11 13:28:31 2013 +0000 summary: #16921 Since Win9x is no longer supported, CREATE_NEW_CONSOLE is no longer implied by shell=True files: Doc/library/subprocess.rst | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -826,8 +826,6 @@ The new process has a new console, instead of inheriting its parent's console (the default). - This flag is always set when :class:`Popen` is created with ``shell=True``. - .. data:: CREATE_NEW_PROCESS_GROUP A :class:`Popen` ``creationflags`` parameter to specify that a new process -- Repository URL: http://hg.python.org/cpython From christian at python.org Fri Jan 11 17:08:30 2013 From: christian at python.org (Christian Heimes) Date: Fri, 11 Jan 2013 17:08:30 +0100 Subject: [Python-checkins] Daily reference leaks (aef7db0d3893): sum=287 In-Reply-To: References:

Message-ID: <50F038FE.208@python.org> Am 11.01.2013 07:09, schrieb Nick Coghlan: > On Fri, Jan 11, 2013 at 2:57 PM, wrote: >> results for aef7db0d3893 on branch "default" >> -------------------------------------------- >> >> test_dbm leaked [2, 0, 0] references, sum=2 >> test_dbm leaked [2, 2, 1] memory blocks, sum=5 > > Hmm, I'm starting to wonder if there's something to this one - it > seems to be popping up a bit lately. > >> test_xml_etree_c leaked [56, 56, 56] references, sum=168 >> test_xml_etree_c leaked [36, 38, 38] memory blocks, sum=112 > > I'm gonna take a wild guess and suggest there may be a problem with > the recent pickling fix in the C extension :) It has more issues. Coverity has sent me some complains, see attachment. -------------- next part -------------- An embedded message was scrubbed... From: scan-admin at coverity.com Subject: New Defects reported by Coverity Scan for Python Date: Thu, 10 Jan 2013 16:35:29 -0800 Size: 4865 URL: From python-checkins at python.org Fri Jan 11 17:18:01 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 11 Jan 2013 17:18:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=233583=3A_mention_t?= =?utf-8?q?hat_testing_whether_a_bad_address_not_triggering?= Message-ID: <3YjTkx3ldXzPHs@mail.python.org> http://hg.python.org/cpython/rev/acce13a6e728 changeset: 81419:acce13a6e728 user: Brett Cannon date: Fri Jan 11 11:17:53 2013 -0500 summary: Issue #3583: mention that testing whether a bad address not triggering an OSError can mean a bad DNS server and not an actual bug. files: Lib/test/test_urllibnet.py | 19 +++++++++---------- 1 files changed, 9 insertions(+), 10 deletions(-) diff --git a/Lib/test/test_urllibnet.py b/Lib/test/test_urllibnet.py --- a/Lib/test/test_urllibnet.py +++ b/Lib/test/test_urllibnet.py @@ -121,16 +121,15 @@ else: # This happens with some overzealous DNS providers such as OpenDNS self.skipTest("%r should not resolve for test to work" % bogus_domain) - self.assertRaises(OSError, - # SF patch 809915: In Sep 2003, VeriSign started - # highjacking invalid .com and .net addresses to - # boost traffic to their own site. This test - # started failing then. One hopes the .invalid - # domain will be spared to serve its defined - # purpose. - # urllib.urlopen, "http://www.sadflkjsasadf.com/") - urllib.request.urlopen, - "http://sadflkjsasf.i.nvali.d/") + failure_explanation = ('opening an invalid URL did not raise OSError; ' + 'can be caused by a broken DNS server ' + '(e.g. returns 404 or hijacks page)') + with self.assertRaises(OSError, msg=failure_explanation): + # SF patch 809915: In Sep 2003, VeriSign started highjacking + # invalid .com and .net addresses to boost traffic to their own + # site. This test started failing then. One hopes the .invalid + # domain will be spared to serve its defined purpose. + urllib.request.urlopen("http://sadflkjsasf.i.nvali.d/") class urlretrieveNetworkTests(unittest.TestCase): -- Repository URL: http://hg.python.org/cpython From brett at python.org Fri Jan 11 18:08:18 2013 From: brett at python.org (Brett Cannon) Date: Fri, 11 Jan 2013 12:08:18 -0500 Subject: [Python-checkins] cpython (merge 3.3 -> default): Issue #15539: Fix a number of bugs in Tools/scripts/pindent.py. In-Reply-To: <3YjLF866ftzS6g@mail.python.org> References: <3YjLF866ftzS6g@mail.python.org> Message-ID: This seems to have caused the Windows buildbots to fail. On Fri, Jan 11, 2013 at 5:40 AM, serhiy.storchaka wrote: > http://hg.python.org/cpython/rev/8452c23139c6 > changeset: 81407:8452c23139c6 > parent: 81399:5ec8daab477a > parent: 81406:01df1f7841b2 > user: Serhiy Storchaka > date: Fri Jan 11 12:12:32 2013 +0200 > summary: > Issue #15539: Fix a number of bugs in Tools/scripts/pindent.py. > Now pindent.py works with a "with" statement. pindent.py no longer produces > improper indentation. pindent.py now works with continued lines broken after > "class" or "def" keywords and with continuations at the start of line. Added > regression tests for pindent.py. Modernized pindent.py. > > files: > Lib/test/test_tools.py | 326 ++++++++++++++++++++++++++- > Misc/NEWS | 7 + > Tools/scripts/pindent.py | 166 +++++-------- > 3 files changed, 393 insertions(+), 106 deletions(-) > > > diff --git a/Lib/test/test_tools.py b/Lib/test/test_tools.py > --- a/Lib/test/test_tools.py > +++ b/Lib/test/test_tools.py > @@ -9,10 +9,13 @@ > import importlib.machinery > import unittest > from unittest import mock > +import shutil > +import subprocess > import sysconfig > import tempfile > +import textwrap > from test import support > -from test.script_helper import assert_python_ok > +from test.script_helper import assert_python_ok, temp_dir > > if not sysconfig.is_python_build(): > # XXX some installers do contain the tools, should we detect that > @@ -36,6 +39,327 @@ > self.assertGreater(err, b'') > > > +class PindentTests(unittest.TestCase): > + script = os.path.join(scriptsdir, 'pindent.py') > + > + def assertFileEqual(self, fn1, fn2): > + with open(fn1) as f1, open(fn2) as f2: > + self.assertEqual(f1.readlines(), f2.readlines()) > + > + def pindent(self, source, *args): > + with subprocess.Popen( > + (sys.executable, self.script) + args, > + stdin=subprocess.PIPE, stdout=subprocess.PIPE, > + universal_newlines=True) as proc: > + out, err = proc.communicate(source) > + self.assertIsNone(err) > + return out > + > + def lstriplines(self, data): > + return '\n'.join(line.lstrip() for line in data.splitlines()) + '\n' > + > + def test_selftest(self): > + with temp_dir() as directory: > + data_path = os.path.join(directory, '_test.py') > + with open(self.script) as f: > + closed = f.read() > + with open(data_path, 'w') as f: > + f.write(closed) > + > + rc, out, err = assert_python_ok(self.script, '-d', data_path) > + self.assertEqual(out, b'') > + self.assertEqual(err, b'') > + backup = data_path + '~' > + self.assertTrue(os.path.exists(backup)) > + with open(backup) as f: > + self.assertEqual(f.read(), closed) > + with open(data_path) as f: > + clean = f.read() > + compile(clean, '_test.py', 'exec') > + self.assertEqual(self.pindent(clean, '-c'), closed) > + self.assertEqual(self.pindent(closed, '-d'), clean) > + > + rc, out, err = assert_python_ok(self.script, '-c', data_path) > + self.assertEqual(out, b'') > + self.assertEqual(err, b'') > + with open(backup) as f: > + self.assertEqual(f.read(), clean) > + with open(data_path) as f: > + self.assertEqual(f.read(), closed) > + > + broken = self.lstriplines(closed) > + with open(data_path, 'w') as f: > + f.write(broken) > + rc, out, err = assert_python_ok(self.script, '-r', data_path) > + self.assertEqual(out, b'') > + self.assertEqual(err, b'') > + with open(backup) as f: > + self.assertEqual(f.read(), broken) > + with open(data_path) as f: > + indented = f.read() > + compile(indented, '_test.py', 'exec') > + self.assertEqual(self.pindent(broken, '-r'), indented) > + > + def pindent_test(self, clean, closed): > + self.assertEqual(self.pindent(clean, '-c'), closed) > + self.assertEqual(self.pindent(closed, '-d'), clean) > + broken = self.lstriplines(closed) > + self.assertEqual(self.pindent(broken, '-r', '-e', '-s', '4'), closed) > + > + def test_statements(self): > + clean = textwrap.dedent("""\ > + if a: > + pass > + > + if a: > + pass > + else: > + pass > + > + if a: > + pass > + elif: > + pass > + else: > + pass > + > + while a: > + break > + > + while a: > + break > + else: > + pass > + > + for i in a: > + break > + > + for i in a: > + break > + else: > + pass > + > + try: > + pass > + finally: > + pass > + > + try: > + pass > + except TypeError: > + pass > + except ValueError: > + pass > + else: > + pass > + > + try: > + pass > + except TypeError: > + pass > + except ValueError: > + pass > + finally: > + pass > + > + with a: > + pass > + > + class A: > + pass > + > + def f(): > + pass > + """) > + > + closed = textwrap.dedent("""\ > + if a: > + pass > + # end if > + > + if a: > + pass > + else: > + pass > + # end if > + > + if a: > + pass > + elif: > + pass > + else: > + pass > + # end if > + > + while a: > + break > + # end while > + > + while a: > + break > + else: > + pass > + # end while > + > + for i in a: > + break > + # end for > + > + for i in a: > + break > + else: > + pass > + # end for > + > + try: > + pass > + finally: > + pass > + # end try > + > + try: > + pass > + except TypeError: > + pass > + except ValueError: > + pass > + else: > + pass > + # end try > + > + try: > + pass > + except TypeError: > + pass > + except ValueError: > + pass > + finally: > + pass > + # end try > + > + with a: > + pass > + # end with > + > + class A: > + pass > + # end class A > + > + def f(): > + pass > + # end def f > + """) > + self.pindent_test(clean, closed) > + > + def test_multilevel(self): > + clean = textwrap.dedent("""\ > + def foobar(a, b): > + if a == b: > + a = a+1 > + elif a < b: > + b = b-1 > + if b > a: a = a-1 > + else: > + print 'oops!' > + """) > + closed = textwrap.dedent("""\ > + def foobar(a, b): > + if a == b: > + a = a+1 > + elif a < b: > + b = b-1 > + if b > a: a = a-1 > + # end if > + else: > + print 'oops!' > + # end if > + # end def foobar > + """) > + self.pindent_test(clean, closed) > + > + def test_preserve_indents(self): > + clean = textwrap.dedent("""\ > + if a: > + if b: > + pass > + """) > + closed = textwrap.dedent("""\ > + if a: > + if b: > + pass > + # end if > + # end if > + """) > + self.assertEqual(self.pindent(clean, '-c'), closed) > + self.assertEqual(self.pindent(closed, '-d'), clean) > + broken = self.lstriplines(closed) > + self.assertEqual(self.pindent(broken, '-r', '-e', '-s', '9'), closed) > + clean = textwrap.dedent("""\ > + if a: > + \tif b: > + \t\tpass > + """) > + closed = textwrap.dedent("""\ > + if a: > + \tif b: > + \t\tpass > + \t# end if > + # end if > + """) > + self.assertEqual(self.pindent(clean, '-c'), closed) > + self.assertEqual(self.pindent(closed, '-d'), clean) > + broken = self.lstriplines(closed) > + self.assertEqual(self.pindent(broken, '-r'), closed) > + > + def test_escaped_newline(self): > + clean = textwrap.dedent("""\ > + class\\ > + \\ > + A: > + def\ > + \\ > + f: > + pass > + """) > + closed = textwrap.dedent("""\ > + class\\ > + \\ > + A: > + def\ > + \\ > + f: > + pass > + # end def f > + # end class A > + """) > + self.assertEqual(self.pindent(clean, '-c'), closed) > + self.assertEqual(self.pindent(closed, '-d'), clean) > + > + def test_empty_line(self): > + clean = textwrap.dedent("""\ > + if a: > + > + pass > + """) > + closed = textwrap.dedent("""\ > + if a: > + > + pass > + # end if > + """) > + self.pindent_test(clean, closed) > + > + def test_oneline(self): > + clean = textwrap.dedent("""\ > + if a: pass > + """) > + closed = textwrap.dedent("""\ > + if a: pass > + # end if > + """) > + self.pindent_test(clean, closed) > + > + > class TestSundryScripts(unittest.TestCase): > # At least make sure the rest don't have syntax errors. When tests are > # added for a script it should be added to the whitelist below. > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -621,6 +621,8 @@ > Tests > ----- > > +- Issue #15539: Added regression tests for Tools/scripts/pindent.py. > + > - Issue #16836: Enable IPv6 support even if IPv6 is disabled on the build host. > > - Issue #16925: test_configparser now works with unittest test discovery. > @@ -777,6 +779,11 @@ > Tools/Demos > ----------- > > +- Issue #15539: Fix a number of bugs in Tools/scripts/pindent.py. Now > + pindent.py works with a "with" statement. pindent.py no longer produces > + improper indentation. pindent.py now works with continued lines broken after > + "class" or "def" keywords and with continuations at the start of line. > + > - Issue #11797: Add a 2to3 fixer that maps reload() to imp.reload(). > > - Issue #10966: Remove the concept of unexpected skipped tests. > diff --git a/Tools/scripts/pindent.py b/Tools/scripts/pindent.py > --- a/Tools/scripts/pindent.py > +++ b/Tools/scripts/pindent.py > @@ -79,8 +79,9 @@ > # Defaults > STEPSIZE = 8 > TABSIZE = 8 > -EXPANDTABS = 0 > +EXPANDTABS = False > > +import io > import re > import sys > > @@ -89,7 +90,8 @@ > next['while'] = next['for'] = 'else', 'end' > next['try'] = 'except', 'finally' > next['except'] = 'except', 'else', 'finally', 'end' > -next['else'] = next['finally'] = next['def'] = next['class'] = 'end' > +next['else'] = next['finally'] = next['with'] = \ > + next['def'] = next['class'] = 'end' > next['end'] = () > start = 'if', 'while', 'for', 'try', 'with', 'def', 'class' > > @@ -105,11 +107,11 @@ > self.expandtabs = expandtabs > self._write = fpo.write > self.kwprog = re.compile( > - r'^\s*(?P[a-z]+)' > - r'(\s+(?P[a-zA-Z_]\w*))?' > + r'^(?:\s|\\\n)*(?P[a-z]+)' > + r'((?:\s|\\\n)+(?P[a-zA-Z_]\w*))?' > r'[^\w]') > self.endprog = re.compile( > - r'^\s*#?\s*end\s+(?P[a-z]+)' > + r'^(?:\s|\\\n)*#?\s*end\s+(?P[a-z]+)' > r'(\s+(?P[a-zA-Z_]\w*))?' > r'[^\w]') > self.wsprog = re.compile(r'^[ \t]*') > @@ -125,7 +127,7 @@ > > def readline(self): > line = self.fpi.readline() > - if line: self.lineno = self.lineno + 1 > + if line: self.lineno += 1 > # end if > return line > # end def readline > @@ -143,27 +145,24 @@ > line2 = self.readline() > if not line2: break > # end if > - line = line + line2 > + line += line2 > # end while > return line > # end def getline > > - def putline(self, line, indent = None): > - if indent is None: > - self.write(line) > - return > + def putline(self, line, indent): > + tabs, spaces = divmod(indent*self.indentsize, self.tabsize) > + i = self.wsprog.match(line).end() > + line = line[i:] > + if line[:1] not in ('\n', '\r', ''): > + line = '\t'*tabs + ' '*spaces + line > # end if > - tabs, spaces = divmod(indent*self.indentsize, self.tabsize) > - i = 0 > - m = self.wsprog.match(line) > - if m: i = m.end() > - # end if > - self.write('\t'*tabs + ' '*spaces + line[i:]) > + self.write(line) > # end def putline > > def reformat(self): > stack = [] > - while 1: > + while True: > line = self.getline() > if not line: break # EOF > # end if > @@ -173,10 +172,9 @@ > kw2 = m.group('kw') > if not stack: > self.error('unexpected end') > - elif stack[-1][0] != kw2: > + elif stack.pop()[0] != kw2: > self.error('unmatched end') > # end if > - del stack[-1:] > self.putline(line, len(stack)) > continue > # end if > @@ -208,23 +206,23 @@ > def delete(self): > begin_counter = 0 > end_counter = 0 > - while 1: > + while True: > line = self.getline() > if not line: break # EOF > # end if > m = self.endprog.match(line) > if m: > - end_counter = end_counter + 1 > + end_counter += 1 > continue > # end if > m = self.kwprog.match(line) > if m: > kw = m.group('kw') > if kw in start: > - begin_counter = begin_counter + 1 > + begin_counter += 1 > # end if > # end if > - self.putline(line) > + self.write(line) > # end while > if begin_counter - end_counter < 0: > sys.stderr.write('Warning: input contained more end tags than expected\n') > @@ -234,17 +232,12 @@ > # end def delete > > def complete(self): > - self.indentsize = 1 > stack = [] > todo = [] > - thisid = '' > - current, firstkw, lastkw, topid = 0, '', '', '' > - while 1: > + currentws = thisid = firstkw = lastkw = topid = '' > + while True: > line = self.getline() > - i = 0 > - m = self.wsprog.match(line) > - if m: i = m.end() > - # end if > + i = self.wsprog.match(line).end() > m = self.endprog.match(line) > if m: > thiskw = 'end' > @@ -269,7 +262,9 @@ > thiskw = '' > # end if > # end if > - indent = len(line[:i].expandtabs(self.tabsize)) > + indentws = line[:i] > + indent = len(indentws.expandtabs(self.tabsize)) > + current = len(currentws.expandtabs(self.tabsize)) > while indent < current: > if firstkw: > if topid: > @@ -278,11 +273,11 @@ > else: > s = '# end %s\n' % firstkw > # end if > - self.putline(s, current) > + self.write(currentws + s) > firstkw = lastkw = '' > # end if > - current, firstkw, lastkw, topid = stack[-1] > - del stack[-1] > + currentws, firstkw, lastkw, topid = stack.pop() > + current = len(currentws.expandtabs(self.tabsize)) > # end while > if indent == current and firstkw: > if thiskw == 'end': > @@ -297,18 +292,18 @@ > else: > s = '# end %s\n' % firstkw > # end if > - self.putline(s, current) > + self.write(currentws + s) > firstkw = lastkw = topid = '' > # end if > # end if > if indent > current: > - stack.append((current, firstkw, lastkw, topid)) > + stack.append((currentws, firstkw, lastkw, topid)) > if thiskw and thiskw not in start: > # error > thiskw = '' > # end if > - current, firstkw, lastkw, topid = \ > - indent, thiskw, thiskw, thisid > + currentws, firstkw, lastkw, topid = \ > + indentws, thiskw, thiskw, thisid > # end if > if thiskw: > if thiskw in start: > @@ -326,7 +321,6 @@ > self.write(line) > # end while > # end def complete > - > # end class PythonIndenter > > # Simplified user interface > @@ -352,76 +346,34 @@ > pi.reformat() > # end def reformat_filter > > -class StringReader: > - def __init__(self, buf): > - self.buf = buf > - self.pos = 0 > - self.len = len(self.buf) > - # end def __init__ > - def read(self, n = 0): > - if n <= 0: > - n = self.len - self.pos > - else: > - n = min(n, self.len - self.pos) > - # end if > - r = self.buf[self.pos : self.pos + n] > - self.pos = self.pos + n > - return r > - # end def read > - def readline(self): > - i = self.buf.find('\n', self.pos) > - return self.read(i + 1 - self.pos) > - # end def readline > - def readlines(self): > - lines = [] > - line = self.readline() > - while line: > - lines.append(line) > - line = self.readline() > - # end while > - return lines > - # end def readlines > - # seek/tell etc. are left as an exercise for the reader > -# end class StringReader > - > -class StringWriter: > - def __init__(self): > - self.buf = '' > - # end def __init__ > - def write(self, s): > - self.buf = self.buf + s > - # end def write > - def getvalue(self): > - return self.buf > - # end def getvalue > -# end class StringWriter > - > def complete_string(source, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): > - input = StringReader(source) > - output = StringWriter() > + input = io.StringIO(source) > + output = io.StringIO() > pi = PythonIndenter(input, output, stepsize, tabsize, expandtabs) > pi.complete() > return output.getvalue() > # end def complete_string > > def delete_string(source, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): > - input = StringReader(source) > - output = StringWriter() > + input = io.StringIO(source) > + output = io.StringIO() > pi = PythonIndenter(input, output, stepsize, tabsize, expandtabs) > pi.delete() > return output.getvalue() > # end def delete_string > > def reformat_string(source, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): > - input = StringReader(source) > - output = StringWriter() > + input = io.StringIO(source) > + output = io.StringIO() > pi = PythonIndenter(input, output, stepsize, tabsize, expandtabs) > pi.reformat() > return output.getvalue() > # end def reformat_string > > def complete_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): > - source = open(filename, 'r').read() > + with open(filename, 'r') as f: > + source = f.read() > + # end with > result = complete_string(source, stepsize, tabsize, expandtabs) > if source == result: return 0 > # end if > @@ -429,14 +381,16 @@ > try: os.rename(filename, filename + '~') > except OSError: pass > # end try > - f = open(filename, 'w') > - f.write(result) > - f.close() > + with open(filename, 'w') as f: > + f.write(result) > + # end with > return 1 > # end def complete_file > > def delete_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): > - source = open(filename, 'r').read() > + with open(filename, 'r') as f: > + source = f.read() > + # end with > result = delete_string(source, stepsize, tabsize, expandtabs) > if source == result: return 0 > # end if > @@ -444,14 +398,16 @@ > try: os.rename(filename, filename + '~') > except OSError: pass > # end try > - f = open(filename, 'w') > - f.write(result) > - f.close() > + with open(filename, 'w') as f: > + f.write(result) > + # end with > return 1 > # end def delete_file > > def reformat_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): > - source = open(filename, 'r').read() > + with open(filename, 'r') as f: > + source = f.read() > + # end with > result = reformat_string(source, stepsize, tabsize, expandtabs) > if source == result: return 0 > # end if > @@ -459,9 +415,9 @@ > try: os.rename(filename, filename + '~') > except OSError: pass > # end try > - f = open(filename, 'w') > - f.write(result) > - f.close() > + with open(filename, 'w') as f: > + f.write(result) > + # end with > return 1 > # end def reformat_file > > @@ -474,7 +430,7 @@ > -r : reformat a completed program (use #end directives) > -s stepsize: indentation step (default %(STEPSIZE)d) > -t tabsize : the worth in spaces of a tab (default %(TABSIZE)d) > --e : expand TABs into spaces (defailt OFF) > +-e : expand TABs into spaces (default OFF) > [file] ... : files are changed in place, with backups in file~ > If no files are specified or a single - is given, > the program acts as a filter (reads stdin, writes stdout). > @@ -517,7 +473,7 @@ > elif o == '-t': > tabsize = int(a) > elif o == '-e': > - expandtabs = 1 > + expandtabs = True > # end if > # end for > if not action: > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > From eliben at gmail.com Fri Jan 11 18:36:06 2013 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 11 Jan 2013 09:36:06 -0800 Subject: [Python-checkins] Daily reference leaks (aef7db0d3893): sum=287 In-Reply-To: References:

Message-ID: On Thu, Jan 10, 2013 at 10:09 PM, Nick Coghlan wrote: > On Fri, Jan 11, 2013 at 2:57 PM, wrote: > > results for aef7db0d3893 on branch "default" > > -------------------------------------------- > > > > test_dbm leaked [2, 0, 0] references, sum=2 > > test_dbm leaked [2, 2, 1] memory blocks, sum=5 > > Hmm, I'm starting to wonder if there's something to this one - it > seems to be popping up a bit lately. > > > test_xml_etree_c leaked [56, 56, 56] references, sum=168 > > test_xml_etree_c leaked [36, 38, 38] memory blocks, sum=112 > > I'm gonna take a wild guess and suggest there may be a problem with > the recent pickling fix in the C extension :) > Yep. We're on it :-) Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Fri Jan 11 18:42:20 2013 From: christian at python.org (Christian Heimes) Date: Fri, 11 Jan 2013 18:42:20 +0100 Subject: [Python-checkins] Daily reference leaks (aef7db0d3893): sum=287 In-Reply-To: References:

<50F038FE.208@python.org> Message-ID: <50F04EFC.80505@python.org> Am 11.01.2013 18:19, schrieb Andrea Griffini: > On Fri, Jan 11, 2013 at 5:08 PM, Christian Heimes wrote: >> It has more issues. Coverity has sent me some complains, see attachment. > > The second complaint seems a false positive; if self->extra is null > then children is set to 0 and following code is not executed. Yes, Coverity's results have lots of false positives. On the other hand it's able to detect hard to find resource leaks. You get used to the false positives and learn to appreciate the good stuff. :) From python-checkins at python.org Fri Jan 11 19:07:31 2013 From: python-checkins at python.org (georg.brandl) Date: Fri, 11 Jan 2013 19:07:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Closes_=2316916=3A_clarify_?= =?utf-8?q?=22slicing_equivalent_to_extended_unpacking=22_example=3A_the?= Message-ID: <3YjX9H2bqTzP5Y@mail.python.org> http://hg.python.org/peps/rev/a0077c1d201d changeset: 4664:a0077c1d201d user: Georg Brandl date: Fri Jan 11 19:08:14 2013 +0100 summary: Closes #16916: clarify "slicing equivalent to extended unpacking" example: the latter always creates a list. files: pep-3132.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-3132.txt b/pep-3132.txt --- a/pep-3132.txt +++ b/pep-3132.txt @@ -68,7 +68,7 @@ For example, if ``seq`` is a slicable sequence, all the following assignments are equivalent if ``seq`` has at least three elements:: - a, b, c = seq[0], seq[1:-1], seq[-1] + a, b, c = seq[0], list(seq[1:-1]), seq[-1] a, *b, c = seq [a, *b, c] = seq -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri Jan 11 20:26:01 2013 From: python-checkins at python.org (georg.brandl) Date: Fri, 11 Jan 2013 20:26:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogQ2xvc2VzICMxNjkz?= =?utf-8?q?6=3A_fix_duplicate/ambiguous_description_of_stat=2ES=5FIFMT_in_?= =?utf-8?q?the_docs=2E?= Message-ID: <3YjYvs4xLlzS9y@mail.python.org> http://hg.python.org/cpython/rev/0d7a8a4d6f30 changeset: 81420:0d7a8a4d6f30 branch: 2.7 parent: 81414:1cfe0f50fd0c user: Georg Brandl date: Fri Jan 11 20:25:54 2013 +0100 summary: Closes #16936: fix duplicate/ambiguous description of stat.S_IFMT in the docs. files: Doc/library/stat.rst | 4 ---- 1 files changed, 0 insertions(+), 4 deletions(-) diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -171,10 +171,6 @@ Use of the functions above is more portable than use of the first set of flags: -.. data:: S_IFMT - - Bit mask for the file type bit fields. - .. data:: S_IFSOCK Socket. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 20:26:03 2013 From: python-checkins at python.org (georg.brandl) Date: Fri, 11 Jan 2013 20:26:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogQ2xvc2VzICMxNjkz?= =?utf-8?q?6=3A_fix_duplicate/ambiguous_description_of_stat=2ES=5FIFMT_in_?= =?utf-8?q?the_docs=2E?= Message-ID: <3YjYvv0j8KzSBS@mail.python.org> http://hg.python.org/cpython/rev/3555391a9909 changeset: 81421:3555391a9909 branch: 3.3 parent: 81416:d3374d393975 user: Georg Brandl date: Fri Jan 11 20:25:54 2013 +0100 summary: Closes #16936: fix duplicate/ambiguous description of stat.S_IFMT in the docs. files: Doc/library/stat.rst | 4 ---- 1 files changed, 0 insertions(+), 4 deletions(-) diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -182,10 +182,6 @@ Use of the functions above is more portable than use of the first set of flags: -.. data:: S_IFMT - - Bit mask for the file type bit fields. - .. data:: S_IFSOCK Socket. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 20:26:04 2013 From: python-checkins at python.org (georg.brandl) Date: Fri, 11 Jan 2013 20:26:04 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_merge_with_3=2E3?= Message-ID: <3YjYvw3kZ6zSCG@mail.python.org> http://hg.python.org/cpython/rev/21abd27ca9ed changeset: 81422:21abd27ca9ed parent: 81419:acce13a6e728 parent: 81421:3555391a9909 user: Georg Brandl date: Fri Jan 11 20:26:54 2013 +0100 summary: merge with 3.3 files: Doc/library/stat.rst | 4 ---- 1 files changed, 0 insertions(+), 4 deletions(-) diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -182,10 +182,6 @@ Use of the functions above is more portable than use of the first set of flags: -.. data:: S_IFMT - - Bit mask for the file type bit fields. - .. data:: S_IFSOCK Socket. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 21:23:43 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 11 Jan 2013 21:23:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE1NTM5?= =?utf-8?q?=3A_Fix_backup_file_creation_in_pindent=2Epy_on_Windows?= Message-ID: <3YjbBR0mzmzS4V@mail.python.org> http://hg.python.org/cpython/rev/2d0c5f97af48 changeset: 81423:2d0c5f97af48 branch: 2.7 parent: 81420:0d7a8a4d6f30 user: Serhiy Storchaka date: Fri Jan 11 22:16:15 2013 +0200 summary: Issue #15539: Fix backup file creation in pindent.py on Windows files: Lib/test/test_tools.py | 1 + Tools/scripts/pindent.py | 34 ++++++++++++++++++--------- 2 files changed, 23 insertions(+), 12 deletions(-) diff --git a/Lib/test/test_tools.py b/Lib/test/test_tools.py --- a/Lib/test/test_tools.py +++ b/Lib/test/test_tools.py @@ -57,6 +57,7 @@ return '\n'.join(line.lstrip() for line in data.splitlines()) + '\n' def test_selftest(self): + self.maxDiff = None with temp_dir() as directory: data_path = os.path.join(directory, '_test.py') with open(self.script) as f: diff --git a/Tools/scripts/pindent.py b/Tools/scripts/pindent.py --- a/Tools/scripts/pindent.py +++ b/Tools/scripts/pindent.py @@ -76,6 +76,8 @@ # - realign comments # - optionally do much more thorough reformatting, a la C indent +from __future__ import print_function + # Defaults STEPSIZE = 8 TABSIZE = 8 @@ -370,6 +372,23 @@ return output.getvalue() # end def reformat_string +def make_backup(filename): + import os, os.path + backup = filename + '~' + if os.path.lexists(backup): + try: + os.remove(backup) + except os.error: + print("Can't remove backup %r" % (backup,), file=sys.stderr) + # end try + # end if + try: + os.rename(filename, backup) + except os.error: + print("Can't rename %r to %r" % (filename, backup), file=sys.stderr) + # end try +# end def make_backup + def complete_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): with open(filename, 'r') as f: source = f.read() @@ -377,10 +396,7 @@ result = complete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if - import os - try: os.rename(filename, filename + '~') - except os.error: pass - # end try + make_backup(filename) with open(filename, 'w') as f: f.write(result) # end with @@ -394,10 +410,7 @@ result = delete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if - import os - try: os.rename(filename, filename + '~') - except os.error: pass - # end try + make_backup(filename) with open(filename, 'w') as f: f.write(result) # end with @@ -411,10 +424,7 @@ result = reformat_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if - import os - try: os.rename(filename, filename + '~') - except os.error: pass - # end try + make_backup(filename) with open(filename, 'w') as f: f.write(result) # end with -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 21:23:44 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 11 Jan 2013 21:23:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE1NTM5?= =?utf-8?q?=3A_Fix_a_backup_file_creation_in_pindent=2Epy_on_Windows=2E?= Message-ID: <3YjbBS3qBjzS8x@mail.python.org> http://hg.python.org/cpython/rev/a9990104f728 changeset: 81424:a9990104f728 branch: 3.2 parent: 81415:0ed68fb13b3a user: Serhiy Storchaka date: Fri Jan 11 22:18:17 2013 +0200 summary: Issue #15539: Fix a backup file creation in pindent.py on Windows. files: Lib/test/test_tools.py | 1 + Tools/scripts/pindent.py | 32 +++++++++++++++++---------- 2 files changed, 21 insertions(+), 12 deletions(-) diff --git a/Lib/test/test_tools.py b/Lib/test/test_tools.py --- a/Lib/test/test_tools.py +++ b/Lib/test/test_tools.py @@ -58,6 +58,7 @@ return '\n'.join(line.lstrip() for line in data.splitlines()) + '\n' def test_selftest(self): + self.maxDiff = None with temp_dir() as directory: data_path = os.path.join(directory, '_test.py') with open(self.script) as f: diff --git a/Tools/scripts/pindent.py b/Tools/scripts/pindent.py --- a/Tools/scripts/pindent.py +++ b/Tools/scripts/pindent.py @@ -370,6 +370,23 @@ return output.getvalue() # end def reformat_string +def make_backup(filename): + import os, os.path + backup = filename + '~' + if os.path.lexists(backup): + try: + os.remove(backup) + except os.error: + print("Can't remove backup %r" % (backup,), file=sys.stderr) + # end try + # end if + try: + os.rename(filename, backup) + except os.error: + print("Can't rename %r to %r" % (filename, backup), file=sys.stderr) + # end try +# end def make_backup + def complete_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): with open(filename, 'r') as f: source = f.read() @@ -377,10 +394,7 @@ result = complete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if - import os - try: os.rename(filename, filename + '~') - except os.error: pass - # end try + make_backup(filename) with open(filename, 'w') as f: f.write(result) # end with @@ -394,10 +408,7 @@ result = delete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if - import os - try: os.rename(filename, filename + '~') - except os.error: pass - # end try + make_backup(filename) with open(filename, 'w') as f: f.write(result) # end with @@ -411,10 +422,7 @@ result = reformat_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if - import os - try: os.rename(filename, filename + '~') - except os.error: pass - # end try + make_backup(filename) with open(filename, 'w') as f: f.write(result) # end with -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 21:23:45 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 11 Jan 2013 21:23:45 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=2315539=3A_Fix_a_backup_file_creation_in_pindent=2Epy_o?= =?utf-8?q?n_Windows=2E?= Message-ID: <3YjbBT6dkrzS9G@mail.python.org> http://hg.python.org/cpython/rev/735361e8709e changeset: 81425:735361e8709e branch: 3.3 parent: 81421:3555391a9909 parent: 81424:a9990104f728 user: Serhiy Storchaka date: Fri Jan 11 22:19:07 2013 +0200 summary: Issue #15539: Fix a backup file creation in pindent.py on Windows. files: Lib/test/test_tools.py | 1 + Tools/scripts/pindent.py | 32 +++++++++++++++++---------- 2 files changed, 21 insertions(+), 12 deletions(-) diff --git a/Lib/test/test_tools.py b/Lib/test/test_tools.py --- a/Lib/test/test_tools.py +++ b/Lib/test/test_tools.py @@ -59,6 +59,7 @@ return '\n'.join(line.lstrip() for line in data.splitlines()) + '\n' def test_selftest(self): + self.maxDiff = None with temp_dir() as directory: data_path = os.path.join(directory, '_test.py') with open(self.script) as f: diff --git a/Tools/scripts/pindent.py b/Tools/scripts/pindent.py --- a/Tools/scripts/pindent.py +++ b/Tools/scripts/pindent.py @@ -370,6 +370,23 @@ return output.getvalue() # end def reformat_string +def make_backup(filename): + import os, os.path + backup = filename + '~' + if os.path.lexists(backup): + try: + os.remove(backup) + except os.error: + print("Can't remove backup %r" % (backup,), file=sys.stderr) + # end try + # end if + try: + os.rename(filename, backup) + except os.error: + print("Can't rename %r to %r" % (filename, backup), file=sys.stderr) + # end try +# end def make_backup + def complete_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): with open(filename, 'r') as f: source = f.read() @@ -377,10 +394,7 @@ result = complete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if - import os - try: os.rename(filename, filename + '~') - except os.error: pass - # end try + make_backup(filename) with open(filename, 'w') as f: f.write(result) # end with @@ -394,10 +408,7 @@ result = delete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if - import os - try: os.rename(filename, filename + '~') - except os.error: pass - # end try + make_backup(filename) with open(filename, 'w') as f: f.write(result) # end with @@ -411,10 +422,7 @@ result = reformat_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if - import os - try: os.rename(filename, filename + '~') - except os.error: pass - # end try + make_backup(filename) with open(filename, 'w') as f: f.write(result) # end with -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 21:23:47 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 11 Jan 2013 21:23:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2315539=3A_Fix_a_backup_file_creation_in_pindent?= =?utf-8?q?=2Epy_on_Windows=2E?= Message-ID: <3YjbBW71tFzS95@mail.python.org> http://hg.python.org/cpython/rev/c46dec051819 changeset: 81426:c46dec051819 parent: 81422:21abd27ca9ed parent: 81425:735361e8709e user: Serhiy Storchaka date: Fri Jan 11 22:21:45 2013 +0200 summary: Issue #15539: Fix a backup file creation in pindent.py on Windows. files: Lib/test/test_tools.py | 1 + Tools/scripts/pindent.py | 32 +++++++++++++++++---------- 2 files changed, 21 insertions(+), 12 deletions(-) diff --git a/Lib/test/test_tools.py b/Lib/test/test_tools.py --- a/Lib/test/test_tools.py +++ b/Lib/test/test_tools.py @@ -59,6 +59,7 @@ return '\n'.join(line.lstrip() for line in data.splitlines()) + '\n' def test_selftest(self): + self.maxDiff = None with temp_dir() as directory: data_path = os.path.join(directory, '_test.py') with open(self.script) as f: diff --git a/Tools/scripts/pindent.py b/Tools/scripts/pindent.py --- a/Tools/scripts/pindent.py +++ b/Tools/scripts/pindent.py @@ -370,6 +370,23 @@ return output.getvalue() # end def reformat_string +def make_backup(filename): + import os, os.path + backup = filename + '~' + if os.path.lexists(backup): + try: + os.remove(backup) + except OSError: + print("Can't remove backup %r" % (backup,), file=sys.stderr) + # end try + # end if + try: + os.rename(filename, backup) + except OSError: + print("Can't rename %r to %r" % (filename, backup), file=sys.stderr) + # end try +# end def make_backup + def complete_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): with open(filename, 'r') as f: source = f.read() @@ -377,10 +394,7 @@ result = complete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if - import os - try: os.rename(filename, filename + '~') - except OSError: pass - # end try + make_backup(filename) with open(filename, 'w') as f: f.write(result) # end with @@ -394,10 +408,7 @@ result = delete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if - import os - try: os.rename(filename, filename + '~') - except OSError: pass - # end try + make_backup(filename) with open(filename, 'w') as f: f.write(result) # end with @@ -411,10 +422,7 @@ result = reformat_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if - import os - try: os.rename(filename, filename + '~') - except OSError: pass - # end try + make_backup(filename) with open(filename, 'w') as f: f.write(result) # end with -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 21:45:13 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 11 Jan 2013 21:45:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE2NzMw?= =?utf-8?q?=3A_Don=27t_raise_an_exception_in?= Message-ID: <3YjbgF2cTqzS8m@mail.python.org> http://hg.python.org/cpython/rev/159967aa24a5 changeset: 81427:159967aa24a5 branch: 3.3 parent: 81416:d3374d393975 user: Brett Cannon date: Fri Jan 11 15:40:12 2013 -0500 summary: Issue #16730: Don't raise an exception in importlib.machinery.FileFinder when the directory has become unreadable or a file. This brings semantics in line with Python 3.2 import. Reported and diagnosed by David Pritchard. files: Lib/importlib/_bootstrap.py | 5 +- Lib/test/test_importlib/source/test_finder.py | 35 + Misc/NEWS | 5 + Python/importlib.h | 1901 +++++---- 4 files changed, 995 insertions(+), 951 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -1395,8 +1395,9 @@ path = self.path try: contents = _os.listdir(path) - except FileNotFoundError: - # Directory has been removed since last import + except (FileNotFoundError, PermissionError, NotADirectoryError): + # Directory has either been removed, turned into a file, or made + # unreadable. contents = [] # We store two cached versions, to handle runtime changes of the # PYTHONCASEOK environment variable. diff --git a/Lib/test/test_importlib/source/test_finder.py b/Lib/test/test_importlib/source/test_finder.py --- a/Lib/test/test_importlib/source/test_finder.py +++ b/Lib/test/test_importlib/source/test_finder.py @@ -6,6 +6,9 @@ import imp import os import py_compile +import stat +import sys +import tempfile from test.support import make_legacy_pyc import unittest import warnings @@ -147,6 +150,38 @@ self.assertIsNotNone(finder.find_module(mod)) self.assertIsNone(finder.find_module(mod)) + @unittest.skipUnless(sys.platform != 'win32', + 'os.chmod() does not support the needed arguments under Windows') + def test_no_read_directory(self): + # Issue #16730 + tempdir = tempfile.TemporaryDirectory() + original_mode = os.stat(tempdir.name).st_mode + def cleanup(tempdir): + """Cleanup function for the temporary directory. + + Since we muck with the permissions, we want to set them back to + their original values to make sure the directory can be properly + cleaned up. + + """ + os.chmod(tempdir.name, original_mode) + # If this is not explicitly called then the __del__ method is used, + # but since already mucking around might as well explicitly clean + # up. + tempdir.__exit__(None, None, None) + self.addCleanup(cleanup, tempdir) + os.chmod(tempdir.name, stat.S_IWUSR | stat.S_IXUSR) + finder = self.get_finder(tempdir.name) + self.assertEqual((None, []), finder.find_loader('doesnotexist')) + + def test_ignore_file(self): + # If a directory got changed to a file from underneath us, then don't + # worry about looking for submodules. + with tempfile.NamedTemporaryFile() as file_obj: + finder = self.get_finder(file_obj.name) + self.assertEqual((None, []), finder.find_loader('doesnotexist')) + + def test_main(): from test.support import run_unittest run_unittest(FinderTests) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -12,6 +12,11 @@ Core and Builtins ----------------- +- Issue #16730: importlib.machinery.FileFinder now no longers raises an + exception when trying to populate its cache and it finds out the directory is + unreadable or has turned into a file. Reported and diagnosed by + David Pritchard. + - Issue #16906: Fix a logic error that prevented most static strings from being cleared. diff --git a/Python/importlib.h b/Python/importlib.h --- a/Python/importlib.h +++ b/Python/importlib.h [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 21:45:14 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 11 Jan 2013 21:45:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge_from_3=2E3_for_fix_for_issue_=2316730?= Message-ID: <3YjbgG6rcZzS8w@mail.python.org> http://hg.python.org/cpython/rev/b94f308e9b47 changeset: 81428:b94f308e9b47 parent: 81419:acce13a6e728 parent: 81427:159967aa24a5 user: Brett Cannon date: Fri Jan 11 15:42:30 2013 -0500 summary: Merge from 3.3 for fix for issue #16730 files: Lib/importlib/_bootstrap.py | 5 +- Lib/test/test_importlib/source/test_finder.py | 35 + Misc/NEWS | 5 + Python/importlib.h | 1889 +++++---- 4 files changed, 989 insertions(+), 945 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -1401,8 +1401,9 @@ path = self.path try: contents = _os.listdir(path) - except FileNotFoundError: - # Directory has been removed since last import + except (FileNotFoundError, PermissionError, NotADirectoryError): + # Directory has either been removed, turned into a file, or made + # unreadable. contents = [] # We store two cached versions, to handle runtime changes of the # PYTHONCASEOK environment variable. diff --git a/Lib/test/test_importlib/source/test_finder.py b/Lib/test/test_importlib/source/test_finder.py --- a/Lib/test/test_importlib/source/test_finder.py +++ b/Lib/test/test_importlib/source/test_finder.py @@ -6,6 +6,9 @@ import imp import os import py_compile +import stat +import sys +import tempfile from test.support import make_legacy_pyc import unittest import warnings @@ -147,6 +150,38 @@ self.assertIsNotNone(finder.find_module(mod)) self.assertIsNone(finder.find_module(mod)) + @unittest.skipUnless(sys.platform != 'win32', + 'os.chmod() does not support the needed arguments under Windows') + def test_no_read_directory(self): + # Issue #16730 + tempdir = tempfile.TemporaryDirectory() + original_mode = os.stat(tempdir.name).st_mode + def cleanup(tempdir): + """Cleanup function for the temporary directory. + + Since we muck with the permissions, we want to set them back to + their original values to make sure the directory can be properly + cleaned up. + + """ + os.chmod(tempdir.name, original_mode) + # If this is not explicitly called then the __del__ method is used, + # but since already mucking around might as well explicitly clean + # up. + tempdir.__exit__(None, None, None) + self.addCleanup(cleanup, tempdir) + os.chmod(tempdir.name, stat.S_IWUSR | stat.S_IXUSR) + finder = self.get_finder(tempdir.name) + self.assertEqual((None, []), finder.find_loader('doesnotexist')) + + def test_ignore_file(self): + # If a directory got changed to a file from underneath us, then don't + # worry about looking for submodules. + with tempfile.NamedTemporaryFile() as file_obj: + finder = self.get_finder(file_obj.name) + self.assertEqual((None, []), finder.find_loader('doesnotexist')) + + def test_main(): from test.support import run_unittest run_unittest(FinderTests) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,11 @@ Core and Builtins ----------------- +- Issue #16730: importlib.machinery.FileFinder now no longers raises an + exception when trying to populate its cache and it finds out the directory is + unreadable or has turned into a file. Reported and diagnosed by + David Pritchard. + - Issue #16906: Fix a logic error that prevented most static strings from being cleared. diff --git a/Python/importlib.h b/Python/importlib.h --- a/Python/importlib.h +++ b/Python/importlib.h [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 21:45:16 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 11 Jan 2013 21:45:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?q?=29=3A_merge?= Message-ID: <3YjbgJ31X2zS9M@mail.python.org> http://hg.python.org/cpython/rev/e3d47d5b9110 changeset: 81429:e3d47d5b9110 parent: 81428:b94f308e9b47 parent: 81426:c46dec051819 user: Brett Cannon date: Fri Jan 11 15:43:34 2013 -0500 summary: merge files: Doc/library/stat.rst | 4 --- Lib/test/test_tools.py | 1 + Tools/scripts/pindent.py | 32 +++++++++++++++++---------- 3 files changed, 21 insertions(+), 16 deletions(-) diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -182,10 +182,6 @@ Use of the functions above is more portable than use of the first set of flags: -.. data:: S_IFMT - - Bit mask for the file type bit fields. - .. data:: S_IFSOCK Socket. diff --git a/Lib/test/test_tools.py b/Lib/test/test_tools.py --- a/Lib/test/test_tools.py +++ b/Lib/test/test_tools.py @@ -59,6 +59,7 @@ return '\n'.join(line.lstrip() for line in data.splitlines()) + '\n' def test_selftest(self): + self.maxDiff = None with temp_dir() as directory: data_path = os.path.join(directory, '_test.py') with open(self.script) as f: diff --git a/Tools/scripts/pindent.py b/Tools/scripts/pindent.py --- a/Tools/scripts/pindent.py +++ b/Tools/scripts/pindent.py @@ -370,6 +370,23 @@ return output.getvalue() # end def reformat_string +def make_backup(filename): + import os, os.path + backup = filename + '~' + if os.path.lexists(backup): + try: + os.remove(backup) + except OSError: + print("Can't remove backup %r" % (backup,), file=sys.stderr) + # end try + # end if + try: + os.rename(filename, backup) + except OSError: + print("Can't rename %r to %r" % (filename, backup), file=sys.stderr) + # end try +# end def make_backup + def complete_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): with open(filename, 'r') as f: source = f.read() @@ -377,10 +394,7 @@ result = complete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if - import os - try: os.rename(filename, filename + '~') - except OSError: pass - # end try + make_backup(filename) with open(filename, 'w') as f: f.write(result) # end with @@ -394,10 +408,7 @@ result = delete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if - import os - try: os.rename(filename, filename + '~') - except OSError: pass - # end try + make_backup(filename) with open(filename, 'w') as f: f.write(result) # end with @@ -411,10 +422,7 @@ result = reformat_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if - import os - try: os.rename(filename, filename + '~') - except OSError: pass - # end try + make_backup(filename) with open(filename, 'w') as f: f.write(result) # end with -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 21:45:17 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 11 Jan 2013 21:45:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuMyk6?= =?utf-8?q?_merge?= Message-ID: <3YjbgK5tPWzSBC@mail.python.org> http://hg.python.org/cpython/rev/9c695fddd10a changeset: 81430:9c695fddd10a branch: 3.3 parent: 81427:159967aa24a5 parent: 81425:735361e8709e user: Brett Cannon date: Fri Jan 11 15:44:00 2013 -0500 summary: merge files: Doc/library/stat.rst | 4 --- Lib/test/test_tools.py | 1 + Tools/scripts/pindent.py | 32 +++++++++++++++++---------- 3 files changed, 21 insertions(+), 16 deletions(-) diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -182,10 +182,6 @@ Use of the functions above is more portable than use of the first set of flags: -.. data:: S_IFMT - - Bit mask for the file type bit fields. - .. data:: S_IFSOCK Socket. diff --git a/Lib/test/test_tools.py b/Lib/test/test_tools.py --- a/Lib/test/test_tools.py +++ b/Lib/test/test_tools.py @@ -59,6 +59,7 @@ return '\n'.join(line.lstrip() for line in data.splitlines()) + '\n' def test_selftest(self): + self.maxDiff = None with temp_dir() as directory: data_path = os.path.join(directory, '_test.py') with open(self.script) as f: diff --git a/Tools/scripts/pindent.py b/Tools/scripts/pindent.py --- a/Tools/scripts/pindent.py +++ b/Tools/scripts/pindent.py @@ -370,6 +370,23 @@ return output.getvalue() # end def reformat_string +def make_backup(filename): + import os, os.path + backup = filename + '~' + if os.path.lexists(backup): + try: + os.remove(backup) + except os.error: + print("Can't remove backup %r" % (backup,), file=sys.stderr) + # end try + # end if + try: + os.rename(filename, backup) + except os.error: + print("Can't rename %r to %r" % (filename, backup), file=sys.stderr) + # end try +# end def make_backup + def complete_file(filename, stepsize = STEPSIZE, tabsize = TABSIZE, expandtabs = EXPANDTABS): with open(filename, 'r') as f: source = f.read() @@ -377,10 +394,7 @@ result = complete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if - import os - try: os.rename(filename, filename + '~') - except os.error: pass - # end try + make_backup(filename) with open(filename, 'w') as f: f.write(result) # end with @@ -394,10 +408,7 @@ result = delete_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if - import os - try: os.rename(filename, filename + '~') - except os.error: pass - # end try + make_backup(filename) with open(filename, 'w') as f: f.write(result) # end with @@ -411,10 +422,7 @@ result = reformat_string(source, stepsize, tabsize, expandtabs) if source == result: return 0 # end if - import os - try: os.rename(filename, filename + '~') - except os.error: pass - # end try + make_backup(filename) with open(filename, 'w') as f: f.write(result) # end with -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Jan 11 21:45:19 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 11 Jan 2013 21:45:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_merge?= Message-ID: <3YjbgM1PS9zS8p@mail.python.org> http://hg.python.org/cpython/rev/578d5ae1c8eb changeset: 81431:578d5ae1c8eb parent: 81429:e3d47d5b9110 parent: 81430:9c695fddd10a user: Brett Cannon date: Fri Jan 11 15:44:57 2013 -0500 summary: merge files: -- Repository URL: http://hg.python.org/cpython From agriff at tin.it Fri Jan 11 18:19:44 2013 From: agriff at tin.it (Andrea Griffini) Date: Fri, 11 Jan 2013 18:19:44 +0100 Subject: [Python-checkins] [Python-Dev] Daily reference leaks (aef7db0d3893): sum=287 In-Reply-To: <50F038FE.208@python.org> References:

<50F038FE.208@python.org> Message-ID: On Fri, Jan 11, 2013 at 5:08 PM, Christian Heimes wrote: > It has more issues. Coverity has sent me some complains, see attachment. The second complaint seems a false positive; if self->extra is null then children is set to 0 and following code is not executed. From python-checkins at python.org Sat Jan 12 00:09:34 2013 From: python-checkins at python.org (brett.cannon) Date: Sat, 12 Jan 2013 00:09:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2315031=3A_Refactor?= =?utf-8?q?_some_code_in_importlib_pertaining_to_validating?= Message-ID: <3Yjfsp6PBWzSBY@mail.python.org> http://hg.python.org/cpython/rev/a4292889e942 changeset: 81432:a4292889e942 user: Brett Cannon date: Fri Jan 11 18:09:25 2013 -0500 summary: Issue #15031: Refactor some code in importlib pertaining to validating and compiling bytecode. Thanks to Ronan Lamy for pointing the redundancy and taking an initial stab at the refactor (as did Nick Coghlan). files: Lib/importlib/_bootstrap.py | 139 +- Misc/NEWS | 4 + Python/importlib.h | 8560 +++++++++++----------- 3 files changed, 4375 insertions(+), 4328 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -623,6 +623,71 @@ return loader +def _validate_bytecode_header(data, source_stats=None, name=None, path=None): + """Validate the header of the passed-in bytecode against source_stats (if + given) and returning the bytecode that can be compiled by compile(). + + All other arguments are used to enhance error reporting. + + ImportError is raised when the magic number is incorrect or the bytecode is + found to be stale. EOFError is raised when the data is found to be + truncated. + + """ + exc_details = {} + if name is not None: + exc_details['name'] = name + else: + # To prevent having to make all messages have a conditional name. + name = 'bytecode' + if path is not None: + exc_details['path'] = path + magic = data[:4] + raw_timestamp = data[4:8] + raw_size = data[8:12] + if magic != _MAGIC_BYTES: + msg = 'bad magic number in {!r}: {!r}'.format(name, magic) + raise ImportError(msg, **exc_details) + elif len(raw_timestamp) != 4: + message = 'bad timestamp in {!r}'.format(name) + _verbose_message(message) + raise EOFError(message) + elif len(raw_size) != 4: + message = 'bad size in {!r}'.format(name) + _verbose_message(message) + raise EOFError(message) + if source_stats is not None: + try: + source_mtime = int(source_stats['mtime']) + except KeyError: + pass + else: + if _r_long(raw_timestamp) != source_mtime: + message = 'bytecode is stale for {!r}'.format(name) + _verbose_message(message) + raise ImportError(message, **exc_details) + try: + source_size = source_stats['size'] & 0xFFFFFFFF + except KeyError: + pass + else: + if _r_long(raw_size) != source_size: + raise ImportError("bytecode is stale for {!r}".format(name), + **exc_details) + return data[12:] + + +def _compile_bytecode(data, name=None, bytecode_path=None, source_path=None): + """Compile bytecode as returned by _validate_bytecode_header().""" + code = marshal.loads(data) + if isinstance(code, _code_type): + _verbose_message('code object from {!r}', bytecode_path) + if source_path is not None: + _imp._fix_co_filename(code, source_path) + return code + else: + raise ImportError("Non-code object in {!r}".format(bytecode_path), + name=name, path=bytecode_path) # Loaders ##################################################################### @@ -801,51 +866,6 @@ tail_name = fullname.rpartition('.')[2] return filename_base == '__init__' and tail_name != '__init__' - def _bytes_from_bytecode(self, fullname, data, bytecode_path, source_stats): - """Return the marshalled bytes from bytecode, verifying the magic - number, timestamp and source size along the way. - - If source_stats is None then skip the timestamp check. - - """ - magic = data[:4] - raw_timestamp = data[4:8] - raw_size = data[8:12] - if magic != _MAGIC_BYTES: - msg = 'bad magic number in {!r}: {!r}'.format(fullname, magic) - raise ImportError(msg, name=fullname, path=bytecode_path) - elif len(raw_timestamp) != 4: - message = 'bad timestamp in {}'.format(fullname) - _verbose_message(message) - raise EOFError(message) - elif len(raw_size) != 4: - message = 'bad size in {}'.format(fullname) - _verbose_message(message) - raise EOFError(message) - if source_stats is not None: - try: - source_mtime = int(source_stats['mtime']) - except KeyError: - pass - else: - if _r_long(raw_timestamp) != source_mtime: - message = 'bytecode is stale for {}'.format(fullname) - _verbose_message(message) - raise ImportError(message, name=fullname, - path=bytecode_path) - try: - source_size = source_stats['size'] & 0xFFFFFFFF - except KeyError: - pass - else: - if _r_long(raw_size) != source_size: - raise ImportError( - "bytecode is stale for {}".format(fullname), - name=fullname, path=bytecode_path) - # Can't return the code object as errors from marshal loading need to - # propagate even when source is available. - return data[12:] - @module_for_loader def _load_module(self, module, *, sourceless=False): """Helper for load_module able to handle either source or sourceless @@ -965,24 +985,17 @@ pass else: try: - bytes_data = self._bytes_from_bytecode(fullname, data, - bytecode_path, - st) + bytes_data = _validate_bytecode_header(data, + source_stats=st, name=fullname, + path=bytecode_path) except (ImportError, EOFError): pass else: _verbose_message('{} matches {}', bytecode_path, source_path) - found = marshal.loads(bytes_data) - if isinstance(found, _code_type): - _imp._fix_co_filename(found, source_path) - _verbose_message('code object from {}', - bytecode_path) - return found - else: - msg = "Non-code object in {}" - raise ImportError(msg.format(bytecode_path), - name=fullname, path=bytecode_path) + return _compile_bytecode(bytes_data, name=fullname, + bytecode_path=bytecode_path, + source_path=source_path) source_bytes = self.get_data(source_path) code_object = self.source_to_code(source_bytes, source_path) _verbose_message('code object from {}', source_path) @@ -1098,14 +1111,8 @@ def get_code(self, fullname): path = self.get_filename(fullname) data = self.get_data(path) - bytes_data = self._bytes_from_bytecode(fullname, data, path, None) - found = marshal.loads(bytes_data) - if isinstance(found, _code_type): - _verbose_message('code object from {!r}', path) - return found - else: - raise ImportError("Non-code object in {}".format(path), - name=fullname, path=path) + bytes_data = _validate_bytecode_header(data, name=fullname, path=path) + return _compile_bytecode(bytes_data, name=fullname, bytecode_path=path) def get_source(self, fullname): """Return None as there is no source code.""" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -217,6 +217,10 @@ Library ------- +- Issue #15031: Refactor some .pyc management code to cut down on code + duplication. Thanks to Ronan Lamy for the report and taking an initial stab + at the problem. + - Issue #16900: Issue a ResourceWarning when an ssl socket is left unclosed. - Issue #13899: \A, \Z, and \B now correctly match the A, Z, and B literals diff --git a/Python/importlib.h b/Python/importlib.h --- a/Python/importlib.h +++ b/Python/importlib.h [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 00:40:07 2013 From: python-checkins at python.org (vinay.sajip) Date: Sat, 12 Jan 2013 00:40:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Closes_=2316613=3A_Added_o?= =?utf-8?q?ptional_mapping_argument_to_ChainMap=2Enew=5Fchild=2E?= Message-ID: <3YjgY358JQzS9N@mail.python.org> http://hg.python.org/cpython/rev/c0ddae67f4df changeset: 81433:c0ddae67f4df user: Vinay Sajip date: Fri Jan 11 23:39:53 2013 +0000 summary: Closes #16613: Added optional mapping argument to ChainMap.new_child. files: Doc/library/collections.rst | 13 ++++++--- Lib/collections/__init__.py | 11 ++++++-- Lib/test/test_collections.py | 32 ++++++++++++++++++++++++ 3 files changed, 49 insertions(+), 7 deletions(-) diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -76,14 +76,19 @@ be modified to change which mappings are searched. The list should always contain at least one mapping. - .. method:: new_child() + .. method:: new_child(m=None) - Returns a new :class:`ChainMap` containing a new :class:`dict` followed by - all of the maps in the current instance. A call to ``d.new_child()`` is - equivalent to: ``ChainMap({}, *d.maps)``. This method is used for + Returns a new :class:`ChainMap` containing a new map followed by + all of the maps in the current instance. If ``m`` is specified, + it becomes the new map at the front of the list of mappings; if not + specified, an empty dict is used, so that a call to ``d.new_child()`` + is equivalent to: ``ChainMap({}, *d.maps)``. This method is used for creating subcontexts that can be updated without altering values in any of the parent mappings. + .. versionchanged:: 3.4 + The optional ``m`` parameter was added. + .. attribute:: parents Property returning a new :class:`ChainMap` containing all of the maps in diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py --- a/Lib/collections/__init__.py +++ b/Lib/collections/__init__.py @@ -821,9 +821,14 @@ __copy__ = copy - def new_child(self): # like Django's Context.push() - 'New ChainMap with a new dict followed by all previous maps.' - return self.__class__({}, *self.maps) + def new_child(self, m=None): # like Django's Context.push() + ''' + New ChainMap with a new map followed by all previous maps. If no + map is provided, an empty dict is used. + ''' + if m is None: + m = {} + return self.__class__(m, *self.maps) @property def parents(self): # like Django's Context.pop() diff --git a/Lib/test/test_collections.py b/Lib/test/test_collections.py --- a/Lib/test/test_collections.py +++ b/Lib/test/test_collections.py @@ -112,6 +112,38 @@ self.assertEqual(dict(d), dict(a=1, b=2, c=30)) self.assertEqual(dict(d.items()), dict(a=1, b=2, c=30)) + def test_new_child(self): + 'Tests for changes for issue #16613.' + c = ChainMap() + c['a'] = 1 + c['b'] = 2 + m = {'b':20, 'c': 30} + d = c.new_child(m) + self.assertEqual(d.maps, [{'b':20, 'c':30}, {'a':1, 'b':2}]) # check internal state + self.assertIs(m, d.maps[0]) + + # Use a different map than a dict + class lowerdict(dict): + def __getitem__(self, key): + if isinstance(key, str): + key = key.lower() + return dict.__getitem__(self, key) + def __contains__(self, key): + if isinstance(key, str): + key = key.lower() + return dict.__contains__(self, key) + + c = ChainMap() + c['a'] = 1 + c['b'] = 2 + m = lowerdict(b=20, c=30) + d = c.new_child(m) + self.assertIs(m, d.maps[0]) + for key in 'abc': # check contains + self.assertIn(key, d) + for k, v in dict(a=1, B=20, C=30, z=100).items(): # check get + self.assertEqual(d.get(k, 100), v) + ################################################################################ ### Named Tuples -- Repository URL: http://hg.python.org/cpython From ncoghlan at gmail.com Sat Jan 12 03:55:25 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 12 Jan 2013 12:55:25 +1000 Subject: [Python-checkins] cpython: Issue #15031: Refactor some code in importlib pertaining to validating In-Reply-To: <3Yjfsp6PBWzSBY@mail.python.org> References: <3Yjfsp6PBWzSBY@mail.python.org> Message-ID: Nice improvement. Just a couple of minor cleanup suggestions. On Sat, Jan 12, 2013 at 9:09 AM, brett.cannon wrote: > + else: > + # To prevent having to make all messages have a conditional name. > + name = 'bytecode' For consistency with other default/implied names, I suggest wrapping this in angle brackets: "") > + if path is not None: > + exc_details['path'] = path > + magic = data[:4] > + raw_timestamp = data[4:8] > + raw_size = data[8:12] > + if magic != _MAGIC_BYTES: > + msg = 'bad magic number in {!r}: {!r}'.format(name, magic) > + raise ImportError(msg, **exc_details) > + elif len(raw_timestamp) != 4: > + message = 'bad timestamp in {!r}'.format(name) > + _verbose_message(message) > + raise EOFError(message) > + elif len(raw_size) != 4: > + message = 'bad size in {!r}'.format(name) > + _verbose_message(message) > + raise EOFError(message) For timestamp and size "incomplete" would probably be a better word than "bad" in the error messages (since we're only checking the length rather than the value). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From python-checkins at python.org Sat Jan 12 04:31:13 2013 From: python-checkins at python.org (chris.jerdonek) Date: Sat, 12 Jan 2013 04:31:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE2OTMz?= =?utf-8?q?=3A_Improve_choices_examples_in_argparse_documentation=2E?= Message-ID: <3Yjmgj20S3zSBM@mail.python.org> http://hg.python.org/cpython/rev/b2bb3219d36d changeset: 81434:b2bb3219d36d branch: 2.7 parent: 81423:2d0c5f97af48 user: Chris Jerdonek date: Fri Jan 11 19:25:28 2013 -0800 summary: Issue #16933: Improve choices examples in argparse documentation. files: Doc/library/argparse.rst | 41 ++++++++++++++------------- 1 files changed, 21 insertions(+), 20 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1005,32 +1005,33 @@ ^^^^^^^ Some command-line arguments should be selected from a restricted set of values. -These can be handled by passing a container object as the ``choices`` keyword +These can be handled by passing a container object as the *choices* keyword argument to :meth:`~ArgumentParser.add_argument`. When the command line is -parsed, argument values will be checked, and an error message will be displayed if -the argument was not one of the acceptable values:: +parsed, argument values will be checked, and an error message will be displayed +if the argument was not one of the acceptable values:: - >>> parser = argparse.ArgumentParser(prog='PROG') - >>> parser.add_argument('foo', choices='abc') - >>> parser.parse_args('c'.split()) - Namespace(foo='c') - >>> parser.parse_args('X'.split()) - usage: PROG [-h] {a,b,c} - PROG: error: argument foo: invalid choice: 'X' (choose from 'a', 'b', 'c') + >>> parser = argparse.ArgumentParser(prog='game.py') + >>> parser.add_argument('move', choices=['rock', 'paper', 'scissors']) + >>> parser.parse_args(['rock']) + Namespace(move='rock') + >>> parser.parse_args(['fire']) + usage: game.py [-h] {rock,paper,scissors} + game.py: error: argument move: invalid choice: 'fire' (choose from 'rock', + 'paper', 'scissors') -Note that inclusion in the ``choices`` container is checked after any type_ -conversions have been performed, so the type of the objects in the ``choices`` +Note that inclusion in the *choices* container is checked after any type_ +conversions have been performed, so the type of the objects in the *choices* container should match the type_ specified:: - >>> parser = argparse.ArgumentParser(prog='PROG') - >>> parser.add_argument('foo', type=complex, choices=[1, 1j]) - >>> parser.parse_args('1j'.split()) - Namespace(foo=1j) - >>> parser.parse_args('-- -4'.split()) - usage: PROG [-h] {1,1j} - PROG: error: argument foo: invalid choice: (-4+0j) (choose from 1, 1j) + >>> parser = argparse.ArgumentParser(prog='doors.py') + >>> parser.add_argument('door', type=int, choices=range(1, 4)) + >>> print(parser.parse_args(['3'])) + Namespace(door=3) + >>> parser.parse_args(['4']) + usage: doors.py [-h] {1,2,3} + doors.py: error: argument door: invalid choice: 4 (choose from 1, 2, 3) -Any object that supports the ``in`` operator can be passed as the ``choices`` +Any object that supports the ``in`` operator can be passed as the *choices* value, so :class:`dict` objects, :class:`set` objects, custom containers, etc. are all supported. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 04:31:14 2013 From: python-checkins at python.org (chris.jerdonek) Date: Sat, 12 Jan 2013 04:31:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE2OTMz?= =?utf-8?q?_=282=2E7_forward-port=29=3A_Improve_choices_examples_in_argpar?= =?utf-8?q?se_docs=2E?= Message-ID: <3Yjmgk4zhzzSBs@mail.python.org> http://hg.python.org/cpython/rev/eaa2a6074741 changeset: 81435:eaa2a6074741 branch: 3.2 parent: 81424:a9990104f728 user: Chris Jerdonek date: Fri Jan 11 19:26:44 2013 -0800 summary: Issue #16933 (2.7 forward-port): Improve choices examples in argparse docs. files: Doc/library/argparse.rst | 41 ++++++++++++++------------- 1 files changed, 21 insertions(+), 20 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1003,32 +1003,33 @@ ^^^^^^^ Some command-line arguments should be selected from a restricted set of values. -These can be handled by passing a container object as the ``choices`` keyword +These can be handled by passing a container object as the *choices* keyword argument to :meth:`~ArgumentParser.add_argument`. When the command line is -parsed, argument values will be checked, and an error message will be displayed if -the argument was not one of the acceptable values:: +parsed, argument values will be checked, and an error message will be displayed +if the argument was not one of the acceptable values:: - >>> parser = argparse.ArgumentParser(prog='PROG') - >>> parser.add_argument('foo', choices='abc') - >>> parser.parse_args('c'.split()) - Namespace(foo='c') - >>> parser.parse_args('X'.split()) - usage: PROG [-h] {a,b,c} - PROG: error: argument foo: invalid choice: 'X' (choose from 'a', 'b', 'c') + >>> parser = argparse.ArgumentParser(prog='game.py') + >>> parser.add_argument('move', choices=['rock', 'paper', 'scissors']) + >>> parser.parse_args(['rock']) + Namespace(move='rock') + >>> parser.parse_args(['fire']) + usage: game.py [-h] {rock,paper,scissors} + game.py: error: argument move: invalid choice: 'fire' (choose from 'rock', + 'paper', 'scissors') -Note that inclusion in the ``choices`` container is checked after any type_ -conversions have been performed, so the type of the objects in the ``choices`` +Note that inclusion in the *choices* container is checked after any type_ +conversions have been performed, so the type of the objects in the *choices* container should match the type_ specified:: - >>> parser = argparse.ArgumentParser(prog='PROG') - >>> parser.add_argument('foo', type=complex, choices=[1, 1j]) - >>> parser.parse_args('1j'.split()) - Namespace(foo=1j) - >>> parser.parse_args('-- -4'.split()) - usage: PROG [-h] {1,1j} - PROG: error: argument foo: invalid choice: (-4+0j) (choose from 1, 1j) + >>> parser = argparse.ArgumentParser(prog='doors.py') + >>> parser.add_argument('door', type=int, choices=range(1, 4)) + >>> print(parser.parse_args(['3'])) + Namespace(door=3) + >>> parser.parse_args(['4']) + usage: doors.py [-h] {1,2,3} + doors.py: error: argument door: invalid choice: 4 (choose from 1, 2, 3) -Any object that supports the ``in`` operator can be passed as the ``choices`` +Any object that supports the ``in`` operator can be passed as the *choices* value, so :class:`dict` objects, :class:`set` objects, custom containers, etc. are all supported. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 04:31:16 2013 From: python-checkins at python.org (chris.jerdonek) Date: Sat, 12 Jan 2013 04:31:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=2316933_=28merge_from_3=2E2=29=3A_Improve_choices_examp?= =?utf-8?q?les_in_argparse_docs=2E?= Message-ID: <3Yjmgm0xB5zSBc@mail.python.org> http://hg.python.org/cpython/rev/de9eb3031f5a changeset: 81436:de9eb3031f5a branch: 3.3 parent: 81430:9c695fddd10a parent: 81435:eaa2a6074741 user: Chris Jerdonek date: Fri Jan 11 19:28:05 2013 -0800 summary: Issue #16933 (merge from 3.2): Improve choices examples in argparse docs. files: Doc/library/argparse.rst | 41 ++++++++++++++------------- 1 files changed, 21 insertions(+), 20 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1022,32 +1022,33 @@ ^^^^^^^ Some command-line arguments should be selected from a restricted set of values. -These can be handled by passing a container object as the ``choices`` keyword +These can be handled by passing a container object as the *choices* keyword argument to :meth:`~ArgumentParser.add_argument`. When the command line is -parsed, argument values will be checked, and an error message will be displayed if -the argument was not one of the acceptable values:: +parsed, argument values will be checked, and an error message will be displayed +if the argument was not one of the acceptable values:: - >>> parser = argparse.ArgumentParser(prog='PROG') - >>> parser.add_argument('foo', choices='abc') - >>> parser.parse_args('c'.split()) - Namespace(foo='c') - >>> parser.parse_args('X'.split()) - usage: PROG [-h] {a,b,c} - PROG: error: argument foo: invalid choice: 'X' (choose from 'a', 'b', 'c') + >>> parser = argparse.ArgumentParser(prog='game.py') + >>> parser.add_argument('move', choices=['rock', 'paper', 'scissors']) + >>> parser.parse_args(['rock']) + Namespace(move='rock') + >>> parser.parse_args(['fire']) + usage: game.py [-h] {rock,paper,scissors} + game.py: error: argument move: invalid choice: 'fire' (choose from 'rock', + 'paper', 'scissors') -Note that inclusion in the ``choices`` container is checked after any type_ -conversions have been performed, so the type of the objects in the ``choices`` +Note that inclusion in the *choices* container is checked after any type_ +conversions have been performed, so the type of the objects in the *choices* container should match the type_ specified:: - >>> parser = argparse.ArgumentParser(prog='PROG') - >>> parser.add_argument('foo', type=complex, choices=[1, 1j]) - >>> parser.parse_args('1j'.split()) - Namespace(foo=1j) - >>> parser.parse_args('-- -4'.split()) - usage: PROG [-h] {1,1j} - PROG: error: argument foo: invalid choice: (-4+0j) (choose from 1, 1j) + >>> parser = argparse.ArgumentParser(prog='doors.py') + >>> parser.add_argument('door', type=int, choices=range(1, 4)) + >>> print(parser.parse_args(['3'])) + Namespace(door=3) + >>> parser.parse_args(['4']) + usage: doors.py [-h] {1,2,3} + doors.py: error: argument door: invalid choice: 4 (choose from 1, 2, 3) -Any object that supports the ``in`` operator can be passed as the ``choices`` +Any object that supports the ``in`` operator can be passed as the *choices* value, so :class:`dict` objects, :class:`set` objects, custom containers, etc. are all supported. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 04:31:17 2013 From: python-checkins at python.org (chris.jerdonek) Date: Sat, 12 Jan 2013 04:31:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2316933_=28merge_from_3=2E3=29=3A_Improve_choices?= =?utf-8?q?_examples_in_argparse_docs=2E?= Message-ID: <3Yjmgn3X0tzSDC@mail.python.org> http://hg.python.org/cpython/rev/bddbaaf332d7 changeset: 81437:bddbaaf332d7 parent: 81433:c0ddae67f4df parent: 81436:de9eb3031f5a user: Chris Jerdonek date: Fri Jan 11 19:29:43 2013 -0800 summary: Issue #16933 (merge from 3.3): Improve choices examples in argparse docs. files: Doc/library/argparse.rst | 41 ++++++++++++++------------- 1 files changed, 21 insertions(+), 20 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1022,32 +1022,33 @@ ^^^^^^^ Some command-line arguments should be selected from a restricted set of values. -These can be handled by passing a container object as the ``choices`` keyword +These can be handled by passing a container object as the *choices* keyword argument to :meth:`~ArgumentParser.add_argument`. When the command line is -parsed, argument values will be checked, and an error message will be displayed if -the argument was not one of the acceptable values:: +parsed, argument values will be checked, and an error message will be displayed +if the argument was not one of the acceptable values:: - >>> parser = argparse.ArgumentParser(prog='PROG') - >>> parser.add_argument('foo', choices='abc') - >>> parser.parse_args('c'.split()) - Namespace(foo='c') - >>> parser.parse_args('X'.split()) - usage: PROG [-h] {a,b,c} - PROG: error: argument foo: invalid choice: 'X' (choose from 'a', 'b', 'c') + >>> parser = argparse.ArgumentParser(prog='game.py') + >>> parser.add_argument('move', choices=['rock', 'paper', 'scissors']) + >>> parser.parse_args(['rock']) + Namespace(move='rock') + >>> parser.parse_args(['fire']) + usage: game.py [-h] {rock,paper,scissors} + game.py: error: argument move: invalid choice: 'fire' (choose from 'rock', + 'paper', 'scissors') -Note that inclusion in the ``choices`` container is checked after any type_ -conversions have been performed, so the type of the objects in the ``choices`` +Note that inclusion in the *choices* container is checked after any type_ +conversions have been performed, so the type of the objects in the *choices* container should match the type_ specified:: - >>> parser = argparse.ArgumentParser(prog='PROG') - >>> parser.add_argument('foo', type=complex, choices=[1, 1j]) - >>> parser.parse_args('1j'.split()) - Namespace(foo=1j) - >>> parser.parse_args('-- -4'.split()) - usage: PROG [-h] {1,1j} - PROG: error: argument foo: invalid choice: (-4+0j) (choose from 1, 1j) + >>> parser = argparse.ArgumentParser(prog='doors.py') + >>> parser.add_argument('door', type=int, choices=range(1, 4)) + >>> print(parser.parse_args(['3'])) + Namespace(door=3) + >>> parser.parse_args(['4']) + usage: doors.py [-h] {1,2,3} + doors.py: error: argument door: invalid choice: 4 (choose from 1, 2, 3) -Any object that supports the ``in`` operator can be passed as the ``choices`` +Any object that supports the ``in`` operator can be passed as the *choices* value, so :class:`dict` objects, :class:`set` objects, custom containers, etc. are all supported. -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sat Jan 12 05:58:35 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 12 Jan 2013 05:58:35 +0100 Subject: [Python-checkins] Daily reference leaks (c0ddae67f4df): sum=280 Message-ID: results for c0ddae67f4df on branch "default" -------------------------------------------- test_xml_etree_c leaked [56, 56, 56] references, sum=168 test_xml_etree_c leaked [36, 38, 38] memory blocks, sum=112 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogBm4Rxs', '-x'] From python-checkins at python.org Sat Jan 12 07:30:12 2013 From: python-checkins at python.org (raymond.hettinger) Date: Sat, 12 Jan 2013 07:30:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2316398=3A_Optimize?= =?utf-8?b?IGRlcXVlLnJvdGF0ZSgp?= Message-ID: <3YjrfD1z9FzSCk@mail.python.org> http://hg.python.org/cpython/rev/9fb793b60e79 changeset: 81438:9fb793b60e79 user: Raymond Hettinger date: Fri Jan 11 22:29:50 2013 -0800 summary: Issue #16398: Optimize deque.rotate() files: Misc/NEWS | 3 + Modules/_collectionsmodule.c | 72 +++++++++++++++++++---- 2 files changed, 61 insertions(+), 14 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -221,6 +221,9 @@ duplication. Thanks to Ronan Lamy for the report and taking an initial stab at the problem. +- Issue #16398: Optimize deque.rotate() so that it only moves pointers + and doesn't touch the underlying data with increfs and decrefs. + - Issue #16900: Issue a ResourceWarning when an ssl socket is left unclosed. - Issue #13899: \A, \Z, and \B now correctly match the A, Z, and B literals diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -414,9 +414,10 @@ _deque_rotate(dequeobject *deque, Py_ssize_t n) { Py_ssize_t i, len=deque->len, halflen=(len+1)>>1; - PyObject *item, *rv; + PyObject *item; + block *prevblock, *leftblock, *rightblock; - if (len == 0) + if (len <= 1) return 0; if (n > halflen || n < -halflen) { n %= len; @@ -426,23 +427,66 @@ n += len; } + assert(deque->len > 1); + deque->state++; + leftblock = deque->leftblock; + rightblock = deque->rightblock; for (i=0 ; idata[deque->rightindex]; assert (item != NULL); - rv = deque_appendleft(deque, item); - Py_DECREF(item); - if (rv == NULL) - return -1; - Py_DECREF(rv); + deque->rightindex--; + if (deque->rightindex == -1) { + assert(rightblock != NULL); + prevblock = rightblock->leftlink; + assert(leftblock != rightblock); + freeblock(rightblock); + prevblock->rightlink = NULL; + deque->rightblock = rightblock = prevblock; + deque->rightindex = BLOCKLEN - 1; + } + if (deque->leftindex == 0) { + block *b = newblock(NULL, leftblock, deque->len); + if (b == NULL) { + deque->len--; + Py_DECREF(item); + return -1; + } + assert(leftblock->leftlink == NULL); + leftblock->leftlink = b; + deque->leftblock = leftblock = b; + deque->leftindex = BLOCKLEN; + } + deque->leftindex--; + leftblock->data[deque->leftindex] = item; } for (i=0 ; i>n ; i--) { - item = deque_popleft(deque, NULL); + assert(leftblock != NULL); + item = leftblock->data[deque->leftindex]; assert (item != NULL); - rv = deque_append(deque, item); - Py_DECREF(item); - if (rv == NULL) - return -1; - Py_DECREF(rv); + deque->leftindex++; + if (deque->leftindex == BLOCKLEN) { + assert(leftblock != rightblock); + prevblock = leftblock->rightlink; + freeblock(leftblock); + assert(prevblock != NULL); + prevblock->leftlink = NULL; + deque->leftblock = leftblock = prevblock; + deque->leftindex = 0; + } + if (deque->rightindex == BLOCKLEN-1) { + block *b = newblock(rightblock, NULL, deque->len); + if (b == NULL) { + deque->len--; + Py_DECREF(item); + return -1; + } + assert(rightblock->rightlink == NULL); + rightblock->rightlink = b; + deque->rightblock = rightblock = b; + deque->rightindex = -1; + } + deque->rightindex++; + rightblock->data[deque->rightindex] = item; } return 0; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 09:05:16 2013 From: python-checkins at python.org (raymond.hettinger) Date: Sat, 12 Jan 2013 09:05:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE2Mzk4?= =?utf-8?q?=3A_Optimize_deque=2Erotate=28=29?= Message-ID: <3Yjtlw6LyNzSCc@mail.python.org> http://hg.python.org/cpython/rev/0d81333bde78 changeset: 81439:0d81333bde78 branch: 2.7 parent: 81434:b2bb3219d36d user: Raymond Hettinger date: Sat Jan 12 00:05:00 2013 -0800 summary: Issue #16398: Optimize deque.rotate() files: Misc/NEWS | 3 + Modules/_collectionsmodule.c | 72 +++++++++++++++++++---- 2 files changed, 61 insertions(+), 14 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -189,6 +189,9 @@ - Issue #13899: \A, \Z, and \B now correctly match the A, Z, and B literals when used inside character classes (e.g. '[\A]'). Patch by Matthew Barnett. +- Issue #16398: Optimize deque.rotate() so that it only moves pointers + and doesn't touch the underlying data with increfs and decrefs. + - Issue #15109: Fix regression in sqlite3's iterdump method where it would die with an encoding error if the database contained string values containing non-ASCII. (Regression was introduced by fix for 9750). diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -414,9 +414,10 @@ _deque_rotate(dequeobject *deque, Py_ssize_t n) { Py_ssize_t i, len=deque->len, halflen=(len+1)>>1; - PyObject *item, *rv; + PyObject *item; + block *prevblock, *leftblock, *rightblock; - if (len == 0) + if (len <= 1) return 0; if (n > halflen || n < -halflen) { n %= len; @@ -426,23 +427,66 @@ n += len; } + assert(deque->len > 1); + deque->state++; + leftblock = deque->leftblock; + rightblock = deque->rightblock; for (i=0 ; idata[deque->rightindex]; assert (item != NULL); - rv = deque_appendleft(deque, item); - Py_DECREF(item); - if (rv == NULL) - return -1; - Py_DECREF(rv); + deque->rightindex--; + if (deque->rightindex == -1) { + assert(rightblock != NULL); + prevblock = rightblock->leftlink; + assert(leftblock != rightblock); + freeblock(rightblock); + prevblock->rightlink = NULL; + deque->rightblock = rightblock = prevblock; + deque->rightindex = BLOCKLEN - 1; + } + if (deque->leftindex == 0) { + block *b = newblock(NULL, leftblock, deque->len); + if (b == NULL) { + deque->len--; + Py_DECREF(item); + return -1; + } + assert(leftblock->leftlink == NULL); + leftblock->leftlink = b; + deque->leftblock = leftblock = b; + deque->leftindex = BLOCKLEN; + } + deque->leftindex--; + leftblock->data[deque->leftindex] = item; } for (i=0 ; i>n ; i--) { - item = deque_popleft(deque, NULL); + assert(leftblock != NULL); + item = leftblock->data[deque->leftindex]; assert (item != NULL); - rv = deque_append(deque, item); - Py_DECREF(item); - if (rv == NULL) - return -1; - Py_DECREF(rv); + deque->leftindex++; + if (deque->leftindex == BLOCKLEN) { + assert(leftblock != rightblock); + prevblock = leftblock->rightlink; + freeblock(leftblock); + assert(prevblock != NULL); + prevblock->leftlink = NULL; + deque->leftblock = leftblock = prevblock; + deque->leftindex = 0; + } + if (deque->rightindex == BLOCKLEN-1) { + block *b = newblock(rightblock, NULL, deque->len); + if (b == NULL) { + deque->len--; + Py_DECREF(item); + return -1; + } + assert(rightblock->rightlink == NULL); + rightblock->rightlink = b; + deque->rightblock = rightblock = b; + deque->rightindex = -1; + } + deque->rightindex++; + rightblock->data[deque->rightindex] = item; } return 0; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 09:41:02 2013 From: python-checkins at python.org (ezio.melotti) Date: Sat, 12 Jan 2013 09:41:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogIzE2OTQwOiBmaXgg?= =?utf-8?q?indentation_in_example=2E?= Message-ID: <3YjvYB5QjdzSCx@mail.python.org> http://hg.python.org/cpython/rev/eae31f2b6f60 changeset: 81440:eae31f2b6f60 branch: 2.7 user: Ezio Melotti date: Sat Jan 12 10:39:45 2013 +0200 summary: #16940: fix indentation in example. files: Doc/library/argparse.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1486,8 +1486,8 @@ positional arguments: {a,b} sub-command help - a a help - b b help + a a help + b b help optional arguments: -h, --help show this help message and exit -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 09:41:04 2013 From: python-checkins at python.org (ezio.melotti) Date: Sat, 12 Jan 2013 09:41:04 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogIzE2OTQwOiBmaXgg?= =?utf-8?q?indentation_in_example=2E?= Message-ID: <3YjvYD0yG7zR1G@mail.python.org> http://hg.python.org/cpython/rev/3d54723c9be6 changeset: 81441:3d54723c9be6 branch: 3.2 parent: 81435:eaa2a6074741 user: Ezio Melotti date: Sat Jan 12 10:39:45 2013 +0200 summary: #16940: fix indentation in example. files: Doc/library/argparse.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1487,8 +1487,8 @@ positional arguments: {a,b} sub-command help - a a help - b b help + a a help + b b help optional arguments: -h, --help show this help message and exit -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 09:41:05 2013 From: python-checkins at python.org (ezio.melotti) Date: Sat, 12 Jan 2013 09:41:05 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_=2316940=3A_merge_with_3=2E2=2E?= Message-ID: <3YjvYF3f0kzS8T@mail.python.org> http://hg.python.org/cpython/rev/b468f6c8eae5 changeset: 81442:b468f6c8eae5 branch: 3.3 parent: 81436:de9eb3031f5a parent: 81441:3d54723c9be6 user: Ezio Melotti date: Sat Jan 12 10:40:24 2013 +0200 summary: #16940: merge with 3.2. files: Doc/library/argparse.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1506,8 +1506,8 @@ positional arguments: {a,b} sub-command help - a a help - b b help + a a help + b b help optional arguments: -h, --help show this help message and exit -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 09:41:06 2013 From: python-checkins at python.org (ezio.melotti) Date: Sat, 12 Jan 2013 09:41:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogIzE2OTQwOiBtZXJnZSB3aXRoIDMuMy4=?= Message-ID: <3YjvYG6GkgzS8T@mail.python.org> http://hg.python.org/cpython/rev/6fe28afa6611 changeset: 81443:6fe28afa6611 parent: 81438:9fb793b60e79 parent: 81442:b468f6c8eae5 user: Ezio Melotti date: Sat Jan 12 10:40:49 2013 +0200 summary: #16940: merge with 3.3. files: Doc/library/argparse.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1506,8 +1506,8 @@ positional arguments: {a,b} sub-command help - a a help - b b help + a a help + b b help optional arguments: -h, --help show this help message and exit -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 12:32:52 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sat, 12 Jan 2013 12:32:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2316876=3A_Revert_b?= =?utf-8?q?e8e6b81284e=2C_which_wasn=27t_thread-safe=3A_wait_until_a?= Message-ID: <3YjzMS3C6nzSDp@mail.python.org> http://hg.python.org/cpython/rev/30eb98c8afef changeset: 81444:30eb98c8afef user: Charles-Fran?ois Natali date: Sat Jan 12 12:31:00 2013 +0100 summary: Issue #16876: Revert be8e6b81284e, which wasn't thread-safe: wait until a solution is found for poll(). files: Misc/NEWS | 3 -- Modules/selectmodule.c | 45 ++++++++--------------------- 2 files changed, 13 insertions(+), 35 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -233,9 +233,6 @@ failing if the connection used a row factory (such as sqlite3.Row) that produced unsortable objects. (Regression was introduced by fix for 9750). -- Issue #16876: Optimize epoll.poll() by keeping a per-instance epoll events - buffer instead of allocating a new one at each poll(). - - Issue #16491: IDLE now prints chained exception tracebacks. - fcntl: add F_DUPFD_CLOEXEC constant, available on Linux 2.6.24+. diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -1056,14 +1056,9 @@ #include #endif -/* default maximum number of events returned by epoll_wait() */ -#define EPOLL_DEFAULT_MAXEVENTS (FD_SETSIZE) - typedef struct { PyObject_HEAD - SOCKET epfd; /* epoll control file descriptor */ - int maxevents; /* maximum number of epoll events */ - struct epoll_event *evs; /* epoll events buffer */ + SOCKET epfd; /* epoll control file descriptor */ } pyEpoll_Object; static PyTypeObject pyEpoll_Type; @@ -1119,15 +1114,6 @@ PyErr_SetFromErrno(PyExc_OSError); return NULL; } - - self->maxevents = EPOLL_DEFAULT_MAXEVENTS; - self->evs = PyMem_New(struct epoll_event, self->maxevents); - if (!self->evs) { - Py_DECREF(self); - PyErr_NoMemory(); - return NULL; - } - return (PyObject *)self; } @@ -1154,10 +1140,6 @@ pyepoll_dealloc(pyEpoll_Object *self) { (void)pyepoll_internal_close(self); - if (self->evs) { - PyMem_Free(self->evs); - self->evs = NULL; - } Py_TYPE(self)->tp_free(self); } @@ -1338,6 +1320,7 @@ int maxevents = -1; int nfds, i; PyObject *elist = NULL, *etuple = NULL; + struct epoll_event *evs = NULL; static char *kwlist[] = {"timeout", "maxevents", NULL}; if (self->epfd < 0) @@ -1361,27 +1344,24 @@ } if (maxevents == -1) { - maxevents = EPOLL_DEFAULT_MAXEVENTS; - } else if (maxevents < 1) { + maxevents = FD_SETSIZE-1; + } + else if (maxevents < 1) { PyErr_Format(PyExc_ValueError, "maxevents must be greater than 0, got %d", maxevents); return NULL; } - if (maxevents > self->maxevents) { - struct epoll_event *orig_evs = self->evs; - PyMem_RESIZE(self->evs, struct epoll_event, maxevents); - if (!self->evs) { - self->evs = orig_evs; - PyErr_NoMemory(); - return NULL; - } - self->maxevents = maxevents; + evs = PyMem_New(struct epoll_event, maxevents); + if (evs == NULL) { + Py_DECREF(self); + PyErr_NoMemory(); + return NULL; } Py_BEGIN_ALLOW_THREADS - nfds = epoll_wait(self->epfd, self->evs, self->maxevents, timeout); + nfds = epoll_wait(self->epfd, evs, maxevents, timeout); Py_END_ALLOW_THREADS if (nfds < 0) { PyErr_SetFromErrno(PyExc_OSError); @@ -1394,7 +1374,7 @@ } for (i = 0; i < nfds; i++) { - etuple = Py_BuildValue("iI", self->evs[i].data.fd, self->evs[i].events); + etuple = Py_BuildValue("iI", evs[i].data.fd, evs[i].events); if (etuple == NULL) { Py_CLEAR(elist); goto error; @@ -1403,6 +1383,7 @@ } error: + PyMem_Free(evs); return elist; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 14:21:32 2013 From: python-checkins at python.org (eli.bendersky) Date: Sat, 12 Jan 2013 14:21:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE2MDc2?= =?utf-8?q?=3A_fix_refleak_in_pickling_of_Element=2E?= Message-ID: <3Yk1mr1TXbzS8w@mail.python.org> http://hg.python.org/cpython/rev/5b36768b9a11 changeset: 81445:5b36768b9a11 branch: 3.3 parent: 81442:b468f6c8eae5 user: Eli Bendersky date: Sat Jan 12 05:20:16 2013 -0800 summary: Issue #16076: fix refleak in pickling of Element. Thanks to Ezio Melotti and Daniel Shahaf for the patch. files: Modules/_elementtree.c | 16 +++++----------- 1 files changed, 5 insertions(+), 11 deletions(-) diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c --- a/Modules/_elementtree.c +++ b/Modules/_elementtree.c @@ -859,8 +859,10 @@ PICKLED_ATTRIB, self->extra->attrib, PICKLED_TEXT, self->text, PICKLED_TAIL, self->tail); - if (instancedict) + if (instancedict) { + Py_DECREF(children); return instancedict; + } else { for (i = 0; i < PyList_GET_SIZE(children); i++) Py_DECREF(PyList_GET_ITEM(children, i)); @@ -884,25 +886,17 @@ PyErr_SetString(PyExc_TypeError, "tag may not be NULL"); return NULL; } - if (!text) { - Py_INCREF(Py_None); - text = Py_None; - } - if (!tail) { - Py_INCREF(Py_None); - tail = Py_None; - } Py_CLEAR(self->tag); self->tag = tag; Py_INCREF(self->tag); Py_CLEAR(self->text); - self->text = text; + self->text = text ? text : Py_None; Py_INCREF(self->text); Py_CLEAR(self->tail); - self->tail = tail; + self->tail = tail ? tail : Py_None; Py_INCREF(self->tail); /* Handle ATTRIB and CHILDREN. */ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 14:21:33 2013 From: python-checkins at python.org (eli.bendersky) Date: Sat, 12 Jan 2013 14:21:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Close_=2316076=3A_fix_refleak_in_pickling_of_Element=2E?= Message-ID: <3Yk1ms4QfjzSBJ@mail.python.org> http://hg.python.org/cpython/rev/848738d3c40f changeset: 81446:848738d3c40f parent: 81444:30eb98c8afef parent: 81445:5b36768b9a11 user: Eli Bendersky date: Sat Jan 12 05:21:06 2013 -0800 summary: Close #16076: fix refleak in pickling of Element. Thanks to Ezio Melotti and Daniel Shahaf for the patch. files: Modules/_elementtree.c | 16 +++++----------- 1 files changed, 5 insertions(+), 11 deletions(-) diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c --- a/Modules/_elementtree.c +++ b/Modules/_elementtree.c @@ -859,8 +859,10 @@ PICKLED_ATTRIB, self->extra->attrib, PICKLED_TEXT, self->text, PICKLED_TAIL, self->tail); - if (instancedict) + if (instancedict) { + Py_DECREF(children); return instancedict; + } else { for (i = 0; i < PyList_GET_SIZE(children); i++) Py_DECREF(PyList_GET_ITEM(children, i)); @@ -884,25 +886,17 @@ PyErr_SetString(PyExc_TypeError, "tag may not be NULL"); return NULL; } - if (!text) { - Py_INCREF(Py_None); - text = Py_None; - } - if (!tail) { - Py_INCREF(Py_None); - tail = Py_None; - } Py_CLEAR(self->tag); self->tag = tag; Py_INCREF(self->tag); Py_CLEAR(self->text); - self->text = text; + self->text = text ? text : Py_None; Py_INCREF(self->text); Py_CLEAR(self->tail); - self->tail = tail; + self->tail = tail ? tail : Py_None; Py_INCREF(self->tail); /* Handle ATTRIB and CHILDREN. */ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 14:43:32 2013 From: python-checkins at python.org (eli.bendersky) Date: Sat, 12 Jan 2013 14:43:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE2MDc2?= =?utf-8?q?=3A_check_for_return_value_of_PyTuple=5FNew_for_args_=28followi?= =?utf-8?q?ng?= Message-ID: <3Yk2GD5GLlzSCN@mail.python.org> http://hg.python.org/cpython/rev/4501813ea676 changeset: 81447:4501813ea676 branch: 3.3 parent: 81445:5b36768b9a11 user: Eli Bendersky date: Sat Jan 12 05:42:38 2013 -0800 summary: Issue #16076: check for return value of PyTuple_New for args (following Coverity report) and cleanup code. files: Modules/_elementtree.c | 23 +++++++++++++---------- 1 files changed, 13 insertions(+), 10 deletions(-) diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c --- a/Modules/_elementtree.c +++ b/Modules/_elementtree.c @@ -950,19 +950,22 @@ PICKLED_TAIL, PICKLED_CHILDREN, 0}; PyObject *args; PyObject *tag, *attrib, *text, *tail, *children; - int error; - - /* More instance dict members than we know to handle? */ + PyObject *retval; + tag = attrib = text = tail = children = NULL; args = PyTuple_New(0); - error = ! PyArg_ParseTupleAndKeywords(args, state, "|$OOOOO", kwlist, &tag, - &attrib, &text, &tail, &children); + if (!args) + return NULL; + + if (PyArg_ParseTupleAndKeywords(args, state, "|$OOOOO", kwlist, &tag, + &attrib, &text, &tail, &children)) + retval = element_setstate_from_attributes(self, tag, attrib, text, + tail, children); + else + retval = NULL; + Py_DECREF(args); - if (error) - return NULL; - else - return element_setstate_from_attributes(self, tag, attrib, text, - tail, children); + return retval; } static PyObject * -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 14:43:34 2013 From: python-checkins at python.org (eli.bendersky) Date: Sat, 12 Jan 2013 14:43:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2316076=3A_check_for_return_value_of_PyTuple=5FNe?= =?utf-8?q?w_for_args_=28following?= Message-ID: <3Yk2GG0jLnzS8H@mail.python.org> http://hg.python.org/cpython/rev/7313096e0bad changeset: 81448:7313096e0bad parent: 81446:848738d3c40f parent: 81447:4501813ea676 user: Eli Bendersky date: Sat Jan 12 05:43:08 2013 -0800 summary: Issue #16076: check for return value of PyTuple_New for args (following Coverity report) and cleanup code. files: Modules/_elementtree.c | 23 +++++++++++++---------- 1 files changed, 13 insertions(+), 10 deletions(-) diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c --- a/Modules/_elementtree.c +++ b/Modules/_elementtree.c @@ -950,19 +950,22 @@ PICKLED_TAIL, PICKLED_CHILDREN, 0}; PyObject *args; PyObject *tag, *attrib, *text, *tail, *children; - int error; - - /* More instance dict members than we know to handle? */ + PyObject *retval; + tag = attrib = text = tail = children = NULL; args = PyTuple_New(0); - error = ! PyArg_ParseTupleAndKeywords(args, state, "|$OOOOO", kwlist, &tag, - &attrib, &text, &tail, &children); + if (!args) + return NULL; + + if (PyArg_ParseTupleAndKeywords(args, state, "|$OOOOO", kwlist, &tag, + &attrib, &text, &tail, &children)) + retval = element_setstate_from_attributes(self, tag, attrib, text, + tail, children); + else + retval = NULL; + Py_DECREF(args); - if (error) - return NULL; - else - return element_setstate_from_attributes(self, tag, attrib, text, - tail, children); + return retval; } static PyObject * -- Repository URL: http://hg.python.org/cpython From eliben at gmail.com Sat Jan 12 14:46:33 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sat, 12 Jan 2013 05:46:33 -0800 Subject: [Python-checkins] Daily reference leaks (aef7db0d3893): sum=287 In-Reply-To: <50F038FE.208@python.org> References:

<50F038FE.208@python.org> Message-ID: On Fri, Jan 11, 2013 at 8:08 AM, Christian Heimes wrote: > Am 11.01.2013 07:09, schrieb Nick Coghlan: > > On Fri, Jan 11, 2013 at 2:57 PM, wrote: > >> results for aef7db0d3893 on branch "default" > >> -------------------------------------------- > >> > >> test_dbm leaked [2, 0, 0] references, sum=2 > >> test_dbm leaked [2, 2, 1] memory blocks, sum=5 > > > > Hmm, I'm starting to wonder if there's something to this one - it > > seems to be popping up a bit lately. > > > >> test_xml_etree_c leaked [56, 56, 56] references, sum=168 > >> test_xml_etree_c leaked [36, 38, 38] memory blocks, sum=112 > > > > I'm gonna take a wild guess and suggest there may be a problem with > > the recent pickling fix in the C extension :) > > It has more issues. Coverity has sent me some complains, see attachment. > The second report is indeed a false positive. Coverity doesn't know that PyList_GET_SIZE returns 0 for PyList_New(0). Maybe it can be taught some project/domain-specific information? The first report is legit, however. PyTuple_New(0) was called and its return value wasn't checked for NULL. I actually think Coverity is very useful for such cases because forgetting to check NULL returns from PyObject constructors is a common mistake and it's not something that would show up in tests. Anyway, this was fixed. Thanks for reporting Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Jan 12 14:57:39 2013 From: brett at python.org (Brett Cannon) Date: Sat, 12 Jan 2013 08:57:39 -0500 Subject: [Python-checkins] cpython: Issue #15031: Refactor some code in importlib pertaining to validating In-Reply-To: References: <3Yjfsp6PBWzSBY@mail.python.org> Message-ID: On Fri, Jan 11, 2013 at 9:55 PM, Nick Coghlan wrote: > Nice improvement. Just a couple of minor cleanup suggestions. > > On Sat, Jan 12, 2013 at 9:09 AM, brett.cannon > wrote: >> + else: >> + # To prevent having to make all messages have a conditional name. >> + name = 'bytecode' > > For consistency with other default/implied names, I suggest wrapping > this in angle brackets: "") Good suggestion. > >> + if path is not None: >> + exc_details['path'] = path >> + magic = data[:4] >> + raw_timestamp = data[4:8] >> + raw_size = data[8:12] >> + if magic != _MAGIC_BYTES: >> + msg = 'bad magic number in {!r}: {!r}'.format(name, magic) >> + raise ImportError(msg, **exc_details) >> + elif len(raw_timestamp) != 4: >> + message = 'bad timestamp in {!r}'.format(name) >> + _verbose_message(message) >> + raise EOFError(message) >> + elif len(raw_size) != 4: >> + message = 'bad size in {!r}'.format(name) >> + _verbose_message(message) >> + raise EOFError(message) > > For timestamp and size "incomplete" would probably be a better word > than "bad" in the error messages (since we're only checking the length > rather than the value). True. Those were the original messages and in hindsight not accurate. Since we don't consider exception messages a backwards-compatible thing I'll update them. From python-checkins at python.org Sat Jan 12 15:13:56 2013 From: python-checkins at python.org (eli.bendersky) Date: Sat, 12 Jan 2013 15:13:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Close_=2315442=3A_Expand_t?= =?utf-8?q?he_list_of_default_directories_ignored_by_filecmp=2Edircmp?= Message-ID: <3Yk2xJ5bqrzS5v@mail.python.org> http://hg.python.org/cpython/rev/a1efab48d8f8 changeset: 81449:a1efab48d8f8 user: Eli Bendersky date: Sat Jan 12 06:13:32 2013 -0800 summary: Close #15442: Expand the list of default directories ignored by filecmp.dircmp and expose it as a module attribute files: Doc/library/filecmp.rst | 14 ++++++++++---- Lib/filecmp.py | 10 +++++++--- Lib/test/test_filecmp.py | 16 +++++++++++----- Misc/NEWS | 4 ++++ 4 files changed, 32 insertions(+), 12 deletions(-) diff --git a/Doc/library/filecmp.rst b/Doc/library/filecmp.rst --- a/Doc/library/filecmp.rst +++ b/Doc/library/filecmp.rst @@ -55,10 +55,10 @@ .. class:: dircmp(a, b, ignore=None, hide=None) - Construct a new directory comparison object, to compare the directories *a* and - *b*. *ignore* is a list of names to ignore, and defaults to ``['RCS', 'CVS', - 'tags']``. *hide* is a list of names to hide, and defaults to ``[os.curdir, - os.pardir]``. + Construct a new directory comparison object, to compare the directories *a* + and *b*. *ignore* is a list of names to ignore, and defaults to + :attr:`filecmp.DEFAULT_IGNORES`. *hide* is a list of names to hide, and + defaults to ``[os.curdir, os.pardir]``. The :class:`dircmp` class compares files by doing *shallow* comparisons as described for :func:`filecmp.cmp`. @@ -164,6 +164,12 @@ A dictionary mapping names in :attr:`common_dirs` to :class:`dircmp` objects. +.. attribute:: DEFAULT_IGNORES + + .. versionadded:: 3.3 + + List of directories ignored by :class:`dircmp` by default. + Here is a simplified example of using the ``subdirs`` attribute to search recursively through two directories to show common different files:: diff --git a/Lib/filecmp.py b/Lib/filecmp.py --- a/Lib/filecmp.py +++ b/Lib/filecmp.py @@ -13,11 +13,15 @@ import stat from itertools import filterfalse -__all__ = ["cmp", "dircmp", "cmpfiles"] +__all__ = ['cmp', 'dircmp', 'cmpfiles', 'DEFAULT_IGNORES'] _cache = {} BUFSIZE = 8*1024 +DEFAULT_IGNORES = [ + 'RCS', 'CVS', 'tags', '.git', '.hg', '.bzr', '_darcs', '__pycache__'] + + def cmp(f1, f2, shallow=True): """Compare two files. @@ -80,7 +84,7 @@ dircmp(a, b, ignore=None, hide=None) A and B are directories. IGNORE is a list of names to ignore, - defaults to ['RCS', 'CVS', 'tags']. + defaults to DEFAULT_IGNORES. HIDE is a list of names to hide, defaults to [os.curdir, os.pardir]. @@ -116,7 +120,7 @@ else: self.hide = hide if ignore is None: - self.ignore = ['RCS', 'CVS', 'tags'] # Names ignored in comparison + self.ignore = DEFAULT_IGNORES else: self.ignore = ignore diff --git a/Lib/test/test_filecmp.py b/Lib/test/test_filecmp.py --- a/Lib/test/test_filecmp.py +++ b/Lib/test/test_filecmp.py @@ -1,4 +1,3 @@ - import os, filecmp, shutil, tempfile import unittest from test import support @@ -46,9 +45,14 @@ self.dir = os.path.join(tmpdir, 'dir') self.dir_same = os.path.join(tmpdir, 'dir-same') self.dir_diff = os.path.join(tmpdir, 'dir-diff') + + # Another dir is created under dir_same, but it has a name from the + # ignored list so it should not affect testing results. + self.dir_ignored = os.path.join(self.dir_same, '.hg') + self.caseinsensitive = os.path.normcase('A') == os.path.normcase('a') data = 'Contents of file go here.\n' - for dir in [self.dir, self.dir_same, self.dir_diff]: + for dir in (self.dir, self.dir_same, self.dir_diff, self.dir_ignored): shutil.rmtree(dir, True) os.mkdir(dir) if self.caseinsensitive and dir is self.dir_same: @@ -64,9 +68,11 @@ output.close() def tearDown(self): - shutil.rmtree(self.dir) - shutil.rmtree(self.dir_same) - shutil.rmtree(self.dir_diff) + for dir in (self.dir, self.dir_same, self.dir_diff): + shutil.rmtree(dir) + + def test_default_ignores(self): + self.assertIn('.hg', filecmp.DEFAULT_IGNORES) def test_cmpfiles(self): self.assertTrue(filecmp.cmpfiles(self.dir, self.dir, ['file']) == diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1387,6 +1387,10 @@ - Issue #15250: Document that `filecmp.dircmp()` compares files shallowly. Patch contributed by Chris Jerdonek. +- Issue #15442: Expose the default list of directories ignored by + `filecmp.dircmp()` as a module attribute, and expand the list to more modern + values. + Tests ----- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 15:48:40 2013 From: python-checkins at python.org (nick.coghlan) Date: Sat, 12 Jan 2013 15:48:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Updates_based_on_implementati?= =?utf-8?q?on_progress?= Message-ID: <3Yk3jN1cWKzS8Y@mail.python.org> http://hg.python.org/peps/rev/89781bc231d9 changeset: 4665:89781bc231d9 user: Nick Coghlan date: Sun Jan 13 00:48:28 2013 +1000 summary: Updates based on implementation progress files: pep-0432.txt | 28 ++++++++++++++++++++++++---- 1 files changed, 24 insertions(+), 4 deletions(-) diff --git a/pep-0432.txt b/pep-0432.txt --- a/pep-0432.txt +++ b/pep-0432.txt @@ -177,6 +177,8 @@ * Whether or not to use randomised hashes (and if used, potentially specify a specific random seed) +* Whether or not to enable the import system (required by CPython's + build process when freezing the importlib._bootstrap bytecode) * The "Where is Python located?" elements in the ``sys`` module: * ``sys.executable`` * ``sys.base_exec_prefix`` @@ -406,6 +408,7 @@ TBD: Cover the initialization of the following in more detail: +* Completely disabling the import system * The initial warning system state: * ``sys.warnoptions`` * (-W option, PYTHONWARNINGS) @@ -612,9 +615,10 @@ int ignore_environment; /* -E switch */ int use_hash_seed; /* PYTHONHASHSEED */ unsigned long hash_seed; /* PYTHONHASHSEED */ + int _disable_importlib; /* Needed by freeze_importlib */ } Py_CoreConfig; - #define Py_CoreConfig_INIT {0, -1, 0} + #define Py_CoreConfig_INIT {0, -1, 0, 0} The core configuration settings pointer may be ``NULL``, in which case the default values are ``ignore_environment = 0`` and ``use_hash_seed = -1``. @@ -661,6 +665,13 @@ ``hash_seed``. On success the function will return zero. A non-zero return value indicates an error (most likely in the conversion to an integer). +The ``_disable_importlib`` setting is used as part of the CPython build +process to create an interpreter with no import capability at all. It is +considered private to the CPython development team (hence the leading +underscore), as the only known use case is to permit compiler changes +that invalidate the previously frozen bytecode for ``importlib._bootstrap`` +without breaking the build process. + The aim is to keep this initial level of configuration as small as possible in order to keep the bootstrapping environment consistent across different embedding applications. If we can create a valid interpreter state @@ -1046,8 +1057,17 @@ Implementation ============== -None as yet. Once I have a reasonably solid plan of attack, I intend to work -on a reference implementation as a feature branch in my BitBucket sandbox [2_] +The reference implementation is being developed as a feature branch in my +BitBucket sandbox [2_]. + +As the number of application binaries created by the build process is now +four, the reference implementation also creates a new top level "Apps" +directory in the CPython source tree. The source files for the main +``python`` binary and the new ``pysystem`` binary will be located in that +directory. The source files for the ``_freeze_importlib`` binary and the +``_testembed`` binary have been moved out of the Modules directory (which +is intended for CPython builtin and extension modules) and into the Tools +directory. References @@ -1057,7 +1077,7 @@ (http://wiki.python.org/moin/CPythonInterpreterInitialization) .. [2] BitBucket Sandbox - (https://bitbucket.org/ncoghlan/cpython_sandbox) + (https://bitbucket.org/ncoghlan/cpython_sandbox/compare/pep432_modular_bootstrap..default#commits) .. [3] \*nix getpath implementation (http://hg.python.org/cpython/file/default/Modules/getpath.c) -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sat Jan 12 16:45:12 2013 From: python-checkins at python.org (eli.bendersky) Date: Sat, 12 Jan 2013 16:45:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWVzICMxNTA4?= =?utf-8?q?3_and_=2316992=3A_port_find=2E*_method_tests_to_unittest?= Message-ID: <3Yk4yc4HhNzRxt@mail.python.org> http://hg.python.org/cpython/rev/f9d1d120c19e changeset: 81450:f9d1d120c19e branch: 3.3 parent: 81447:4501813ea676 user: Eli Bendersky date: Sat Jan 12 07:42:46 2013 -0800 summary: Issues #15083 and #16992: port find.* method tests to unittest files: Lib/test/test_xml_etree.py | 228 ++++++++++-------------- 1 files changed, 94 insertions(+), 134 deletions(-) diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -329,128 +329,6 @@ 'hello' """ -def find(): - """ - Test find methods (including xpath syntax). - - >>> elem = ET.XML(SAMPLE_XML) - >>> elem.find("tag").tag - 'tag' - >>> ET.ElementTree(elem).find("tag").tag - 'tag' - >>> elem.find("section/tag").tag - 'tag' - >>> elem.find("./tag").tag - 'tag' - >>> ET.ElementTree(elem).find("./tag").tag - 'tag' - >>> ET.ElementTree(elem).find("/tag").tag - 'tag' - >>> elem[2] = ET.XML(SAMPLE_SECTION) - >>> elem.find("section/nexttag").tag - 'nexttag' - >>> ET.ElementTree(elem).find("section/tag").tag - 'tag' - >>> ET.ElementTree(elem).find("tog") - >>> ET.ElementTree(elem).find("tog/foo") - >>> elem.findtext("tag") - 'text' - >>> elem.findtext("section/nexttag") - '' - >>> elem.findtext("section/nexttag", "default") - '' - >>> elem.findtext("tog") - >>> elem.findtext("tog", "default") - 'default' - >>> ET.ElementTree(elem).findtext("tag") - 'text' - >>> ET.ElementTree(elem).findtext("tog/foo") - >>> ET.ElementTree(elem).findtext("tog/foo", "default") - 'default' - >>> ET.ElementTree(elem).findtext("./tag") - 'text' - >>> ET.ElementTree(elem).findtext("/tag") - 'text' - >>> elem.findtext("section/tag") - 'subtext' - >>> ET.ElementTree(elem).findtext("section/tag") - 'subtext' - >>> summarize_list(elem.findall(".")) - ['body'] - >>> summarize_list(elem.findall("tag")) - ['tag', 'tag'] - >>> summarize_list(elem.findall("tog")) - [] - >>> summarize_list(elem.findall("tog/foo")) - [] - >>> summarize_list(elem.findall("*")) - ['tag', 'tag', 'section'] - >>> summarize_list(elem.findall(".//tag")) - ['tag', 'tag', 'tag', 'tag'] - >>> summarize_list(elem.findall("section/tag")) - ['tag'] - >>> summarize_list(elem.findall("section//tag")) - ['tag', 'tag'] - >>> summarize_list(elem.findall("section/*")) - ['tag', 'nexttag', 'nextsection'] - >>> summarize_list(elem.findall("section//*")) - ['tag', 'nexttag', 'nextsection', 'tag'] - >>> summarize_list(elem.findall("section/.//*")) - ['tag', 'nexttag', 'nextsection', 'tag'] - >>> summarize_list(elem.findall("*/*")) - ['tag', 'nexttag', 'nextsection'] - >>> summarize_list(elem.findall("*//*")) - ['tag', 'nexttag', 'nextsection', 'tag'] - >>> summarize_list(elem.findall("*/tag")) - ['tag'] - >>> summarize_list(elem.findall("*/./tag")) - ['tag'] - >>> summarize_list(elem.findall("./tag")) - ['tag', 'tag'] - >>> summarize_list(elem.findall(".//tag")) - ['tag', 'tag', 'tag', 'tag'] - >>> summarize_list(elem.findall("././tag")) - ['tag', 'tag'] - >>> summarize_list(elem.findall(".//tag[@class]")) - ['tag', 'tag', 'tag'] - >>> summarize_list(elem.findall(".//tag[@class='a']")) - ['tag'] - >>> summarize_list(elem.findall(".//tag[@class='b']")) - ['tag', 'tag'] - >>> summarize_list(elem.findall(".//tag[@id]")) - ['tag'] - >>> summarize_list(elem.findall(".//section[tag]")) - ['section'] - >>> summarize_list(elem.findall(".//section[element]")) - [] - >>> summarize_list(elem.findall("../tag")) - [] - >>> summarize_list(elem.findall("section/../tag")) - ['tag', 'tag'] - >>> summarize_list(ET.ElementTree(elem).findall("./tag")) - ['tag', 'tag'] - - Following example is invalid in 1.2. - A leading '*' is assumed in 1.3. - - >>> elem.findall("section//") == elem.findall("section//*") - True - - ET's Path module handles this case incorrectly; this gives - a warning in 1.3, and the behaviour will be modified in 1.4. - - >>> summarize_list(ET.ElementTree(elem).findall("/tag")) - ['tag', 'tag'] - - >>> elem = ET.XML(SAMPLE_XML_NS) - >>> summarize_list(elem.findall("tag")) - [] - >>> summarize_list(elem.findall("{http://effbot.org/ns}tag")) - ['{http://effbot.org/ns}tag', '{http://effbot.org/ns}tag'] - >>> summarize_list(elem.findall(".//{http://effbot.org/ns}tag")) - ['{http://effbot.org/ns}tag', '{http://effbot.org/ns}tag', '{http://effbot.org/ns}tag'] - """ - def file_init(): """ >>> import io @@ -469,16 +347,6 @@ 'empty-element' """ -def bad_find(): - """ - Check bad or unsupported path expressions. - - >>> elem = ET.XML(SAMPLE_XML) - >>> elem.findall("/tag") - Traceback (most recent call last): - SyntaxError: cannot use absolute path on element - """ - def path_cache(): """ Check that the path cache behaves sanely. @@ -1883,6 +1751,98 @@ method='html') self.assertEqual(serialized, expected) + +class ElementFindTest(unittest.TestCase): + def test_find_simple(self): + e = ET.XML(SAMPLE_XML) + self.assertEqual(e.find('tag').tag, 'tag') + self.assertEqual(e.find('section/tag').tag, 'tag') + self.assertEqual(e.find('./tag').tag, 'tag') + + e[2] = ET.XML(SAMPLE_SECTION) + self.assertEqual(e.find('section/nexttag').tag, 'nexttag') + + self.assertEqual(e.findtext('./tag'), 'text') + self.assertEqual(e.findtext('section/tag'), 'subtext') + + # section/nexttag is found but has no text + self.assertEqual(e.findtext('section/nexttag'), '') + self.assertEqual(e.findtext('section/nexttag', 'default'), '') + + # tog doesn't exist and 'default' kicks in + self.assertIsNone(e.findtext('tog')) + self.assertEqual(e.findtext('tog', 'default'), 'default') + + def test_findall(self): + e = ET.XML(SAMPLE_XML) + e[2] = ET.XML(SAMPLE_SECTION) + self.assertEqual(summarize_list(e.findall('.')), ['body']) + self.assertEqual(summarize_list(e.findall('tag')), ['tag', 'tag']) + self.assertEqual(summarize_list(e.findall('tog')), []) + self.assertEqual(summarize_list(e.findall('tog/foo')), []) + self.assertEqual(summarize_list(e.findall('*')), + ['tag', 'tag', 'section']) + self.assertEqual(summarize_list(e.findall('.//tag')), + ['tag'] * 4) + self.assertEqual(summarize_list(e.findall('section/tag')), ['tag']) + self.assertEqual(summarize_list(e.findall('section//tag')), ['tag'] * 2) + self.assertEqual(summarize_list(e.findall('section/*')), + ['tag', 'nexttag', 'nextsection']) + self.assertEqual(summarize_list(e.findall('section//*')), + ['tag', 'nexttag', 'nextsection', 'tag']) + self.assertEqual(summarize_list(e.findall('section/.//*')), + ['tag', 'nexttag', 'nextsection', 'tag']) + self.assertEqual(summarize_list(e.findall('*/*')), + ['tag', 'nexttag', 'nextsection']) + self.assertEqual(summarize_list(e.findall('*//*')), + ['tag', 'nexttag', 'nextsection', 'tag']) + self.assertEqual(summarize_list(e.findall('*/tag')), ['tag']) + self.assertEqual(summarize_list(e.findall('*/./tag')), ['tag']) + self.assertEqual(summarize_list(e.findall('./tag')), ['tag'] * 2) + self.assertEqual(summarize_list(e.findall('././tag')), ['tag'] * 2) + + self.assertEqual(summarize_list(e.findall('.//tag[@class]')), + ['tag'] * 3) + self.assertEqual(summarize_list(e.findall('.//tag[@class="a"]')), + ['tag']) + self.assertEqual(summarize_list(e.findall('.//tag[@class="b"]')), + ['tag'] * 2) + self.assertEqual(summarize_list(e.findall('.//tag[@id]')), + ['tag']) + self.assertEqual(summarize_list(e.findall('.//section[tag]')), + ['section']) + self.assertEqual(summarize_list(e.findall('.//section[element]')), []) + self.assertEqual(summarize_list(e.findall('../tag')), []) + self.assertEqual(summarize_list(e.findall('section/../tag')), + ['tag'] * 2) + self.assertEqual(e.findall('section//'), e.findall('section//*')) + + def test_test_find_with_ns(self): + e = ET.XML(SAMPLE_XML_NS) + self.assertEqual(summarize_list(e.findall('tag')), []) + self.assertEqual( + summarize_list(e.findall("{http://effbot.org/ns}tag")), + ['{http://effbot.org/ns}tag'] * 2) + self.assertEqual( + summarize_list(e.findall(".//{http://effbot.org/ns}tag")), + ['{http://effbot.org/ns}tag'] * 3) + + def test_bad_find(self): + e = ET.XML(SAMPLE_XML) + with self.assertRaisesRegex(SyntaxError, 'cannot use absolute path'): + e.findall('/tag') + + def test_find_through_ElementTree(self): + e = ET.XML(SAMPLE_XML) + self.assertEqual(ET.ElementTree(e).find('tag').tag, 'tag') + self.assertEqual(ET.ElementTree(e).findtext('tag'), 'text') + self.assertEqual(summarize_list(ET.ElementTree(e).findall('tag')), + ['tag'] * 2) + # this produces a warning + self.assertEqual(summarize_list(ET.ElementTree(e).findall('//tag')), + ['tag'] * 3) + + class ElementIterTest(unittest.TestCase): def _ilist(self, elem, tag=None): return summarize_list(elem.iter(tag)) @@ -2547,6 +2507,7 @@ ParseErrorTest, XincludeTest, ElementTreeTest, + ElementFindTest, ElementIterTest, TreeBuilderTest, ] @@ -2560,10 +2521,9 @@ ]) try: - support.run_unittest(*test_classes) - # XXX the C module should give the same warnings as the Python module with CleanContext(quiet=(pyET is not ET)): + support.run_unittest(*test_classes) support.run_doctest(sys.modules[__name__], verbosity=True) finally: # don't interfere with subsequent tests -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 16:45:14 2013 From: python-checkins at python.org (eli.bendersky) Date: Sat, 12 Jan 2013 16:45:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issues_=2315083_and_=2316992=3A_port_find=2E*_method_tes?= =?utf-8?q?ts_to_unittest?= Message-ID: <3Yk4yf1J1RzRnh@mail.python.org> http://hg.python.org/cpython/rev/18b16104166c changeset: 81451:18b16104166c parent: 81449:a1efab48d8f8 parent: 81450:f9d1d120c19e user: Eli Bendersky date: Sat Jan 12 07:43:35 2013 -0800 summary: Issues #15083 and #16992: port find.* method tests to unittest files: Lib/test/test_xml_etree.py | 228 ++++++++++-------------- 1 files changed, 94 insertions(+), 134 deletions(-) diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -329,128 +329,6 @@ 'hello' """ -def find(): - """ - Test find methods (including xpath syntax). - - >>> elem = ET.XML(SAMPLE_XML) - >>> elem.find("tag").tag - 'tag' - >>> ET.ElementTree(elem).find("tag").tag - 'tag' - >>> elem.find("section/tag").tag - 'tag' - >>> elem.find("./tag").tag - 'tag' - >>> ET.ElementTree(elem).find("./tag").tag - 'tag' - >>> ET.ElementTree(elem).find("/tag").tag - 'tag' - >>> elem[2] = ET.XML(SAMPLE_SECTION) - >>> elem.find("section/nexttag").tag - 'nexttag' - >>> ET.ElementTree(elem).find("section/tag").tag - 'tag' - >>> ET.ElementTree(elem).find("tog") - >>> ET.ElementTree(elem).find("tog/foo") - >>> elem.findtext("tag") - 'text' - >>> elem.findtext("section/nexttag") - '' - >>> elem.findtext("section/nexttag", "default") - '' - >>> elem.findtext("tog") - >>> elem.findtext("tog", "default") - 'default' - >>> ET.ElementTree(elem).findtext("tag") - 'text' - >>> ET.ElementTree(elem).findtext("tog/foo") - >>> ET.ElementTree(elem).findtext("tog/foo", "default") - 'default' - >>> ET.ElementTree(elem).findtext("./tag") - 'text' - >>> ET.ElementTree(elem).findtext("/tag") - 'text' - >>> elem.findtext("section/tag") - 'subtext' - >>> ET.ElementTree(elem).findtext("section/tag") - 'subtext' - >>> summarize_list(elem.findall(".")) - ['body'] - >>> summarize_list(elem.findall("tag")) - ['tag', 'tag'] - >>> summarize_list(elem.findall("tog")) - [] - >>> summarize_list(elem.findall("tog/foo")) - [] - >>> summarize_list(elem.findall("*")) - ['tag', 'tag', 'section'] - >>> summarize_list(elem.findall(".//tag")) - ['tag', 'tag', 'tag', 'tag'] - >>> summarize_list(elem.findall("section/tag")) - ['tag'] - >>> summarize_list(elem.findall("section//tag")) - ['tag', 'tag'] - >>> summarize_list(elem.findall("section/*")) - ['tag', 'nexttag', 'nextsection'] - >>> summarize_list(elem.findall("section//*")) - ['tag', 'nexttag', 'nextsection', 'tag'] - >>> summarize_list(elem.findall("section/.//*")) - ['tag', 'nexttag', 'nextsection', 'tag'] - >>> summarize_list(elem.findall("*/*")) - ['tag', 'nexttag', 'nextsection'] - >>> summarize_list(elem.findall("*//*")) - ['tag', 'nexttag', 'nextsection', 'tag'] - >>> summarize_list(elem.findall("*/tag")) - ['tag'] - >>> summarize_list(elem.findall("*/./tag")) - ['tag'] - >>> summarize_list(elem.findall("./tag")) - ['tag', 'tag'] - >>> summarize_list(elem.findall(".//tag")) - ['tag', 'tag', 'tag', 'tag'] - >>> summarize_list(elem.findall("././tag")) - ['tag', 'tag'] - >>> summarize_list(elem.findall(".//tag[@class]")) - ['tag', 'tag', 'tag'] - >>> summarize_list(elem.findall(".//tag[@class='a']")) - ['tag'] - >>> summarize_list(elem.findall(".//tag[@class='b']")) - ['tag', 'tag'] - >>> summarize_list(elem.findall(".//tag[@id]")) - ['tag'] - >>> summarize_list(elem.findall(".//section[tag]")) - ['section'] - >>> summarize_list(elem.findall(".//section[element]")) - [] - >>> summarize_list(elem.findall("../tag")) - [] - >>> summarize_list(elem.findall("section/../tag")) - ['tag', 'tag'] - >>> summarize_list(ET.ElementTree(elem).findall("./tag")) - ['tag', 'tag'] - - Following example is invalid in 1.2. - A leading '*' is assumed in 1.3. - - >>> elem.findall("section//") == elem.findall("section//*") - True - - ET's Path module handles this case incorrectly; this gives - a warning in 1.3, and the behaviour will be modified in 1.4. - - >>> summarize_list(ET.ElementTree(elem).findall("/tag")) - ['tag', 'tag'] - - >>> elem = ET.XML(SAMPLE_XML_NS) - >>> summarize_list(elem.findall("tag")) - [] - >>> summarize_list(elem.findall("{http://effbot.org/ns}tag")) - ['{http://effbot.org/ns}tag', '{http://effbot.org/ns}tag'] - >>> summarize_list(elem.findall(".//{http://effbot.org/ns}tag")) - ['{http://effbot.org/ns}tag', '{http://effbot.org/ns}tag', '{http://effbot.org/ns}tag'] - """ - def file_init(): """ >>> import io @@ -469,16 +347,6 @@ 'empty-element' """ -def bad_find(): - """ - Check bad or unsupported path expressions. - - >>> elem = ET.XML(SAMPLE_XML) - >>> elem.findall("/tag") - Traceback (most recent call last): - SyntaxError: cannot use absolute path on element - """ - def path_cache(): """ Check that the path cache behaves sanely. @@ -1883,6 +1751,98 @@ method='html') self.assertEqual(serialized, expected) + +class ElementFindTest(unittest.TestCase): + def test_find_simple(self): + e = ET.XML(SAMPLE_XML) + self.assertEqual(e.find('tag').tag, 'tag') + self.assertEqual(e.find('section/tag').tag, 'tag') + self.assertEqual(e.find('./tag').tag, 'tag') + + e[2] = ET.XML(SAMPLE_SECTION) + self.assertEqual(e.find('section/nexttag').tag, 'nexttag') + + self.assertEqual(e.findtext('./tag'), 'text') + self.assertEqual(e.findtext('section/tag'), 'subtext') + + # section/nexttag is found but has no text + self.assertEqual(e.findtext('section/nexttag'), '') + self.assertEqual(e.findtext('section/nexttag', 'default'), '') + + # tog doesn't exist and 'default' kicks in + self.assertIsNone(e.findtext('tog')) + self.assertEqual(e.findtext('tog', 'default'), 'default') + + def test_findall(self): + e = ET.XML(SAMPLE_XML) + e[2] = ET.XML(SAMPLE_SECTION) + self.assertEqual(summarize_list(e.findall('.')), ['body']) + self.assertEqual(summarize_list(e.findall('tag')), ['tag', 'tag']) + self.assertEqual(summarize_list(e.findall('tog')), []) + self.assertEqual(summarize_list(e.findall('tog/foo')), []) + self.assertEqual(summarize_list(e.findall('*')), + ['tag', 'tag', 'section']) + self.assertEqual(summarize_list(e.findall('.//tag')), + ['tag'] * 4) + self.assertEqual(summarize_list(e.findall('section/tag')), ['tag']) + self.assertEqual(summarize_list(e.findall('section//tag')), ['tag'] * 2) + self.assertEqual(summarize_list(e.findall('section/*')), + ['tag', 'nexttag', 'nextsection']) + self.assertEqual(summarize_list(e.findall('section//*')), + ['tag', 'nexttag', 'nextsection', 'tag']) + self.assertEqual(summarize_list(e.findall('section/.//*')), + ['tag', 'nexttag', 'nextsection', 'tag']) + self.assertEqual(summarize_list(e.findall('*/*')), + ['tag', 'nexttag', 'nextsection']) + self.assertEqual(summarize_list(e.findall('*//*')), + ['tag', 'nexttag', 'nextsection', 'tag']) + self.assertEqual(summarize_list(e.findall('*/tag')), ['tag']) + self.assertEqual(summarize_list(e.findall('*/./tag')), ['tag']) + self.assertEqual(summarize_list(e.findall('./tag')), ['tag'] * 2) + self.assertEqual(summarize_list(e.findall('././tag')), ['tag'] * 2) + + self.assertEqual(summarize_list(e.findall('.//tag[@class]')), + ['tag'] * 3) + self.assertEqual(summarize_list(e.findall('.//tag[@class="a"]')), + ['tag']) + self.assertEqual(summarize_list(e.findall('.//tag[@class="b"]')), + ['tag'] * 2) + self.assertEqual(summarize_list(e.findall('.//tag[@id]')), + ['tag']) + self.assertEqual(summarize_list(e.findall('.//section[tag]')), + ['section']) + self.assertEqual(summarize_list(e.findall('.//section[element]')), []) + self.assertEqual(summarize_list(e.findall('../tag')), []) + self.assertEqual(summarize_list(e.findall('section/../tag')), + ['tag'] * 2) + self.assertEqual(e.findall('section//'), e.findall('section//*')) + + def test_test_find_with_ns(self): + e = ET.XML(SAMPLE_XML_NS) + self.assertEqual(summarize_list(e.findall('tag')), []) + self.assertEqual( + summarize_list(e.findall("{http://effbot.org/ns}tag")), + ['{http://effbot.org/ns}tag'] * 2) + self.assertEqual( + summarize_list(e.findall(".//{http://effbot.org/ns}tag")), + ['{http://effbot.org/ns}tag'] * 3) + + def test_bad_find(self): + e = ET.XML(SAMPLE_XML) + with self.assertRaisesRegex(SyntaxError, 'cannot use absolute path'): + e.findall('/tag') + + def test_find_through_ElementTree(self): + e = ET.XML(SAMPLE_XML) + self.assertEqual(ET.ElementTree(e).find('tag').tag, 'tag') + self.assertEqual(ET.ElementTree(e).findtext('tag'), 'text') + self.assertEqual(summarize_list(ET.ElementTree(e).findall('tag')), + ['tag'] * 2) + # this produces a warning + self.assertEqual(summarize_list(ET.ElementTree(e).findall('//tag')), + ['tag'] * 3) + + class ElementIterTest(unittest.TestCase): def _ilist(self, elem, tag=None): return summarize_list(elem.iter(tag)) @@ -2547,6 +2507,7 @@ ParseErrorTest, XincludeTest, ElementTreeTest, + ElementFindTest, ElementIterTest, TreeBuilderTest, ] @@ -2560,10 +2521,9 @@ ]) try: - support.run_unittest(*test_classes) - # XXX the C module should give the same warnings as the Python module with CleanContext(quiet=(pyET is not ET)): + support.run_unittest(*test_classes) support.run_doctest(sys.modules[__name__], verbosity=True) finally: # don't interfere with subsequent tests -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 16:45:15 2013 From: python-checkins at python.org (eli.bendersky) Date: Sat, 12 Jan 2013 16:45:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_clean_trailing?= =?utf-8?q?_whitespace?= Message-ID: <3Yk4yg41Z1zS5v@mail.python.org> http://hg.python.org/cpython/rev/bbfc8f62cb67 changeset: 81452:bbfc8f62cb67 branch: 3.3 parent: 81450:f9d1d120c19e user: Eli Bendersky date: Sat Jan 12 07:44:32 2013 -0800 summary: clean trailing whitespace files: Lib/test/test_xml_etree.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -1831,7 +1831,7 @@ e = ET.XML(SAMPLE_XML) with self.assertRaisesRegex(SyntaxError, 'cannot use absolute path'): e.findall('/tag') - + def test_find_through_ElementTree(self): e = ET.XML(SAMPLE_XML) self.assertEqual(ET.ElementTree(e).find('tag').tag, 'tag') @@ -1841,7 +1841,7 @@ # this produces a warning self.assertEqual(summarize_list(ET.ElementTree(e).findall('//tag')), ['tag'] * 3) - + class ElementIterTest(unittest.TestCase): def _ilist(self, elem, tag=None): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 16:45:16 2013 From: python-checkins at python.org (eli.bendersky) Date: Sat, 12 Jan 2013 16:45:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_clean_trailing_whitespace?= Message-ID: <3Yk4yh6d29zS7q@mail.python.org> http://hg.python.org/cpython/rev/7bdeeed5960c changeset: 81453:7bdeeed5960c parent: 81451:18b16104166c parent: 81452:bbfc8f62cb67 user: Eli Bendersky date: Sat Jan 12 07:44:47 2013 -0800 summary: clean trailing whitespace files: Lib/test/test_xml_etree.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -1831,7 +1831,7 @@ e = ET.XML(SAMPLE_XML) with self.assertRaisesRegex(SyntaxError, 'cannot use absolute path'): e.findall('/tag') - + def test_find_through_ElementTree(self): e = ET.XML(SAMPLE_XML) self.assertEqual(ET.ElementTree(e).find('tag').tag, 'tag') @@ -1841,7 +1841,7 @@ # this produces a warning self.assertEqual(summarize_list(ET.ElementTree(e).findall('//tag')), ['tag'] * 3) - + class ElementIterTest(unittest.TestCase): def _ilist(self, elem, tag=None): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 16:56:24 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sat, 12 Jan 2013 16:56:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE2NzYy?= =?utf-8?q?=3A_Fix_some_test=5Fsubprocess_failures_on_NetBSD_and_OpenBSD?= =?utf-8?b?OiBraWxsKCk=?= Message-ID: <3Yk5CX1kYWzS65@mail.python.org> http://hg.python.org/cpython/rev/61d6b34af419 changeset: 81454:61d6b34af419 branch: 2.7 parent: 81440:eae31f2b6f60 user: Charles-Fran?ois Natali date: Sat Jan 12 16:52:20 2013 +0100 summary: Issue #16762: Fix some test_subprocess failures on NetBSD and OpenBSD: kill() returns ESRCH for a zombie process, which is not POSIX-compliant. files: Lib/test/test_subprocess.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -892,6 +892,8 @@ getattr(p, method)(*args) return p + @unittest.skipIf(sys.platform.startswith(('netbsd', 'openbsd')), + "Due to known OS bug (issue #16762)") def _kill_dead_process(self, method, *args): # Do not inherit file handles from the parent. # It should fix failures on some platforms. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 16:56:25 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sat, 12 Jan 2013 16:56:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE2NzYy?= =?utf-8?q?=3A_Fix_some_test=5Fsubprocess_failures_on_NetBSD_and_OpenBSD?= =?utf-8?b?OiBraWxsKCk=?= Message-ID: <3Yk5CY4kZszSDd@mail.python.org> http://hg.python.org/cpython/rev/58ce6ac61ada changeset: 81455:58ce6ac61ada branch: 3.2 parent: 81441:3d54723c9be6 user: Charles-Fran?ois Natali date: Sat Jan 12 16:52:20 2013 +0100 summary: Issue #16762: Fix some test_subprocess failures on NetBSD and OpenBSD: kill() returns ESRCH for a zombie process, which is not POSIX-compliant. files: Lib/test/test_subprocess.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -1206,6 +1206,8 @@ getattr(p, method)(*args) return p + @unittest.skipIf(sys.platform.startswith(('netbsd', 'openbsd')), + "Due to known OS bug (issue #16762)") def _kill_dead_process(self, method, *args): # Do not inherit file handles from the parent. # It should fix failures on some platforms. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 16:56:27 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sat, 12 Jan 2013 16:56:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=2316762=3A_Fix_some_test=5Fsubprocess_failures_on_NetBS?= =?utf-8?q?D_and_OpenBSD=3A_kill=28=29?= Message-ID: <3Yk5Cb0vJQzSF9@mail.python.org> http://hg.python.org/cpython/rev/a3f0414af55b changeset: 81456:a3f0414af55b branch: 3.3 parent: 81452:bbfc8f62cb67 parent: 81455:58ce6ac61ada user: Charles-Fran?ois Natali date: Sat Jan 12 16:54:45 2013 +0100 summary: Issue #16762: Fix some test_subprocess failures on NetBSD and OpenBSD: kill() returns ESRCH for a zombie process, which is not POSIX-compliant. files: Lib/test/test_subprocess.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -1360,6 +1360,8 @@ getattr(p, method)(*args) return p + @unittest.skipIf(sys.platform.startswith(('netbsd', 'openbsd')), + "Due to known OS bug (issue #16762)") def _kill_dead_process(self, method, *args): # Do not inherit file handles from the parent. # It should fix failures on some platforms. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 16:56:28 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sat, 12 Jan 2013 16:56:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2316762=3A_Fix_some_test=5Fsubprocess_failures_on?= =?utf-8?q?_NetBSD_and_OpenBSD=3A_kill=28=29?= Message-ID: <3Yk5Cc3d1zzSFK@mail.python.org> http://hg.python.org/cpython/rev/487ed428f0ba changeset: 81457:487ed428f0ba parent: 81453:7bdeeed5960c parent: 81456:a3f0414af55b user: Charles-Fran?ois Natali date: Sat Jan 12 16:55:31 2013 +0100 summary: Issue #16762: Fix some test_subprocess failures on NetBSD and OpenBSD: kill() returns ESRCH for a zombie process, which is not POSIX-compliant. files: Lib/test/test_subprocess.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -1360,6 +1360,8 @@ getattr(p, method)(*args) return p + @unittest.skipIf(sys.platform.startswith(('netbsd', 'openbsd')), + "Due to known OS bug (issue #16762)") def _kill_dead_process(self, method, *args): # Do not inherit file handles from the parent. # It should fix failures on some platforms. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 17:20:26 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 12 Jan 2013 17:20:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE2ODI5?= =?utf-8?q?=3A_IDLE_printing_no_longer_fails_if_there_are_spaces_or_other?= Message-ID: <3Yk5lG1nl0zS2D@mail.python.org> http://hg.python.org/cpython/rev/e651d96e6b07 changeset: 81458:e651d96e6b07 branch: 2.7 parent: 81454:61d6b34af419 user: Serhiy Storchaka date: Sat Jan 12 18:12:27 2013 +0200 summary: Issue #16829: IDLE printing no longer fails if there are spaces or other special characters in the file path. files: Lib/idlelib/IOBinding.py | 3 ++- Misc/NEWS | 3 +++ 2 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Lib/idlelib/IOBinding.py b/Lib/idlelib/IOBinding.py --- a/Lib/idlelib/IOBinding.py +++ b/Lib/idlelib/IOBinding.py @@ -7,6 +7,7 @@ import os import types +import pipes import sys import codecs import tempfile @@ -503,7 +504,7 @@ else: #no printing for this platform printPlatform = False if printPlatform: #we can try to print for this platform - command = command % filename + command = command % pipes.quote(filename) pipe = os.popen(command, "r") # things can get ugly on NT if there is no printer available. output = pipe.read().strip() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -186,6 +186,9 @@ Library ------- +- Issue #16829: IDLE printing no longer fails if there are spaces or other + special characters in the file path. + - Issue #13899: \A, \Z, and \B now correctly match the A, Z, and B literals when used inside character classes (e.g. '[\A]'). Patch by Matthew Barnett. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 17:20:27 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 12 Jan 2013 17:20:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE2ODI5?= =?utf-8?q?=3A_IDLE_printing_no_longer_fails_if_there_are_spaces_or_other?= Message-ID: <3Yk5lH4t8wzS8d@mail.python.org> http://hg.python.org/cpython/rev/20065626c0b5 changeset: 81459:20065626c0b5 branch: 3.2 parent: 81455:58ce6ac61ada user: Serhiy Storchaka date: Sat Jan 12 18:13:24 2013 +0200 summary: Issue #16829: IDLE printing no longer fails if there are spaces or other special characters in the file path. files: Lib/idlelib/IOBinding.py | 3 ++- Misc/NEWS | 3 +++ 2 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Lib/idlelib/IOBinding.py b/Lib/idlelib/IOBinding.py --- a/Lib/idlelib/IOBinding.py +++ b/Lib/idlelib/IOBinding.py @@ -1,5 +1,6 @@ import os import types +import pipes import sys import codecs import tempfile @@ -458,7 +459,7 @@ else: #no printing for this platform printPlatform = False if printPlatform: #we can try to print for this platform - command = command % filename + command = command % pipes.quote(filename) pipe = os.popen(command, "r") # things can get ugly on NT if there is no printer available. output = pipe.read().strip() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -199,6 +199,9 @@ Library ------- +- Issue #16829: IDLE printing no longer fails if there are spaces or other + special characters in the file path. + - Issue #13899: \A, \Z, and \B now correctly match the A, Z, and B literals when used inside character classes (e.g. '[\A]'). Patch by Matthew Barnett. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 17:20:29 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 12 Jan 2013 17:20:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=2316829=3A_IDLE_printing_no_longer_fails_if_there_are_s?= =?utf-8?q?paces_or_other?= Message-ID: <3Yk5lK0p4pzS8d@mail.python.org> http://hg.python.org/cpython/rev/778bead39825 changeset: 81460:778bead39825 branch: 3.3 parent: 81456:a3f0414af55b parent: 81459:20065626c0b5 user: Serhiy Storchaka date: Sat Jan 12 18:16:18 2013 +0200 summary: Issue #16829: IDLE printing no longer fails if there are spaces or other special characters in the file path. files: Lib/idlelib/IOBinding.py | 3 ++- Misc/NEWS | 3 +++ 2 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Lib/idlelib/IOBinding.py b/Lib/idlelib/IOBinding.py --- a/Lib/idlelib/IOBinding.py +++ b/Lib/idlelib/IOBinding.py @@ -1,5 +1,6 @@ import os import types +import shlex import sys import codecs import tempfile @@ -458,7 +459,7 @@ else: #no printing for this platform printPlatform = False if printPlatform: #we can try to print for this platform - command = command % filename + command = command % shlex.quote(filename) pipe = os.popen(command, "r") # things can get ugly on NT if there is no printer available. output = pipe.read().strip() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -147,6 +147,9 @@ Library ------- +- Issue #16829: IDLE printing no longer fails if there are spaces or other + special characters in the file path. + - Issue #16900: Issue a ResourceWarning when an ssl socket is left unclosed. - Issue #13899: \A, \Z, and \B now correctly match the A, Z, and B literals -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 17:20:30 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 12 Jan 2013 17:20:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2316829=3A_IDLE_printing_no_longer_fails_if_there?= =?utf-8?q?_are_spaces_or_other?= Message-ID: <3Yk5lL3hyszS9S@mail.python.org> http://hg.python.org/cpython/rev/529b5ced59e0 changeset: 81461:529b5ced59e0 parent: 81457:487ed428f0ba parent: 81460:778bead39825 user: Serhiy Storchaka date: Sat Jan 12 18:17:24 2013 +0200 summary: Issue #16829: IDLE printing no longer fails if there are spaces or other special characters in the file path. files: Lib/idlelib/IOBinding.py | 3 ++- Misc/NEWS | 3 +++ 2 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Lib/idlelib/IOBinding.py b/Lib/idlelib/IOBinding.py --- a/Lib/idlelib/IOBinding.py +++ b/Lib/idlelib/IOBinding.py @@ -1,5 +1,6 @@ import os import types +import shlex import sys import codecs import tempfile @@ -458,7 +459,7 @@ else: #no printing for this platform printPlatform = False if printPlatform: #we can try to print for this platform - command = command % filename + command = command % shlex.quote(filename) pipe = os.popen(command, "r") # things can get ugly on NT if there is no printer available. output = pipe.read().strip() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -217,6 +217,9 @@ Library ------- +- Issue #16829: IDLE printing no longer fails if there are spaces or other + special characters in the file path. + - Issue #15031: Refactor some .pyc management code to cut down on code duplication. Thanks to Ronan Lamy for the report and taking an initial stab at the problem. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 17:34:55 2013 From: python-checkins at python.org (r.david.murray) Date: Sat, 12 Jan 2013 17:34:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_=2316259=3A_delete_some_no?= =?utf-8?q?-longer-used_code_from_regrtest=2E?= Message-ID: <3Yk63z5nlLzS5j@mail.python.org> http://hg.python.org/cpython/rev/e22c09f636d4 changeset: 81462:e22c09f636d4 user: R David Murray date: Sat Jan 12 11:34:38 2013 -0500 summary: #16259: delete some no-longer-used code from regrtest. dash_R is only called from one location, and from that location an indirect_test is passed. files: Lib/test/regrtest.py | 10 +--------- 1 files changed, 1 insertions(+), 9 deletions(-) diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py --- a/Lib/test/regrtest.py +++ b/Lib/test/regrtest.py @@ -1386,14 +1386,6 @@ for obj in abc.__subclasses__() + [abc]: abcs[obj] = obj._abc_registry.copy() - if indirect_test: - def run_the_test(): - indirect_test() - else: - def run_the_test(): - del sys.modules[the_module.__name__] - exec('import ' + the_module.__name__) - nwarmup, ntracked, fname = huntrleaks fname = os.path.join(support.SAVEDCWD, fname) repcount = nwarmup + ntracked @@ -1404,7 +1396,7 @@ print(("1234567890"*(repcount//10 + 1))[:repcount], file=sys.stderr) sys.stderr.flush() for i in range(repcount): - run_the_test() + indirect_test() alloc_after, rc_after = dash_R_cleanup(fs, ps, pic, zdc, abcs) sys.stderr.write('.') sys.stderr.flush() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 21:17:12 2013 From: python-checkins at python.org (victor.stinner) Date: Sat, 12 Jan 2013 21:17:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Add_a_new_PEP=3A_333=3A_Add_c?= =?utf-8?q?loexec_argument_to_functions_creating_file_descriptors?= Message-ID: <3YkC0S160GzS8d@mail.python.org> http://hg.python.org/peps/rev/2033ba1eeabd changeset: 4666:2033ba1eeabd user: Victor Stinner date: Sat Jan 12 21:15:11 2013 +0100 summary: Add a new PEP: 333: Add cloexec argument to functions creating file descriptors files: pep-0433.txt | 519 +++++++++++++++++++++++++++++++++++++++ 1 files changed, 519 insertions(+), 0 deletions(-) diff --git a/pep-0433.txt b/pep-0433.txt new file mode 100644 --- /dev/null +++ b/pep-0433.txt @@ -0,0 +1,519 @@ +PEP: 433 +Title: Add cloexec argument to functions creating file descriptors +Version: $Revision$ +Last-Modified: $Date$ +Author: Victor Stinner +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 10-January-2013 +Python-Version: 3.4 + + +Abstract +======== + +This PEP proposes to add a new optional argument ``cloexec`` on functions +creating file descriptors in the Python standard library. If the argument is +``True``, the close-on-exec flag will be set on the new file descriptor. + + +Rationale +========= + +On UNIX, subprocess closes file descriptors greater than 2 by default since +Python 3.2 [#subprocess_close]_. All file descriptors created by the parent +process are automatically closed. + +There are other cases creating a subprocess or executing a new program where +file descriptors are not closed: functions of the os.spawn*() family and third +party modules calling ``exec()`` or ``fork()`` + ``exec()``. In this case, file +descriptors are shared between the parent and the child processes which is +usually unexpected and causes various issues. + + +Inherited file descriptors issues +--------------------------------- + +Closing the file descriptor in the parent process does not close the related +resource (file, socket, ...) because it is still open in the child process. + +The listening socket of TCPServer is not closed on ``exec()``: the child +process is able to get connection from new clients; if the parent closes the +listening socket and create a new listening socket on the same address, it +would get an "address already is used" error. + +Not closing file descriptors can lead to resource exhaustion: even if the +parent closes all files, creating a new file descriptor may fail with "too many +files" because files are still open in the child process. + + +Security +-------- + +Leaking file descriptors is a major security vulnerability. An untrusted child +process can read sensitive data like passwords and take control of the parent +process though leaked file descriptors. It is for example a known vulnerability +to escape from a chroot. + + +Atomicity +--------- + +Using ``fcntl()`` to set the close-on-exec flag is not safe in a multithreaded +application. If a thread calls ``fork()`` and ``exec()`` between the creation +of the file descriptor and the call to ``fcntl(fd, F_SETFD, new_flags)``: the +file descriptor will be inherited by the child process. Modern operating +systems offer functions to set the flag during the creation of the file +descriptor, which avoids the race condition. + + +Portability +----------- + +Python 3.2 added ``socket.SOCK_CLOEXEC`` flag, Python 3.3 added +``os.O_CLOEXEC`` flag and ``os.pipe2()`` function. It is already possible to +set atomically close-on-exec flag in Python 3.3 when opening a file and +creating a pipe or socket. + +The problem is that these flags and functions are not portable: only recent +versions of operating systems support them. ``O_CLOEXEC`` and ``SOCK_CLOEXEC`` +flags are ignored by old Linux versions and so ``FD_CLOEXEC`` flag must be +checked using ``fcntl(fd, F_GETFD)``. If the kernel ignores ``O_CLOEXEC`` or +``SOCK_CLOEXEC`` flag, a call to ``fcntl(fd, F_SETFD, flags)`` is required to +set close-on-exec flag. + +Note: OpenBSD older 5.2 does not close the file descriptor with close-on-exec +flag set if ``fork()`` is used before ``exec()``, but it works correctly if +``exec()`` is called without ``fork()``. + + +Scope +----- + +Applications still have to close explicitly file descriptors after a +``fork()``. The close-on-exec flag only closes file descriptors after +``exec()``, and so after ``fork()`` + ``exec()``. + +Many functions of the Python standard library creating file descriptors are not +changed by the PEP, and so will not have the close-on-exec flag set. Some +examples: + + * ``os.urandom()``: on UNIX, it creates a file descriptor on UNIX to read + ``/dev/urandom``. Adding an ``cloexec`` argument to ``os.urandom()`` does + not make sense on Windows. + * ``curses.windows.getwin()`` and ``curses.windows.putwin()`` creates a temporary file using ``fdopen(fd, "wb+");`` + * ``mmap.mmap()`` opens ``/dev/null`` using ``open("/dev/zero", O_RDWR);`` if + ``MAP_ANONYMOUS`` is not defined. + * If the ``PYTHONSTARTUP`` environment variable is set, the corresponding file + is opened using ``fopen(startup, "r");`` + * ``python script.py`` opens ``script.py`` using ``fopen(filename, "r");`` + * etc. + +Third party modules creating file descriptors may not set close-on-exec flag. + +Impacted functions: + + * ``os.forkpty()`` + * ``http.server.CGIHTTPRequestHandler.run_cgi()`` + +Impacted modules: + + * ``multiprocessing`` + * ``socketserver`` + * ``subprocess`` + * ``tempfile`` + * ``xmlrpc.server`` + * Maybe: ``signal``, ``threading`` + +XXX Should ``subprocess.Popen`` set the close-on-exec flag on file XXX +XXX descriptors of the constructor the ``pass_fds`` argument? XXX + + +Proposition +=========== + +This PEP proposes to add a new optional argument ``cloexec`` on functions +creating file descriptors in the Python standard library. If the argument is +``True``, the close-on-exec flag will be set on the new file descriptor. + +Add a new function: + + * ``os.set_cloexec(fd: int, cloexec: bool)``: set or unset the close-on-exec + flag of a file descriptor + +Add a new optional ``cloexec`` argument to: + + * ``open()``: ``os.fdopen()`` is indirectly modified + * ``os.dup()``, ``os.dup2()`` + * ``os.pipe()`` + * ``socket.socket()``, ``socket.socketpair()`` ``socket.socket.accept()`` + * Maybe also: ``os.open()``, ``os.openpty()`` + * TODO: + + * ``select.devpoll()`` + * ``select.poll()`` + * ``select.epoll()`` + * ``select.kqueue()`` + * ``socket.socket.recvmsg()``: use ``MSG_CMSG_CLOEXEC``, or ``os.set_cloexec()`` + +The default value of the ``cloexec`` argument is ``False`` to keep the backward +compatibility. + + + + + + +Applications using inherance of file descriptors +================================================ + +Network servers using fork may want to pass the client socket to the child +process. For example, a CGI server pass the socket client through file +descriptors 0 (stdin) and 1 (stdout) using ``dup2()``. + +Example of programs taking file descriptors from the parent process: + + * valgrind: ``--log-fd=``, ``--input-fd=``, etc. + * qemu: ``-add-fd `` command line option + * GnuPG: ``--status-fd ``, ``--logger-fd ``, etc. + * openssl command: ``-pass fd:`` + * xterm: ``-S `` + +On Linux, it is possible to use ``/dev/fd/`` filename to pass a file +descriptor to a program expecting a filename. It can be used to pass a password +for example. + +These applications only pass a few file descriptors, usually only one. +Fixing these applications to unset close-on-exec flag should be easy. + +If the ``subprocess`` module is used, inherited file descriptors must be specified +using the ``pass_fds`` argument (except if the ``close_fds`` argument is set +explicitly to ``False``). So the ``subprocess`` module knows the list of file +descriptors on which close-on-exec flag must be unset. + +File descriptors 0 (stdin), 1 (stdout) and 2 (stderr) are expected to be +inherited and so should not have the close-on-exec flag. So a CGI server should +not be impacted by this PEP. + + +Performances +============ + +Setting close-on-exec flag may require additional system calls for each +creation of new file descriptors. The number of additional system calls +depends on the method used to set the flag: + + * ``O_NOINHERIT``: no additionnal system call + * ``O_CLOEXEC``: one addition system call, but only at the creation of the + first file descriptor, to check if the flag is supported. If no, Python has + to fallback to the next method. + * ``ioctl(fd, FIOCLEX)``: one addition system call per file descriptor + * ``fcntl(fd, F_SETFD, flags)``: two addition system calls per file + descriptor, one to get old flags and one to set new flags + +XXX Benchmark the overhead for these 4 methods. XXX + + +Implementation +============== + +os.set_cloexec(fd, cloexec) +--------------------------- + +Best-effort by definition. Pseudo-code:: + + if os.name == 'nt': + def set_cloexec(fd, cloexec=True): + SetHandleInformation(fd, HANDLE_FLAG_INHERIT, int(cloexec)) + else: + fnctl = None + ioctl = None + try: + import ioctl + except ImportError: + try: + import fcntl + except ImportError: + pass + if ioctl is not None and hasattr('FIOCLEX', ioctl): + def set_cloexec(fd, cloexec=True): + if cloexec: + ioctl.ioctl(fd, ioctl.FIOCLEX) + else: + ioctl.ioctl(fd, ioctl.FIONCLEX) + elif fnctl is not None: + def set_cloexec(fd, cloexec=True): + flags = fcntl.fcntl(fd, fcntl.F_GETFD) + if cloexec: + flags |= FD_CLOEXEC + else: + flags &= ~FD_CLOEXEC + fcntl.fcntl(fd, fcntl.F_SETFD, flags) + else: + def set_cloexec(fd, cloexec=True): + raise NotImplementedError("close-on-exec flag is not supported on your platform") + +ioctl is preferred over fcntl because it requires only one syscall, instead of +two syscalls for fcntl. + +Note: ``fcntl(fd, F_SETFD, flags)`` only supports one flag (``FD_CLOEXEC``), so +it would be possible to avoid ``fcntl(fd, F_GETFD)``. But it may drop other +flags in the future, and so it is safer to keep the two functions calls. + +open() +------ + + * Windows: ``open()`` with ``O_NOINHERIT`` flag [atomic] + * ``open()`` with ``O_CLOEXEC flag`` [atomic] + * ``open()`` + ``os.set_cloexec(fd, True)`` [best-effort] + +os.dup() +-------- + + * ``fcntl(fd, F_DUPFD_CLOEXEC)`` [atomic] + * ``dup()`` + ``os.set_cloexec(fd, True)`` [best-effort] + +os.dup2() +--------- + + * ``dup3()`` with ``O_CLOEXEC`` flag [atomic] + * ``dup2()`` + ``os.set_cloexec(fd, True)`` [best-effort] + +os.pipe() +--------- + + * Windows: ``_pipe()`` with ``O_NOINHERIT`` flag [atomic] + * ``pipe2()`` with ``O_CLOEXEC`` flag [atomic] + * ``pipe()`` + ``os.set_cloexec(fd, True)`` [best-effort] + +socket.socket() +--------------- + + * ``socket()`` with ``SOCK_CLOEXEC`` flag [atomic] + * ``socket()`` + ``os.set_cloexec(fd, True)`` [best-effort] + +socket.socketpair() +------------------- + + * ``socketpair()`` with ``SOCK_CLOEXEC`` flag [atomic] + * ``socketpair()`` + ``os.set_cloexec(fd, True)`` [best-effort] + +socket.socket.accept() +---------------------- + + * ``accept4()`` with ``SOCK_CLOEXEC`` flag [atomic] + * ``accept()`` + ``os.set_cloexec(fd, True)`` [best-effort] + + +Backward compatibility +====================== + +There is no backward incompatible change. The default behaviour is unchanged: +the close-on-exec flag is not set by default. + + +Alternatives +============ + +Always set close-on-exec flag +----------------------------- + +Always set close-on-exec flag on new file descriptors created by Python. This +alternative just changes the default value of the new ``cloexec`` argument. + +``subprocess.Popen`` constructor has an ``pass_fds`` argument to specify which +file descriptors must be inherited. The close-on-exec flag of these file +descriptors must be changed with ``os.set_cloexec()``. + +If the close-on-exec flag must not be set, ``cloexec=False`` can be specified. + +Advantages of setting close-on-exec flag by default: + + * There are far more programs that are bitten by FD inheritance upon + exec (see `Inherited file descriptors issues`_ and `Security`_) than + programs relying on it. + * Checking if a module creates file descriptors is difficult. For example, + ``os.urandom()`` creates a file descriptor on UNIX to read ``/dev/urandom`` + (and closes it at exit), whereas it is implemented using a function call on + Windows. It is not possible to control close-on-exec flag of the file + descriptor used by ``os.urandom()``, because ``os.urandom()`` API does not + allow it. + * No need to add a new ``cloexec`` argument everywhere: functions creating + file descriptors will read ``sys.getdefaultcloexec()`` to decide if the + close-on-exec must be set or not. For example, adding an ``cloexec`` + argument to ``os.urandom()`` does not make sense on Windows. + +Drawbacks of setting close-on-exec flag by default: + + * The os module is written as a thin wrapper to system calls (to functions of + the C standard library). If atomic flags are not supported, a single Python + function call may now call 2 or 3 system calls (see `Performances`_ + section). + * Extra system calls, if any, may slow down Python: see `Performances`_. + * It violates the principle of least surprise. Developers using the os module + may expect that Python respects the POSIX standard and so that close-on-exec + flag is not set by default. + * Only file descriptors created by the Python standard library will comply to + ``sys.setdefaultcloexec()``. The close-on-exec flag is unchanged for file + descriptors created by third party modules calling directly C functions. + Third party modules will have to be modified to read + ``sys.getdefaultcloexec()`` to make them comply to this PEP. + + +Add a function to set close-on-exec flag by default +--------------------------------------------------- + +An alternative is to add also a function to change globally the default +behaviour. It would be possible to set close-on-exec flag for the whole +application including all modules and the Python standard library. This +alternative is based on the PEP but adds extra changes. + +Add new functions: + + * ``sys.getdefaultcloexec() -> bool``: get the default value of the + close-on-exec flag for new file descriptor + * ``sys.setdefaultcloexec(cloexec: bool)``: enable or disable close-on-exec + flag, the state of the flag can be overriden in each function creating a + file descriptor + +The major change is that the default value of the ``cloexec`` argument is +``sys.getdefaultcloexec()``, instead of ``False``. + +When ``sys.setdefaultcloexec(True)`` is called to set close-on-exec by default, +we have the same drawbacks than `Always set close-on-exec +flag`_ alternative. + +There are additionnal drawbacks of having two behaviours depending on +``sys.getdefaultcloexec()`` value: + + * It is not more possible to know if the close-on-exec flag will be set or not + just by reading the source code. + + +open(): add "e" flag to mode +---------------------------- + +A new "e" mode would set close-on-exec flag (best-effort). + +This API does not allow to disable explictly close-on-exec flag if it was +enabled globally with ``sys.setdefaultcloexec()``. + +Note: Since its version 2.7, the GNU libc supports ``"e"`` flag for ``fopen()``. +It uses ``O_CLOEXEC`` if available, or use ``fcntl(fd, F_SETFD, FD_CLOEXEC)``. + + +Appendix: Operating system support +================================== + +Windows +------- + +Windows has an ``O_NOINHERIT`` flag: "Do not inherit in child processes". + +For example, it is supported by ``open()`` and ``_pipe()``. + +The value of the flag can be modified using: +``SetHandleInformation(fd, HANDLE_FLAG_INHERIT, 1)``. + +``CreateProcess()`` has an ``bInheritHandles`` argument: if it is FALSE, the +handles are not inherited. It is used by ``subprocess.Popen`` with +``close_fds`` option. + +fcntl +----- + +Functions: + + * ``fcntl(fd, F_GETFD)`` + * ``fcntl(fd, F_SETFD, flags | FD_CLOEXEC)`` + +Availability: AIX, Digital UNIX, FreeBSD, HP-UX, IRIX, Linux, Mac OS X, +OpenBSD, Solaris, SunOS, Unicos. + +ioctl +----- + +Functions: + + * ``ioctl(fd, FIOCLEX, 0)`` sets close-on-exec flag + * ``ioctl(fd, FIONCLEX, 0)`` unsets close-on-exec flag + +Availability: Linux, Mac OS X, QNX, NetBSD, OpenBSD, FreeBSD. + + +Atomic flags +------------ + +New flags: + + * ``O_CLOEXEC``: available on Linux (2.6.23+), FreeBSD (8.3+), OpenBSD 5.0, + will be part of the next NetBSD release (6.1?). This flag is part of + POSIX.1-2008. + * ``socket()``: ``SOCK_CLOEXEC`` flag, available on Linux 2.6.27+, + OpenBSD 5.2, NetBSD 6.0. + * ``fcntl()``: ``F_DUPFD_CLOEXEC`` flag, available on Linux 2.6.24+, + OpenBSD 5.0, FreeBSD 9.1, NetBSD 6.0. This flag is part of POSIX.1-2008. + * ``recvmsg()``: ``MSG_CMSG_CLOEXEC``, available on Linux 2.6.23+, NetBSD 6.0. + +On Linux older than 2.6.23, ``O_CLOEXEC`` flag is simply ignored. So we have to +check that the flag is supported by calling ``fcntl()``. If it does not work, +we have to set the flag using ``fcntl()``. + +XXX what is the behaviour on Linux older than 2.6.27 with SOCK_CLOEXEC? XXX + +New functions: + + * ``dup3()``: available on Linux 2.6.27+ (and glibc 2.9) + * ``pipe2()``: available on Linux 2.6.27+ (and glibc 2.9) + * ``accept4()``: available on Linux 2.6.28+ (and glibc 2.10) + +If ``accept4()`` is called on Linux older than 2.6.28, ``accept4()`` returns +``-1`` (fail) and errno is set to ``ENOSYS``. + + +Links +===== + +Links: + + * `Secure File Descriptor Handling + `_ (Ulrich Drepper, 2008) + * `win32_support.py of the Tornado project + `_: + emulate fcntl(fd, F_SETFD, FD_CLOEXEC) using + ``SetHandleInformation(fd, HANDLE_FLAG_INHERIT, 1)`` + +Python issues: + + * `open() does not able to set flags, such as O_CLOEXEC + `_ + * `Add "e" mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT + `_ + * `TCP listening sockets created without FD_CLOEXEC flag + `_ + * `Use O_CLOEXEC in the tempfile module + `_ + * `Support accept4() for atomic setting of flags at socket creation + `_ + * `Add an 'afterfork' module + `_ + +Ruby: + + * `Set FD_CLOEXEC for all fds (except 0, 1, 2) + `_ + * `O_CLOEXEC flag missing for Kernel::open + `_: + `commit reverted + `_ later + +Footnotes +========= + +.. [#subprocess_close] On UNIX since Python 3.2, subprocess.Popen() closes all file descriptors by + default: ``close_fds=True``. It closes file descriptors in range 3 inclusive + to ``local_max_fd`` exclusive, where ``local_max_fd`` is ``fcntl(0, + F_MAXFD)`` on NetBSD, or ``sysconf(_SC_OPEN_MAX)`` otherwise. If the error + pipe has a descriptor smaller than 3, ``ValueError`` is raised. + -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sat Jan 12 21:47:14 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 12 Jan 2013 21:47:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogU1NMQ29udGV4dC5s?= =?utf-8?q?oad=5Fdh=5Fparams=28=29_now_properly_closes_the_input_file=2E?= Message-ID: <3YkCg61WZwzS2B@mail.python.org> http://hg.python.org/cpython/rev/523d5b1e3fd5 changeset: 81463:523d5b1e3fd5 branch: 3.3 parent: 81460:778bead39825 user: Antoine Pitrou date: Sat Jan 12 21:43:45 2013 +0100 summary: SSLContext.load_dh_params() now properly closes the input file. files: Misc/NEWS | 2 ++ Modules/_ssl.c | 1 + 2 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -147,6 +147,8 @@ Library ------- +- SSLContext.load_dh_params() now properly closes the input file. + - Issue #16829: IDLE printing no longer fails if there are spaces or other special characters in the file path. diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -2177,6 +2177,7 @@ errno = 0; PySSL_BEGIN_ALLOW_THREADS dh = PEM_read_DHparams(f, NULL, NULL, NULL); + fclose(f); PySSL_END_ALLOW_THREADS if (dh == NULL) { if (errno != 0) { -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 21:47:15 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 12 Jan 2013 21:47:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_SSLContext=2Eload=5Fdh=5Fparams=28=29_now_properly_close?= =?utf-8?q?s_the_input_file=2E?= Message-ID: <3YkCg74bV0zS47@mail.python.org> http://hg.python.org/cpython/rev/44e614446f78 changeset: 81464:44e614446f78 parent: 81462:e22c09f636d4 parent: 81463:523d5b1e3fd5 user: Antoine Pitrou date: Sat Jan 12 21:44:33 2013 +0100 summary: SSLContext.load_dh_params() now properly closes the input file. files: Misc/NEWS | 2 ++ Modules/_ssl.c | 1 + 2 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -217,6 +217,8 @@ Library ------- +- SSLContext.load_dh_params() now properly closes the input file. + - Issue #16829: IDLE printing no longer fails if there are spaces or other special characters in the file path. diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -2248,6 +2248,7 @@ errno = 0; PySSL_BEGIN_ALLOW_THREADS dh = PEM_read_DHparams(f, NULL, NULL, NULL); + fclose(f); PySSL_END_ALLOW_THREADS if (dh == NULL) { if (errno != 0) { -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 21:59:28 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 12 Jan 2013 21:59:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE2OTIz?= =?utf-8?q?=3A_Fix_ResourceWarnings_in_test=5Fssl=2E?= Message-ID: <3YkCxD3LGczS3g@mail.python.org> http://hg.python.org/cpython/rev/2c3f5ed7a5c9 changeset: 81465:2c3f5ed7a5c9 branch: 3.3 parent: 81463:523d5b1e3fd5 user: Antoine Pitrou date: Sat Jan 12 21:54:44 2013 +0100 summary: Issue #16923: Fix ResourceWarnings in test_ssl. files: Lib/test/test_ssl.py | 68 ++++++++++++++++--------------- Misc/NEWS | 2 + 2 files changed, 37 insertions(+), 33 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -208,20 +208,21 @@ s = socket.socket(socket.AF_INET) ss = ssl.wrap_socket(s) wr = weakref.ref(ss) - del ss - self.assertEqual(wr(), None) + with support.check_warnings(("", ResourceWarning)): + del ss + self.assertEqual(wr(), None) def test_wrapped_unconnected(self): # Methods on an unconnected SSLSocket propagate the original # socket.error raise by the underlying socket object. s = socket.socket(socket.AF_INET) - ss = ssl.wrap_socket(s) - self.assertRaises(socket.error, ss.recv, 1) - self.assertRaises(socket.error, ss.recv_into, bytearray(b'x')) - self.assertRaises(socket.error, ss.recvfrom, 1) - self.assertRaises(socket.error, ss.recvfrom_into, bytearray(b'x'), 1) - self.assertRaises(socket.error, ss.send, b'x') - self.assertRaises(socket.error, ss.sendto, b'x', ('0.0.0.0', 0)) + with ssl.wrap_socket(s) as ss: + self.assertRaises(socket.error, ss.recv, 1) + self.assertRaises(socket.error, ss.recv_into, bytearray(b'x')) + self.assertRaises(socket.error, ss.recvfrom, 1) + self.assertRaises(socket.error, ss.recvfrom_into, bytearray(b'x'), 1) + self.assertRaises(socket.error, ss.send, b'x') + self.assertRaises(socket.error, ss.sendto, b'x', ('0.0.0.0', 0)) def test_timeout(self): # Issue #8524: when creating an SSL socket, the timeout of the @@ -229,8 +230,8 @@ for timeout in (None, 0.0, 5.0): s = socket.socket(socket.AF_INET) s.settimeout(timeout) - ss = ssl.wrap_socket(s) - self.assertEqual(timeout, ss.gettimeout()) + with ssl.wrap_socket(s) as ss: + self.assertEqual(timeout, ss.gettimeout()) def test_errors(self): sock = socket.socket() @@ -243,9 +244,9 @@ self.assertRaisesRegex(ValueError, "certfile must be specified for server-side operations", ssl.wrap_socket, sock, server_side=True, certfile="") - s = ssl.wrap_socket(sock, server_side=True, certfile=CERTFILE) - self.assertRaisesRegex(ValueError, "can't connect in server-side mode", - s.connect, (HOST, 8080)) + with ssl.wrap_socket(sock, server_side=True, certfile=CERTFILE) as s: + self.assertRaisesRegex(ValueError, "can't connect in server-side mode", + s.connect, (HOST, 8080)) with self.assertRaises(IOError) as cm: with socket.socket() as sock: ssl.wrap_socket(sock, certfile=WRONGCERT) @@ -358,21 +359,21 @@ def test_unknown_channel_binding(self): # should raise ValueError for unknown type s = socket.socket(socket.AF_INET) - ss = ssl.wrap_socket(s) - with self.assertRaises(ValueError): - ss.get_channel_binding("unknown-type") + with ssl.wrap_socket(s) as ss: + with self.assertRaises(ValueError): + ss.get_channel_binding("unknown-type") @unittest.skipUnless("tls-unique" in ssl.CHANNEL_BINDING_TYPES, "'tls-unique' channel binding not available") def test_tls_unique_channel_binding(self): # unconnected should return None for known type s = socket.socket(socket.AF_INET) - ss = ssl.wrap_socket(s) - self.assertIsNone(ss.get_channel_binding("tls-unique")) + with ssl.wrap_socket(s) as ss: + self.assertIsNone(ss.get_channel_binding("tls-unique")) # the same for server-side s = socket.socket(socket.AF_INET) - ss = ssl.wrap_socket(s, server_side=True, certfile=CERTFILE) - self.assertIsNone(ss.get_channel_binding("tls-unique")) + with ssl.wrap_socket(s, server_side=True, certfile=CERTFILE) as ss: + self.assertIsNone(ss.get_channel_binding("tls-unique")) def test_dealloc_warn(self): ss = ssl.wrap_socket(socket.socket(socket.AF_INET)) @@ -623,10 +624,10 @@ with socket.socket() as s: s.bind(("127.0.0.1", 0)) s.listen(5) - with socket.socket() as c: - c.connect(s.getsockname()) - c.setblocking(False) - c = ctx.wrap_socket(c, False, do_handshake_on_connect=False) + c = socket.socket() + c.connect(s.getsockname()) + c.setblocking(False) + with ctx.wrap_socket(c, False, do_handshake_on_connect=False) as c: with self.assertRaises(ssl.SSLWantReadError) as cm: c.do_handshake() s = str(cm.exception) @@ -867,12 +868,12 @@ def test_ciphers(self): remote = ("svn.python.org", 443) with support.transient_internet(remote[0]): - s = ssl.wrap_socket(socket.socket(socket.AF_INET), - cert_reqs=ssl.CERT_NONE, ciphers="ALL") - s.connect(remote) - s = ssl.wrap_socket(socket.socket(socket.AF_INET), - cert_reqs=ssl.CERT_NONE, ciphers="DEFAULT") - s.connect(remote) + with ssl.wrap_socket(socket.socket(socket.AF_INET), + cert_reqs=ssl.CERT_NONE, ciphers="ALL") as s: + s.connect(remote) + with ssl.wrap_socket(socket.socket(socket.AF_INET), + cert_reqs=ssl.CERT_NONE, ciphers="DEFAULT") as s: + s.connect(remote) # Error checking can happen at instantiation or when connecting with self.assertRaisesRegex(ssl.SSLError, "No cipher can be selected"): with socket.socket(socket.AF_INET) as sock: @@ -1847,6 +1848,8 @@ client_addr = client.getsockname() client.close() t.join() + remote.close() + server.close() # Sanity checks. self.assertIsInstance(remote, ssl.SSLSocket) self.assertEqual(peer, client_addr) @@ -1861,8 +1864,7 @@ with ThreadedEchoServer(CERTFILE, ssl_version=ssl.PROTOCOL_SSLv23, chatty=False) as server: - with socket.socket() as sock: - s = context.wrap_socket(sock) + with context.wrap_socket(socket.socket()) as s: with self.assertRaises((OSError, ssl.SSLError)): s.connect((HOST, server.port)) self.assertIn("no shared cipher", str(server.conn_errors[0])) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -434,6 +434,8 @@ Tests ----- +- Issue #16923: Fix ResourceWarnings in test_ssl. + - Issue #15539: Added regression tests for Tools/scripts/pindent.py. - Issue #16925: test_configparser now works with unittest test discovery. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 21:59:30 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 12 Jan 2013 21:59:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2316923=3A_Fix_ResourceWarnings_in_test=5Fssl=2E?= Message-ID: <3YkCxG0DXFzSFS@mail.python.org> http://hg.python.org/cpython/rev/f79d282c8147 changeset: 81466:f79d282c8147 parent: 81464:44e614446f78 parent: 81465:2c3f5ed7a5c9 user: Antoine Pitrou date: Sat Jan 12 21:56:56 2013 +0100 summary: Issue #16923: Fix ResourceWarnings in test_ssl. files: Lib/test/test_ssl.py | 68 ++++++++++++++++--------------- Misc/NEWS | 2 + 2 files changed, 37 insertions(+), 33 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -217,20 +217,21 @@ s = socket.socket(socket.AF_INET) ss = ssl.wrap_socket(s) wr = weakref.ref(ss) - del ss - self.assertEqual(wr(), None) + with support.check_warnings(("", ResourceWarning)): + del ss + self.assertEqual(wr(), None) def test_wrapped_unconnected(self): # Methods on an unconnected SSLSocket propagate the original # OSError raise by the underlying socket object. s = socket.socket(socket.AF_INET) - ss = ssl.wrap_socket(s) - self.assertRaises(OSError, ss.recv, 1) - self.assertRaises(OSError, ss.recv_into, bytearray(b'x')) - self.assertRaises(OSError, ss.recvfrom, 1) - self.assertRaises(OSError, ss.recvfrom_into, bytearray(b'x'), 1) - self.assertRaises(OSError, ss.send, b'x') - self.assertRaises(OSError, ss.sendto, b'x', ('0.0.0.0', 0)) + with ssl.wrap_socket(s) as ss: + self.assertRaises(OSError, ss.recv, 1) + self.assertRaises(OSError, ss.recv_into, bytearray(b'x')) + self.assertRaises(OSError, ss.recvfrom, 1) + self.assertRaises(OSError, ss.recvfrom_into, bytearray(b'x'), 1) + self.assertRaises(OSError, ss.send, b'x') + self.assertRaises(OSError, ss.sendto, b'x', ('0.0.0.0', 0)) def test_timeout(self): # Issue #8524: when creating an SSL socket, the timeout of the @@ -238,8 +239,8 @@ for timeout in (None, 0.0, 5.0): s = socket.socket(socket.AF_INET) s.settimeout(timeout) - ss = ssl.wrap_socket(s) - self.assertEqual(timeout, ss.gettimeout()) + with ssl.wrap_socket(s) as ss: + self.assertEqual(timeout, ss.gettimeout()) def test_errors(self): sock = socket.socket() @@ -252,9 +253,9 @@ self.assertRaisesRegex(ValueError, "certfile must be specified for server-side operations", ssl.wrap_socket, sock, server_side=True, certfile="") - s = ssl.wrap_socket(sock, server_side=True, certfile=CERTFILE) - self.assertRaisesRegex(ValueError, "can't connect in server-side mode", - s.connect, (HOST, 8080)) + with ssl.wrap_socket(sock, server_side=True, certfile=CERTFILE) as s: + self.assertRaisesRegex(ValueError, "can't connect in server-side mode", + s.connect, (HOST, 8080)) with self.assertRaises(OSError) as cm: with socket.socket() as sock: ssl.wrap_socket(sock, certfile=WRONGCERT) @@ -367,21 +368,21 @@ def test_unknown_channel_binding(self): # should raise ValueError for unknown type s = socket.socket(socket.AF_INET) - ss = ssl.wrap_socket(s) - with self.assertRaises(ValueError): - ss.get_channel_binding("unknown-type") + with ssl.wrap_socket(s) as ss: + with self.assertRaises(ValueError): + ss.get_channel_binding("unknown-type") @unittest.skipUnless("tls-unique" in ssl.CHANNEL_BINDING_TYPES, "'tls-unique' channel binding not available") def test_tls_unique_channel_binding(self): # unconnected should return None for known type s = socket.socket(socket.AF_INET) - ss = ssl.wrap_socket(s) - self.assertIsNone(ss.get_channel_binding("tls-unique")) + with ssl.wrap_socket(s) as ss: + self.assertIsNone(ss.get_channel_binding("tls-unique")) # the same for server-side s = socket.socket(socket.AF_INET) - ss = ssl.wrap_socket(s, server_side=True, certfile=CERTFILE) - self.assertIsNone(ss.get_channel_binding("tls-unique")) + with ssl.wrap_socket(s, server_side=True, certfile=CERTFILE) as ss: + self.assertIsNone(ss.get_channel_binding("tls-unique")) def test_dealloc_warn(self): ss = ssl.wrap_socket(socket.socket(socket.AF_INET)) @@ -660,10 +661,10 @@ with socket.socket() as s: s.bind(("127.0.0.1", 0)) s.listen(5) - with socket.socket() as c: - c.connect(s.getsockname()) - c.setblocking(False) - c = ctx.wrap_socket(c, False, do_handshake_on_connect=False) + c = socket.socket() + c.connect(s.getsockname()) + c.setblocking(False) + with ctx.wrap_socket(c, False, do_handshake_on_connect=False) as c: with self.assertRaises(ssl.SSLWantReadError) as cm: c.do_handshake() s = str(cm.exception) @@ -904,12 +905,12 @@ def test_ciphers(self): remote = ("svn.python.org", 443) with support.transient_internet(remote[0]): - s = ssl.wrap_socket(socket.socket(socket.AF_INET), - cert_reqs=ssl.CERT_NONE, ciphers="ALL") - s.connect(remote) - s = ssl.wrap_socket(socket.socket(socket.AF_INET), - cert_reqs=ssl.CERT_NONE, ciphers="DEFAULT") - s.connect(remote) + with ssl.wrap_socket(socket.socket(socket.AF_INET), + cert_reqs=ssl.CERT_NONE, ciphers="ALL") as s: + s.connect(remote) + with ssl.wrap_socket(socket.socket(socket.AF_INET), + cert_reqs=ssl.CERT_NONE, ciphers="DEFAULT") as s: + s.connect(remote) # Error checking can happen at instantiation or when connecting with self.assertRaisesRegex(ssl.SSLError, "No cipher can be selected"): with socket.socket(socket.AF_INET) as sock: @@ -1886,6 +1887,8 @@ client_addr = client.getsockname() client.close() t.join() + remote.close() + server.close() # Sanity checks. self.assertIsInstance(remote, ssl.SSLSocket) self.assertEqual(peer, client_addr) @@ -1900,8 +1903,7 @@ with ThreadedEchoServer(CERTFILE, ssl_version=ssl.PROTOCOL_SSLv23, chatty=False) as server: - with socket.socket() as sock: - s = context.wrap_socket(sock) + with context.wrap_socket(socket.socket()) as s: with self.assertRaises(OSError): s.connect((HOST, server.port)) self.assertIn("no shared cipher", str(server.conn_errors[0])) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -635,6 +635,8 @@ Tests ----- +- Issue #16923: Fix ResourceWarnings in test_ssl. + - Issue #15539: Added regression tests for Tools/scripts/pindent.py. - Issue #16836: Enable IPv6 support even if IPv6 is disabled on the build host. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 22:03:23 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 12 Jan 2013 22:03:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSW4gdGVzdF9zc2ws?= =?utf-8?q?_threaded_tests_shouldn=27t_need_the_=22network=22_resource_to_?= =?utf-8?q?be_enabled?= Message-ID: <3YkD1l5ZsJzSFS@mail.python.org> http://hg.python.org/cpython/rev/c276fbd95182 changeset: 81467:c276fbd95182 branch: 3.3 parent: 81465:2c3f5ed7a5c9 user: Antoine Pitrou date: Sat Jan 12 22:00:09 2013 +0100 summary: In test_ssl, threaded tests shouldn't need the "network" resource to be enabled files: Lib/test/test_ssl.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -2032,7 +2032,7 @@ if _have_threads: thread_info = support.threading_setup() - if thread_info and support.is_resource_enabled('network'): + if thread_info: tests.append(ThreadedTests) try: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 22:03:25 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 12 Jan 2013 22:03:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_In_test=5Fssl=2C_threaded_tests_shouldn=27t_need_the_net?= =?utf-8?q?work_resource_to_be_enabled?= Message-ID: <3YkD1n245RzSFs@mail.python.org> http://hg.python.org/cpython/rev/5697373c56c1 changeset: 81468:5697373c56c1 parent: 81466:f79d282c8147 parent: 81467:c276fbd95182 user: Antoine Pitrou date: Sat Jan 12 22:00:54 2013 +0100 summary: In test_ssl, threaded tests shouldn't need the network resource to be enabled files: Lib/test/test_ssl.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -2166,7 +2166,7 @@ if _have_threads: thread_info = support.threading_setup() - if thread_info and support.is_resource_enabled('network'): + if thread_info: tests.append(ThreadedTests) try: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Jan 12 23:02:53 2013 From: python-checkins at python.org (eli.bendersky) Date: Sat, 12 Jan 2013 23:02:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_fix_versionadded_to_3=2E4?= Message-ID: <3YkFLP16n4zSDr@mail.python.org> http://hg.python.org/cpython/rev/72ddb250f058 changeset: 81469:72ddb250f058 user: Eli Bendersky date: Sat Jan 12 14:02:29 2013 -0800 summary: fix versionadded to 3.4 files: Doc/library/filecmp.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/filecmp.rst b/Doc/library/filecmp.rst --- a/Doc/library/filecmp.rst +++ b/Doc/library/filecmp.rst @@ -166,7 +166,7 @@ .. attribute:: DEFAULT_IGNORES - .. versionadded:: 3.3 + .. versionadded:: 3.4 List of directories ignored by :class:`dircmp` by default. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 00:06:30 2013 From: python-checkins at python.org (victor.stinner) Date: Sun, 13 Jan 2013 00:06:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_433=3A_cleanup=2C_complet?= =?utf-8?q?e=2C_reorganize?= Message-ID: <3YkGlp1znMzSBy@mail.python.org> http://hg.python.org/peps/rev/7fd26f1f2ce1 changeset: 4667:7fd26f1f2ce1 user: Victor Stinner date: Sun Jan 13 00:04:29 2013 +0100 summary: PEP 433: cleanup, complete, reorganize files: pep-0433.txt | 322 +++++++++++++++++++++++--------------- 1 files changed, 192 insertions(+), 130 deletions(-) diff --git a/pep-0433.txt b/pep-0433.txt --- a/pep-0433.txt +++ b/pep-0433.txt @@ -23,7 +23,9 @@ On UNIX, subprocess closes file descriptors greater than 2 by default since Python 3.2 [#subprocess_close]_. All file descriptors created by the parent -process are automatically closed. +process are automatically closed. ``xmlrpc.server.SimpleXMLRPCServer`` sets +the close-on-exec flag of the listening socket, the parent class +``socketserver.BaseServer`` does not set this flag. There are other cases creating a subprocess or executing a new program where file descriptors are not closed: functions of the os.spawn*() family and third @@ -31,6 +33,9 @@ descriptors are shared between the parent and the child processes which is usually unexpected and causes various issues. +This PEP proposes to continue the work started with the change in the +subprocess, to fix the issue in any code, and not just code using subprocess. + Inherited file descriptors issues --------------------------------- @@ -95,22 +100,10 @@ ``fork()``. The close-on-exec flag only closes file descriptors after ``exec()``, and so after ``fork()`` + ``exec()``. -Many functions of the Python standard library creating file descriptors are not -changed by the PEP, and so will not have the close-on-exec flag set. Some -examples: - - * ``os.urandom()``: on UNIX, it creates a file descriptor on UNIX to read - ``/dev/urandom``. Adding an ``cloexec`` argument to ``os.urandom()`` does - not make sense on Windows. - * ``curses.windows.getwin()`` and ``curses.windows.putwin()`` creates a temporary file using ``fdopen(fd, "wb+");`` - * ``mmap.mmap()`` opens ``/dev/null`` using ``open("/dev/zero", O_RDWR);`` if - ``MAP_ANONYMOUS`` is not defined. - * If the ``PYTHONSTARTUP`` environment variable is set, the corresponding file - is opened using ``fopen(startup, "r");`` - * ``python script.py`` opens ``script.py`` using ``fopen(filename, "r");`` - * etc. - -Third party modules creating file descriptors may not set close-on-exec flag. +This PEP only change the close-on-exec flag of file descriptors created by the +Python standard library, or by modules using the standard library. Third party +modules not using the standard library should be modified to conform to this +PEP. The new ``os.set_cloexec()`` function can be used for example. Impacted functions: @@ -129,9 +122,13 @@ XXX Should ``subprocess.Popen`` set the close-on-exec flag on file XXX XXX descriptors of the constructor the ``pass_fds`` argument? XXX +.. note:: + See `Close file descriptors after fork`_ for a possible solution for + ``fork()`` without ``exec()``. -Proposition -=========== + +Proposal +======== This PEP proposes to add a new optional argument ``cloexec`` on functions creating file descriptors in the Python standard library. If the argument is @@ -160,41 +157,196 @@ The default value of the ``cloexec`` argument is ``False`` to keep the backward compatibility. +The close-on-exec flag will not be set on file descriptors 0 (stdin), 1 +(stdout) and 2 (stderr), because these files are expected to be inherited. It +would still be possible to set close-on-exec flag explicitly using +``os.set_cloexec()``. +Drawbacks: + * Many functions of the Python standard library creating file descriptors are + cannot be changed by this proposal, because adding a ``cloexec`` optional + argument would be surprising and too many functions would need it. For + example, ``os.urandom()`` uses a temporary file on UNIX, but it calls a + function of Windows API on Windows. Adding a ``cloexec`` argument to + ``os.urandom()`` would not make sense. See `Always set close-on-exec flag`_ + for an incomplete list of functions creating file descriptors. + * Checking if a module creates file descriptors is difficult. For example, + ``os.urandom()`` creates a file descriptor on UNIX to read ``/dev/urandom`` + (and closes it at exit), whereas it is implemented using a function call on + Windows. It is not possible to control close-on-exec flag of the file + descriptor used by ``os.urandom()``, because ``os.urandom()`` API does not + allow it. +Alternatives +============ + +Always set close-on-exec flag +----------------------------- + +Always set close-on-exec flag on new file descriptors created by Python. This +alternative just changes the default value of the new ``cloexec`` argument. + +If a file must be inherited by child processes, ``cloexec=False`` argument can +be used. + +``subprocess.Popen`` constructor has an ``pass_fds`` argument to specify which +file descriptors must be inherited. The close-on-exec flag of these file +descriptors must be changed with ``os.set_cloexec()``. + +Example of functions creating file descriptors which will be modified to +set close-on-exec flag: + + * ``os.urandom()`` (on UNIX) + * ``curses.window.getwin()``, ``curses.window.putwin()`` + * ``mmap.mmap()`` (if ``MAP_ANONYMOUS`` is not defined) + * ``oss.open()`` + * ``Modules/main.c``: ``RunStartupFile()`` + * ``Python/pythonrun.c``: ``PyRun_SimpleFileExFlags()`` + * ``Modules/getpath.c``: ``search_for_exec_prefix()`` + * ``Modules/zipimport.c``: ``read_directory()`` + * ``Modules/_ssl.c``: ``load_dh_params()`` + * ``PC/getpathp.c``: ``calculate_path()`` + * ``Python/errors.c``: ``PyErr_ProgramText()`` + * ``Python/import.c``: ``imp_load_dynamic()`` + * TODO: ``PC/_msi.c`` + +Many functions are impacted indirectly by this alternative. Examples: + + * ``logging.FileHandler`` + +Advantages of setting close-on-exec flag by default: + + * There are far more programs that are bitten by FD inheritance upon + exec (see `Inherited file descriptors issues`_ and `Security`_) than + programs relying on it + (see `Applications using inherance of file descriptors`_). + +Drawbacks of setting close-on-exec flag by default: + + * The os module is written as a thin wrapper to system calls (to functions of + the C standard library). If atomic flags to set close-on-exec flag are not + supported (see `Appendix: Operating system support`_), a single Python + function call may call 2 or 3 system calls (see `Performances`_ section). + * Extra system calls, if any, may slow down Python: see `Performances`_. + * It violates the principle of least surprise. Developers using the os module + may expect that Python respects the POSIX standard and so that close-on-exec + flag is not set by default. + +Backward compatibility: only a few programs rely on inherance of file +descriptors, and they only pass a few file descriptors, usually just one. +These programs will fail immediatly with ``EBADF`` error, and it will be simple +to fix them: add ``cloexec=False`` argument or use +``os.set_cloexec(fd, False)``. + +The ``subprocess`` module will be changed anyway to unset close-on-exec flag on +file descriptors listed in the ``pass_fds`` argument of Popen constructor. So +it possible that these programs will not need any fix if they use the +``subprocess`` module. + + +Add a function to set close-on-exec flag by default +--------------------------------------------------- + +An alternative is to add also a function to change globally the default +behaviour. It would be possible to set close-on-exec flag for the whole +application including all modules and the Python standard library. This +alternative is based on the `Proposal`_ and adds extra changes. + +Add new functions: + + * ``sys.getdefaultcloexec() -> bool``: get the default value of the + close-on-exec flag for new file descriptor + * ``sys.setdefaultcloexec(cloexec: bool)``: enable or disable close-on-exec + flag, the state of the flag can be overriden in each function creating a + file descriptor + +The major change is that the default value of the ``cloexec`` argument is +``sys.getdefaultcloexec()``, instead of ``False``. + +When ``sys.setdefaultcloexec(True)`` is called to set close-on-exec by default, +we have the same drawbacks than `Always set close-on-exec +flag`_ alternative. + +There are additionnal drawbacks of having two behaviours depending on +``sys.getdefaultcloexec()`` value: + + * It is not more possible to know if the close-on-exec flag will be set or not + just by reading the source code. + + +Close file descriptors after fork +--------------------------------- + +This PEP does not fix issues with applications using ``fork()`` without +``exec()``. Python needs a generic process to register callbacks which +would be called after a fork, see `Add an 'afterfork' module`_. Such +registry could be used to close file descriptors just after a ``fork()``. + +Drawbacks: + + * This alternative does not solve the problem for programs using ``exec()`` + without ``fork()``. + * A third party module may call directly the C function ``fork()`` which will + not call "atfork" callbacks. + * All functions creating file descriptors must be changed to register a + callback and then unregister their callback when the file is closed. Or a + list of *all* open file descriptors must be maintained. + * The operating system is a better place than Python to close automatically + file descriptors. For example, it is not easy to avoid a race condition + between closing the file and unregistering the callback closing the file. + + +open(): add "e" flag to mode +---------------------------- + +A new "e" mode would set close-on-exec flag (best-effort). + +This alternative only solves the problem for ``open()``. socket.socket() and +os.pipe() do not have a ``mode`` argument for example. + +Since its version 2.7, the GNU libc supports ``"e"`` flag for ``fopen()``. It +uses ``O_CLOEXEC`` if available, or use ``fcntl(fd, F_SETFD, FD_CLOEXEC)``. +With Visual Studio, fopen() accepts a "N" flag which uses ``O_NOINHERIT``. + Applications using inherance of file descriptors ================================================ +Most developers don't know that file descriptors are inherited by default. Most +programs do not rely on inherance of file descriptors. For example, +``subprocess.Popen`` was changed in Python 3.2 to close all file descriptors +greater than 2 in the child process by default. No user complained about this +behavior change. + Network servers using fork may want to pass the client socket to the child -process. For example, a CGI server pass the socket client through file -descriptors 0 (stdin) and 1 (stdout) using ``dup2()``. +process. For example, on UNIX a CGI server pass the socket client through file +descriptors 0 (stdin) and 1 (stdout) using ``dup2()``. This specific case is +not impacted by this PEP because the close-on-exec flag is never set on file +descriptors smaller than 3. -Example of programs taking file descriptors from the parent process: +To access a restricted resource like creating a socket listening on a TCP port +lower than 1024 or reading a file containing sensitive data like passwords, a +common practice is: start as the root user, create a file descriptor, create +a child process, pass the file descriptor to the child process and exit. +Security is very important in such use case: leaking another file descriptor +would be a critical security vulnerability (see `Security`_). The root process +may not exit but monitors the child process instead, and restarts a new child +process and pass the same file descriptor if the previous child process +crashed. +Example of programs taking file descriptors from the parent process using a +command line option: + + * gpg: ``--status-fd ``, ``--logger-fd ``, etc. + * openssl: ``-pass fd:`` + * qemu: ``-add-fd `` * valgrind: ``--log-fd=``, ``--input-fd=``, etc. - * qemu: ``-add-fd `` command line option - * GnuPG: ``--status-fd ``, ``--logger-fd ``, etc. - * openssl command: ``-pass fd:`` * xterm: ``-S `` -On Linux, it is possible to use ``/dev/fd/`` filename to pass a file -descriptor to a program expecting a filename. It can be used to pass a password -for example. - -These applications only pass a few file descriptors, usually only one. -Fixing these applications to unset close-on-exec flag should be easy. - -If the ``subprocess`` module is used, inherited file descriptors must be specified -using the ``pass_fds`` argument (except if the ``close_fds`` argument is set -explicitly to ``False``). So the ``subprocess`` module knows the list of file -descriptors on which close-on-exec flag must be unset. - -File descriptors 0 (stdin), 1 (stdout) and 2 (stderr) are expected to be -inherited and so should not have the close-on-exec flag. So a CGI server should -not be impacted by this PEP. +On Linux, it is possible to use ``"/dev/fd/"`` filename to pass a file +descriptor to a program expecting a filename. Performances @@ -313,96 +465,6 @@ the close-on-exec flag is not set by default. -Alternatives -============ - -Always set close-on-exec flag ------------------------------ - -Always set close-on-exec flag on new file descriptors created by Python. This -alternative just changes the default value of the new ``cloexec`` argument. - -``subprocess.Popen`` constructor has an ``pass_fds`` argument to specify which -file descriptors must be inherited. The close-on-exec flag of these file -descriptors must be changed with ``os.set_cloexec()``. - -If the close-on-exec flag must not be set, ``cloexec=False`` can be specified. - -Advantages of setting close-on-exec flag by default: - - * There are far more programs that are bitten by FD inheritance upon - exec (see `Inherited file descriptors issues`_ and `Security`_) than - programs relying on it. - * Checking if a module creates file descriptors is difficult. For example, - ``os.urandom()`` creates a file descriptor on UNIX to read ``/dev/urandom`` - (and closes it at exit), whereas it is implemented using a function call on - Windows. It is not possible to control close-on-exec flag of the file - descriptor used by ``os.urandom()``, because ``os.urandom()`` API does not - allow it. - * No need to add a new ``cloexec`` argument everywhere: functions creating - file descriptors will read ``sys.getdefaultcloexec()`` to decide if the - close-on-exec must be set or not. For example, adding an ``cloexec`` - argument to ``os.urandom()`` does not make sense on Windows. - -Drawbacks of setting close-on-exec flag by default: - - * The os module is written as a thin wrapper to system calls (to functions of - the C standard library). If atomic flags are not supported, a single Python - function call may now call 2 or 3 system calls (see `Performances`_ - section). - * Extra system calls, if any, may slow down Python: see `Performances`_. - * It violates the principle of least surprise. Developers using the os module - may expect that Python respects the POSIX standard and so that close-on-exec - flag is not set by default. - * Only file descriptors created by the Python standard library will comply to - ``sys.setdefaultcloexec()``. The close-on-exec flag is unchanged for file - descriptors created by third party modules calling directly C functions. - Third party modules will have to be modified to read - ``sys.getdefaultcloexec()`` to make them comply to this PEP. - - -Add a function to set close-on-exec flag by default ---------------------------------------------------- - -An alternative is to add also a function to change globally the default -behaviour. It would be possible to set close-on-exec flag for the whole -application including all modules and the Python standard library. This -alternative is based on the PEP but adds extra changes. - -Add new functions: - - * ``sys.getdefaultcloexec() -> bool``: get the default value of the - close-on-exec flag for new file descriptor - * ``sys.setdefaultcloexec(cloexec: bool)``: enable or disable close-on-exec - flag, the state of the flag can be overriden in each function creating a - file descriptor - -The major change is that the default value of the ``cloexec`` argument is -``sys.getdefaultcloexec()``, instead of ``False``. - -When ``sys.setdefaultcloexec(True)`` is called to set close-on-exec by default, -we have the same drawbacks than `Always set close-on-exec -flag`_ alternative. - -There are additionnal drawbacks of having two behaviours depending on -``sys.getdefaultcloexec()`` value: - - * It is not more possible to know if the close-on-exec flag will be set or not - just by reading the source code. - - -open(): add "e" flag to mode ----------------------------- - -A new "e" mode would set close-on-exec flag (best-effort). - -This API does not allow to disable explictly close-on-exec flag if it was -enabled globally with ``sys.setdefaultcloexec()``. - -Note: Since its version 2.7, the GNU libc supports ``"e"`` flag for ``fopen()``. -It uses ``O_CLOEXEC`` if available, or use ``fcntl(fd, F_SETFD, FD_CLOEXEC)``. - - Appendix: Operating system support ================================== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Jan 13 01:32:00 2013 From: python-checkins at python.org (victor.stinner) Date: Sun, 13 Jan 2013 01:32:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_433=3A_reformat_to_70_col?= =?utf-8?q?umns?= Message-ID: <3YkJfS0RL6zRwm@mail.python.org> http://hg.python.org/peps/rev/ac3351088370 changeset: 4668:ac3351088370 user: Victor Stinner date: Sun Jan 13 01:30:02 2013 +0100 summary: PEP 433: reformat to 70 columns files: pep-0433.txt | 423 +++++++++++++++++++++----------------- 1 files changed, 232 insertions(+), 191 deletions(-) diff --git a/pep-0433.txt b/pep-0433.txt --- a/pep-0433.txt +++ b/pep-0433.txt @@ -13,84 +13,92 @@ Abstract ======== -This PEP proposes to add a new optional argument ``cloexec`` on functions -creating file descriptors in the Python standard library. If the argument is -``True``, the close-on-exec flag will be set on the new file descriptor. +This PEP proposes to add a new optional argument ``cloexec`` on +functions creating file descriptors in the Python standard library. If +the argument is ``True``, the close-on-exec flag will be set on the +new file descriptor. Rationale ========= -On UNIX, subprocess closes file descriptors greater than 2 by default since -Python 3.2 [#subprocess_close]_. All file descriptors created by the parent -process are automatically closed. ``xmlrpc.server.SimpleXMLRPCServer`` sets -the close-on-exec flag of the listening socket, the parent class -``socketserver.BaseServer`` does not set this flag. +On UNIX, subprocess closes file descriptors greater than 2 by default +since Python 3.2 [#subprocess_close]_. All file descriptors created by +the parent process are automatically closed. +``xmlrpc.server.SimpleXMLRPCServer`` sets the close-on-exec flag of +the listening socket, the parent class ``socketserver.BaseServer`` +does not set this flag. -There are other cases creating a subprocess or executing a new program where -file descriptors are not closed: functions of the os.spawn*() family and third -party modules calling ``exec()`` or ``fork()`` + ``exec()``. In this case, file -descriptors are shared between the parent and the child processes which is -usually unexpected and causes various issues. +There are other cases creating a subprocess or executing a new program +where file descriptors are not closed: functions of the os.spawn*() +family and third party modules calling ``exec()`` or ``fork()`` + +``exec()``. In this case, file descriptors are shared between the +parent and the child processes which is usually unexpected and causes +various issues. This PEP proposes to continue the work started with the change in the -subprocess, to fix the issue in any code, and not just code using subprocess. +subprocess, to fix the issue in any code, and not just code using +subprocess. Inherited file descriptors issues --------------------------------- -Closing the file descriptor in the parent process does not close the related -resource (file, socket, ...) because it is still open in the child process. +Closing the file descriptor in the parent process does not close the +related resource (file, socket, ...) because it is still open in the +child process. -The listening socket of TCPServer is not closed on ``exec()``: the child -process is able to get connection from new clients; if the parent closes the -listening socket and create a new listening socket on the same address, it -would get an "address already is used" error. +The listening socket of TCPServer is not closed on ``exec()``: the +child process is able to get connection from new clients; if the +parent closes the listening socket and create a new listening socket +on the same address, it would get an "address already is used" error. -Not closing file descriptors can lead to resource exhaustion: even if the -parent closes all files, creating a new file descriptor may fail with "too many -files" because files are still open in the child process. +Not closing file descriptors can lead to resource exhaustion: even if +the parent closes all files, creating a new file descriptor may fail +with "too many files" because files are still open in the child +process. Security -------- -Leaking file descriptors is a major security vulnerability. An untrusted child -process can read sensitive data like passwords and take control of the parent -process though leaked file descriptors. It is for example a known vulnerability -to escape from a chroot. +Leaking file descriptors is a major security vulnerability. An +untrusted child process can read sensitive data like passwords and +take control of the parent process though leaked file descriptors. It +is for example a known vulnerability to escape from a chroot. Atomicity --------- -Using ``fcntl()`` to set the close-on-exec flag is not safe in a multithreaded -application. If a thread calls ``fork()`` and ``exec()`` between the creation -of the file descriptor and the call to ``fcntl(fd, F_SETFD, new_flags)``: the -file descriptor will be inherited by the child process. Modern operating -systems offer functions to set the flag during the creation of the file -descriptor, which avoids the race condition. +Using ``fcntl()`` to set the close-on-exec flag is not safe in a +multithreaded application. If a thread calls ``fork()`` and ``exec()`` +between the creation of the file descriptor and the call to +``fcntl(fd, F_SETFD, new_flags)``: the file descriptor will be +inherited by the child process. Modern operating systems offer +functions to set the flag during the creation of the file descriptor, +which avoids the race condition. Portability ----------- Python 3.2 added ``socket.SOCK_CLOEXEC`` flag, Python 3.3 added -``os.O_CLOEXEC`` flag and ``os.pipe2()`` function. It is already possible to -set atomically close-on-exec flag in Python 3.3 when opening a file and -creating a pipe or socket. +``os.O_CLOEXEC`` flag and ``os.pipe2()`` function. It is already +possible to set atomically close-on-exec flag in Python 3.3 when +opening a file and creating a pipe or socket. -The problem is that these flags and functions are not portable: only recent -versions of operating systems support them. ``O_CLOEXEC`` and ``SOCK_CLOEXEC`` -flags are ignored by old Linux versions and so ``FD_CLOEXEC`` flag must be -checked using ``fcntl(fd, F_GETFD)``. If the kernel ignores ``O_CLOEXEC`` or -``SOCK_CLOEXEC`` flag, a call to ``fcntl(fd, F_SETFD, flags)`` is required to -set close-on-exec flag. +The problem is that these flags and functions are not portable: only +recent versions of operating systems support them. ``O_CLOEXEC`` and +``SOCK_CLOEXEC`` flags are ignored by old Linux versions and so +``FD_CLOEXEC`` flag must be checked using ``fcntl(fd, F_GETFD)``. If +the kernel ignores ``O_CLOEXEC`` or ``SOCK_CLOEXEC`` flag, a call to +``fcntl(fd, F_SETFD, flags)`` is required to set close-on-exec flag. -Note: OpenBSD older 5.2 does not close the file descriptor with close-on-exec -flag set if ``fork()`` is used before ``exec()``, but it works correctly if -``exec()`` is called without ``fork()``. +.. note:: + OpenBSD older 5.2 does not close the file descriptor with + close-on-exec flag set if ``fork()`` is used before ``exec()``, but + it works correctly if ``exec()`` is called without ``fork()``. Scope @@ -100,10 +108,11 @@ ``fork()``. The close-on-exec flag only closes file descriptors after ``exec()``, and so after ``fork()`` + ``exec()``. -This PEP only change the close-on-exec flag of file descriptors created by the -Python standard library, or by modules using the standard library. Third party -modules not using the standard library should be modified to conform to this -PEP. The new ``os.set_cloexec()`` function can be used for example. +This PEP only change the close-on-exec flag of file descriptors +created by the Python standard library, or by modules using the +standard library. Third party modules not using the standard library +should be modified to conform to this PEP. The new +``os.set_cloexec()`` function can be used for example. Impacted functions: @@ -144,7 +153,8 @@ * ``open()``: ``os.fdopen()`` is indirectly modified * ``os.dup()``, ``os.dup2()`` * ``os.pipe()`` - * ``socket.socket()``, ``socket.socketpair()`` ``socket.socket.accept()`` + * ``socket.socket()``, ``socket.socketpair()``, + ``socket.socket.accept()`` * Maybe also: ``os.open()``, ``os.openpty()`` * TODO: @@ -152,31 +162,33 @@ * ``select.poll()`` * ``select.epoll()`` * ``select.kqueue()`` - * ``socket.socket.recvmsg()``: use ``MSG_CMSG_CLOEXEC``, or ``os.set_cloexec()`` + * ``socket.socket.recvmsg()``: use ``MSG_CMSG_CLOEXEC``, + or ``os.set_cloexec()`` -The default value of the ``cloexec`` argument is ``False`` to keep the backward -compatibility. +The default value of the ``cloexec`` argument is ``False`` to keep the +backward compatibility. -The close-on-exec flag will not be set on file descriptors 0 (stdin), 1 -(stdout) and 2 (stderr), because these files are expected to be inherited. It -would still be possible to set close-on-exec flag explicitly using -``os.set_cloexec()``. +The close-on-exec flag will not be set on file descriptors 0 (stdin), +1 (stdout) and 2 (stderr), because these files are expected to be +inherited. It would still be possible to set close-on-exec flag +explicitly using ``os.set_cloexec()``. Drawbacks: - * Many functions of the Python standard library creating file descriptors are - cannot be changed by this proposal, because adding a ``cloexec`` optional - argument would be surprising and too many functions would need it. For - example, ``os.urandom()`` uses a temporary file on UNIX, but it calls a - function of Windows API on Windows. Adding a ``cloexec`` argument to - ``os.urandom()`` would not make sense. See `Always set close-on-exec flag`_ - for an incomplete list of functions creating file descriptors. - * Checking if a module creates file descriptors is difficult. For example, - ``os.urandom()`` creates a file descriptor on UNIX to read ``/dev/urandom`` - (and closes it at exit), whereas it is implemented using a function call on - Windows. It is not possible to control close-on-exec flag of the file - descriptor used by ``os.urandom()``, because ``os.urandom()`` API does not - allow it. + * Many functions of the Python standard library creating file + descriptors are cannot be changed by this proposal, because adding + a ``cloexec`` optional argument would be surprising and too many + functions would need it. For example, ``os.urandom()`` uses a + temporary file on UNIX, but it calls a function of Windows API on + Windows. Adding a ``cloexec`` argument to ``os.urandom()`` would + not make sense. See `Always set close-on-exec flag`_ for an + incomplete list of functions creating file descriptors. + * Checking if a module creates file descriptors is difficult. For + example, ``os.urandom()`` creates a file descriptor on UNIX to read + ``/dev/urandom`` (and closes it at exit), whereas it is implemented + using a function call on Windows. It is not possible to control + close-on-exec flag of the file descriptor used by ``os.urandom()``, + because ``os.urandom()`` API does not allow it. Alternatives @@ -185,18 +197,20 @@ Always set close-on-exec flag ----------------------------- -Always set close-on-exec flag on new file descriptors created by Python. This -alternative just changes the default value of the new ``cloexec`` argument. +Always set close-on-exec flag on new file descriptors created by +Python. This alternative just changes the default value of the new +``cloexec`` argument. -If a file must be inherited by child processes, ``cloexec=False`` argument can -be used. +If a file must be inherited by child processes, ``cloexec=False`` +argument can be used. -``subprocess.Popen`` constructor has an ``pass_fds`` argument to specify which -file descriptors must be inherited. The close-on-exec flag of these file -descriptors must be changed with ``os.set_cloexec()``. +``subprocess.Popen`` constructor has an ``pass_fds`` argument to +specify which file descriptors must be inherited. The close-on-exec +flag of these file descriptors must be changed with +``os.set_cloexec()``. -Example of functions creating file descriptors which will be modified to -set close-on-exec flag: +Example of functions creating file descriptors which will be modified +to set close-on-exec flag: * ``os.urandom()`` (on UNIX) * ``curses.window.getwin()``, ``curses.window.putwin()`` @@ -219,83 +233,89 @@ Advantages of setting close-on-exec flag by default: * There are far more programs that are bitten by FD inheritance upon - exec (see `Inherited file descriptors issues`_ and `Security`_) than - programs relying on it - (see `Applications using inherance of file descriptors`_). + exec (see `Inherited file descriptors issues`_ and `Security`_) + than programs relying on it (see `Applications using inherance of + file descriptors`_). Drawbacks of setting close-on-exec flag by default: - * The os module is written as a thin wrapper to system calls (to functions of - the C standard library). If atomic flags to set close-on-exec flag are not - supported (see `Appendix: Operating system support`_), a single Python - function call may call 2 or 3 system calls (see `Performances`_ section). - * Extra system calls, if any, may slow down Python: see `Performances`_. - * It violates the principle of least surprise. Developers using the os module - may expect that Python respects the POSIX standard and so that close-on-exec - flag is not set by default. + * The os module is written as a thin wrapper to system calls (to + functions of the C standard library). If atomic flags to set + close-on-exec flag are not supported (see `Appendix: Operating + system support`_), a single Python function call may call 2 or 3 + system calls (see `Performances`_ section). + * Extra system calls, if any, may slow down Python: see + `Performances`_. + * It violates the principle of least surprise. Developers using the + os module may expect that Python respects the POSIX standard and so + that close-on-exec flag is not set by default. Backward compatibility: only a few programs rely on inherance of file -descriptors, and they only pass a few file descriptors, usually just one. -These programs will fail immediatly with ``EBADF`` error, and it will be simple -to fix them: add ``cloexec=False`` argument or use +descriptors, and they only pass a few file descriptors, usually just +one. These programs will fail immediatly with ``EBADF`` error, and it +will be simple to fix them: add ``cloexec=False`` argument or use ``os.set_cloexec(fd, False)``. -The ``subprocess`` module will be changed anyway to unset close-on-exec flag on -file descriptors listed in the ``pass_fds`` argument of Popen constructor. So -it possible that these programs will not need any fix if they use the -``subprocess`` module. +The ``subprocess`` module will be changed anyway to unset +close-on-exec flag on file descriptors listed in the ``pass_fds`` +argument of Popen constructor. So it possible that these programs will +not need any fix if they use the ``subprocess`` module. Add a function to set close-on-exec flag by default --------------------------------------------------- -An alternative is to add also a function to change globally the default -behaviour. It would be possible to set close-on-exec flag for the whole -application including all modules and the Python standard library. This -alternative is based on the `Proposal`_ and adds extra changes. +An alternative is to add also a function to change globally the +default behaviour. It would be possible to set close-on-exec flag for +the whole application including all modules and the Python standard +library. This alternative is based on the `Proposal`_ and adds extra +changes. Add new functions: * ``sys.getdefaultcloexec() -> bool``: get the default value of the close-on-exec flag for new file descriptor - * ``sys.setdefaultcloexec(cloexec: bool)``: enable or disable close-on-exec - flag, the state of the flag can be overriden in each function creating a - file descriptor + * ``sys.setdefaultcloexec(cloexec: bool)``: enable or disable + close-on-exec flag, the state of the flag can be overriden in each + function creating a file descriptor -The major change is that the default value of the ``cloexec`` argument is -``sys.getdefaultcloexec()``, instead of ``False``. +The major change is that the default value of the ``cloexec`` argument +is ``sys.getdefaultcloexec()``, instead of ``False``. -When ``sys.setdefaultcloexec(True)`` is called to set close-on-exec by default, -we have the same drawbacks than `Always set close-on-exec +When ``sys.setdefaultcloexec(True)`` is called to set close-on-exec by +default, we have the same drawbacks than `Always set close-on-exec flag`_ alternative. There are additionnal drawbacks of having two behaviours depending on ``sys.getdefaultcloexec()`` value: - * It is not more possible to know if the close-on-exec flag will be set or not - just by reading the source code. + * It is not more possible to know if the close-on-exec flag will be + set or not just by reading the source code. Close file descriptors after fork --------------------------------- -This PEP does not fix issues with applications using ``fork()`` without -``exec()``. Python needs a generic process to register callbacks which -would be called after a fork, see `Add an 'afterfork' module`_. Such -registry could be used to close file descriptors just after a ``fork()``. +This PEP does not fix issues with applications using ``fork()`` +without ``exec()``. Python needs a generic process to register +callbacks which would be called after a fork, see `Add an 'afterfork' +module`_. Such registry could be used to close file descriptors just +after a ``fork()``. Drawbacks: - * This alternative does not solve the problem for programs using ``exec()`` - without ``fork()``. - * A third party module may call directly the C function ``fork()`` which will - not call "atfork" callbacks. - * All functions creating file descriptors must be changed to register a - callback and then unregister their callback when the file is closed. Or a - list of *all* open file descriptors must be maintained. - * The operating system is a better place than Python to close automatically - file descriptors. For example, it is not easy to avoid a race condition - between closing the file and unregistering the callback closing the file. + * This alternative does not solve the problem for programs using + ``exec()`` without ``fork()``. + * A third party module may call directly the C function ``fork()`` + which will not call "atfork" callbacks. + * All functions creating file descriptors must be changed to register + a callback and then unregister their callback when the file is + closed. Or a list of *all* open file descriptors must be + maintained. + * The operating system is a better place than Python to close + automatically file descriptors. For example, it is not easy to + avoid a race condition between closing the file and unregistering + the callback closing the file. open(): add "e" flag to mode @@ -303,41 +323,44 @@ A new "e" mode would set close-on-exec flag (best-effort). -This alternative only solves the problem for ``open()``. socket.socket() and -os.pipe() do not have a ``mode`` argument for example. +This alternative only solves the problem for ``open()``. +socket.socket() and os.pipe() do not have a ``mode`` argument for +example. -Since its version 2.7, the GNU libc supports ``"e"`` flag for ``fopen()``. It -uses ``O_CLOEXEC`` if available, or use ``fcntl(fd, F_SETFD, FD_CLOEXEC)``. -With Visual Studio, fopen() accepts a "N" flag which uses ``O_NOINHERIT``. +Since its version 2.7, the GNU libc supports ``"e"`` flag for +``fopen()``. It uses ``O_CLOEXEC`` if available, or use ``fcntl(fd, +F_SETFD, FD_CLOEXEC)``. With Visual Studio, fopen() accepts a "N" +flag which uses ``O_NOINHERIT``. Applications using inherance of file descriptors ================================================ -Most developers don't know that file descriptors are inherited by default. Most -programs do not rely on inherance of file descriptors. For example, -``subprocess.Popen`` was changed in Python 3.2 to close all file descriptors -greater than 2 in the child process by default. No user complained about this -behavior change. +Most developers don't know that file descriptors are inherited by +default. Most programs do not rely on inherance of file descriptors. +For example, ``subprocess.Popen`` was changed in Python 3.2 to close +all file descriptors greater than 2 in the child process by default. +No user complained about this behavior change. -Network servers using fork may want to pass the client socket to the child -process. For example, on UNIX a CGI server pass the socket client through file -descriptors 0 (stdin) and 1 (stdout) using ``dup2()``. This specific case is -not impacted by this PEP because the close-on-exec flag is never set on file -descriptors smaller than 3. +Network servers using fork may want to pass the client socket to the +child process. For example, on UNIX a CGI server pass the socket +client through file descriptors 0 (stdin) and 1 (stdout) using +``dup2()``. This specific case is not impacted by this PEP because the +close-on-exec flag is never set on file descriptors smaller than 3. -To access a restricted resource like creating a socket listening on a TCP port -lower than 1024 or reading a file containing sensitive data like passwords, a -common practice is: start as the root user, create a file descriptor, create -a child process, pass the file descriptor to the child process and exit. -Security is very important in such use case: leaking another file descriptor -would be a critical security vulnerability (see `Security`_). The root process -may not exit but monitors the child process instead, and restarts a new child -process and pass the same file descriptor if the previous child process +To access a restricted resource like creating a socket listening on a +TCP port lower than 1024 or reading a file containing sensitive data +like passwords, a common practice is: start as the root user, create a +file descriptor, create a child process, pass the file descriptor to +the child process and exit. Security is very important in such use +case: leaking another file descriptor would be a critical security +vulnerability (see `Security`_). The root process may not exit but +monitors the child process instead, and restarts a new child process +and pass the same file descriptor if the previous child process crashed. -Example of programs taking file descriptors from the parent process using a -command line option: +Example of programs taking file descriptors from the parent process +using a command line option: * gpg: ``--status-fd ``, ``--logger-fd ``, etc. * openssl: ``-pass fd:`` @@ -345,22 +368,23 @@ * valgrind: ``--log-fd=``, ``--input-fd=``, etc. * xterm: ``-S `` -On Linux, it is possible to use ``"/dev/fd/"`` filename to pass a file -descriptor to a program expecting a filename. +On Linux, it is possible to use ``"/dev/fd/"`` filename to pass a +file descriptor to a program expecting a filename. Performances ============ -Setting close-on-exec flag may require additional system calls for each -creation of new file descriptors. The number of additional system calls -depends on the method used to set the flag: +Setting close-on-exec flag may require additional system calls for +each creation of new file descriptors. The number of additional system +calls depends on the method used to set the flag: * ``O_NOINHERIT``: no additionnal system call - * ``O_CLOEXEC``: one addition system call, but only at the creation of the - first file descriptor, to check if the flag is supported. If no, Python has - to fallback to the next method. - * ``ioctl(fd, FIOCLEX)``: one addition system call per file descriptor + * ``O_CLOEXEC``: one addition system call, but only at the creation + of the first file descriptor, to check if the flag is supported. If + no, Python has to fallback to the next method. + * ``ioctl(fd, FIOCLEX)``: one addition system call per file + descriptor * ``fcntl(fd, F_SETFD, flags)``: two addition system calls per file descriptor, one to get old flags and one to set new flags @@ -377,7 +401,8 @@ if os.name == 'nt': def set_cloexec(fd, cloexec=True): - SetHandleInformation(fd, HANDLE_FLAG_INHERIT, int(cloexec)) + SetHandleInformation(fd, HANDLE_FLAG_INHERIT, + int(cloexec)) else: fnctl = None ioctl = None @@ -404,14 +429,22 @@ fcntl.fcntl(fd, fcntl.F_SETFD, flags) else: def set_cloexec(fd, cloexec=True): - raise NotImplementedError("close-on-exec flag is not supported on your platform") + raise NotImplementedError( + "close-on-exec flag is not supported " + "on your platform") -ioctl is preferred over fcntl because it requires only one syscall, instead of -two syscalls for fcntl. +ioctl is preferred over fcntl because it requires only one syscall, +instead of two syscalls for fcntl. -Note: ``fcntl(fd, F_SETFD, flags)`` only supports one flag (``FD_CLOEXEC``), so -it would be possible to avoid ``fcntl(fd, F_GETFD)``. But it may drop other -flags in the future, and so it is safer to keep the two functions calls. +.. note:: + ``fcntl(fd, F_SETFD, flags)`` only supports one flag + (``FD_CLOEXEC``), so it would be possible to avoid ``fcntl(fd, + F_GETFD)``. But it may drop other flags in the future, and so it is + safer to keep the two functions calls. + +.. note:: + ``fopen()`` function of the GNU libc ignores the error if + ``fcntl(fd, F_SETFD, flags)`` failed. open() ------ @@ -461,8 +494,8 @@ Backward compatibility ====================== -There is no backward incompatible change. The default behaviour is unchanged: -the close-on-exec flag is not set by default. +There is no backward incompatible change. The default behaviour is +unchanged: the close-on-exec flag is not set by default. Appendix: Operating system support @@ -471,16 +504,17 @@ Windows ------- -Windows has an ``O_NOINHERIT`` flag: "Do not inherit in child processes". +Windows has an ``O_NOINHERIT`` flag: "Do not inherit in child +processes". For example, it is supported by ``open()`` and ``_pipe()``. The value of the flag can be modified using: ``SetHandleInformation(fd, HANDLE_FLAG_INHERIT, 1)``. -``CreateProcess()`` has an ``bInheritHandles`` argument: if it is FALSE, the -handles are not inherited. It is used by ``subprocess.Popen`` with -``close_fds`` option. +``CreateProcess()`` has an ``bInheritHandles`` argument: if it is +FALSE, the handles are not inherited. It is used by +``subprocess.Popen`` with ``close_fds`` option. fcntl ----- @@ -490,8 +524,8 @@ * ``fcntl(fd, F_GETFD)`` * ``fcntl(fd, F_SETFD, flags | FD_CLOEXEC)`` -Availability: AIX, Digital UNIX, FreeBSD, HP-UX, IRIX, Linux, Mac OS X, -OpenBSD, Solaris, SunOS, Unicos. +Availability: AIX, Digital UNIX, FreeBSD, HP-UX, IRIX, Linux, Mac OS +X, OpenBSD, Solaris, SunOS, Unicos. ioctl ----- @@ -509,20 +543,23 @@ New flags: - * ``O_CLOEXEC``: available on Linux (2.6.23+), FreeBSD (8.3+), OpenBSD 5.0, - will be part of the next NetBSD release (6.1?). This flag is part of - POSIX.1-2008. + * ``O_CLOEXEC``: available on Linux (2.6.23+), FreeBSD (8.3+), + OpenBSD 5.0, QNX, BeOS, next NetBSD release (6.1?). This flag is + part of POSIX.1-2008. * ``socket()``: ``SOCK_CLOEXEC`` flag, available on Linux 2.6.27+, OpenBSD 5.2, NetBSD 6.0. * ``fcntl()``: ``F_DUPFD_CLOEXEC`` flag, available on Linux 2.6.24+, - OpenBSD 5.0, FreeBSD 9.1, NetBSD 6.0. This flag is part of POSIX.1-2008. - * ``recvmsg()``: ``MSG_CMSG_CLOEXEC``, available on Linux 2.6.23+, NetBSD 6.0. + OpenBSD 5.0, FreeBSD 9.1, NetBSD 6.0. This flag is part of + POSIX.1-2008. + * ``recvmsg()``: ``MSG_CMSG_CLOEXEC``, available on Linux 2.6.23+, + NetBSD 6.0. -On Linux older than 2.6.23, ``O_CLOEXEC`` flag is simply ignored. So we have to -check that the flag is supported by calling ``fcntl()``. If it does not work, -we have to set the flag using ``fcntl()``. +On Linux older than 2.6.23, ``O_CLOEXEC`` flag is simply ignored. So +we have to check that the flag is supported by calling ``fcntl()``. If +it does not work, we have to set the flag using ``fcntl()``. -XXX what is the behaviour on Linux older than 2.6.27 with SOCK_CLOEXEC? XXX +XXX what is the behaviour on Linux older than 2.6.27 +XXX with SOCK_CLOEXEC? XXX New functions: @@ -530,8 +567,8 @@ * ``pipe2()``: available on Linux 2.6.27+ (and glibc 2.9) * ``accept4()``: available on Linux 2.6.28+ (and glibc 2.10) -If ``accept4()`` is called on Linux older than 2.6.28, ``accept4()`` returns -``-1`` (fail) and errno is set to ``ENOSYS``. +If ``accept4()`` is called on Linux older than 2.6.28, ``accept4()`` +returns ``-1`` (fail) and errno is set to ``ENOSYS``. Links @@ -540,7 +577,8 @@ Links: * `Secure File Descriptor Handling - `_ (Ulrich Drepper, 2008) + `_ (Ulrich Drepper, + 2008) * `win32_support.py of the Tornado project `_: emulate fcntl(fd, F_SETFD, FD_CLOEXEC) using @@ -568,14 +606,17 @@ * `O_CLOEXEC flag missing for Kernel::open `_: `commit reverted - `_ later + `_ + later Footnotes ========= -.. [#subprocess_close] On UNIX since Python 3.2, subprocess.Popen() closes all file descriptors by - default: ``close_fds=True``. It closes file descriptors in range 3 inclusive - to ``local_max_fd`` exclusive, where ``local_max_fd`` is ``fcntl(0, - F_MAXFD)`` on NetBSD, or ``sysconf(_SC_OPEN_MAX)`` otherwise. If the error - pipe has a descriptor smaller than 3, ``ValueError`` is raised. +.. [#subprocess_close] On UNIX since Python 3.2, subprocess.Popen() + closes all file descriptors by default: ``close_fds=True``. It + closes file descriptors in range 3 inclusive to ``local_max_fd`` + exclusive, where ``local_max_fd`` is ``fcntl(0, F_MAXFD)`` on + NetBSD, or ``sysconf(_SC_OPEN_MAX)`` otherwise. If the error pipe + has a descriptor smaller than 3, ``ValueError`` is raised. + -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Jan 13 01:36:38 2013 From: python-checkins at python.org (victor.stinner) Date: Sun, 13 Jan 2013 01:36:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_433=3A_finish_to_format_t?= =?utf-8?q?o_70_columns?= Message-ID: <3YkJlp2HFkzS8Z@mail.python.org> http://hg.python.org/peps/rev/a6879f3056ad changeset: 4669:a6879f3056ad user: Victor Stinner date: Sun Jan 13 01:34:36 2013 +0100 summary: PEP 433: finish to format to 70 columns files: pep-0433.txt | 15 ++++++++------- 1 files changed, 8 insertions(+), 7 deletions(-) diff --git a/pep-0433.txt b/pep-0433.txt --- a/pep-0433.txt +++ b/pep-0433.txt @@ -132,21 +132,22 @@ XXX descriptors of the constructor the ``pass_fds`` argument? XXX .. note:: - See `Close file descriptors after fork`_ for a possible solution for - ``fork()`` without ``exec()``. + See `Close file descriptors after fork`_ for a possible solution + for ``fork()`` without ``exec()``. Proposal ======== -This PEP proposes to add a new optional argument ``cloexec`` on functions -creating file descriptors in the Python standard library. If the argument is -``True``, the close-on-exec flag will be set on the new file descriptor. +This PEP proposes to add a new optional argument ``cloexec`` on +functions creating file descriptors in the Python standard library. If +the argument is ``True``, the close-on-exec flag will be set on the +new file descriptor. Add a new function: - * ``os.set_cloexec(fd: int, cloexec: bool)``: set or unset the close-on-exec - flag of a file descriptor + * ``os.set_cloexec(fd: int, cloexec: bool)``: set or unset the + close-on-exec flag of a file descriptor Add a new optional ``cloexec`` argument to: -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Jan 13 03:23:09 2013 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 13 Jan 2013 03:23:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogbWFrZSBkZXF1ZV9j?= =?utf-8?q?lear_void=2C_since_it=27s_infallible?= Message-ID: <3YkM6j4wTLzSG6@mail.python.org> http://hg.python.org/cpython/rev/623a7de80432 changeset: 81470:623a7de80432 branch: 3.3 parent: 81467:c276fbd95182 user: Benjamin Peterson date: Sat Jan 12 21:22:18 2013 -0500 summary: make deque_clear void, since it's infallible files: Modules/_collectionsmodule.c | 8 ++------ 1 files changed, 2 insertions(+), 6 deletions(-) diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -588,7 +588,7 @@ PyDoc_STRVAR(remove_doc, "D.remove(value) -- remove first occurrence of value."); -static int +static void deque_clear(dequeobject *deque) { PyObject *item; @@ -601,7 +601,6 @@ assert(deque->leftblock == deque->rightblock && deque->leftindex - 1 == deque->rightindex && deque->len == 0); - return 0; } static PyObject * @@ -704,10 +703,7 @@ static PyObject * deque_clearmethod(dequeobject *deque) { - int rv; - - rv = deque_clear(deque); - assert (rv != -1); + deque_clear(deque); Py_RETURN_NONE; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 03:23:11 2013 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 13 Jan 2013 03:23:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4z?= Message-ID: <3YkM6l0KQ3zSGL@mail.python.org> http://hg.python.org/cpython/rev/016953aabc47 changeset: 81471:016953aabc47 parent: 81469:72ddb250f058 parent: 81470:623a7de80432 user: Benjamin Peterson date: Sat Jan 12 21:22:33 2013 -0500 summary: merge 3.3 files: Modules/_collectionsmodule.c | 8 ++------ 1 files changed, 2 insertions(+), 6 deletions(-) diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -632,7 +632,7 @@ PyDoc_STRVAR(remove_doc, "D.remove(value) -- remove first occurrence of value."); -static int +static void deque_clear(dequeobject *deque) { PyObject *item; @@ -645,7 +645,6 @@ assert(deque->leftblock == deque->rightblock && deque->leftindex - 1 == deque->rightindex && deque->len == 0); - return 0; } static PyObject * @@ -748,10 +747,7 @@ static PyObject * deque_clearmethod(dequeobject *deque) { - int rv; - - rv = deque_clear(deque); - assert (rv != -1); + deque_clear(deque); Py_RETURN_NONE; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 03:23:12 2013 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 13 Jan 2013 03:23:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogbWFrZSBkZXF1ZV9j?= =?utf-8?q?lear_void=2C_since_it=27s_infallible?= Message-ID: <3YkM6m2sr4zSFw@mail.python.org> http://hg.python.org/cpython/rev/ad585b439d97 changeset: 81472:ad585b439d97 branch: 2.7 parent: 81458:e651d96e6b07 user: Benjamin Peterson date: Sat Jan 12 21:22:18 2013 -0500 summary: make deque_clear void, since it's infallible files: Modules/_collectionsmodule.c | 8 ++------ 1 files changed, 2 insertions(+), 6 deletions(-) diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -632,7 +632,7 @@ PyDoc_STRVAR(remove_doc, "D.remove(value) -- remove first occurrence of value."); -static int +static void deque_clear(dequeobject *deque) { PyObject *item; @@ -645,7 +645,6 @@ assert(deque->leftblock == deque->rightblock && deque->leftindex - 1 == deque->rightindex && deque->len == 0); - return 0; } static PyObject * @@ -748,10 +747,7 @@ static PyObject * deque_clearmethod(dequeobject *deque) { - int rv; - - rv = deque_clear(deque); - assert (rv != -1); + deque_clear(deque); Py_RETURN_NONE; } -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sun Jan 13 05:55:53 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 13 Jan 2013 05:55:53 +0100 Subject: [Python-checkins] Daily reference leaks (016953aabc47): sum=6 Message-ID: results for 016953aabc47 on branch "default" -------------------------------------------- test_dbm leaked [0, 2, 0] references, sum=2 test_dbm leaked [0, 2, 2] memory blocks, sum=4 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog81Sac8', '-x'] From python-checkins at python.org Sun Jan 13 07:35:15 2013 From: python-checkins at python.org (nick.coghlan) Date: Sun, 13 Jan 2013 07:35:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_432=3A_Flesh_out_a_design?= =?utf-8?q?_for_main_execution?= Message-ID: <3YkSjb01nPzS5y@mail.python.org> http://hg.python.org/peps/rev/ecbd58cf5115 changeset: 4670:ecbd58cf5115 user: Nick Coghlan date: Sun Jan 13 16:35:06 2013 +1000 summary: PEP 432: Flesh out a design for main execution files: pep-0432.txt | 231 +++++++++++++++++++++++++++++--------- 1 files changed, 175 insertions(+), 56 deletions(-) diff --git a/pep-0432.txt b/pep-0432.txt --- a/pep-0432.txt +++ b/pep-0432.txt @@ -43,9 +43,11 @@ * Initializing - interpreter partially available * Initialized - interpreter available, __main__ related metadata incomplete -* Main Execution - __main__ related metadata populated, bytecode - executing in the __main__ module namespace (embedding applications - may choose not to use this phase) +* Main Preparation - __main__ related metadata populated +* Main Execution - bytecode executing in the __main__ module namespace + +(Embedding applications may choose not to use the Main Preparation and +Execution phases) As a concrete use case to help guide any design changes, and to solve a known problem where the appropriate defaults for system utilities differ from those @@ -78,21 +80,21 @@ (such as configuring the Unicode settings for OS interfaces in Python 3 as well as bootstrapping a pure Python implementation of the import system). -Much of this complexity is accessible only through the ``Py_Main`` and -``Py_Initialize`` APIs, offering embedding applications little opportunity -for customisation. This creeping complexity also makes life difficult for -maintainers, as much of the configuration needs to take place prior to the -``Py_Initialize`` call, meaning much of the Python C API cannot be used -safely. +Much of this complexity is formally accessible only through the ``Py_Main`` +and ``Py_Initialize`` APIs, offering embedding applications little +opportunity for customisation. This creeping complexity also makes life +difficult for maintainers, as much of the configuration needs to take +place prior to the ``Py_Initialize`` call, meaning much of the Python C +API cannot be used safely. A number of proposals are on the table for even *more* sophisticated startup behaviour, such as an isolated mode equivalent to that described in this PEP as a "system Python" [6_], better control over ``sys.path`` initialization (easily adding additional directories on the command line in a cross-platform fashion [7_], as well as controlling the configuration of -``sys.path[0]`` [8_]), easier configuration of utilities like coverage tracing -when launching Python subprocesses [9_], and easier control of the encoding -used for the standard IO streams when embedding CPython in a larger +``sys.path[0]`` [8_]), easier configuration of utilities like coverage +tracing when launching Python subprocesses [9_], and easier control of the +encoding used for the standard IO streams when embedding CPython in a larger application [10_]. Rather than attempting to bolt such behaviour onto an already complicated @@ -118,8 +120,8 @@ By moving to an explicitly multi-phase startup sequence, developers should only need to understand which features are not available in the core -bootstrapping state, as the vast majority of the configuration process -will now take place in that state. +bootstrapping phase, as the vast majority of the configuration process +will now take place during that phase. By basing the new design on a combination of C structures and Python data types, it should also be easier to modify the system in the @@ -504,14 +506,13 @@ Interpreter Initialization Phases --------------------------------- -Four distinct phases are proposed: +Five distinct phases are proposed: * Pre-Initialization: * no interpreter is available. * ``Py_IsInitializing()`` returns ``0`` * ``Py_IsInitialized()`` returns ``0`` - * ``Py_IsRunningMain()`` returns ``0`` * The embedding application determines the settings required to create the main interpreter and moves to the next phase by calling ``Py_BeginInitialization``. @@ -521,7 +522,6 @@ * the main interpreter is available, but only partially configured. * ``Py_IsInitializing()`` returns ``1`` * ``Py_IsInitialized()`` returns ``0`` - * ``Py_RunningMain()`` returns ``0`` * The embedding application determines and applies the settings required to complete the initialization process by calling ``Py_ReadConfiguration`` and ``Py_EndInitialization``. @@ -529,26 +529,28 @@ * Initialized: * the main interpreter is available and fully operational, but - ``__main__`` related metadata is incomplete and the site module may - not have been imported. + ``__main__`` related metadata is incomplete * ``Py_IsInitializing()`` returns ``0`` * ``Py_IsInitialized()`` returns ``1`` - * ``Py_IsRunningMain()`` returns ``0`` * Optionally, the embedding application may identify and begin executing code in the ``__main__`` module namespace by calling - ``Py_RunPathAsMain``, ``Py_RunModuleAsMain`` or ``Py_RunStreamAsMain``. + ``PyRun_PrepareMain`` and ``PyRun_ExecMain``. + +* Main Preparation: + + * subphase of Initialized (not separately identified at runtime) + * fully populates ``__main__`` related metadata + * may execute code in ``__main__`` namespace (e.g. ``PYTHONSTARTUP``) * Main Execution: - * bytecode is being executed in the ``__main__`` namespace - * ``Py_IsInitializing()`` returns ``0`` - * ``Py_IsInitialized()`` returns ``1`` - * ``Py_IsRunningMain()`` returns ``1`` + * subphase of Initialized (not separately identified at runtime) + * user supplied bytecode is being executed in the ``__main__`` namespace -As indicated by the phase reporting functions, main module execution is -an optional subphase of Initialized rather than a completely distinct phase. +As noted above, main module preparation and execution are optional subphases +of Initialized rather than completely distinct phases. -All 4 phases will be used by the standard CPython interpreter and the +All listed phases will be used by the standard CPython interpreter and the proposed System Python interpreter. Other embedding applications may choose to skip the step of executing code in the ``__main__`` namespace. @@ -817,15 +819,9 @@ /* Filesystem access */ PyUnicodeObject *fs_encoding; - /* Interactive interpreter */ - int stdin_is_interactive; /* Force interactive behaviour */ - int inspect_main; /* -i switch, PYTHONINSPECT */ - PyUnicodeObject *startup_file; /* PYTHONSTARTUP */ - /* Debugging output */ int debug_parser; /* -d switch, PYTHONDEBUG */ int verbosity; /* -v switch */ - int suppress_banner; /* -q switch */ /* Code generation */ int bytes_warnings; /* -b switch */ @@ -833,6 +829,32 @@ /* Signal handling */ int install_sig_handlers; + + /* Implicit execution */ + PyUnicodeObject *startup_file; /* PYTHONSTARTUP */ + + /* Main module + * + * If prepare_main is set, at most one of the main_* settings should + * be set before calling PyRun_PrepareMain (Py_ReadConfiguration will + * set one of them based on the command line arguments if prepare_main + * is non-zero when that API is called). + int prepare_main; + PyUnicodeObject *main_source; /* -c switch */ + PyUnicodeObject *main_path; /* filesystem path */ + PyUnicodeObject *main_module; /* -m switch */ + PyCodeObject *main_code; /* Run directly from a code object */ + PyObject *main_stream; /* Run from stream */ + int run_implicit_code; /* Run implicit code during prep */ + + /* Interactive main + * + * Note: Settings related to interactive mode are very much in flux. + */ + PyObject *prompt_stream; /* Output interactive prompt */ + int show_banner; /* -q switch (inverted) */ + int inspect_main; /* -i switch, PYTHONINSPECT */ + } Py_Config; @@ -844,17 +866,19 @@ #define _Py_ImportConfig_INIT -1, -1, NULL #define _Py_StreamConfig_INIT -1, NULL, NULL, NULL, NULL, NULL, NULL #define _Py_FilesystemConfig_INIT NULL - #define _Py_InteractiveConfig_INIT -1, -1, NULL #define _Py_DebuggingConfig_INIT -1, -1, -1 #define _Py_CodeGenConfig_INIT -1, -1 #define _Py_SignalConfig_INIT -1 + #define _Py_ImplicitConfig_INIT NULL + #define _Py_MainConfig_INIT -1, NULL, NULL, NULL, NULL, NULL, -1 + #define _Py_InteractiveConfig_INIT NULL, -1, -1 #define Py_Config_INIT {_Py_ArgConfig_INIT, _Py_LocationConfig_INIT, _Py_SiteConfig_INIT, _Py_ImportConfig_INIT, _Py_StreamConfig_INIT, _Py_FilesystemConfig_INIT, - _Py_InteractiveConfig_INIT, _Py_DebuggingConfig_INIT, _Py_CodeGenConfig_INIT, - _Py_SignalConfig_INIT} + _Py_SignalConfig_INIT, _Py_ImplicitConfig_INIT, + _Py_MainConfig_INIT, _Py_InteractiveConfig_INIT} @@ -893,6 +917,11 @@ * it will be the same as ``sys.path[0]`` rather than the location of the ``__main__`` module when executing a valid ``sys.path`` entry (typically a zipfile or directory) + * otherwise, it will be accurate: + + * the script name if running an ordinary script + * ``-c`` if executing a supplied string + * ``-`` or the empty string if running from stdin * the metadata in the ``__main__`` module will still indicate it is a builtin module @@ -904,21 +933,103 @@ ``import site`` is later explicitly executed in the process. +Preparing the main module +------------------------- + +This subphase completes the population of the ``__main__`` module +related metadata, without actually starting execution of the ``__main__`` +module code. + +It is handled by calling the following API:: + + int PyRun_PrepareMain(); + +The actual processing is driven by the main related settings stored in +the interpreter state as part of the configuration struct. + +If ``prepare_main`` is zero, this call does nothing. + +If all of ``main_source``, ``main_path``, ``main_module``, +``main_stream`` and ``main_code`` are NULL, this call does nothing. + +If more than one of ``main_source``, ``main_path``, ``main_module``, +``main_stream`` or ``main_code`` are set, ``RuntimeError`` will be reported. + +If ``main_code`` is already set, then this call does nothing. + +If ``main_stream`` is set, and ``run_implicit_code`` is also set, then +the file identified in ``startup_file`` will be read, compiled and +executed in the ``__main__`` namespace. + +If ``main_source``, ``main_path`` or ``main_module`` are set, then this +call will take whatever steps are needed to populate ``main_code``: + +* For ``main_source``, the supplied string will be compiled and saved to + ``main_code``. + +* For ``main_path``: + * if the supplied path is recognised as a valid ``sys.path`` entry, it + is inserted as ``sys.path[0]``, ``main_module`` is set + to ``__main__`` and processing continues as for ``main_module`` below. + * otherwise, path is read as a CPython bytecode file + * if that fails, it is read as a Python source file and compiled + * in the latter two cases, the code object is saved to ``main_code`` + and ``__main__.__file__`` is set appropriately + +* For ``main_module``: + * any parent package is imported + * the loader for the module is determined + * if the loader indicates the module is a package, add ``.__main__`` to + the end of ``main_module`` and try again (if the final name segment + is already ``.__main__`` then fail immediately) + * once the module source code is located, save the compiled module code + as ``main_code`` and populate the following attributes in ``__main__`` + appropriately: ``__name__``, ``__loader__``, ``__file__``, + ``__cached__``, ``__package__``. + + +(Note: the behaviour described in this section isn't new, it's a write-up +of the current behaviour of the CPython interpreter adjusted for the new +configuration system) + + Executing the main module ------------------------- - +This subphase covers the execution of the actual ``__main__`` module code. -Initial thought is that hiding the various options behind a single API -would make that API too complicated, so 3 separate APIs is more likely:: +It is handled by calling the following API:: - Py_RunPathAsMain - Py_RunModuleAsMain - Py_RunStreamAsMain + int PyRun_ExecMain(); -Query API to indicate that ``sys.argv[0]`` is fully populated:: +The actual processing is driven by the main related settings stored in +the interpreter state as part of the configuration struct. - Py_IsRunningMain() +If both ``main_stream`` and ``main_code`` are NULL, this call does nothing. + +If both ``main_stream`` and ``main_code`` are set, ``RuntimeError`` will +be reported. + +If ``main_stream`` and ``prompt_stream`` are both set, main execution will +be delegated to a new API:: + + int PyRun_InteractiveMain(PyObject *input, PyObject* output); + +If ``main_stream`` is set and ``prompt_stream`` is NULL, main execution will +be delegated to a new API:: + + int PyRun_StreamInMain(PyObject *input); + +If ``main_code`` is set, main execution will be delegated to a new +API:: + + int PyRun_CodeInMain(PyCodeObject *code); + +After execution of main completes, if ``inspect_main`` is set, or +the ``PYTHONINSPECT`` environment variable has been set, then +``PyRun_ExecMain`` will invoke +``PyRun_InteractiveMain(sys.__stdin__, sys.__stdout__)``. + Internal Storage of Configuration Data -------------------------------------- @@ -931,7 +1042,7 @@ For debugging purposes, the configuration settings will be exposed as a ``sys._configuration`` simple namespace (similar to ``sys.flags`` and ``sys.implementation``. Field names will match those in the configuration -structs, exception for ``hash_seed``, which will be deliberately excluded. +structs, except for ``hash_seed``, which will be deliberately excluded. An underscored attribute is chosen deliberately, as these configuration settings are part of the CPython implementation, rather than part of the @@ -941,16 +1052,19 @@ attributes on ``sys.implementation``, as described in PEP 421. These are *snapshots* of the initial configuration settings. They are not -consulted by the interpreter during runtime. +modified by the interpreter during runtime (except as noted above). Stable ABI ---------- -All of the APIs proposed in this PEP are excluded from the stable ABI, as +Most of the APIs proposed in this PEP are excluded from the stable ABI, as embedding a Python interpreter involves a much higher degree of coupling than merely writing an extension. +The only newly exposed API that will be part of the stable ABI is the +``Py_IsInitializing()`` query. + Build time configuration ------------------------ @@ -1038,27 +1152,32 @@ * Error details for Py_ReadConfiguration and Py_EndInitialization (these should become clear as the implementation progresses) -* Is ``Py_IsRunningMain()`` worth keeping? -* Should the answers to ``Py_IsInitialized()`` and ``Py_IsRunningMain()`` be - exposed via the ``sys`` module? -* Is the ``Py_Config`` struct too unwieldy to be practical? Would a Python - dictionary be a better choice? +* Should there be ``Py_PreparingMain()`` and ``Py_RunningMain()`` query APIs? +* Should the answer to ``Py_IsInitialized()`` be exposed via the ``sys`` + module? +* Is initialisation of the ``Py_Config`` struct too unwieldy to be + maintainable? Would a Python dictionary be a better choice, despite + being harder to work with from C code? * Would it be better to manage the flag variables in ``Py_Config`` as Python integers or as "negative means false, positive means true, zero means not set" so the struct can be initialized with a simple ``memset(&config, 0, sizeof(*config))``, eliminating the need to update both Py_Config and Py_Config_INIT when adding new fields? -* The name of the system Python executable is a bikeshed waiting to be +* The name of the new system Python executable is a bikeshed waiting to be painted. The 3 options considered so far are ``spython``, ``pysystem`` and ``python-minimal``. The PEP text reflects my current preferred choice - i.e. ``pysystem``. + (``pysystem``). Implementation ============== The reference implementation is being developed as a feature branch in my -BitBucket sandbox [2_]. +BitBucket sandbox [2_]. Pull requests to fix the inevitably broken +Windows builds are welcome, but the basic design is still in too much flux +for other pull requests to be feasible just yet. Once the overall design +settles down and it's a matter of migrating individual settings over to +the new design, that level of collaboration should become more practical. As the number of application binaries created by the build process is now four, the reference implementation also creates a new top level "Apps" -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Jan 13 08:24:18 2013 From: python-checkins at python.org (nick.coghlan) Date: Sun, 13 Jan 2013 08:24:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_No_need_for_the_abbreviation?= Message-ID: <3YkTpB5p8CzS5R@mail.python.org> http://hg.python.org/peps/rev/4f996a14d315 changeset: 4671:4f996a14d315 user: Nick Coghlan date: Sun Jan 13 17:24:10 2013 +1000 summary: No need for the abbreviation files: pep-0432.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0432.txt b/pep-0432.txt --- a/pep-0432.txt +++ b/pep-0432.txt @@ -828,7 +828,7 @@ int optimize; /* -O switch */ /* Signal handling */ - int install_sig_handlers; + int install_signal_handlers; /* Implicit execution */ PyUnicodeObject *startup_file; /* PYTHONSTARTUP */ -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Jan 13 11:26:31 2013 From: python-checkins at python.org (georg.brandl) Date: Sun, 13 Jan 2013 11:26:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Closes_=2316949=3A_update_wor?= =?utf-8?q?ding_about_string_exceptions=2E?= Message-ID: <3YkYrR594RzS0n@mail.python.org> http://hg.python.org/peps/rev/bf68a42a43bf changeset: 4672:bf68a42a43bf user: Georg Brandl date: Sun Jan 13 11:27:31 2013 +0100 summary: Closes #16949: update wording about string exceptions. files: pep-0008.txt | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0008.txt b/pep-0008.txt --- a/pep-0008.txt +++ b/pep-0008.txt @@ -763,8 +763,8 @@ - Use class-based exceptions. - String exceptions in new code are forbidden, because this language - feature is being removed in Python 2.6. + String exceptions in new code are forbidden, and this language + feature has been removed in Python 2.6. Modules or packages should define their own domain-specific base exception class, which should be subclassed from the built-in -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Jan 13 11:27:10 2013 From: python-checkins at python.org (georg.brandl) Date: Sun, 13 Jan 2013 11:27:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Closes_=2316950=3A_update_wor?= =?utf-8?q?ding_about_raise_syntax=2E?= Message-ID: <3YkYsB3HjgzRxn@mail.python.org> http://hg.python.org/peps/rev/89db18c77152 changeset: 4673:89db18c77152 user: Georg Brandl date: Sun Jan 13 11:28:10 2013 +0100 summary: Closes #16950: update wording about raise syntax. files: pep-0008.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0008.txt b/pep-0008.txt --- a/pep-0008.txt +++ b/pep-0008.txt @@ -783,7 +783,7 @@ The paren-using form is preferred because when the exception arguments are long or include string formatting, you don't need to use line continuation characters thanks to the containing - parentheses. The older form will be removed in Python 3. + parentheses. The older form is not legal syntax in Python 3. - When catching exceptions, mention specific exceptions whenever possible instead of using a bare ``except:`` clause. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Jan 13 14:15:54 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sun, 13 Jan 2013 14:15:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogdGVzdF9hc3luY29y?= =?utf-8?q?e=3A_wait_explicitly_for_a_thread_termination_=28this_dangling_?= =?utf-8?q?thread?= Message-ID: <3Ykdbt4GpFzS7k@mail.python.org> http://hg.python.org/cpython/rev/6e8788353ddf changeset: 81473:6e8788353ddf branch: 2.7 user: Charles-Fran?ois Natali date: Sun Jan 13 13:55:13 2013 +0100 summary: test_asyncore: wait explicitly for a thread termination (this dangling thread could be the cause of a random failure). files: Lib/test/test_asyncore.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_asyncore.py b/Lib/test/test_asyncore.py --- a/Lib/test/test_asyncore.py +++ b/Lib/test/test_asyncore.py @@ -712,6 +712,7 @@ server = TCPServer() t = threading.Thread(target=lambda: asyncore.loop(timeout=0.1, count=500)) t.start() + self.addCleanup(t.join) for x in xrange(20): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 14:15:56 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sun, 13 Jan 2013 14:15:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogdGVzdF9hc3luY29y?= =?utf-8?q?e=3A_wait_explicitly_for_a_thread_termination_=28this_dangling_?= =?utf-8?q?thread?= Message-ID: <3Ykdbw06RyzSD1@mail.python.org> http://hg.python.org/cpython/rev/d55a06f7ccbf changeset: 81474:d55a06f7ccbf branch: 3.2 parent: 81459:20065626c0b5 user: Charles-Fran?ois Natali date: Sun Jan 13 13:56:52 2013 +0100 summary: test_asyncore: wait explicitly for a thread termination (this dangling thread could be the cause of a random failure). files: Lib/test/test_asyncore.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_asyncore.py b/Lib/test/test_asyncore.py --- a/Lib/test/test_asyncore.py +++ b/Lib/test/test_asyncore.py @@ -738,6 +738,7 @@ server = TCPServer() t = threading.Thread(target=lambda: asyncore.loop(timeout=0.1, count=500)) t.start() + self.addCleanup(t.join) for x in range(20): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 14:15:57 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sun, 13 Jan 2013 14:15:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_test=5Fasyncore=3A_wait_explicitly_for_a_thread_termination_?= =?utf-8?q?=28this_dangling_thread?= Message-ID: <3Ykdbx2gR2zSDF@mail.python.org> http://hg.python.org/cpython/rev/0545ecc24b3a changeset: 81475:0545ecc24b3a branch: 3.3 parent: 81470:623a7de80432 parent: 81474:d55a06f7ccbf user: Charles-Fran?ois Natali date: Sun Jan 13 14:08:01 2013 +0100 summary: test_asyncore: wait explicitly for a thread termination (this dangling thread could be the cause of a random failure). files: Lib/test/test_asyncore.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_asyncore.py b/Lib/test/test_asyncore.py --- a/Lib/test/test_asyncore.py +++ b/Lib/test/test_asyncore.py @@ -789,7 +789,7 @@ t = threading.Thread(target=lambda: asyncore.loop(timeout=0.1, count=500)) t.start() - + self.addCleanup(t.join) s = socket.socket(self.family, socket.SOCK_STREAM) s.settimeout(.2) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 14:15:58 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sun, 13 Jan 2013 14:15:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogRml4IHRlc3RfcG9z?= =?utf-8?q?ix_failure_on_NetBSD_buildbots=3A_sched=5Fsetparam=28=29_and?= Message-ID: <3Ykdby5FrnzS9S@mail.python.org> http://hg.python.org/cpython/rev/1719136ca357 changeset: 81476:1719136ca357 branch: 3.3 user: Charles-Fran?ois Natali date: Sun Jan 13 14:10:37 2013 +0100 summary: Fix test_posix failure on NetBSD buildbots: sched_setparam() and sched_setscheduler() can fail with EINVAL if the process scheduling policy is neither SCHED_FIFO nor SCHED_RR. files: Lib/test/test_posix.py | 20 ++++++++++---------- 1 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -930,17 +930,17 @@ self.assertRaises(OSError, posix.sched_getparam, -1) param = posix.sched_getparam(0) self.assertIsInstance(param.sched_priority, int) - try: - posix.sched_setscheduler(0, mine, param) - except OSError as e: - if e.errno != errno.EPERM: - raise - # POSIX states that calling sched_setparam() on a process with a - # scheduling policy other than SCHED_FIFO or SCHED_RR is - # implementation-defined: FreeBSD returns EINVAL. - if not sys.platform.startswith('freebsd'): - posix.sched_setparam(0, param) + # POSIX states that calling sched_setparam() or sched_setscheduler() on + # a process with a scheduling policy other than SCHED_FIFO or SCHED_RR + # is implementation-defined: NetBSD and FreeBSD can return EINVAL. + if not sys.platform.startswith(('freebsd', 'netbsd')): + try: + posix.sched_setscheduler(0, mine, param) + posix.sched_setparam(0, param) + except OSError as e: + if e.errno != errno.EPERM: + raise self.assertRaises(OSError, posix.sched_setparam, -1, param) self.assertRaises(OSError, posix.sched_setscheduler, -1, mine, param) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 14:16:00 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sun, 13 Jan 2013 14:16:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_test=5Fasyncore=3A_wait_explicitly_for_a_thread_terminat?= =?utf-8?q?ion_=28this_dangling_thread?= Message-ID: <3Ykdc015zWzSGw@mail.python.org> http://hg.python.org/cpython/rev/aa26568e4739 changeset: 81477:aa26568e4739 parent: 81471:016953aabc47 parent: 81476:1719136ca357 user: Charles-Fran?ois Natali date: Sun Jan 13 14:12:35 2013 +0100 summary: test_asyncore: wait explicitly for a thread termination (this dangling thread could be the cause of a random failure). files: Lib/test/test_asyncore.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_asyncore.py b/Lib/test/test_asyncore.py --- a/Lib/test/test_asyncore.py +++ b/Lib/test/test_asyncore.py @@ -789,7 +789,7 @@ t = threading.Thread(target=lambda: asyncore.loop(timeout=0.1, count=500)) t.start() - + self.addCleanup(t.join) s = socket.socket(self.family, socket.SOCK_STREAM) s.settimeout(.2) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 14:16:01 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sun, 13 Jan 2013 14:16:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_test=5Fposix_failure_o?= =?utf-8?q?n_NetBSD_buildbots=3A_sched=5Fsetparam=28=29_and?= Message-ID: <3Ykdc13gSFzSGs@mail.python.org> http://hg.python.org/cpython/rev/3c67bd5abe38 changeset: 81478:3c67bd5abe38 user: Charles-Fran?ois Natali date: Sun Jan 13 14:13:25 2013 +0100 summary: Fix test_posix failure on NetBSD buildbots: sched_setparam() and sched_setscheduler() can fail with EINVAL if the process scheduling policy is neither SCHED_FIFO nor SCHED_RR. files: Lib/test/test_posix.py | 20 ++++++++++---------- 1 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -930,17 +930,17 @@ self.assertRaises(OSError, posix.sched_getparam, -1) param = posix.sched_getparam(0) self.assertIsInstance(param.sched_priority, int) - try: - posix.sched_setscheduler(0, mine, param) - except OSError as e: - if e.errno != errno.EPERM: - raise - # POSIX states that calling sched_setparam() on a process with a - # scheduling policy other than SCHED_FIFO or SCHED_RR is - # implementation-defined: FreeBSD returns EINVAL. - if not sys.platform.startswith('freebsd'): - posix.sched_setparam(0, param) + # POSIX states that calling sched_setparam() or sched_setscheduler() on + # a process with a scheduling policy other than SCHED_FIFO or SCHED_RR + # is implementation-defined: NetBSD and FreeBSD can return EINVAL. + if not sys.platform.startswith(('freebsd', 'netbsd')): + try: + posix.sched_setscheduler(0, mine, param) + posix.sched_setparam(0, param) + except OSError as e: + if e.errno != errno.EPERM: + raise self.assertRaises(OSError, posix.sched_setparam, -1, param) self.assertRaises(OSError, posix.sched_setscheduler, -1, mine, param) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 14:18:19 2013 From: python-checkins at python.org (eli.bendersky) Date: Sun, 13 Jan 2013 14:18:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Clean_trailing?= =?utf-8?q?_whitespace_in_=5Felementtree=2Ec?= Message-ID: <3Ykdfg2FFgzS9R@mail.python.org> http://hg.python.org/cpython/rev/7e072816b197 changeset: 81479:7e072816b197 branch: 3.3 parent: 81470:623a7de80432 user: Eli Bendersky date: Sun Jan 13 05:14:47 2013 -0800 summary: Clean trailing whitespace in _elementtree.c files: Modules/_elementtree.c | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c --- a/Modules/_elementtree.c +++ b/Modules/_elementtree.c @@ -310,7 +310,7 @@ /* Helper function for extracting the attrib dictionary from a keywords dict. * This is required by some constructors/functions in this module that can - * either accept attrib as a keyword argument or all attributes splashed + * either accept attrib as a keyword argument or all attributes splashed * directly into *kwds. * If there is no 'attrib' keyword, return an empty dict. */ @@ -1977,7 +1977,7 @@ elementiter_next(ElementIterObject *it) { /* Sub-element iterator. - * + * * A short note on gettext: this function serves both the iter() and * itertext() methods to avoid code duplication. However, there are a few * small differences in the way these iterations work. Namely: @@ -3039,7 +3039,7 @@ } } -static void +static void expat_start_doctype_handler(XMLParserObject *self, const XML_Char *doctype_name, const XML_Char *sysid, @@ -3244,7 +3244,7 @@ self_xp->handle_doctype = PyObject_GetAttrString(target, "doctype"); PyErr_Clear(); - + /* configure parser */ EXPAT(SetUserData)(self_xp->parser, self_xp); EXPAT(SetElementHandler)( -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 14:18:20 2013 From: python-checkins at python.org (eli.bendersky) Date: Sun, 13 Jan 2013 14:18:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Clean_trailing_whitespace_in_=5Felementtree=2Ec?= Message-ID: <3Ykdfh58CszSGs@mail.python.org> http://hg.python.org/cpython/rev/0b3ebb344a8a changeset: 81480:0b3ebb344a8a parent: 81471:016953aabc47 parent: 81479:7e072816b197 user: Eli Bendersky date: Sun Jan 13 05:15:52 2013 -0800 summary: Clean trailing whitespace in _elementtree.c files: Modules/_elementtree.c | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c --- a/Modules/_elementtree.c +++ b/Modules/_elementtree.c @@ -310,7 +310,7 @@ /* Helper function for extracting the attrib dictionary from a keywords dict. * This is required by some constructors/functions in this module that can - * either accept attrib as a keyword argument or all attributes splashed + * either accept attrib as a keyword argument or all attributes splashed * directly into *kwds. * If there is no 'attrib' keyword, return an empty dict. */ @@ -1977,7 +1977,7 @@ elementiter_next(ElementIterObject *it) { /* Sub-element iterator. - * + * * A short note on gettext: this function serves both the iter() and * itertext() methods to avoid code duplication. However, there are a few * small differences in the way these iterations work. Namely: @@ -3039,7 +3039,7 @@ } } -static void +static void expat_start_doctype_handler(XMLParserObject *self, const XML_Char *doctype_name, const XML_Char *sysid, @@ -3244,7 +3244,7 @@ self_xp->handle_doctype = PyObject_GetAttrString(target, "doctype"); PyErr_Clear(); - + /* configure parser */ EXPAT(SetUserData)(self_xp->parser, self_xp); EXPAT(SetElementHandler)( -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 14:18:22 2013 From: python-checkins at python.org (eli.bendersky) Date: Sun, 13 Jan 2013 14:18:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?q?=29=3A_merge_heads?= Message-ID: <3Ykdfk0dWNzS9S@mail.python.org> http://hg.python.org/cpython/rev/05cacbd2f545 changeset: 81481:05cacbd2f545 parent: 81480:0b3ebb344a8a parent: 81478:3c67bd5abe38 user: Eli Bendersky date: Sun Jan 13 05:16:36 2013 -0800 summary: merge heads files: Lib/test/test_asyncore.py | 2 +- Lib/test/test_posix.py | 20 ++++++++++---------- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/Lib/test/test_asyncore.py b/Lib/test/test_asyncore.py --- a/Lib/test/test_asyncore.py +++ b/Lib/test/test_asyncore.py @@ -789,7 +789,7 @@ t = threading.Thread(target=lambda: asyncore.loop(timeout=0.1, count=500)) t.start() - + self.addCleanup(t.join) s = socket.socket(self.family, socket.SOCK_STREAM) s.settimeout(.2) diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -930,17 +930,17 @@ self.assertRaises(OSError, posix.sched_getparam, -1) param = posix.sched_getparam(0) self.assertIsInstance(param.sched_priority, int) - try: - posix.sched_setscheduler(0, mine, param) - except OSError as e: - if e.errno != errno.EPERM: - raise - # POSIX states that calling sched_setparam() on a process with a - # scheduling policy other than SCHED_FIFO or SCHED_RR is - # implementation-defined: FreeBSD returns EINVAL. - if not sys.platform.startswith('freebsd'): - posix.sched_setparam(0, param) + # POSIX states that calling sched_setparam() or sched_setscheduler() on + # a process with a scheduling policy other than SCHED_FIFO or SCHED_RR + # is implementation-defined: NetBSD and FreeBSD can return EINVAL. + if not sys.platform.startswith(('freebsd', 'netbsd')): + try: + posix.sched_setscheduler(0, mine, param) + posix.sched_setparam(0, param) + except OSError as e: + if e.errno != errno.EPERM: + raise self.assertRaises(OSError, posix.sched_setparam, -1, param) self.assertRaises(OSError, posix.sched_setscheduler, -1, mine, param) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 14:18:23 2013 From: python-checkins at python.org (eli.bendersky) Date: Sun, 13 Jan 2013 14:18:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuMyk6?= =?utf-8?q?_merge_heads?= Message-ID: <3Ykdfl3JK1zSHG@mail.python.org> http://hg.python.org/cpython/rev/9202a716673b changeset: 81482:9202a716673b branch: 3.3 parent: 81479:7e072816b197 parent: 81476:1719136ca357 user: Eli Bendersky date: Sun Jan 13 05:17:19 2013 -0800 summary: merge heads files: Lib/test/test_asyncore.py | 2 +- Lib/test/test_posix.py | 20 ++++++++++---------- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/Lib/test/test_asyncore.py b/Lib/test/test_asyncore.py --- a/Lib/test/test_asyncore.py +++ b/Lib/test/test_asyncore.py @@ -789,7 +789,7 @@ t = threading.Thread(target=lambda: asyncore.loop(timeout=0.1, count=500)) t.start() - + self.addCleanup(t.join) s = socket.socket(self.family, socket.SOCK_STREAM) s.settimeout(.2) diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -930,17 +930,17 @@ self.assertRaises(OSError, posix.sched_getparam, -1) param = posix.sched_getparam(0) self.assertIsInstance(param.sched_priority, int) - try: - posix.sched_setscheduler(0, mine, param) - except OSError as e: - if e.errno != errno.EPERM: - raise - # POSIX states that calling sched_setparam() on a process with a - # scheduling policy other than SCHED_FIFO or SCHED_RR is - # implementation-defined: FreeBSD returns EINVAL. - if not sys.platform.startswith('freebsd'): - posix.sched_setparam(0, param) + # POSIX states that calling sched_setparam() or sched_setscheduler() on + # a process with a scheduling policy other than SCHED_FIFO or SCHED_RR + # is implementation-defined: NetBSD and FreeBSD can return EINVAL. + if not sys.platform.startswith(('freebsd', 'netbsd')): + try: + posix.sched_setscheduler(0, mine, param) + posix.sched_setparam(0, param) + except OSError as e: + if e.errno != errno.EPERM: + raise self.assertRaises(OSError, posix.sched_setparam, -1, param) self.assertRaises(OSError, posix.sched_setscheduler, -1, mine, param) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 14:18:25 2013 From: python-checkins at python.org (eli.bendersky) Date: Sun, 13 Jan 2013 14:18:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_null_merge?= Message-ID: <3Ykdfn1BtSzSDR@mail.python.org> http://hg.python.org/cpython/rev/3e0739b30370 changeset: 81483:3e0739b30370 parent: 81481:05cacbd2f545 parent: 81482:9202a716673b user: Eli Bendersky date: Sun Jan 13 05:17:52 2013 -0800 summary: null merge files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 14:26:59 2013 From: python-checkins at python.org (eli.bendersky) Date: Sun, 13 Jan 2013 14:26:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE2OTIy?= =?utf-8?q?=3A_fixed_findtext=28=29_to_return_empty_Unicode_string_instead?= =?utf-8?q?_of_empty?= Message-ID: <3Ykdrg4G94zP6D@mail.python.org> http://hg.python.org/cpython/rev/849eb27baf1c changeset: 81484:849eb27baf1c branch: 3.2 parent: 81474:d55a06f7ccbf user: Eli Bendersky date: Sun Jan 13 05:22:05 2013 -0800 summary: Issue #16922: fixed findtext() to return empty Unicode string instead of empty bytes object when there's no text. Patch by Serhiy Storchaka. files: Lib/test/test_xml_etree.py | 2 ++ Modules/_elementtree.c | 2 +- 2 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -352,6 +352,8 @@ 'subtext' >>> ET.ElementTree(elem).findtext("section/tag") 'subtext' + >>> ET.XML('').findtext('empty') + '' >>> summarize_list(elem.findall(".")) ['body'] >>> summarize_list(elem.findall("tag")) diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c --- a/Modules/_elementtree.c +++ b/Modules/_elementtree.c @@ -840,7 +840,7 @@ PyObject* text = element_get_text(item); if (text == Py_None) - return PyBytes_FromString(""); + return PyUnicode_FromString(""); Py_XINCREF(text); return text; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 14:27:01 2013 From: python-checkins at python.org (eli.bendersky) Date: Sun, 13 Jan 2013 14:27:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Merge_3=2E2_and_revert_changes_since_the_changes_in_3=2E3_are_?= =?utf-8?q?a_bit_different?= Message-ID: <3Ykdrj01QWzSGW@mail.python.org> http://hg.python.org/cpython/rev/d50e259f5482 changeset: 81485:d50e259f5482 branch: 3.3 parent: 81482:9202a716673b parent: 81484:849eb27baf1c user: Eli Bendersky date: Sun Jan 13 05:25:17 2013 -0800 summary: Merge 3.2 and revert changes since the changes in 3.3 are a bit different files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 14:27:02 2013 From: python-checkins at python.org (eli.bendersky) Date: Sun, 13 Jan 2013 14:27:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE2OTIy?= =?utf-8?q?=3A_fixed_findtext=28=29_to_return_empty_Unicode_string_instead?= =?utf-8?q?_of_empty?= Message-ID: <3Ykdrk2cqnzSHp@mail.python.org> http://hg.python.org/cpython/rev/6323e5f1ed81 changeset: 81486:6323e5f1ed81 branch: 3.3 user: Eli Bendersky date: Sun Jan 13 05:26:07 2013 -0800 summary: Issue #16922: fixed findtext() to return empty Unicode string instead of empty bytes object when there's no text. Patch by Serhiy Storchaka. files: Lib/test/test_xml_etree.py | 3 +++ Modules/_elementtree.c | 2 +- 2 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -1773,6 +1773,9 @@ self.assertIsNone(e.findtext('tog')) self.assertEqual(e.findtext('tog', 'default'), 'default') + # Issue #16922 + self.assertEqual(ET.XML('').findtext('empty'), '') + def test_findall(self): e = ET.XML(SAMPLE_XML) e[2] = ET.XML(SAMPLE_SECTION) diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c --- a/Modules/_elementtree.c +++ b/Modules/_elementtree.c @@ -1129,7 +1129,7 @@ PyObject* text = element_get_text(item); if (text == Py_None) - return PyBytes_FromString(""); + return PyUnicode_New(0, 0); Py_XINCREF(text); return text; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 14:27:03 2013 From: python-checkins at python.org (eli.bendersky) Date: Sun, 13 Jan 2013 14:27:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2316922=3A_fixed_findtext=28=29_to_return_empty_U?= =?utf-8?q?nicode_string_instead_of_empty?= Message-ID: <3Ykdrl5NjJzSHl@mail.python.org> http://hg.python.org/cpython/rev/c38423931724 changeset: 81487:c38423931724 parent: 81483:3e0739b30370 parent: 81486:6323e5f1ed81 user: Eli Bendersky date: Sun Jan 13 05:26:31 2013 -0800 summary: Issue #16922: fixed findtext() to return empty Unicode string instead of empty bytes object when there's no text. Patch by Serhiy Storchaka. files: Lib/test/test_xml_etree.py | 3 +++ Modules/_elementtree.c | 2 +- 2 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -1773,6 +1773,9 @@ self.assertIsNone(e.findtext('tog')) self.assertEqual(e.findtext('tog', 'default'), 'default') + # Issue #16922 + self.assertEqual(ET.XML('').findtext('empty'), '') + def test_findall(self): e = ET.XML(SAMPLE_XML) e[2] = ET.XML(SAMPLE_SECTION) diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c --- a/Modules/_elementtree.c +++ b/Modules/_elementtree.c @@ -1129,7 +1129,7 @@ PyObject* text = element_get_text(item); if (text == Py_None) - return PyBytes_FromString(""); + return PyUnicode_New(0, 0); Py_XINCREF(text); return text; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 14:57:46 2013 From: python-checkins at python.org (victor.stinner) Date: Sun, 13 Jan 2013 14:57:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_433=3A?= Message-ID: <3YkfXB3dP9zSHM@mail.python.org> http://hg.python.org/peps/rev/5e580a0eebcd changeset: 4674:5e580a0eebcd user: Victor Stinner date: Sun Jan 13 14:53:59 2013 +0100 summary: PEP 433: * argument=>parameter * "always set" => "set by default" * unset => clear * add more example of inherance issues * add more examples of security issues files: pep-0433.txt | 100 +++++++++++++++++++++++++------------- 1 files changed, 66 insertions(+), 34 deletions(-) diff --git a/pep-0433.txt b/pep-0433.txt --- a/pep-0433.txt +++ b/pep-0433.txt @@ -1,5 +1,5 @@ PEP: 433 -Title: Add cloexec argument to functions creating file descriptors +Title: Add cloexec parameter to functions creating file descriptors Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner @@ -13,18 +13,21 @@ Abstract ======== -This PEP proposes to add a new optional argument ``cloexec`` on +This PEP proposes to add a new optional parameter ``cloexec`` on functions creating file descriptors in the Python standard library. If -the argument is ``True``, the close-on-exec flag will be set on the +the parameter is ``True``, the close-on-exec flag will be set on the new file descriptor. Rationale ========= +XXX recap briefly what the close-on-exec flag does + On UNIX, subprocess closes file descriptors greater than 2 by default since Python 3.2 [#subprocess_close]_. All file descriptors created by -the parent process are automatically closed. +the parent process are automatically closed in the child process. + ``xmlrpc.server.SimpleXMLRPCServer`` sets the close-on-exec flag of the listening socket, the parent class ``socketserver.BaseServer`` does not set this flag. @@ -58,6 +61,17 @@ with "too many files" because files are still open in the child process. +See also the following issues: + + * `Issue #2320: Race condition in subprocess using stdin + `_ (2008) + * `Issue #3006: subprocess.Popen causes socket to remain open after + close `_ (2008) + * `Issue #7213: subprocess leaks open file descriptors between Popen + instances causing hangs `_ (2009) + * `Issue #12786: subprocess wait() hangs when stdin is closed + `_ (2011) + Security -------- @@ -67,6 +81,22 @@ take control of the parent process though leaked file descriptors. It is for example a known vulnerability to escape from a chroot. +See also the CERT recommandation: +`FIO42-C. Ensure files are properly closed when they are no longer needed +`_. + + +Example of vulnerabilities: + + + * `OpenSSH Security Advisory: portable-keysign-rand-helper.adv + `_ + (April 2011) + * `CWE-403: Exposure of File Descriptor to Unintended Control Sphere + `_ (2008) + * `Hijacking Apache https by mod_php + `_ (Dec 2003) + Atomicity --------- @@ -100,6 +130,8 @@ close-on-exec flag set if ``fork()`` is used before ``exec()``, but it works correctly if ``exec()`` is called without ``fork()``. +XXX recheck this OpenBSD bug using a C program. XXX + Scope ----- @@ -128,8 +160,8 @@ * ``xmlrpc.server`` * Maybe: ``signal``, ``threading`` -XXX Should ``subprocess.Popen`` set the close-on-exec flag on file XXX -XXX descriptors of the constructor the ``pass_fds`` argument? XXX +XXX Should ``subprocess.Popen`` clear the close-on-exec flag on file +XXX descriptors of the constructor the ``pass_fds`` parameter? .. note:: See `Close file descriptors after fork`_ for a possible solution @@ -139,17 +171,17 @@ Proposal ======== -This PEP proposes to add a new optional argument ``cloexec`` on +This PEP proposes to add a new optional parameter ``cloexec`` on functions creating file descriptors in the Python standard library. If -the argument is ``True``, the close-on-exec flag will be set on the +the parameter is ``True``, the close-on-exec flag will be set on the new file descriptor. Add a new function: - * ``os.set_cloexec(fd: int, cloexec: bool)``: set or unset the + * ``os.set_cloexec(fd: int, cloexec: bool)``: set or clear the close-on-exec flag of a file descriptor -Add a new optional ``cloexec`` argument to: +Add a new optional ``cloexec`` parameter to: * ``open()``: ``os.fdopen()`` is indirectly modified * ``os.dup()``, ``os.dup2()`` @@ -166,7 +198,7 @@ * ``socket.socket.recvmsg()``: use ``MSG_CMSG_CLOEXEC``, or ``os.set_cloexec()`` -The default value of the ``cloexec`` argument is ``False`` to keep the +The default value of the ``cloexec`` parameter is ``False`` to keep the backward compatibility. The close-on-exec flag will not be set on file descriptors 0 (stdin), @@ -177,12 +209,12 @@ Drawbacks: * Many functions of the Python standard library creating file - descriptors are cannot be changed by this proposal, because adding - a ``cloexec`` optional argument would be surprising and too many + descriptors cannot be changed by this proposal, because adding + a ``cloexec`` optional parameter would be surprising and too many functions would need it. For example, ``os.urandom()`` uses a temporary file on UNIX, but it calls a function of Windows API on - Windows. Adding a ``cloexec`` argument to ``os.urandom()`` would - not make sense. See `Always set close-on-exec flag`_ for an + Windows. Adding a ``cloexec`` parameter to ``os.urandom()`` would + not make sense. See `Set the close-on-exec flag by default`_ for an incomplete list of functions creating file descriptors. * Checking if a module creates file descriptors is difficult. For example, ``os.urandom()`` creates a file descriptor on UNIX to read @@ -195,17 +227,17 @@ Alternatives ============ -Always set close-on-exec flag ------------------------------ +Set the close-on-exec flag by default +------------------------------------- -Always set close-on-exec flag on new file descriptors created by -Python. This alternative just changes the default value of the new -``cloexec`` argument. +Set the close-on-exec flag by default on new file descriptors created +by Python. This alternative just changes the default value of the new +``cloexec`` parameter. If a file must be inherited by child processes, ``cloexec=False`` -argument can be used. +parameter can be used. -``subprocess.Popen`` constructor has an ``pass_fds`` argument to +``subprocess.Popen`` constructor has an ``pass_fds`` parameter to specify which file descriptors must be inherited. The close-on-exec flag of these file descriptors must be changed with ``os.set_cloexec()``. @@ -254,12 +286,12 @@ Backward compatibility: only a few programs rely on inherance of file descriptors, and they only pass a few file descriptors, usually just one. These programs will fail immediatly with ``EBADF`` error, and it -will be simple to fix them: add ``cloexec=False`` argument or use +will be simple to fix them: add ``cloexec=False`` parameter or use ``os.set_cloexec(fd, False)``. -The ``subprocess`` module will be changed anyway to unset +The ``subprocess`` module will be changed anyway to clear close-on-exec flag on file descriptors listed in the ``pass_fds`` -argument of Popen constructor. So it possible that these programs will +parameter of Popen constructor. So it possible that these programs will not need any fix if they use the ``subprocess`` module. @@ -280,12 +312,12 @@ close-on-exec flag, the state of the flag can be overriden in each function creating a file descriptor -The major change is that the default value of the ``cloexec`` argument +The major change is that the default value of the ``cloexec`` parameter is ``sys.getdefaultcloexec()``, instead of ``False``. When ``sys.setdefaultcloexec(True)`` is called to set close-on-exec by -default, we have the same drawbacks than `Always set close-on-exec -flag`_ alternative. +default, we have the same drawbacks than `Set the close-on-exec flag +by default`_ alternative. There are additionnal drawbacks of having two behaviours depending on ``sys.getdefaultcloexec()`` value: @@ -299,7 +331,7 @@ This PEP does not fix issues with applications using ``fork()`` without ``exec()``. Python needs a generic process to register -callbacks which would be called after a fork, see `Add an 'afterfork' +callbacks which would be called after a fork, see `Add an 'atfork' module`_. Such registry could be used to close file descriptors just after a ``fork()``. @@ -325,7 +357,7 @@ A new "e" mode would set close-on-exec flag (best-effort). This alternative only solves the problem for ``open()``. -socket.socket() and os.pipe() do not have a ``mode`` argument for +socket.socket() and os.pipe() do not have a ``mode`` parameter for example. Since its version 2.7, the GNU libc supports ``"e"`` flag for @@ -341,7 +373,7 @@ default. Most programs do not rely on inherance of file descriptors. For example, ``subprocess.Popen`` was changed in Python 3.2 to close all file descriptors greater than 2 in the child process by default. -No user complained about this behavior change. +No user complained about this behavior change yet. Network servers using fork may want to pass the client socket to the child process. For example, on UNIX a CGI server pass the socket @@ -513,7 +545,7 @@ The value of the flag can be modified using: ``SetHandleInformation(fd, HANDLE_FLAG_INHERIT, 1)``. -``CreateProcess()`` has an ``bInheritHandles`` argument: if it is +``CreateProcess()`` has an ``bInheritHandles`` parameter: if it is FALSE, the handles are not inherited. It is used by ``subprocess.Popen`` with ``close_fds`` option. @@ -534,7 +566,7 @@ Functions: * ``ioctl(fd, FIOCLEX, 0)`` sets close-on-exec flag - * ``ioctl(fd, FIONCLEX, 0)`` unsets close-on-exec flag + * ``ioctl(fd, FIONCLEX, 0)`` clears close-on-exec flag Availability: Linux, Mac OS X, QNX, NetBSD, OpenBSD, FreeBSD. @@ -597,7 +629,7 @@ `_ * `Support accept4() for atomic setting of flags at socket creation `_ - * `Add an 'afterfork' module + * `Add an 'atfork' module `_ Ruby: -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun Jan 13 15:05:10 2013 From: python-checkins at python.org (eli.bendersky) Date: Sun, 13 Jan 2013 15:05:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Close_=2314377=3A_Add_a_ne?= =?utf-8?q?w_parameter_to_ElementTree=2Ewrite_and_some_module-level?= Message-ID: <3Ykfhk63djzSH8@mail.python.org> http://hg.python.org/cpython/rev/58168d69b496 changeset: 81488:58168d69b496 user: Eli Bendersky date: Sun Jan 13 06:04:43 2013 -0800 summary: Close #14377: Add a new parameter to ElementTree.write and some module-level serialization functions - short_empty_elements. It controls how elements without contents are emitted. Patch by Serhiy Storchaka. Feature initially proposed by Ariel Poliak. files: Doc/library/xml.etree.elementtree.rst | 23 +++++++++- Lib/test/test_xml_etree.py | 12 +++++ Lib/xml/etree/ElementTree.py | 31 +++++++++----- Misc/NEWS | 4 + 4 files changed, 56 insertions(+), 14 deletions(-) diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -428,29 +428,39 @@ arguments. Returns an element instance. -.. function:: tostring(element, encoding="us-ascii", method="xml") +.. function:: tostring(element, encoding="us-ascii", method="xml", *, \ + short_empty_elements=True) Generates a string representation of an XML element, including all subelements. *element* is an :class:`Element` instance. *encoding* [1]_ is the output encoding (default is US-ASCII). Use ``encoding="unicode"`` to generate a Unicode string (otherwise, a bytestring is generated). *method* is either ``"xml"``, ``"html"`` or ``"text"`` (default is ``"xml"``). + *short_empty_elements* has the same meaning as in :meth:`ElementTree.write`. Returns an (optionally) encoded string containing the XML data. + .. versionadded:: 3.4 + The *short_empty_elements* parameter. -.. function:: tostringlist(element, encoding="us-ascii", method="xml") + +.. function:: tostringlist(element, encoding="us-ascii", method="xml", *, \ + short_empty_elements=True) Generates a string representation of an XML element, including all subelements. *element* is an :class:`Element` instance. *encoding* [1]_ is the output encoding (default is US-ASCII). Use ``encoding="unicode"`` to generate a Unicode string (otherwise, a bytestring is generated). *method* is either ``"xml"``, ``"html"`` or ``"text"`` (default is ``"xml"``). + *short_empty_elements* has the same meaning as in :meth:`ElementTree.write`. Returns a list of (optionally) encoded strings containing the XML data. It does not guarantee any specific sequence, except that ``"".join(tostringlist(element)) == tostring(element)``. .. versionadded:: 3.2 + .. versionadded:: 3.4 + The *short_empty_elements* parameter. + .. function:: XML(text, parser=None) @@ -742,7 +752,7 @@ .. method:: write(file, encoding="us-ascii", xml_declaration=None, \ - method="xml") + method="xml", *, short_empty_elements=True) Writes the element tree to a file, as XML. *file* is a file name, or a :term:`file object` opened for writing. *encoding* [1]_ is the output @@ -752,6 +762,10 @@ for only if not US-ASCII or UTF-8 or Unicode (default is ``None``). *method* is either ``"xml"``, ``"html"`` or ``"text"`` (default is ``"xml"``). + The keyword-only *short_empty_elements* parameter controls the formatting + of elements that contain no content. If *True* (the default), they are + emitted as a single self-closed tag, otherwise they are emitted as a pair + of start/end tags. The output is either a string (:class:`str`) or binary (:class:`bytes`). This is controlled by the *encoding* argument. If *encoding* is @@ -760,6 +774,9 @@ :term:`file object`; make sure you do not try to write a string to a binary stream and vice versa. + .. versionadded:: 3.4 + The *short_empty_elements* parameter. + This is the XML file that is going to be manipulated:: diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -2380,6 +2380,18 @@ ET.tostring(root, 'utf-16'), b''.join(ET.tostringlist(root, 'utf-16'))) + def test_short_empty_elements(self): + root = ET.fromstring('abc') + self.assertEqual( + ET.tostring(root, 'unicode'), + 'abc') + self.assertEqual( + ET.tostring(root, 'unicode', short_empty_elements=True), + 'abc') + self.assertEqual( + ET.tostring(root, 'unicode', short_empty_elements=False), + 'abc') + class ParseErrorTest(unittest.TestCase): def test_subclass(self): diff --git a/Lib/xml/etree/ElementTree.py b/Lib/xml/etree/ElementTree.py --- a/Lib/xml/etree/ElementTree.py +++ b/Lib/xml/etree/ElementTree.py @@ -797,7 +797,8 @@ encoding=None, xml_declaration=None, default_namespace=None, - method=None): + method=None, *, + short_empty_elements=True): if not method: method = "xml" elif method not in _serialize: @@ -825,7 +826,8 @@ else: qnames, namespaces = _namespaces(self._root, default_namespace) serialize = _serialize[method] - serialize(write, self._root, qnames, namespaces) + serialize(write, self._root, qnames, namespaces, + short_empty_elements=short_empty_elements) def write_c14n(self, file): # lxml.etree compatibility. use output method instead @@ -947,7 +949,8 @@ add_qname(text.text) return qnames, namespaces -def _serialize_xml(write, elem, qnames, namespaces): +def _serialize_xml(write, elem, qnames, namespaces, + short_empty_elements, **kwargs): tag = elem.tag text = elem.text if tag is Comment: @@ -960,7 +963,8 @@ if text: write(_escape_cdata(text)) for e in elem: - _serialize_xml(write, e, qnames, None) + _serialize_xml(write, e, qnames, None, + short_empty_elements=short_empty_elements) else: write("<" + tag) items = list(elem.items()) @@ -982,12 +986,13 @@ else: v = _escape_attrib(v) write(" %s=\"%s\"" % (qnames[k], v)) - if text or len(elem): + if text or len(elem) or not short_empty_elements: write(">") if text: write(_escape_cdata(text)) for e in elem: - _serialize_xml(write, e, qnames, None) + _serialize_xml(write, e, qnames, None, + short_empty_elements=short_empty_elements) write("") else: write(" />") @@ -1002,7 +1007,7 @@ except NameError: pass -def _serialize_html(write, elem, qnames, namespaces): +def _serialize_html(write, elem, qnames, namespaces, **kwargs): tag = elem.tag text = elem.text if tag is Comment: @@ -1166,9 +1171,11 @@ # @return An (optionally) encoded string containing the XML data. # @defreturn string -def tostring(element, encoding=None, method=None): +def tostring(element, encoding=None, method=None, *, + short_empty_elements=True): stream = io.StringIO() if encoding == 'unicode' else io.BytesIO() - ElementTree(element).write(stream, encoding, method=method) + ElementTree(element).write(stream, encoding, method=method, + short_empty_elements=short_empty_elements) return stream.getvalue() ## @@ -1202,10 +1209,12 @@ def tell(self): return len(self.lst) -def tostringlist(element, encoding=None, method=None): +def tostringlist(element, encoding=None, method=None, *, + short_empty_elements=True): lst = [] stream = _ListDataStream(lst) - ElementTree(element).write(stream, encoding, method=method) + ElementTree(element).write(stream, encoding, method=method, + short_empty_elements=short_empty_elements) return lst ## diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -559,6 +559,10 @@ - Issue #16123: IDLE - deprecate running without a subprocess. Patch by Roger Serwy. +- Issue #14377: ElementTree.write and some of the module-level functions have + a new parameter - *short_empty_elements*. It controls how elements with no + contents are emitted. + - Issue #16089: Allow ElementTree.TreeBuilder to work again with a non-Element element_factory (fixes a regression in SimpleTAL). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 15:05:59 2013 From: python-checkins at python.org (eli.bendersky) Date: Sun, 13 Jan 2013 15:05:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_Ariel_to_Misc/ACKS?= Message-ID: <3Ykfjg2NqkzSG0@mail.python.org> http://hg.python.org/cpython/rev/3f202d4954d6 changeset: 81489:3f202d4954d6 user: Eli Bendersky date: Sun Jan 13 06:05:34 2013 -0800 summary: Add Ariel to Misc/ACKS files: Misc/ACKS | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -944,6 +944,7 @@ Jean-Fran?ois Pi?ronne Oleg Plakhotnyuk Remi Pointel +Ariel Poliak Guilherme Polo Michael Pomraning Iustin Pop -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 15:28:17 2013 From: python-checkins at python.org (eli.bendersky) Date: Sun, 13 Jan 2013 15:28:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Cleanup_the_docs_ElementTr?= =?utf-8?q?ee_a_bit_and_describe_the_default=5Fnamespace?= Message-ID: <3YkgCP49RlzS4W@mail.python.org> http://hg.python.org/cpython/rev/f0e80c7404a5 changeset: 81490:f0e80c7404a5 user: Eli Bendersky date: Sun Jan 13 06:27:51 2013 -0800 summary: Cleanup the docs ElementTree a bit and describe the default_namespace parameter. In the code, replace the old outdated Doxygen-ish comment above ElementTree.write by a proper docstring. files: Doc/library/xml.etree.elementtree.rst | 12 +++-- Lib/xml/etree/ElementTree.py | 28 +++++++------- 2 files changed, 21 insertions(+), 19 deletions(-) diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -428,7 +428,7 @@ arguments. Returns an element instance. -.. function:: tostring(element, encoding="us-ascii", method="xml", *, \ +.. function:: tostring(element, encoding="us-ascii", method="xml", *,\ short_empty_elements=True) Generates a string representation of an XML element, including all @@ -443,7 +443,7 @@ The *short_empty_elements* parameter. -.. function:: tostringlist(element, encoding="us-ascii", method="xml", *, \ +.. function:: tostringlist(element, encoding="us-ascii", method="xml", *,\ short_empty_elements=True) Generates a string representation of an XML element, including all @@ -751,8 +751,9 @@ section root element. - .. method:: write(file, encoding="us-ascii", xml_declaration=None, \ - method="xml", *, short_empty_elements=True) + .. method:: write(file, encoding="us-ascii", xml_declaration=None,\ + default_namespace=None, method="xml", *,\ + short_empty_elements=True) Writes the element tree to a file, as XML. *file* is a file name, or a :term:`file object` opened for writing. *encoding* [1]_ is the output @@ -761,7 +762,8 @@ file. Use ``False`` for never, ``True`` for always, ``None`` for only if not US-ASCII or UTF-8 or Unicode (default is ``None``). *method* is either ``"xml"``, ``"html"`` or ``"text"`` (default is - ``"xml"``). + ``"xml"``). *default_namespace* sets the default XML namespace (for + "xmlns"). The keyword-only *short_empty_elements* parameter controls the formatting of elements that contain no content. If *True* (the default), they are emitted as a single self-closed tag, otherwise they are emitted as a pair diff --git a/Lib/xml/etree/ElementTree.py b/Lib/xml/etree/ElementTree.py --- a/Lib/xml/etree/ElementTree.py +++ b/Lib/xml/etree/ElementTree.py @@ -779,26 +779,26 @@ ) return self._root.iterfind(path, namespaces) - ## - # Writes the element tree to a file, as XML. - # - # @def write(file, **options) - # @param file A file name, or a file object opened for writing. - # @param **options Options, given as keyword arguments. - # @keyparam encoding Optional output encoding (default is US-ASCII). - # Use "unicode" to return a Unicode string. - # @keyparam method Optional output method ("xml", "html", "text" or - # "c14n"; default is "xml"). - # @keyparam xml_declaration Controls if an XML declaration should - # be added to the file. Use False for never, True for always, - # None for only if not US-ASCII or UTF-8 or Unicode. None is default. - def write(self, file_or_filename, encoding=None, xml_declaration=None, default_namespace=None, method=None, *, short_empty_elements=True): + """Write the element tree to a file, as XML. 'file_or_filename' is a + file name or a file object opened for writing. 'encoding' is the + output encoding (default is US-ASCII). 'xml_declaration' controls + if an XML declaration should be added to the output. Use False + for never, True for always, None for only if not US-ASCII or + UTF-8 or Unicode (default is None). 'method' is either "xml" + (default), "html", "text" or "c14n". + 'default_namespace' sets the default XML namespace (for "xmlns"). + The keyword-only 'short_empty_elements' parameter controls the + formatting of elements that contain no content. If True (default), + they are emitted as a single self-closed tag, otherwise they are + emitted as a pair of start/end tags. + + """ if not method: method = "xml" elif method not in _serialize: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 21:31:48 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 13 Jan 2013 21:31:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Describe_the_d?= =?utf-8?q?efault=5Fnamespace_parameter_of_ElemetTree=2Ewrite=2E?= Message-ID: <3YkqGr5qrPzMbW@mail.python.org> http://hg.python.org/cpython/rev/c5451d4a0cdd changeset: 81491:c5451d4a0cdd branch: 2.7 parent: 81473:6e8788353ddf user: Serhiy Storchaka date: Sun Jan 13 21:57:14 2013 +0200 summary: Describe the default_namespace parameter of ElemetTree.write. files: Doc/library/xml.etree.elementtree.rst | 6 ++++-- Lib/xml/etree/ElementTree.py | 5 +++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -767,13 +767,15 @@ root element. - .. method:: write(file, encoding="us-ascii", xml_declaration=None, method="xml") + .. method:: write(file, encoding="us-ascii", xml_declaration=None, \ + default_namespace=None, method="xml") Writes the element tree to a file, as XML. *file* is a file name, or a file object opened for writing. *encoding* [1]_ is the output encoding (default is US-ASCII). *xml_declaration* controls if an XML declaration should be added to the file. Use False for never, True for always, None - for only if not US-ASCII or UTF-8 (default is None). *method* is either + for only if not US-ASCII or UTF-8 (default is None). *default_namespace* + sets the default XML namespace (for "xmlns"). *method* is either ``"xml"``, ``"html"`` or ``"text"`` (default is ``"xml"``). Returns an encoded string. diff --git a/Lib/xml/etree/ElementTree.py b/Lib/xml/etree/ElementTree.py --- a/Lib/xml/etree/ElementTree.py +++ b/Lib/xml/etree/ElementTree.py @@ -779,11 +779,12 @@ # @param file A file name, or a file object opened for writing. # @param **options Options, given as keyword arguments. # @keyparam encoding Optional output encoding (default is US-ASCII). - # @keyparam method Optional output method ("xml", "html", "text" or - # "c14n"; default is "xml"). # @keyparam xml_declaration Controls if an XML declaration should # be added to the file. Use False for never, True for always, # None for only if not US-ASCII or UTF-8. None is default. + # @keyparam default_namespace Sets the default XML namespace (for "xmlns"). + # @keyparam method Optional output method ("xml", "html", "text" or + # "c14n"; default is "xml"). def write(self, file_or_filename, # keyword arguments -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 21:31:50 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 13 Jan 2013 21:31:50 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E2=29=3A_Describe_the_d?= =?utf-8?q?efault=5Fnamespace_parameter_of_ElemetTree=2Ewrite=2E?= Message-ID: <3YkqGt1TkmzMbW@mail.python.org> http://hg.python.org/cpython/rev/df8609f1854d changeset: 81492:df8609f1854d branch: 3.2 parent: 81484:849eb27baf1c user: Serhiy Storchaka date: Sun Jan 13 21:58:04 2013 +0200 summary: Describe the default_namespace parameter of ElemetTree.write. files: Doc/library/xml.etree.elementtree.rst | 16 ++++++++------ Lib/xml/etree/ElementTree.py | 5 ++- 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -468,16 +468,18 @@ root element. - .. method:: write(file, encoding="us-ascii", xml_declaration=None, method="xml") + .. method:: write(file, encoding="us-ascii", xml_declaration=None, \ + default_namespace=None, method="xml") Writes the element tree to a file, as XML. *file* is a file name, or a - :term:`file object` opened for writing. *encoding* [1]_ is the output encoding - (default is US-ASCII). Use ``encoding="unicode"`` to write a Unicode string. - *xml_declaration* controls if an XML declaration + :term:`file object` opened for writing. *encoding* [1]_ is the output + encoding (default is US-ASCII). Use ``encoding="unicode"`` to write a + Unicode string. *xml_declaration* controls if an XML declaration should be added to the file. Use False for never, True for always, None - for only if not US-ASCII or UTF-8 or Unicode (default is None). *method* is - either ``"xml"``, ``"html"`` or ``"text"`` (default is ``"xml"``). - Returns an (optionally) encoded string. + for only if not US-ASCII or UTF-8 or Unicode (default is None). + *default_namespace* sets the default XML namespace (for "xmlns"). + *method* is either ``"xml"``, ``"html"`` or ``"text"`` (default is + ``"xml"``). Returns an (optionally) encoded string. This is the XML file that is going to be manipulated:: diff --git a/Lib/xml/etree/ElementTree.py b/Lib/xml/etree/ElementTree.py --- a/Lib/xml/etree/ElementTree.py +++ b/Lib/xml/etree/ElementTree.py @@ -802,11 +802,12 @@ # @param **options Options, given as keyword arguments. # @keyparam encoding Optional output encoding (default is US-ASCII). # Use "unicode" to return a Unicode string. - # @keyparam method Optional output method ("xml", "html", "text" or - # "c14n"; default is "xml"). # @keyparam xml_declaration Controls if an XML declaration should # be added to the file. Use False for never, True for always, # None for only if not US-ASCII or UTF-8 or Unicode. None is default. + # @keyparam default_namespace Sets the default XML namespace (for "xmlns"). + # @keyparam method Optional output method ("xml", "html", "text" or + # "c14n"; default is "xml"). def write(self, file_or_filename, # keyword arguments -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 21:31:51 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 13 Jan 2013 21:31:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Describe_the_default=5Fnamespace_parameter_of_ElemetTree=2Ewri?= =?utf-8?b?dGUu?= Message-ID: <3YkqGv4WQ3zRws@mail.python.org> http://hg.python.org/cpython/rev/159276962d16 changeset: 81493:159276962d16 branch: 3.3 parent: 81486:6323e5f1ed81 parent: 81492:df8609f1854d user: Serhiy Storchaka date: Sun Jan 13 22:04:43 2013 +0200 summary: Describe the default_namespace parameter of ElemetTree.write. files: Doc/library/xml.etree.elementtree.rst | 3 ++- Lib/xml/etree/ElementTree.py | 5 +++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -742,7 +742,7 @@ .. method:: write(file, encoding="us-ascii", xml_declaration=None, \ - method="xml") + default_namespace=None, method="xml") Writes the element tree to a file, as XML. *file* is a file name, or a :term:`file object` opened for writing. *encoding* [1]_ is the output @@ -750,6 +750,7 @@ *xml_declaration* controls if an XML declaration should be added to the file. Use ``False`` for never, ``True`` for always, ``None`` for only if not US-ASCII or UTF-8 or Unicode (default is ``None``). + *default_namespace* sets the default XML namespace (for "xmlns"). *method* is either ``"xml"``, ``"html"`` or ``"text"`` (default is ``"xml"``). diff --git a/Lib/xml/etree/ElementTree.py b/Lib/xml/etree/ElementTree.py --- a/Lib/xml/etree/ElementTree.py +++ b/Lib/xml/etree/ElementTree.py @@ -789,11 +789,12 @@ # @param **options Options, given as keyword arguments. # @keyparam encoding Optional output encoding (default is US-ASCII). # Use "unicode" to return a Unicode string. - # @keyparam method Optional output method ("xml", "html", "text" or - # "c14n"; default is "xml"). # @keyparam xml_declaration Controls if an XML declaration should # be added to the file. Use False for never, True for always, # None for only if not US-ASCII or UTF-8 or Unicode. None is default. + # @keyparam default_namespace Sets the default XML namespace (for "xmlns"). + # @keyparam method Optional output method ("xml", "html", "text" or + # "c14n"; default is "xml"). def write(self, file_or_filename, encoding=None, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 21:31:53 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 13 Jan 2013 21:31:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Cleanup_the_docs_ElementTree_a_bit=2E?= Message-ID: <3YkqGx1WCxzSHZ@mail.python.org> http://hg.python.org/cpython/rev/50606131a987 changeset: 81494:50606131a987 parent: 81490:f0e80c7404a5 parent: 81493:159276962d16 user: Serhiy Storchaka date: Sun Jan 13 22:24:27 2013 +0200 summary: Cleanup the docs ElementTree a bit. files: Doc/library/xml.etree.elementtree.rst | 12 +++++----- Lib/xml/etree/ElementTree.py | 16 +++++++------- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -428,7 +428,7 @@ arguments. Returns an element instance. -.. function:: tostring(element, encoding="us-ascii", method="xml", *,\ +.. function:: tostring(element, encoding="us-ascii", method="xml", *, \ short_empty_elements=True) Generates a string representation of an XML element, including all @@ -443,7 +443,7 @@ The *short_empty_elements* parameter. -.. function:: tostringlist(element, encoding="us-ascii", method="xml", *,\ +.. function:: tostringlist(element, encoding="us-ascii", method="xml", *, \ short_empty_elements=True) Generates a string representation of an XML element, including all @@ -751,8 +751,8 @@ section root element. - .. method:: write(file, encoding="us-ascii", xml_declaration=None,\ - default_namespace=None, method="xml", *,\ + .. method:: write(file, encoding="us-ascii", xml_declaration=None, \ + default_namespace=None, method="xml", *, \ short_empty_elements=True) Writes the element tree to a file, as XML. *file* is a file name, or a @@ -761,9 +761,9 @@ *xml_declaration* controls if an XML declaration should be added to the file. Use ``False`` for never, ``True`` for always, ``None`` for only if not US-ASCII or UTF-8 or Unicode (default is ``None``). + *default_namespace* sets the default XML namespace (for "xmlns"). *method* is either ``"xml"``, ``"html"`` or ``"text"`` (default is - ``"xml"``). *default_namespace* sets the default XML namespace (for - "xmlns"). + ``"xml"``). The keyword-only *short_empty_elements* parameter controls the formatting of elements that contain no content. If *True* (the default), they are emitted as a single self-closed tag, otherwise they are emitted as a pair diff --git a/Lib/xml/etree/ElementTree.py b/Lib/xml/etree/ElementTree.py --- a/Lib/xml/etree/ElementTree.py +++ b/Lib/xml/etree/ElementTree.py @@ -785,16 +785,16 @@ default_namespace=None, method=None, *, short_empty_elements=True): - """Write the element tree to a file, as XML. 'file_or_filename' is a - file name or a file object opened for writing. 'encoding' is the - output encoding (default is US-ASCII). 'xml_declaration' controls - if an XML declaration should be added to the output. Use False - for never, True for always, None for only if not US-ASCII or - UTF-8 or Unicode (default is None). 'method' is either "xml" - (default), "html", "text" or "c14n". + """Write the element tree to a file, as XML. 'file_or_filename' is a + file name or a file object opened for writing. + 'encoding' is the output encoding (default is US-ASCII). + 'xml_declaration' controls if an XML declaration should be added + to the output. Use False for never, True for always, None for only + if not US-ASCII or UTF-8 or Unicode (default is None). 'default_namespace' sets the default XML namespace (for "xmlns"). + 'method' is either "xml" (default), "html", "text" or "c14n". The keyword-only 'short_empty_elements' parameter controls the - formatting of elements that contain no content. If True (default), + formatting of elements that contain no content. If True (default), they are emitted as a single self-closed tag, otherwise they are emitted as a pair of start/end tags. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 23:54:28 2013 From: python-checkins at python.org (richard.oudkerk) Date: Sun, 13 Jan 2013 23:54:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE2OTU1?= =?utf-8?q?=3A_Fix_the_poll=28=29_method_for_multiprocessing=27s_socket?= Message-ID: <3YktRS2wgtzSH7@mail.python.org> http://hg.python.org/cpython/rev/e1c81ab5ad97 changeset: 81495:e1c81ab5ad97 branch: 3.3 parent: 81493:159276962d16 user: Richard Oudkerk date: Sun Jan 13 22:46:48 2013 +0000 summary: Issue #16955: Fix the poll() method for multiprocessing's socket connections on Windows. files: Lib/multiprocessing/connection.py | 2 +- Lib/test/test_multiprocessing.py | 11 +++++++++++ Misc/NEWS | 3 +++ 3 files changed, 15 insertions(+), 1 deletions(-) diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -405,7 +405,7 @@ return self._recv(size) def _poll(self, timeout): - r = wait([self._handle], timeout) + r = wait([self], timeout) return bool(r) diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -2383,6 +2383,17 @@ p.join() l.close() + def test_issue16955(self): + for fam in self.connection.families: + l = self.connection.Listener(family=fam) + c = self.connection.Client(l.address) + a = l.accept() + a.send_bytes(b"hello") + self.assertTrue(c.poll(1)) + a.close() + c.close() + l.close() + class _TestPoll(unittest.TestCase): ALLOWED_TYPES = ('processes', 'threads') diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -147,6 +147,9 @@ Library ------- +- Issue #16955: Fix the poll() method for multiprocessing's socket + connections on Windows. + - SSLContext.load_dh_params() now properly closes the input file. - Issue #16829: IDLE printing no longer fails if there are spaces or other -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Jan 13 23:54:29 2013 From: python-checkins at python.org (richard.oudkerk) Date: Sun, 13 Jan 2013 23:54:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2316955=3A_Fix_the_poll=28=29_method_for_multipro?= =?utf-8?q?cessing=27s_socket?= Message-ID: <3YktRT65grzSHH@mail.python.org> http://hg.python.org/cpython/rev/d904a741afde changeset: 81496:d904a741afde parent: 81494:50606131a987 parent: 81495:e1c81ab5ad97 user: Richard Oudkerk date: Sun Jan 13 22:52:13 2013 +0000 summary: Issue #16955: Fix the poll() method for multiprocessing's socket connections on Windows. files: Lib/multiprocessing/connection.py | 2 +- Lib/test/test_multiprocessing.py | 11 +++++++++++ Misc/NEWS | 3 +++ 3 files changed, 15 insertions(+), 1 deletions(-) diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -405,7 +405,7 @@ return self._recv(size) def _poll(self, timeout): - r = wait([self._handle], timeout) + r = wait([self], timeout) return bool(r) diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -2358,6 +2358,17 @@ p.join() l.close() + def test_issue16955(self): + for fam in self.connection.families: + l = self.connection.Listener(family=fam) + c = self.connection.Client(l.address) + a = l.accept() + a.send_bytes(b"hello") + self.assertTrue(c.poll(1)) + a.close() + c.close() + l.close() + class _TestPoll(unittest.TestCase): ALLOWED_TYPES = ('processes', 'threads') diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -217,6 +217,9 @@ Library ------- +- Issue #16955: Fix the poll() method for multiprocessing's socket + connections on Windows. + - SSLContext.load_dh_params() now properly closes the input file. - Issue #16829: IDLE printing no longer fails if there are spaces or other -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Jan 14 02:24:19 2013 From: python-checkins at python.org (giampaolo.rodola) Date: Mon, 14 Jan 2013 02:24:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_fix_for_previous_commit_re?= =?utf-8?q?lated_to_issue_10527_which_didn=27t_have_the_intended?= Message-ID: <3YkxmM0JwNzSJb@mail.python.org> http://hg.python.org/cpython/rev/831f49cc00fc changeset: 81497:831f49cc00fc user: Giampaolo Rodola' date: Mon Jan 14 02:24:05 2013 +0100 summary: fix for previous commit related to issue 10527 which didn't have the intended effect as per http://bugs.python.org/issue10527#msg179895 files: Lib/multiprocessing/connection.py | 48 +++++++++--------- Lib/test/test_multiprocessing.py | 2 +- 2 files changed, 26 insertions(+), 24 deletions(-) diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -509,27 +509,6 @@ return c1, c2 else: - if hasattr(select, 'poll'): - def _poll(fds, timeout): - if timeout is not None: - timeout = int(timeout) * 1000 # timeout is in milliseconds - fd_map = {} - pollster = select.poll() - for fd in fds: - pollster.register(fd, select.POLLIN) - if hasattr(fd, 'fileno'): - fd_map[fd.fileno()] = fd - else: - fd_map[fd] = fd - ls = [] - for fd, event in pollster.poll(timeout): - if event & select.POLLNVAL: - raise ValueError('invalid file descriptor %i' % fd) - ls.append(fd_map[fd]) - return ls - else: - def _poll(fds, timeout): - return select.select(fds, [], [], timeout)[0] def Pipe(duplex=True): ''' @@ -883,6 +862,29 @@ else: + if hasattr(select, 'poll'): + def _poll(fds, timeout): + if timeout is not None: + timeout = int(timeout) * 1000 # timeout is in milliseconds + fd_map = {} + pollster = select.poll() + for fd in fds: + pollster.register(fd, select.POLLIN) + if hasattr(fd, 'fileno'): + fd_map[fd.fileno()] = fd + else: + fd_map[fd] = fd + ls = [] + for fd, event in pollster.poll(timeout): + if event & select.POLLNVAL: + raise ValueError('invalid file descriptor %i' % fd) + ls.append(fd_map[fd]) + return ls + else: + def _poll(fds, timeout): + return select.select(fds, [], [], timeout)[0] + + def wait(object_list, timeout=None): ''' Wait till an object in object_list is ready/readable. @@ -891,12 +893,12 @@ ''' if timeout is not None: if timeout <= 0: - return select.select(object_list, [], [], 0)[0] + return _poll(object_list, 0) else: deadline = time.time() + timeout while True: try: - return select.select(object_list, [], [], timeout)[0] + return _poll(object_list, timeout) except OSError as e: if e.errno != errno.EINTR: raise diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -3263,7 +3263,7 @@ from multiprocessing.connection import wait expected = 3 - sorted_ = lambda l: sorted(l, key=lambda x: isinstance(x, int)) + sorted_ = lambda l: sorted(l, key=lambda x: id(x)) sem = multiprocessing.Semaphore(0) a, b = multiprocessing.Pipe() p = multiprocessing.Process(target=self.signal_and_sleep, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Jan 14 02:24:35 2013 From: python-checkins at python.org (giampaolo.rodola) Date: Mon, 14 Jan 2013 02:24:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_fix_for_previo?= =?utf-8?q?us_commit_related_to_issue_10527_which_didn=27t_have_the_intend?= =?utf-8?q?ed?= Message-ID: <3Ykxmg1NdXzSKN@mail.python.org> http://hg.python.org/cpython/rev/88fadc0d7b20 changeset: 81498:88fadc0d7b20 branch: 3.3 parent: 81495:e1c81ab5ad97 user: Giampaolo Rodola' date: Mon Jan 14 02:24:25 2013 +0100 summary: fix for previous commit related to issue 10527 which didn't have the intended effect as per http://bugs.python.org/issue10527#msg179895 files: Lib/multiprocessing/connection.py | 48 +++++++++--------- Lib/test/test_multiprocessing.py | 2 +- 2 files changed, 26 insertions(+), 24 deletions(-) diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -509,27 +509,6 @@ return c1, c2 else: - if hasattr(select, 'poll'): - def _poll(fds, timeout): - if timeout is not None: - timeout = int(timeout) * 1000 # timeout is in milliseconds - fd_map = {} - pollster = select.poll() - for fd in fds: - pollster.register(fd, select.POLLIN) - if hasattr(fd, 'fileno'): - fd_map[fd.fileno()] = fd - else: - fd_map[fd] = fd - ls = [] - for fd, event in pollster.poll(timeout): - if event & select.POLLNVAL: - raise ValueError('invalid file descriptor %i' % fd) - ls.append(fd_map[fd]) - return ls - else: - def _poll(fds, timeout): - return select.select(fds, [], [], timeout)[0] def Pipe(duplex=True): ''' @@ -883,6 +862,29 @@ else: + if hasattr(select, 'poll'): + def _poll(fds, timeout): + if timeout is not None: + timeout = int(timeout) * 1000 # timeout is in milliseconds + fd_map = {} + pollster = select.poll() + for fd in fds: + pollster.register(fd, select.POLLIN) + if hasattr(fd, 'fileno'): + fd_map[fd.fileno()] = fd + else: + fd_map[fd] = fd + ls = [] + for fd, event in pollster.poll(timeout): + if event & select.POLLNVAL: + raise ValueError('invalid file descriptor %i' % fd) + ls.append(fd_map[fd]) + return ls + else: + def _poll(fds, timeout): + return select.select(fds, [], [], timeout)[0] + + def wait(object_list, timeout=None): ''' Wait till an object in object_list is ready/readable. @@ -891,12 +893,12 @@ ''' if timeout is not None: if timeout <= 0: - return select.select(object_list, [], [], 0)[0] + return _poll(object_list, 0) else: deadline = time.time() + timeout while True: try: - return select.select(object_list, [], [], timeout)[0] + return _poll(object_list, timeout) except OSError as e: if e.errno != errno.EINTR: raise diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -3244,7 +3244,7 @@ from multiprocessing.connection import wait expected = 3 - sorted_ = lambda l: sorted(l, key=lambda x: isinstance(x, int)) + sorted_ = lambda l: sorted(l, key=lambda x: id(x)) sem = multiprocessing.Semaphore(0) a, b = multiprocessing.Pipe() p = multiprocessing.Process(target=self.signal_and_sleep, -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Mon Jan 14 05:59:31 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 14 Jan 2013 05:59:31 +0100 Subject: [Python-checkins] Daily reference leaks (831f49cc00fc): sum=0 Message-ID: results for 831f49cc00fc on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogrkl4w6', '-x'] From python-checkins at python.org Mon Jan 14 18:27:49 2013 From: python-checkins at python.org (andrew.svetlov) Date: Mon, 14 Jan 2013 18:27:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=235066=3A_Update_ID?= =?utf-8?q?LE_docs?= Message-ID: <3YlM854hJfzR26@mail.python.org> http://hg.python.org/cpython/rev/d1ef91025d70 changeset: 81499:d1ef91025d70 parent: 81497:831f49cc00fc user: Andrew Svetlov date: Mon Jan 14 19:27:36 2013 +0200 summary: Issue #5066: Update IDLE docs Patch by Todd Rovito files: Doc/library/idle.rst | 343 ++++++++++++++++---- Lib/idlelib/NEWS.txt | 2 + Lib/idlelib/help.txt | 498 +++++++++++++++++------------- Misc/NEWS | 2 + 4 files changed, 558 insertions(+), 287 deletions(-) diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -16,70 +16,82 @@ * coded in 100% pure Python, using the :mod:`tkinter` GUI toolkit -* cross-platform: works on Windows and Unix +* cross-platform: works on Windows, Unix, and Mac OS X -* multi-window text editor with multiple undo, Python colorizing and many other - features, e.g. smart indent and call tips +* multi-window text editor with multiple undo, Python colorizing, + smart indent, call tips, and many other features * Python shell window (a.k.a. interactive interpreter) -* debugger (not complete, but you can set breakpoints, view and step) +* debugger (not complete, but you can set breakpoints, view and step) Menus ----- +IDLE has two window types, the Shell window and the Editor window. It is +possible to have multiple editor windows simultaneously. IDLE's +menus dynamically change based on which window is currently selected. Each menu +documented below indicates which window type it is associated with. Click on +the dotted line at the top of a menu to "tear it off": a separate window +containing the menu is created (for Unix and Windows only). -File menu -^^^^^^^^^ +File menu (Shell and Editor) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ New window - create a new editing window + Create a new editing window Open... - open an existing file + Open an existing file Open module... - open an existing module (searches sys.path) + Open an existing module (searches sys.path) + +Recent Files + Open a list of recent files Class browser - show classes and methods in current file + Show classes and methods in current file Path browser - show sys.path directories, modules, classes and methods + Show sys.path directories, modules, classes and methods .. index:: single: Class browser single: Path browser Save - save current window to the associated file (unsaved windows have a \* before and - after the window title) + Save current window to the associated file (unsaved windows have a + \* before and after the window title) Save As... - save current window to new file, which becomes the associated file + Save current window to new file, which becomes the associated file Save Copy As... - save current window to different file without changing the associated file + Save current window to different file without changing the associated file + +Print Window + Print the current window Close - close current window (asks to save if unsaved) + Close current window (asks to save if unsaved) Exit - close all windows and quit IDLE (asks to save if unsaved) + Close all windows and quit IDLE (asks to save if unsaved) -Edit menu -^^^^^^^^^ +Edit menu (Shell and Editor) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Undo - Undo last change to current window (max 1000 changes) + Undo last change to current window (a maximum of 1000 changes may be undone) Redo Redo last undone change to current window Cut - Copy selection into system-wide clipboard; then delete selection + Copy selection into system-wide clipboard; then delete the selection Copy Copy selection into system-wide clipboard @@ -108,11 +120,30 @@ Go to line Ask for a line number and show that line +Expand word + Expand the word you have typed to match another word in the same buffer; + repeat to get a different expansion + +Show call tip + After an unclosed parenthesis for a function, open a small window with + function parameter hints + +Show surrounding parens + Highlight the surrounding parenthesis + +Show Completions + Open a scroll window allowing selection keywords and attributes. See + Completions below. + + +Format menu (Editor window only) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + Indent region - Shift selected lines right 4 spaces + Shift selected lines right by the indent width (default 4 spaces) Dedent region - Shift selected lines left 4 spaces + Shift selected lines left by the indent width (default 4 spaces) Comment out region Insert ## in front of selected lines @@ -121,67 +152,121 @@ Remove leading # or ## from selected lines Tabify region - Turns *leading* stretches of spaces into tabs + Turns *leading* stretches of spaces into tabs. (Note: We recommend using + 4 space blocks to indent Python code.) Untabify region - Turn *all* tabs into the right number of spaces + Turn *all* tabs into the correct number of spaces -Expand word - Expand the word you have typed to match another word in the same buffer; repeat - to get a different expansion +Toggle tabs + Open a dialog to switch between indenting with spaces and tabs. + +New Indent Width + Open a dialog to change indent width. The accepted default by the Python + community is 4 spaces. Format Paragraph - Reformat the current blank-line-separated paragraph + Reformat the current blank-line-separated paragraph. All lines in the + paragraph will be formatted to less than 80 columns. -Import module - Import or reload the current module - -Run script - Execute the current file in the __main__ namespace +Strip trailing whitespace + Removes any space characters after the end of the last non-space character .. index:: single: Import module single: Run script -Windows menu -^^^^^^^^^^^^ +Run menu (Editor window only) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Zoom Height - toggles the window between normal size (24x80) and maximum height. +Python Shell + Open or wake up the Python Shell window -The rest of this menu lists the names of all open windows; select one to bring -it to the foreground (deiconifying it if necessary). +Check module + Check the syntax of the module currently open in the Editor window. If the + module has not been saved IDLE will prompt the user to save the code. +Run module + Restart the shell to clean the environment, then execute the currently + open module. If the module has not been saved IDLE will prompt the user + to save the code. -Debug menu -^^^^^^^^^^ +Shell menu (Shell window only) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -* in the Python Shell window only +View Last Restart + Scroll the shell window to the last Shell restart + +Restart Shell + Restart the shell to clean the environment + +Debug menu (Shell window only) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Go to file/line Look around the insert point for a filename and line number, open the file, and show the line. Useful to view the source lines referenced in an - exception traceback. + exception traceback. Available in the context menu of the Shell window. -Debugger - Run commands in the shell under the debugger. +Debugger (toggle) + This feature is not complete and considered experimental. Run commands in + the shell under the debugger Stack viewer - Show the stack traceback of the last exception. + Show the stack traceback of the last exception Auto-open Stack Viewer - Open stack viewer on traceback. + Toggle automatically opening the stack viewer on unhandled exception .. index:: single: stack viewer single: debugger +Options menu (Shell and Editor) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Edit context menu -^^^^^^^^^^^^^^^^^ +Configure IDLE + Open a configuration dialog. Fonts, indentation, keybindings, and color + themes may be altered. Startup Preferences may be set, and additional + help sources can be specified. -* Right-click in Edit window (Control-click on OS X) +Code Context (toggle)(Editor Window only) + Open a pane at the top of the edit window which shows the block context + of the section of code which is scrolling off the top of the window. + +Windows menu (Shell and Editor) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Zoom Height + Toggles the window between normal size (40x80 initial setting) and maximum + height. The initial size is in the Configure IDLE dialog under the general + tab. + +The rest of this menu lists the names of all open windows; select one to bring +it to the foreground (deiconifying it if necessary). + +Help menu (Shell and Editor) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +About IDLE + Version, copyright, license, credits + +IDLE Help + Display a help file for IDLE detailing the menu options, basic editing and + navigation, and other tips. + +Python Docs + Access local Python documentation, if installed. Or will start a web browser + and open docs.python.org showing the latest Python documentation. + +Additional help sources may be added here with the Configure IDLE dialog under +the General tab. + +Editor Window context menu +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* Right-click in Editor window (Control-click on OS X) Cut Copy selection into system-wide clipboard; then delete selection @@ -207,8 +292,8 @@ single: breakpoints -Shell context menu -^^^^^^^^^^^^^^^^^^ +Shell Window context menu +^^^^^^^^^^^^^^^^^^^^^^^^^ * Right-click in Python Shell window (Control-click on OS X) @@ -225,19 +310,44 @@ Same as in Debug menu. -Basic editing and navigation ----------------------------- +Editing and navigation +---------------------- * :kbd:`Backspace` deletes to the left; :kbd:`Del` deletes to the right +* :kbd:`C-Backspace` delete word left; :kbd:`C-Del` delete word to the right + * Arrow keys and :kbd:`Page Up`/:kbd:`Page Down` to move around +* :kbd:`C-LeftArrow` and :kbd:`C-RightArrow` moves by words + * :kbd:`Home`/:kbd:`End` go to begin/end of line * :kbd:`C-Home`/:kbd:`C-End` go to begin/end of file -* Some :program:`Emacs` bindings may also work, including :kbd:`C-B`, - :kbd:`C-P`, :kbd:`C-A`, :kbd:`C-E`, :kbd:`C-D`, :kbd:`C-L` +* Some useful Emacs bindings are inherited from Tcl/Tk: + + * :kbd:`C-a` beginning of line + + * :kbd:`C-e` end of line + + * :kbd:`C-k` kill line (but doesn't put it in clipboard) + + * :kbd:`C-l` center window around the insertion point + + * :kbd:`C-b` go backwards one character without deleting (usually you can + also use the cursor key for this) + + * :kbd:`C-f` go forward one character without deleting (usually you can + also use the cursor key for this) + + * :kbd:`C-p` go up one line (usually you can also use the cursor key for + this) + + * :kbd:`C-d` delete next character + +Standard keybindings (like :kbd:`C-c` to copy and :kbd:`C-v` to paste) +may work. Keybindings are selected in the Configure IDLE dialog. Automatic indentation @@ -246,27 +356,76 @@ After a block-opening statement, the next line is indented by 4 spaces (in the Python Shell window by one tab). After certain keywords (break, return etc.) the next line is dedented. In leading indentation, :kbd:`Backspace` deletes up -to 4 spaces if they are there. :kbd:`Tab` inserts 1-4 spaces (in the Python -Shell window one tab). See also the indent/dedent region commands in the edit -menu. +to 4 spaces if they are there. :kbd:`Tab` inserts spaces (in the Python +Shell window one tab), number depends on Indent width. Currently tabs +are restricted to four spaces due to Tcl/Tk limitations. +See also the indent/dedent region commands in the edit menu. + +Completions +^^^^^^^^^^^ + +Completions are supplied for functions, classes, and attributes of classes, +both built-in and user-defined. Completions are also provided for +filenames. + +The AutoCompleteWindow (ACW) will open after a predefined delay (default is +two seconds) after a '.' or (in a string) an os.sep is typed. If after one +of those characters (plus zero or more other characters) a tab is typed +the ACW will open immediately if a possible continuation is found. + +If there is only one possible completion for the characters entered, a +:kbd:`Tab` will supply that completion without opening the ACW. + +'Show Completions' will force open a completions window, by default the +:kbd:`C-space` will open a completions window. In an empty +string, this will contain the files in the current directory. On a +blank line, it will contain the built-in and user-defined functions and +classes in the current name spaces, plus any modules imported. If some +characters have been entered, the ACW will attempt to be more specific. + +If a string of characters is typed, the ACW selection will jump to the +entry most closely matching those characters. Entering a :kbd:`tab` will +cause the longest non-ambiguous match to be entered in the Editor window or +Shell. Two :kbd:`tab` in a row will supply the current ACW selection, as +will return or a double click. Cursor keys, Page Up/Down, mouse selection, +and the scroll wheel all operate on the ACW. + +"Hidden" attributes can be accessed by typing the beginning of hidden +name after a '.', e.g. '_'. This allows access to modules with +``__all__`` set, or to class-private attributes. + +Completions and the 'Expand Word' facility can save a lot of typing! + +Completions are currently limited to those in the namespaces. Names in +an Editor window which are not via ``__main__`` and :data:`sys.modules` will +not be found. Run the module once with your imports to correct this situation. +Note that IDLE itself places quite a few modules in sys.modules, so +much can be found by default, e.g. the re module. + +If you don't like the ACW popping up unbidden, simply make the delay +longer or disable the extension. Or another option is the delay could +be set to zero. Another alternative to preventing ACW popups is to +disable the call tips extension. Python Shell window ^^^^^^^^^^^^^^^^^^^ -* :kbd:`C-C` interrupts executing command +* :kbd:`C-c` interrupts executing command -* :kbd:`C-D` sends end-of-file; closes window if typed at a ``>>>`` prompt +* :kbd:`C-d` sends end-of-file; closes window if typed at a ``>>>`` prompt + (this is :kbd:`C-z` on Windows). -* :kbd:`Alt-p` retrieves previous command matching what you have typed +* :kbd:`Alt-/` (Expand word) is also useful to reduce typing -* :kbd:`Alt-n` retrieves next + Command history -* :kbd:`Return` while on any previous command retrieves that command + * :kbd:`Alt-p` retrieves previous command matching what you have typed. On + OS X use :kbd:`C-p`. -* :kbd:`Alt-/` (Expand word) is also useful here + * :kbd:`Alt-n` retrieves next. On OS X use :kbd:`C-n`. -.. index:: single: indentation + * :kbd:`Return` while on any previous command retrieves that command Syntax colors @@ -308,17 +467,17 @@ Upon startup with the ``-s`` option, IDLE will execute the file referenced by the environment variables :envvar:`IDLESTARTUP` or :envvar:`PYTHONSTARTUP`. -Idle first checks for ``IDLESTARTUP``; if ``IDLESTARTUP`` is present the file -referenced is run. If ``IDLESTARTUP`` is not present, Idle checks for +IDLE first checks for ``IDLESTARTUP``; if ``IDLESTARTUP`` is present the file +referenced is run. If ``IDLESTARTUP`` is not present, IDLE checks for ``PYTHONSTARTUP``. Files referenced by these environment variables are -convenient places to store functions that are used frequently from the Idle +convenient places to store functions that are used frequently from the IDLE shell, or for executing import statements to import common modules. In addition, ``Tk`` also loads a startup file if it is present. Note that the Tk file is loaded unconditionally. This additional file is ``.Idle.py`` and is looked for in the user's home directory. Statements in this file will be executed in the Tk namespace, so this file is not useful for importing functions -to be used from Idle's Python shell. +to be used from IDLE's Python shell. Command line usage @@ -349,3 +508,45 @@ the arguments are still available in ``sys.argv``. +Additional help sources +----------------------- + +IDLE includes a help menu entry called "Python Docs" that will open the +extensive sources of help, including tutorials, available at docs.python.org. +Selected URLs can be added or removed from the help menu at any time using the +Configure IDLE dialog. See the IDLE help option in the help menu of IDLE for +more information. + + +Other preferences +----------------- + +The font preferences, highlighting, keys, and general preferences can be +changed via the Configure IDLE menu option. Be sure to note that +keys can be user defined, IDLE ships with four built in key sets. In +addition a user can create a custom key set in the Configure IDLE dialog +under the keys tab. + +Extensions +---------- + +IDLE contains an extension facility. See the beginning of +config-extensions.def in the idlelib directory for further information. The +default extensions are currently: + +* FormatParagraph + +* AutoExpand + +* ZoomHeight + +* ScriptBinding + +* CallTips + +* ParenMatch + +* AutoComplete + +* CodeContext + diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt --- a/Lib/idlelib/NEWS.txt +++ b/Lib/idlelib/NEWS.txt @@ -1,6 +1,8 @@ What's New in IDLE 3.4.0? ========================= +- Issue #5066: Update IDLE docs. Patch by Todd Rovito. + - Issue #16226: Fix IDLE Path Browser crash. (Patch by Roger Serwy) diff --git a/Lib/idlelib/help.txt b/Lib/idlelib/help.txt --- a/Lib/idlelib/help.txt +++ b/Lib/idlelib/help.txt @@ -1,142 +1,185 @@ [See the end of this file for ** TIPS ** on using IDLE !!] -Click on the dotted line at the top of a menu to "tear it off": a -separate window containing the menu is created. +IDLE is the Python IDE built with the tkinter GUI toolkit. -File Menu: +IDLE has the following features: +-coded in 100% pure Python, using the tkinter GUI toolkit +-cross-platform: works on Windows, Unix, and OS X +-multi-window text editor with multiple undo, Python colorizing, smart indent, +call tips, and many other features +-Python shell window (a.k.a interactive interpreter) +-debugger (not complete, but you can set breakpoints, view and step) - New Window -- Create a new editing window - Open... -- Open an existing file - Recent Files... -- Open a list of recent files - Open Module... -- Open an existing module (searches sys.path) - Class Browser -- Show classes and methods in current file - Path Browser -- Show sys.path directories, modules, classes +Menus: + +IDLE has two window types the Shell window and the Editor window. It is +possible to have multiple editor windows simultaneously. IDLE's +menus dynamically change based on which window is currently selected. Each menu +documented below indicates which window type it is associated with. Click on +the dotted line at the top of a menu to "tear it off": a separate window +containing the menu is created (for Unix and Windows only). + +File Menu (Shell and Editor): + + New Window -- Create a new editing window + Open... -- Open an existing file + Open Module... -- Open an existing module (searches sys.path) + Recent Files... -- Open a list of recent files + Class Browser -- Show classes and methods in current file + Path Browser -- Show sys.path directories, modules, classes, and methods - --- - Save -- Save current window to the associated file (unsaved - windows have a * before and after the window title) + --- + Save -- Save current window to the associated file (unsaved + windows have a * before and after the window title) - Save As... -- Save current window to new file, which becomes - the associated file - Save Copy As... -- Save current window to different file - without changing the associated file - --- - Print Window -- Print the current window - --- - Close -- Close current window (asks to save if unsaved) - Exit -- Close all windows, quit (asks to save if unsaved) + Save As... -- Save current window to new file, which becomes + the associated file + Save Copy As... -- Save current window to different file + without changing the associated file + --- + Print Window -- Print the current window + --- + Close -- Close current window (asks to save if unsaved) + Exit -- Close all windows, quit (asks to save if unsaved) -Edit Menu: +Edit Menu (Shell and Editor): - Undo -- Undo last change to current window - (A maximum of 1000 changes may be undone) - Redo -- Redo last undone change to current window - --- - Cut -- Copy a selection into system-wide clipboard, + Undo -- Undo last change to current window + (a maximum of 1000 changes may be undone) + Redo -- Redo last undone change to current window + --- + Cut -- Copy a selection into system-wide clipboard, then delete the selection - Copy -- Copy selection into system-wide clipboard - Paste -- Insert system-wide clipboard into window - Select All -- Select the entire contents of the edit buffer - --- - Find... -- Open a search dialog box with many options - Find Again -- Repeat last search - Find Selection -- Search for the string in the selection - Find in Files... -- Open a search dialog box for searching files - Replace... -- Open a search-and-replace dialog box - Go to Line -- Ask for a line number and show that line - Show Calltip -- Open a small window with function param hints - Show Completions -- Open a scroll window allowing selection keywords - and attributes. (see '*TIPS*', below) - Show Parens -- Highlight the surrounding parenthesis - Expand Word -- Expand the word you have typed to match another - word in the same buffer; repeat to get a + Copy -- Copy selection into system-wide clipboard + Paste -- Insert system-wide clipboard into window + Select All -- Select the entire contents of the edit buffer + --- + Find... -- Open a search dialog box with many options + Find Again -- Repeat last search + Find Selection -- Search for the string in the selection + Find in Files... -- Open a search dialog box for searching files + Replace... -- Open a search-and-replace dialog box + Go to Line -- Ask for a line number and show that line + Expand Word -- Expand the word you have typed to match another + word in the same buffer; repeat to get a different expansion + Show Calltip -- After an unclosed parenthesis for a function, open + a small window with function parameter hints + Show Parens -- Highlight the surrounding parenthesis + Show Completions -- Open a scroll window allowing selection keywords + and attributes. (see '*TIPS*', below) -Format Menu (only in Edit window): +Format Menu (Editor window only): - Indent Region -- Shift selected lines right 4 spaces - Dedent Region -- Shift selected lines left 4 spaces - Comment Out Region -- Insert ## in front of selected lines - Uncomment Region -- Remove leading # or ## from selected lines - Tabify Region -- Turns *leading* stretches of spaces into tabs - (Note: We recommend using 4 space blocks to indent Python code.) - Untabify Region -- Turn *all* tabs into the right number of spaces - New Indent Width... -- Open dialog to change indent width - Format Paragraph -- Reformat the current blank-line-separated - paragraph + Indent Region -- Shift selected lines right by the indent width + (default 4 spaces) + Dedent Region -- Shift selected lines left by the indent width + (default 4 spaces) + Comment Out Region -- Insert ## in front of selected lines + Uncomment Region -- Remove leading # or ## from selected lines + Tabify Region -- Turns *leading* stretches of spaces into tabs. + (Note: We recommend using 4 space blocks to indent Python code.) + Untabify Region -- Turn *all* tabs into the corrent number of spaces + Toggle tabs -- Open a dialog to switch between indenting with + spaces and tabs. + New Indent Width... -- Open a dialog to change indent width. The + accepted default by the Python community is 4 + spaces. + Format Paragraph -- Reformat the current blank-line-separated + paragraph. All lines in the paragraph will be + formatted to less than 80 columns. + --- + Strip trailing whitespace -- Removed any space characters after the end + of the last non-space character -Run Menu (only in Edit window): +Run Menu (Editor window only): - Python Shell -- Open or wake up the Python shell window - --- - Check Module -- Run a syntax check on the module - Run Module -- Execute the current file in the __main__ namespace + Python Shell -- Open or wake up the Python shell window + --- + Check Module -- Check the syntax of the module currently open in the + Editor window. If the module has not been saved IDLE + will prompt the user to save the code. + Run Module -- Restart the shell to clean the environment, then + execute the currently open module. If the module has + not been saved IDLE will prompt the user to save the + code. -Shell Menu (only in Shell window): +Shell Menu (Shell window only): - View Last Restart -- Scroll the shell window to the last restart - Restart Shell -- Restart the interpreter with a fresh environment + View Last Restart -- Scroll the shell window to the last Shell restart + Restart Shell -- Restart the shell to clean the environment -Debug Menu (only in Shell window): +Debug Menu (Shell window only): - Go to File/Line -- look around the insert point for a filename - and line number, open the file, and show the line - Debugger (toggle) -- Run commands in the shell under the debugger - Stack Viewer -- Show the stack traceback of the last exception - Auto-open Stack Viewer (toggle) -- Open stack viewer on traceback + Go to File/Line -- Look around the insert point for a filename + and line number, open the file, and show the line. + Useful to view the source lines referenced in an + exception traceback. Available in the context + menu of the Shell window. + Debugger (toggle) -- This feature is not complete and considered + experimental. Run commands in the shell under the + debugger. + Stack Viewer -- Show the stack traceback of the last exception + Auto-open Stack Viewer (toggle) -- Toggle automatically opening the + stack viewer on unhandled + exception -Options Menu: +Options Menu (Shell and Editor): - Configure IDLE -- Open a configuration dialog. Fonts, indentation, + Configure IDLE -- Open a configuration dialog. Fonts, indentation, keybindings, and color themes may be altered. - Startup Preferences may be set, and Additional Help - Sources can be specified. - - On OS X this menu is not present, use - menu 'IDLE -> Preferences...' instead. - --- - Code Context -- Open a pane at the top of the edit window which - shows the block context of the section of code - which is scrolling off the top or the window. - (Not present in Shell window.) + Startup Preferences may be set, and additional Help + sources can be specified. -Windows Menu: + --- + Code Context (toggle) -- Open a pane at the top of the edit window + which shows the block context of the section + of code which is scrolling off the top or the + window. This is not present in the Shell + window only the Editor window. - Zoom Height -- toggles the window between configured size - and maximum height. - --- - The rest of this menu lists the names of all open windows; - select one to bring it to the foreground (deiconifying it if - necessary). +Windows Menu (Shell and Editor): + + Zoom Height -- Toggles the window between normal size (40x80 initial + setting) and maximum height. The initial size is in the Configure + IDLE dialog under the general tab. + --- + The rest of this menu lists the names of all open windows; + select one to bring it to the foreground (deiconifying it if + necessary). Help Menu: - About IDLE -- Version, copyright, license, credits - IDLE Readme -- Background discussion and change details - --- - IDLE Help -- Display this file - Python Docs -- Access local Python documentation, if - installed. Otherwise, access www.python.org. - --- - (Additional Help Sources may be added here) + About IDLE -- Version, copyright, license, credits + --- + IDLE Help -- Display this file which is a help file for IDLE + detailing the menu options, basic editing and navigation, + and other tips. + Python Docs -- Access local Python documentation, if + installed. Or will start a web browser and open + docs.python.org showing the latest Python documentation. + --- + Additional help sources may be added here with the Configure IDLE + dialog under the General tab. -Edit context menu (Right-click / Control-click on OS X in Edit window): +Editor context menu (Right-click / Control-click on OS X in Edit window): - Cut -- Copy a selection into system-wide clipboard, + Cut -- Copy a selection into system-wide clipboard, then delete the selection - Copy -- Copy selection into system-wide clipboard - Paste -- Insert system-wide clipboard into window - Set Breakpoint -- Sets a breakpoint (when debugger open) - Clear Breakpoint -- Clears the breakpoint on that line + Copy -- Copy selection into system-wide clipboard + Paste -- Insert system-wide clipboard into window + Set Breakpoint -- Sets a breakpoint. Breakpoints are only enabled + when the debugger is open. + Clear Breakpoint -- Clears the breakpoint on that line Shell context menu (Right-click / Control-click on OS X in Shell window): - Cut -- Copy a selection into system-wide clipboard, + Cut -- Copy a selection into system-wide clipboard, then delete the selection - Copy -- Copy selection into system-wide clipboard - Paste -- Insert system-wide clipboard into window - --- - Go to file/line -- Same as in Debug menu + Copy -- Copy selection into system-wide clipboard + Paste -- Insert system-wide clipboard into window + --- + Go to file/line -- Same as in Debug menu ** TIPS ** @@ -144,160 +187,183 @@ Additional Help Sources: - Windows users can Google on zopeshelf.chm to access Zope help files in - the Windows help format. The Additional Help Sources feature of the - configuration GUI supports .chm, along with any other filetypes - supported by your browser. Supply a Menu Item title, and enter the - location in the Help File Path slot of the New Help Source dialog. Use - http:// and/or www. to identify external URLs, or download the file and - browse for its path on your machine using the Browse button. + Windows users can Google on zopeshelf.chm to access Zope help files in + the Windows help format. The Additional Help Sources feature of the + configuration GUI supports .chm, along with any other filetypes + supported by your browser. Supply a Menu Item title, and enter the + location in the Help File Path slot of the New Help Source dialog. Use + http:// and/or www. to identify external URLs, or download the file and + browse for its path on your machine using the Browse button. - All users can access the extensive sources of help, including - tutorials, available at www.python.org/doc. Selected URLs can be added - or removed from the Help menu at any time using Configure IDLE. + All users can access the extensive sources of help, including + tutorials, available at docs.python.org. Selected URLs can be added + or removed from the Help menu at any time using Configure IDLE. Basic editing and navigation: - Backspace deletes char to the left; DEL deletes char to the right. - Control-backspace deletes word left, Control-DEL deletes word right. - Arrow keys and Page Up/Down move around. - Control-left/right Arrow moves by words in a strange but useful way. - Home/End go to begin/end of line. - Control-Home/End go to begin/end of file. - Some useful Emacs bindings are inherited from Tcl/Tk: - Control-a beginning of line - Control-e end of line - Control-k kill line (but doesn't put it in clipboard) - Control-l center window around the insertion point - Standard Windows bindings may work on that platform. - Keybindings are selected in the Settings Dialog, look there. + Backspace deletes char to the left; DEL deletes char to the right. + Control-backspace deletes word left, Control-DEL deletes word right. + Arrow keys and Page Up/Down move around. + Control-left/right Arrow moves by words in a strange but useful way. + Home/End go to begin/end of line. + Control-Home/End go to begin/end of file. + Some useful Emacs bindings are inherited from Tcl/Tk: + Control-a beginning of line + Control-e end of line + Control-k kill line (but doesn't put it in clipboard) + Control-l center window around the insertion point + Standard keybindings (like Control-c to copy and Control-v to + paste) may work. Keybindings are selected in the Configure IDLE + dialog. Automatic indentation: - After a block-opening statement, the next line is indented by 4 spaces - (in the Python Shell window by one tab). After certain keywords - (break, return etc.) the next line is dedented. In leading - indentation, Backspace deletes up to 4 spaces if they are there. Tab - inserts spaces (in the Python Shell window one tab), number depends on - Indent Width. (N.B. Currently tabs are restricted to four spaces due - to Tcl/Tk issues.) + After a block-opening statement, the next line is indented by 4 spaces + (in the Python Shell window by one tab). After certain keywords + (break, return etc.) the next line is dedented. In leading + indentation, Backspace deletes up to 4 spaces if they are there. Tab + inserts spaces (in the Python Shell window one tab), number depends on + Indent Width. Currently tabs are restricted to four spaces due + to Tcl/Tk limitations. See also the indent/dedent region commands in the edit menu. Completions: - Completions are supplied for functions, classes, and attributes of - classes, both built-in and user-defined. Completions are also provided - for filenames. + Completions are supplied for functions, classes, and attributes of + classes, both built-in and user-defined. Completions are also provided + for filenames. - The AutoCompleteWindow (ACW) will open after a predefined delay - (default is two seconds) after a '.' or (in a string) an os.sep is - typed. If after one of those characters (plus zero or more other - characters) you type a Tab the ACW will open immediately if a possible - continuation is found. + The AutoCompleteWindow (ACW) will open after a predefined delay + (default is two seconds) after a '.' or (in a string) an os.sep is + typed. If after one of those characters (plus zero or more other + characters) a tab is typed the ACW will open immediately if a possible + continuation is found. - If there is only one possible completion for the characters entered, a - Tab will supply that completion without opening the ACW. + If there is only one possible completion for the characters entered, a + tab will supply that completion without opening the ACW. - 'Show Completions' will force open a completions window. In an empty - string, this will contain the files in the current directory. On a - blank line, it will contain the built-in and user-defined functions and - classes in the current name spaces, plus any modules imported. If some - characters have been entered, the ACW will attempt to be more specific. + 'Show Completions' will force open a completions window, by default the + Control-space keys will open a completions window. In an empty + string, this will contain the files in the current directory. On a + blank line, it will contain the built-in and user-defined functions and + classes in the current name spaces, plus any modules imported. If some + characters have been entered, the ACW will attempt to be more specific. - If string of characters is typed, the ACW selection will jump to the - entry most closely matching those characters. Entering a Tab will cause - the longest non-ambiguous match to be entered in the Edit window or - Shell. Two Tabs in a row will supply the current ACW selection, as - will Return or a double click. Cursor keys, Page Up/Down, mouse - selection, and the scrollwheel all operate on the ACW. + If string of characters is typed, the ACW selection will jump to the + entry most closely matching those characters. Entering a tab will cause + the longest non-ambiguous match to be entered in the Edit window or + Shell. Two tabs in a row will supply the current ACW selection, as + will return or a double click. Cursor keys, Page Up/Down, mouse + selection, and the scroll wheel all operate on the ACW. - 'Hidden' attributes can be accessed by typing the beginning of hidden - name after a '.'. e.g. '_'. This allows access to modules with - '__all__' set, or to class-private attributes. + "Hidden" attributes can be accessed by typing the beginning of hidden + name after a '.', e.g. '_'. This allows access to modules with + '__all__' set, or to class-private attributes. - Completions and the 'Expand Word' facility can save a lot of typing! + Completions and the 'Expand Word' facility can save a lot of typing! - Completions are currently limited to those in the namespaces. Names in - an Edit window which are not via __main__ or sys.modules will not be - found. Run the module once with your imports to correct this - situation. Note that IDLE itself places quite a few modules in - sys.modules, so much can be found by default, e.g. the re module. + Completions are currently limited to those in the namespaces. Names in + an Editor window which are not via __main__ or sys.modules will not be + found. Run the module once with your imports to correct this + situation. Note that IDLE itself places quite a few modules in + sys.modules, so much can be found by default, e.g. the re module. - If you don't like the ACW popping up unbidden, simply make the delay - longer or disable the extension. OTOH, you could make the delay zero. - - You could also switch off the CallTips extension. (We will be adding - a delay to the call tip window.) + If you don't like the ACW popping up unbidden, simply make the delay + longer or disable the extension. Or another option is the delay could + be set to zero. Another alternative to preventing ACW popups is to + disable the call tips extension. Python Shell window: - Control-c interrupts executing command. - Control-d sends end-of-file; closes window if typed at >>> prompt - (this is Control-z on Windows). + Control-c interrupts executing command. + Control-d sends end-of-file; closes window if typed at >>> prompt + (this is Control-z on Windows). + Alt-/ expand word is also useful to reduce typing. Command history: - Alt-p retrieves previous command matching what you have typed. - Alt-n retrieves next. - (These are Control-p, Control-n on OS X) - Return while cursor is on a previous command retrieves that command. - Expand word is also useful to reduce typing. + Alt-p retrieves previous command matching what you have typed. On OS X + use Control-p. + Alt-n retrieves next. On OS X use Control-n. + Return while cursor is on a previous command retrieves that command. Syntax colors: - The coloring is applied in a background "thread", so you may - occasionally see uncolorized text. To change the color - scheme, use the Configure IDLE / Highlighting dialog. + The coloring is applied in a background "thread", so you may + occasionally see uncolorized text. To change the color + scheme, use the Configure IDLE / Highlighting dialog. Python default syntax colors: - Keywords orange - Builtins royal purple - Strings green - Comments red - Definitions blue + Keywords orange + Builtins royal purple + Strings green + Comments red + Definitions blue Shell default colors: - Console output brown - stdout blue - stderr red - stdin black + Console output brown + stdout blue + stderr red + stdin black Other preferences: - The font preferences, keybinding, and startup preferences can - be changed using the Settings dialog. + The font preferences, highlighting, keys, and general preferences can + be changed via the Configure IDLE menu option. Be sure to note that + keys can be user defined, IDLE ships with four built in key sets. In + addition a user can create a custom key set in the Configure IDLE + dialog under the keys tab. Command line usage: - Enter idle -h at the command prompt to get a usage message. + Enter idle -h at the command prompt to get a usage message. -Running without a subprocess: (DEPRECATED) + idle.py [-c command] [-d] [-e] [-s] [-t title] [arg] ... - If IDLE is started with the -n command line switch it will run in a - single process and will not create the subprocess which runs the RPC - Python execution server. This can be useful if Python cannot create - the subprocess or the RPC socket interface on your platform. However, - in this mode user code is not isolated from IDLE itself. Also, the - environment is not restarted when Run/Run Module (F5) is selected. If - your code has been modified, you must reload() the affected modules and - re-import any specific items (e.g. from foo import baz) if the changes - are to take effect. For these reasons, it is preferable to run IDLE - with the default subprocess if at all possible. + -c command run this command + -d enable debugger + -e edit mode; arguments are files to be edited + -s run $IDLESTARTUP or $PYTHONSTARTUP first + -t title set title of shell window + + If there are arguments: + 1. If -e is used, arguments are files opened for editing and sys.argv + reflects the arguments passed to IDLE itself. + 2. Otherwise, if -c is used, all arguments are placed in + sys.argv[1:...], with sys.argv[0] set to -c. + 3. Otherwise, if neither -e nor -c is used, the first argument is a + script which is executed with the remaining arguments in + sys.argv[1:...] and sys.argv[0] set to the script name. If the + script name is -, no script is executed but an interactive Python + session is started; the arguments are still available in sys.argv. + +Running without a subprocess: (DEPRECATED in Python 3.5 see Issue 16123) + + If IDLE is started with the -n command line switch it will run in a + single process and will not create the subprocess which runs the RPC + Python execution server. This can be useful if Python cannot create + the subprocess or the RPC socket interface on your platform. However, + in this mode user code is not isolated from IDLE itself. Also, the + environment is not restarted when Run/Run Module (F5) is selected. If + your code has been modified, you must reload() the affected modules and + re-import any specific items (e.g. from foo import baz) if the changes + are to take effect. For these reasons, it is preferable to run IDLE + with the default subprocess if at all possible. Extensions: - IDLE contains an extension facility. See the beginning of - config-extensions.def in the idlelib directory for further information. - The default extensions are currently: + IDLE contains an extension facility. See the beginning of + config-extensions.def in the idlelib directory for further information. + The default extensions are currently: - FormatParagraph - AutoExpand - ZoomHeight - ScriptBinding - CallTips - ParenMatch - AutoComplete - CodeContext + FormatParagraph + AutoExpand + ZoomHeight + ScriptBinding + CallTips + ParenMatch + AutoComplete + CodeContext diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -217,6 +217,8 @@ Library ------- +- Issue #5066: Update IDLE docs. Patch by Todd Rovito. + - Issue #16955: Fix the poll() method for multiprocessing's socket connections on Windows. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Jan 14 23:12:03 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 14 Jan 2013 23:12:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Null_merge?= Message-ID: <3YlTS351dfzS9K@mail.python.org> http://hg.python.org/cpython/rev/084c46ff80ae changeset: 81500:084c46ff80ae parent: 81499:d1ef91025d70 parent: 81498:88fadc0d7b20 user: Serhiy Storchaka date: Tue Jan 15 00:11:19 2013 +0200 summary: Null merge files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Jan 14 23:45:10 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 14 Jan 2013 23:45:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzk3MjA6?= =?utf-8?q?_zipfile_now_writes_correct_local_headers_for_files_larger_than?= =?utf-8?q?_4?= Message-ID: <3YlVBG2rJlzS0r@mail.python.org> http://hg.python.org/cpython/rev/ce869b05762c changeset: 81501:ce869b05762c branch: 2.7 parent: 81491:c5451d4a0cdd user: Serhiy Storchaka date: Tue Jan 15 00:29:51 2013 +0200 summary: Issue #9720: zipfile now writes correct local headers for files larger than 4 GiB. files: Lib/zipfile.py | 47 +++++++++++++++++++++++++------------ Misc/NEWS | 3 ++ 2 files changed, 35 insertions(+), 15 deletions(-) diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -316,7 +316,7 @@ # compress_size Size of the compressed file # file_size Size of the uncompressed file - def FileHeader(self): + def FileHeader(self, zip64=None): """Return the per-file header as a string.""" dt = self.date_time dosdate = (dt[0] - 1980) << 9 | dt[1] << 5 | dt[2] @@ -331,12 +331,17 @@ extra = self.extra - if file_size > ZIP64_LIMIT or compress_size > ZIP64_LIMIT: - # File is larger than what fits into a 4 byte integer, - # fall back to the ZIP64 extension + if zip64 is None: + zip64 = file_size > ZIP64_LIMIT or compress_size > ZIP64_LIMIT + if zip64: fmt = ' ZIP64_LIMIT or compress_size > ZIP64_LIMIT: + if not zip64: + raise LargeZipFile("Filesize would require ZIP64 extensions") + # File is larger than what fits into a 4 byte integer, + # fall back to the ZIP64 extension file_size = 0xffffffff compress_size = 0xffffffff self.extract_version = max(45, self.extract_version) @@ -1113,20 +1118,23 @@ zinfo.CRC = 0 self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo - self.fp.write(zinfo.FileHeader()) + self.fp.write(zinfo.FileHeader(False)) return with open(filename, "rb") as fp: # Must overwrite CRC and sizes with correct data later zinfo.CRC = CRC = 0 zinfo.compress_size = compress_size = 0 - zinfo.file_size = file_size = 0 - self.fp.write(zinfo.FileHeader()) + # Compressed size can be larger than uncompressed size + zip64 = self._allowZip64 and \ + zinfo.file_size * 1.05 > ZIP64_LIMIT + self.fp.write(zinfo.FileHeader(zip64)) if zinfo.compress_type == ZIP_DEFLATED: cmpr = zlib.compressobj(zlib.Z_DEFAULT_COMPRESSION, zlib.DEFLATED, -15) else: cmpr = None + file_size = 0 while 1: buf = fp.read(1024 * 8) if not buf: @@ -1146,11 +1154,16 @@ zinfo.compress_size = file_size zinfo.CRC = CRC zinfo.file_size = file_size - # Seek backwards and write CRC and file sizes + if not zip64 and self._allowZip64: + if file_size > ZIP64_LIMIT: + raise RuntimeError('File size has increased during compressing') + if compress_size > ZIP64_LIMIT: + raise RuntimeError('Compressed size larger than uncompressed size') + # Seek backwards and write file header (which will now include + # correct CRC and file sizes) position = self.fp.tell() # Preserve current position in file - self.fp.seek(zinfo.header_offset + 14, 0) - self.fp.write(struct.pack(" ZIP64_LIMIT or \ + zinfo.compress_size > ZIP64_LIMIT + if zip64 and not self._allowZip64: + raise LargeZipFile("Filesize would require ZIP64 extensions") + self.fp.write(zinfo.FileHeader(zip64)) self.fp.write(bytes) - self.fp.flush() if zinfo.flag_bits & 0x08: # Write CRC and file sizes after the file data - self.fp.write(struct.pack(" http://hg.python.org/cpython/rev/b93848ca7760 changeset: 81502:b93848ca7760 branch: 3.2 parent: 81492:df8609f1854d user: Serhiy Storchaka date: Tue Jan 15 00:31:39 2013 +0200 summary: Issue #9720: zipfile now writes correct local headers for files larger than 4 GiB. files: Lib/zipfile.py | 47 +++++++++++++++++++++++++------------ Misc/NEWS | 3 ++ 2 files changed, 35 insertions(+), 15 deletions(-) diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -326,7 +326,7 @@ # compress_size Size of the compressed file # file_size Size of the uncompressed file - def FileHeader(self): + def FileHeader(self, zip64=None): """Return the per-file header as a string.""" dt = self.date_time dosdate = (dt[0] - 1980) << 9 | dt[1] << 5 | dt[2] @@ -341,12 +341,17 @@ extra = self.extra - if file_size > ZIP64_LIMIT or compress_size > ZIP64_LIMIT: - # File is larger than what fits into a 4 byte integer, - # fall back to the ZIP64 extension + if zip64 is None: + zip64 = file_size > ZIP64_LIMIT or compress_size > ZIP64_LIMIT + if zip64: fmt = ' ZIP64_LIMIT or compress_size > ZIP64_LIMIT: + if not zip64: + raise LargeZipFile("Filesize would require ZIP64 extensions") + # File is larger than what fits into a 4 byte integer, + # fall back to the ZIP64 extension file_size = 0xffffffff compress_size = 0xffffffff self.extract_version = max(45, self.extract_version) @@ -1135,20 +1140,23 @@ zinfo.CRC = 0 self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo - self.fp.write(zinfo.FileHeader()) + self.fp.write(zinfo.FileHeader(False)) return with open(filename, "rb") as fp: # Must overwrite CRC and sizes with correct data later zinfo.CRC = CRC = 0 zinfo.compress_size = compress_size = 0 - zinfo.file_size = file_size = 0 - self.fp.write(zinfo.FileHeader()) + # Compressed size can be larger than uncompressed size + zip64 = self._allowZip64 and \ + zinfo.file_size * 1.05 > ZIP64_LIMIT + self.fp.write(zinfo.FileHeader(zip64)) if zinfo.compress_type == ZIP_DEFLATED: cmpr = zlib.compressobj(zlib.Z_DEFAULT_COMPRESSION, zlib.DEFLATED, -15) else: cmpr = None + file_size = 0 while 1: buf = fp.read(1024 * 8) if not buf: @@ -1168,11 +1176,16 @@ zinfo.compress_size = file_size zinfo.CRC = CRC zinfo.file_size = file_size - # Seek backwards and write CRC and file sizes + if not zip64 and self._allowZip64: + if file_size > ZIP64_LIMIT: + raise RuntimeError('File size has increased during compressing') + if compress_size > ZIP64_LIMIT: + raise RuntimeError('Compressed size larger than uncompressed size') + # Seek backwards and write file header (which will now include + # correct CRC and file sizes) position = self.fp.tell() # Preserve current position in file - self.fp.seek(zinfo.header_offset + 14, 0) - self.fp.write(struct.pack(" ZIP64_LIMIT or \ + zinfo.compress_size > ZIP64_LIMIT + if zip64 and not self._allowZip64: + raise LargeZipFile("Filesize would require ZIP64 extensions") + self.fp.write(zinfo.FileHeader(zip64)) self.fp.write(data) - self.fp.flush() if zinfo.flag_bits & 0x08: # Write CRC and file sizes after the file data - self.fp.write(struct.pack(" http://hg.python.org/cpython/rev/656a45738e5e changeset: 81503:656a45738e5e branch: 3.3 parent: 81498:88fadc0d7b20 parent: 81502:b93848ca7760 user: Serhiy Storchaka date: Tue Jan 15 00:38:17 2013 +0200 summary: Issue #9720: zipfile now writes correct local headers for files larger than 4 GiB. files: Lib/zipfile.py | 47 +++++++++++++++++++++++++------------ Misc/NEWS | 3 ++ 2 files changed, 35 insertions(+), 15 deletions(-) diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -346,7 +346,7 @@ # compress_size Size of the compressed file # file_size Size of the uncompressed file - def FileHeader(self): + def FileHeader(self, zip64=None): """Return the per-file header as a string.""" dt = self.date_time dosdate = (dt[0] - 1980) << 9 | dt[1] << 5 | dt[2] @@ -362,12 +362,17 @@ extra = self.extra min_version = 0 - if file_size > ZIP64_LIMIT or compress_size > ZIP64_LIMIT: - # File is larger than what fits into a 4 byte integer, - # fall back to the ZIP64 extension + if zip64 is None: + zip64 = file_size > ZIP64_LIMIT or compress_size > ZIP64_LIMIT + if zip64: fmt = ' ZIP64_LIMIT or compress_size > ZIP64_LIMIT: + if not zip64: + raise LargeZipFile("Filesize would require ZIP64 extensions") + # File is larger than what fits into a 4 byte integer, + # fall back to the ZIP64 extension file_size = 0xffffffff compress_size = 0xffffffff min_version = ZIP64_VERSION @@ -1301,7 +1306,7 @@ zinfo.CRC = 0 self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo - self.fp.write(zinfo.FileHeader()) + self.fp.write(zinfo.FileHeader(False)) return cmpr = _get_compressor(zinfo.compress_type) @@ -1309,8 +1314,11 @@ # Must overwrite CRC and sizes with correct data later zinfo.CRC = CRC = 0 zinfo.compress_size = compress_size = 0 - zinfo.file_size = file_size = 0 - self.fp.write(zinfo.FileHeader()) + # Compressed size can be larger than uncompressed size + zip64 = self._allowZip64 and \ + zinfo.file_size * 1.05 > ZIP64_LIMIT + self.fp.write(zinfo.FileHeader(zip64)) + file_size = 0 while 1: buf = fp.read(1024 * 8) if not buf: @@ -1330,11 +1338,16 @@ zinfo.compress_size = file_size zinfo.CRC = CRC zinfo.file_size = file_size - # Seek backwards and write CRC and file sizes + if not zip64 and self._allowZip64: + if file_size > ZIP64_LIMIT: + raise RuntimeError('File size has increased during compressing') + if compress_size > ZIP64_LIMIT: + raise RuntimeError('Compressed size larger than uncompressed size') + # Seek backwards and write file header (which will now include + # correct CRC and file sizes) position = self.fp.tell() # Preserve current position in file - self.fp.seek(zinfo.header_offset + 14, 0) - self.fp.write(struct.pack(" ZIP64_LIMIT or \ + zinfo.compress_size > ZIP64_LIMIT + if zip64 and not self._allowZip64: + raise LargeZipFile("Filesize would require ZIP64 extensions") + self.fp.write(zinfo.FileHeader(zip64)) self.fp.write(data) - self.fp.flush() if zinfo.flag_bits & 0x08: # Write CRC and file sizes after the file data - self.fp.write(struct.pack(" http://hg.python.org/cpython/rev/628a6af64a46 changeset: 81504:628a6af64a46 parent: 81500:084c46ff80ae parent: 81503:656a45738e5e user: Serhiy Storchaka date: Tue Jan 15 00:41:43 2013 +0200 summary: Issue #9720: zipfile now writes correct local headers for files larger than 4 GiB. files: Lib/zipfile.py | 46 ++++++++++++++++++++++++++----------- Misc/NEWS | 3 ++ 2 files changed, 35 insertions(+), 14 deletions(-) diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -346,7 +346,7 @@ # compress_size Size of the compressed file # file_size Size of the uncompressed file - def FileHeader(self): + def FileHeader(self, zip64=None): """Return the per-file header as a string.""" dt = self.date_time dosdate = (dt[0] - 1980) << 9 | dt[1] << 5 | dt[2] @@ -362,12 +362,17 @@ extra = self.extra min_version = 0 - if file_size > ZIP64_LIMIT or compress_size > ZIP64_LIMIT: - # File is larger than what fits into a 4 byte integer, - # fall back to the ZIP64 extension + if zip64 is None: + zip64 = file_size > ZIP64_LIMIT or compress_size > ZIP64_LIMIT + if zip64: fmt = ' ZIP64_LIMIT or compress_size > ZIP64_LIMIT: + if not zip64: + raise LargeZipFile("Filesize would require ZIP64 extensions") + # File is larger than what fits into a 4 byte integer, + # fall back to the ZIP64 extension file_size = 0xffffffff compress_size = 0xffffffff min_version = ZIP64_VERSION @@ -1301,7 +1306,7 @@ zinfo.CRC = 0 self.filelist.append(zinfo) self.NameToInfo[zinfo.filename] = zinfo - self.fp.write(zinfo.FileHeader()) + self.fp.write(zinfo.FileHeader(False)) return cmpr = _get_compressor(zinfo.compress_type) @@ -1309,8 +1314,11 @@ # Must overwrite CRC and sizes with correct data later zinfo.CRC = CRC = 0 zinfo.compress_size = compress_size = 0 - zinfo.file_size = file_size = 0 - self.fp.write(zinfo.FileHeader()) + # Compressed size can be larger than uncompressed size + zip64 = self._allowZip64 and \ + zinfo.file_size * 1.05 > ZIP64_LIMIT + self.fp.write(zinfo.FileHeader(zip64)) + file_size = 0 while 1: buf = fp.read(1024 * 8) if not buf: @@ -1330,11 +1338,16 @@ zinfo.compress_size = file_size zinfo.CRC = CRC zinfo.file_size = file_size - # Seek backwards and write CRC and file sizes + if not zip64 and self._allowZip64: + if file_size > ZIP64_LIMIT: + raise RuntimeError('File size has increased during compressing') + if compress_size > ZIP64_LIMIT: + raise RuntimeError('Compressed size larger than uncompressed size') + # Seek backwards and write file header (which will now include + # correct CRC and file sizes) position = self.fp.tell() # Preserve current position in file - self.fp.seek(zinfo.header_offset + 14, 0) - self.fp.write(struct.pack(" ZIP64_LIMIT or \ + zinfo.compress_size > ZIP64_LIMIT + if zip64 and not self._allowZip64: + raise LargeZipFile("Filesize would require ZIP64 extensions") + self.fp.write(zinfo.FileHeader(zip64)) self.fp.write(data) - self.fp.flush() if zinfo.flag_bits & 0x08: # Write CRC and file sizes after the file data - self.fp.write(struct.pack(" http://hg.python.org/cpython/rev/13e2e44db99d changeset: 81505:13e2e44db99d user: Serhiy Storchaka date: Tue Jan 15 01:12:17 2013 +0200 summary: Issue #15989: Fix several occurrences of integer overflow when result of PyLong_AsLong() narrowed to int without checks. files: Include/longobject.h | 3 ++ Lib/ctypes/test/test_structures.py | 9 ++++++ Lib/test/string_tests.py | 11 ++++++++ Lib/test/test_fcntl.py | 21 +++++++++++++++ Lib/test/test_fileio.py | 4 +++ Lib/test/test_io.py | 9 ++++++ Lib/test/test_poll.py | 10 +++++++ Lib/test/test_posix.py | 5 +++ Lib/test/test_socket.py | 22 ++++++++++++++-- Modules/_ctypes/stgdict.c | 2 +- Modules/_io/fileio.c | 4 +- Modules/_io/textio.c | 2 +- Modules/parsermodule.c | 24 ++++++++++++++--- Modules/posixmodule.c | 2 +- Modules/selectmodule.c | 12 ++++++--- Modules/socketmodule.c | 6 ++-- Objects/fileobject.c | 4 +- Objects/longobject.c | 18 +++++++++++++ Objects/unicodeobject.c | 4 +- 19 files changed, 148 insertions(+), 24 deletions(-) diff --git a/Include/longobject.h b/Include/longobject.h --- a/Include/longobject.h +++ b/Include/longobject.h @@ -26,6 +26,9 @@ PyAPI_FUNC(size_t) PyLong_AsSize_t(PyObject *); PyAPI_FUNC(unsigned long) PyLong_AsUnsignedLong(PyObject *); PyAPI_FUNC(unsigned long) PyLong_AsUnsignedLongMask(PyObject *); +#ifndef Py_LIMITED_API +PyAPI_FUNC(int) _PyLong_AsInt(PyObject *); +#endif PyAPI_FUNC(PyObject *) PyLong_GetInfo(void); /* It may be useful in the future. I've added it in the PyInt -> PyLong diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py --- a/Lib/ctypes/test/test_structures.py +++ b/Lib/ctypes/test/test_structures.py @@ -1,6 +1,7 @@ import unittest from ctypes import * from struct import calcsize +import _testcapi class SubclassesTest(unittest.TestCase): def test_subclass(self): @@ -199,6 +200,14 @@ "_pack_": -1} self.assertRaises(ValueError, type(Structure), "X", (Structure,), d) + # Issue 15989 + d = {"_fields_": [("a", c_byte)], + "_pack_": _testcapi.INT_MAX + 1} + self.assertRaises(ValueError, type(Structure), "X", (Structure,), d) + d = {"_fields_": [("a", c_byte)], + "_pack_": _testcapi.UINT_MAX + 2} + self.assertRaises(ValueError, type(Structure), "X", (Structure,), d) + def test_initializers(self): class Person(Structure): _fields_ = [("name", c_char*6), diff --git a/Lib/test/string_tests.py b/Lib/test/string_tests.py --- a/Lib/test/string_tests.py +++ b/Lib/test/string_tests.py @@ -5,6 +5,7 @@ import unittest, string, sys, struct from test import support from collections import UserList +import _testcapi class Sequence: def __init__(self, seq='wxyz'): self.seq = seq @@ -1206,6 +1207,16 @@ self.checkraises(ValueError, '%%%df' % (2**64), '__mod__', (3.2)) self.checkraises(ValueError, '%%.%df' % (2**64), '__mod__', (3.2)) + self.checkraises(OverflowError, '%*s', '__mod__', + (_testcapi.PY_SSIZE_T_MAX + 1, '')) + self.checkraises(OverflowError, '%.*f', '__mod__', + (_testcapi.INT_MAX + 1, 1. / 7)) + # Issue 15989 + self.checkraises(OverflowError, '%*s', '__mod__', + (1 << (_testcapi.PY_SSIZE_T_MAX.bit_length() + 1), '')) + self.checkraises(OverflowError, '%.*f', '__mod__', + (_testcapi.UINT_MAX + 1, 1. / 7)) + class X(object): pass self.checkraises(TypeError, 'abc', '__mod__', X()) diff --git a/Lib/test/test_fcntl.py b/Lib/test/test_fcntl.py --- a/Lib/test/test_fcntl.py +++ b/Lib/test/test_fcntl.py @@ -3,6 +3,7 @@ import os import struct import sys +import _testcapi import unittest from test.support import verbose, TESTFN, unlink, run_unittest, import_module @@ -69,6 +70,26 @@ rv = fcntl.fcntl(self.f, fcntl.F_SETLKW, lockdata) self.f.close() + def test_fcntl_bad_file(self): + class F: + def __init__(self, fn): + self.fn = fn + def fileno(self): + return self.fn + self.assertRaises(ValueError, fcntl.fcntl, -1, fcntl.F_SETFL, os.O_NONBLOCK) + self.assertRaises(ValueError, fcntl.fcntl, F(-1), fcntl.F_SETFL, os.O_NONBLOCK) + self.assertRaises(TypeError, fcntl.fcntl, 'spam', fcntl.F_SETFL, os.O_NONBLOCK) + self.assertRaises(TypeError, fcntl.fcntl, F('spam'), fcntl.F_SETFL, os.O_NONBLOCK) + # Issue 15989 + self.assertRaises(OverflowError, fcntl.fcntl, _testcapi.INT_MAX + 1, + fcntl.F_SETFL, os.O_NONBLOCK) + self.assertRaises(OverflowError, fcntl.fcntl, F(_testcapi.INT_MAX + 1), + fcntl.F_SETFL, os.O_NONBLOCK) + self.assertRaises(OverflowError, fcntl.fcntl, _testcapi.INT_MIN - 1, + fcntl.F_SETFL, os.O_NONBLOCK) + self.assertRaises(OverflowError, fcntl.fcntl, F(_testcapi.INT_MIN - 1), + fcntl.F_SETFL, os.O_NONBLOCK) + def test_fcntl_64_bit(self): # Issue #1309352: fcntl shouldn't fail when the third arg fits in a # C 'long' but not in a C 'int'. diff --git a/Lib/test/test_fileio.py b/Lib/test/test_fileio.py --- a/Lib/test/test_fileio.py +++ b/Lib/test/test_fileio.py @@ -8,6 +8,7 @@ from array import array from weakref import proxy from functools import wraps +import _testcapi from test.support import TESTFN, check_warnings, run_unittest, make_bad_fd from collections import UserList @@ -347,6 +348,9 @@ if sys.platform == 'win32': import msvcrt self.assertRaises(OSError, msvcrt.get_osfhandle, make_bad_fd()) + # Issue 15989 + self.assertRaises(TypeError, _FileIO, _testcapi.INT_MAX + 1) + self.assertRaises(TypeError, _FileIO, _testcapi.INT_MIN - 1) def testBadModeArgument(self): # verify that we get a sensible error message for bad mode argument diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -32,6 +32,7 @@ import unittest import warnings import weakref +import _testcapi from collections import deque, UserList from itertools import cycle, count from test import support @@ -1970,6 +1971,14 @@ os.environ.clear() os.environ.update(old_environ) + # Issue 15989 + def test_device_encoding(self): + b = self.BytesIO() + b.fileno = lambda: _testcapi.INT_MAX + 1 + self.assertRaises(OverflowError, self.TextIOWrapper, b) + b.fileno = lambda: _testcapi.UINT_MAX + 1 + self.assertRaises(OverflowError, self.TextIOWrapper, b) + def test_encoding(self): # Check the encoding attribute is always set, and valid b = self.BytesIO() diff --git a/Lib/test/test_poll.py b/Lib/test/test_poll.py --- a/Lib/test/test_poll.py +++ b/Lib/test/test_poll.py @@ -1,6 +1,7 @@ # Test case for the os.poll() function import os, select, random, unittest, subprocess +import _testcapi from test.support import TESTFN, run_unittest try: @@ -151,6 +152,15 @@ if x != 5: self.fail('Overflow must have occurred') + pollster = select.poll() + # Issue 15989 + self.assertRaises(OverflowError, pollster.register, 0, + _testcapi.SHRT_MAX + 1) + self.assertRaises(OverflowError, pollster.register, 0, + _testcapi.USHRT_MAX + 1) + self.assertRaises(OverflowError, pollster.poll, _testcapi.INT_MAX + 1) + self.assertRaises(OverflowError, pollster.poll, _testcapi.UINT_MAX + 1) + def test_main(): run_unittest(PollTests) diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -17,6 +17,7 @@ import tempfile import unittest import warnings +import _testcapi _DUMMY_SYMLINK = os.path.join(tempfile.gettempdir(), support.TESTFN + '-dummy-symlink') @@ -537,6 +538,10 @@ except OSError: pass + # Issue 15989 + self.assertRaises(OverflowError, os.pipe2, _testcapi.INT_MAX + 1) + self.assertRaises(OverflowError, os.pipe2, _testcapi.UINT_MAX + 1) + def test_utime(self): if hasattr(posix, 'utime'): now = time.time() diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py --- a/Lib/test/test_socket.py +++ b/Lib/test/test_socket.py @@ -1262,11 +1262,17 @@ for protocol in range(pickle.HIGHEST_PROTOCOL + 1): self.assertRaises(TypeError, pickle.dumps, sock, protocol) - def test_listen_backlog0(self): + def test_listen_backlog(self): + for backlog in 0, -1: + srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + srv.bind((HOST, 0)) + srv.listen(backlog) + srv.close() + + # Issue 15989 srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM) srv.bind((HOST, 0)) - # backlog = 0 - srv.listen(0) + self.assertRaises(OverflowError, srv.listen, _testcapi.INT_MAX + 1) srv.close() @unittest.skipUnless(support.IPV6_ENABLED, 'IPv6 required for this test.') @@ -1582,6 +1588,11 @@ def _testShutdown(self): self.serv_conn.send(MSG) + # Issue 15989 + self.assertRaises(OverflowError, self.serv_conn.shutdown, + _testcapi.INT_MAX + 1) + self.assertRaises(OverflowError, self.serv_conn.shutdown, + 2 + (_testcapi.UINT_MAX + 1)) self.serv_conn.shutdown(2) def testDetach(self): @@ -3563,6 +3574,11 @@ pass end = time.time() self.assertTrue((end - start) < 1.0, "Error setting non-blocking mode.") + # Issue 15989 + self.assertRaises(OverflowError, self.serv.setblocking, + _testcapi.INT_MAX + 1) + self.assertRaises(OverflowError, self.serv.setblocking, + _testcapi.UINT_MAX + 1) def _testSetBlocking(self): pass diff --git a/Modules/_ctypes/stgdict.c b/Modules/_ctypes/stgdict.c --- a/Modules/_ctypes/stgdict.c +++ b/Modules/_ctypes/stgdict.c @@ -335,7 +335,7 @@ isPacked = PyObject_GetAttrString(type, "_pack_"); if (isPacked) { - pack = PyLong_AsLong(isPacked); + pack = _PyLong_AsInt(isPacked); if (pack < 0 || PyErr_Occurred()) { Py_XDECREF(isPacked); PyErr_SetString(PyExc_ValueError, diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c --- a/Modules/_io/fileio.c +++ b/Modules/_io/fileio.c @@ -244,7 +244,7 @@ return -1; } - fd = PyLong_AsLong(nameobj); + fd = _PyLong_AsInt(nameobj); if (fd < 0) { if (!PyErr_Occurred()) { PyErr_SetString(PyExc_ValueError, @@ -382,7 +382,7 @@ goto error; } - self->fd = PyLong_AsLong(fdobj); + self->fd = _PyLong_AsInt(fdobj); Py_DECREF(fdobj); if (self->fd == -1) { goto error; diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c --- a/Modules/_io/textio.c +++ b/Modules/_io/textio.c @@ -881,7 +881,7 @@ } } else { - int fd = (int) PyLong_AsLong(fileno); + int fd = _PyLong_AsInt(fileno); Py_DECREF(fileno); if (fd == -1 && PyErr_Occurred()) { goto error; diff --git a/Modules/parsermodule.c b/Modules/parsermodule.c --- a/Modules/parsermodule.c +++ b/Modules/parsermodule.c @@ -725,7 +725,7 @@ /* elem must always be a sequence, however simple */ PyObject* elem = PySequence_GetItem(tuple, i); int ok = elem != NULL; - long type = 0; + int type = 0; char *strn = 0; if (ok) @@ -736,8 +736,14 @@ ok = 0; else { ok = PyLong_Check(temp); - if (ok) - type = PyLong_AS_LONG(temp); + if (ok) { + type = _PyLong_AsInt(temp); + if (type == -1 && PyErr_Occurred()) { + Py_DECREF(temp); + Py_DECREF(elem); + return 0; + } + } Py_DECREF(temp); } } @@ -773,8 +779,16 @@ if (len == 3) { PyObject *o = PySequence_GetItem(elem, 2); if (o != NULL) { - if (PyLong_Check(o)) - *line_num = PyLong_AS_LONG(o); + if (PyLong_Check(o)) { + int num = _PyLong_AsInt(o); + if (num == -1 && PyErr_Occurred()) { + Py_DECREF(o); + Py_DECREF(temp); + Py_DECREF(elem); + return 0; + } + *line_num = num; + } else { PyErr_Format(parser_error, "third item in terminal node must be an" diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -7647,7 +7647,7 @@ int fds[2]; int res; - flags = PyLong_AsLong(arg); + flags = _PyLong_AsInt(arg); if (flags == -1 && PyErr_Occurred()) return NULL; diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -352,10 +352,13 @@ i = pos = 0; while (PyDict_Next(self->dict, &pos, &key, &value)) { - self->ufds[i].fd = PyLong_AsLong(key); + assert(i < self->ufd_len); + /* Never overflow */ + self->ufds[i].fd = (int)PyLong_AsLong(key); self->ufds[i].events = (short)PyLong_AsLong(value); i++; } + assert(i == self->ufd_len); self->ufd_uptodate = 1; return 1; } @@ -371,10 +374,11 @@ poll_register(pollObject *self, PyObject *args) { PyObject *o, *key, *value; - int fd, events = POLLIN | POLLPRI | POLLOUT; + int fd; + short events = POLLIN | POLLPRI | POLLOUT; int err; - if (!PyArg_ParseTuple(args, "O|i:register", &o, &events)) { + if (!PyArg_ParseTuple(args, "O|h:register", &o, &events)) { return NULL; } @@ -513,7 +517,7 @@ tout = PyNumber_Long(tout); if (!tout) return NULL; - timeout = PyLong_AsLong(tout); + timeout = _PyLong_AsInt(tout); Py_DECREF(tout); if (timeout == -1 && PyErr_Occurred()) return NULL; diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -2013,7 +2013,7 @@ static PyObject * sock_setblocking(PySocketSockObject *s, PyObject *arg) { - int block; + long block; block = PyLong_AsLong(arg); if (block == -1 && PyErr_Occurred()) @@ -2495,7 +2495,7 @@ int backlog; int res; - backlog = PyLong_AsLong(arg); + backlog = _PyLong_AsInt(arg); if (backlog == -1 && PyErr_Occurred()) return NULL; Py_BEGIN_ALLOW_THREADS @@ -3647,7 +3647,7 @@ int how; int res; - how = PyLong_AsLong(arg); + how = _PyLong_AsInt(arg); if (how == -1 && PyErr_Occurred()) return NULL; Py_BEGIN_ALLOW_THREADS diff --git a/Objects/fileobject.c b/Objects/fileobject.c --- a/Objects/fileobject.c +++ b/Objects/fileobject.c @@ -200,7 +200,7 @@ _Py_IDENTIFIER(fileno); if (PyLong_Check(o)) { - fd = PyLong_AsLong(o); + fd = _PyLong_AsInt(o); } else if ((meth = _PyObject_GetAttrId(o, &PyId_fileno)) != NULL) { @@ -210,7 +210,7 @@ return -1; if (PyLong_Check(fno)) { - fd = PyLong_AsLong(fno); + fd = _PyLong_AsInt(fno); Py_DECREF(fno); } else { diff --git a/Objects/longobject.c b/Objects/longobject.c --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -434,6 +434,24 @@ return result; } +/* Get a C int from a long int object or any object that has an __int__ + method. Return -1 and set an error if overflow occurs. */ + +int +_PyLong_AsInt(PyObject *obj) +{ + int overflow; + long result = PyLong_AsLongAndOverflow(obj, &overflow); + if (overflow || result > INT_MAX || result < INT_MIN) { + /* XXX: could be cute and give a different + message for overflow == -1 */ + PyErr_SetString(PyExc_OverflowError, + "Python int too large to convert to C int"); + return -1; + } + return (int)result; +} + /* Get a Py_ssize_t from a long int object. Returns -1 and sets an error condition if overflow occurs. */ diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13521,7 +13521,7 @@ "* wants int"); return -1; } - arg->width = PyLong_AsLong(v); + arg->width = PyLong_AsSsize_t(v); if (arg->width == -1 && PyErr_Occurred()) return -1; if (arg->width < 0) { @@ -13568,7 +13568,7 @@ "* wants int"); return -1; } - arg->prec = PyLong_AsLong(v); + arg->prec = _PyLong_AsInt(v); if (arg->prec == -1 && PyErr_Occurred()) return -1; if (arg->prec < 0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 01:50:13 2013 From: python-checkins at python.org (richard.oudkerk) Date: Tue, 15 Jan 2013 01:50:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEwNTI3?= =?utf-8?q?=3A_Use_poll=28=29_instead_of_select=28=29_for_multiprocessing_?= =?utf-8?q?pipes?= Message-ID: <3YlXyY1fNnzRyf@mail.python.org> http://hg.python.org/cpython/rev/da5e520a7ba5 changeset: 81506:da5e520a7ba5 branch: 2.7 parent: 81501:ce869b05762c user: Richard Oudkerk date: Mon Jan 14 23:09:14 2013 +0000 summary: Issue #10527: Use poll() instead of select() for multiprocessing pipes files: Misc/NEWS | 2 + Modules/_multiprocessing/socket_connection.c | 33 ++++++++++ 2 files changed, 35 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -186,6 +186,8 @@ Library ------- +- Issue #10527: Use poll() instead of select() for multiprocessing pipes. + - Issue #9720: zipfile now writes correct local headers for files larger than 4 GiB. diff --git a/Modules/_multiprocessing/socket_connection.c b/Modules/_multiprocessing/socket_connection.c --- a/Modules/_multiprocessing/socket_connection.c +++ b/Modules/_multiprocessing/socket_connection.c @@ -8,6 +8,10 @@ #include "multiprocessing.h" +#if defined(HAVE_POLL) && !defined(HAVE_BROKEN_POLL) +# include "poll.h" +#endif + #ifdef MS_WINDOWS # define WRITE(h, buffer, length) send((SOCKET)h, buffer, length, 0) # define READ(h, buffer, length) recv((SOCKET)h, buffer, length, 0) @@ -158,6 +162,34 @@ static int conn_poll(ConnectionObject *conn, double timeout, PyThreadState *_save) { +#if defined(HAVE_POLL) && !defined(HAVE_BROKEN_POLL) + int res; + struct pollfd p; + + p.fd = (int)conn->handle; + p.events = POLLIN | POLLPRI; + p.revents = 0; + + if (timeout < 0) { + res = poll(&p, 1, -1); + } else { + res = poll(&p, 1, (int)(timeout * 1000 + 0.5)); + } + + if (res < 0) { + return MP_SOCKET_ERROR; + } else if (p.revents & (POLLNVAL|POLLERR)) { + Py_BLOCK_THREADS + PyErr_SetString(PyExc_IOError, "poll() gave POLLNVAL or POLLERR"); + Py_UNBLOCK_THREADS + return MP_EXCEPTION_HAS_BEEN_SET; + } else if (p.revents != 0) { + return TRUE; + } else { + assert(res == 0); + return FALSE; + } +#else int res; fd_set rfds; @@ -193,6 +225,7 @@ assert(res == 0); return FALSE; } +#endif } /* -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 02:08:30 2013 From: python-checkins at python.org (richard.oudkerk) Date: Tue, 15 Jan 2013 02:08:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEwNTI3?= =?utf-8?q?=3A_Use_poll=28=29_instead_of_select=28=29_for_multiprocessing_?= =?utf-8?q?pipes?= Message-ID: <3YlYMf34nxzS0r@mail.python.org> http://hg.python.org/cpython/rev/abf111b9a464 changeset: 81507:abf111b9a464 branch: 3.2 parent: 81502:b93848ca7760 user: Richard Oudkerk date: Tue Jan 15 01:01:01 2013 +0000 summary: Issue #10527: Use poll() instead of select() for multiprocessing pipes files: Misc/NEWS | 2 + Modules/_multiprocessing/socket_connection.c | 33 ++++++++++ 2 files changed, 35 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -199,6 +199,8 @@ Library ------- +- Issue #10527: Use poll() instead of select() for multiprocessing pipes. + - Issue #9720: zipfile now writes correct local headers for files larger than 4 GiB. diff --git a/Modules/_multiprocessing/socket_connection.c b/Modules/_multiprocessing/socket_connection.c --- a/Modules/_multiprocessing/socket_connection.c +++ b/Modules/_multiprocessing/socket_connection.c @@ -8,6 +8,10 @@ #include "multiprocessing.h" +#if defined(HAVE_POLL) && !defined(HAVE_BROKEN_POLL) +# include "poll.h" +#endif + #ifdef MS_WINDOWS # define WRITE(h, buffer, length) send((SOCKET)h, buffer, length, 0) # define READ(h, buffer, length) recv((SOCKET)h, buffer, length, 0) @@ -158,6 +162,34 @@ static int conn_poll(ConnectionObject *conn, double timeout, PyThreadState *_save) { +#if defined(HAVE_POLL) && !defined(HAVE_BROKEN_POLL) + int res; + struct pollfd p; + + p.fd = (int)conn->handle; + p.events = POLLIN | POLLPRI; + p.revents = 0; + + if (timeout < 0) { + res = poll(&p, 1, -1); + } else { + res = poll(&p, 1, (int)(timeout * 1000 + 0.5)); + } + + if (res < 0) { + return MP_SOCKET_ERROR; + } else if (p.revents & (POLLNVAL|POLLERR)) { + Py_BLOCK_THREADS + PyErr_SetString(PyExc_IOError, "poll() gave POLLNVAL or POLLERR"); + Py_UNBLOCK_THREADS + return MP_EXCEPTION_HAS_BEEN_SET; + } else if (p.revents != 0) { + return TRUE; + } else { + assert(res == 0); + return FALSE; + } +#else int res; fd_set rfds; @@ -193,6 +225,7 @@ assert(res == 0); return FALSE; } +#endif } /* -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 02:08:31 2013 From: python-checkins at python.org (richard.oudkerk) Date: Tue, 15 Jan 2013 02:08:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Dummy_merge?= Message-ID: <3YlYMg5nNTzS4w@mail.python.org> http://hg.python.org/cpython/rev/27600ad15931 changeset: 81508:27600ad15931 branch: 3.3 parent: 81503:656a45738e5e parent: 81507:abf111b9a464 user: Richard Oudkerk date: Tue Jan 15 01:04:03 2013 +0000 summary: Dummy merge files: Misc/NEWS | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -147,6 +147,8 @@ Library ------- +- Issue #10527: Use poll() instead of select() for multiprocessing pipes. + - Issue #9720: zipfile now writes correct local headers for files larger than 4 GiB. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 02:08:33 2013 From: python-checkins at python.org (richard.oudkerk) Date: Tue, 15 Jan 2013 02:08:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge?= Message-ID: <3YlYMj2YjbzS5X@mail.python.org> http://hg.python.org/cpython/rev/de719ed20c74 changeset: 81509:de719ed20c74 parent: 81505:13e2e44db99d parent: 81508:27600ad15931 user: Richard Oudkerk date: Tue Jan 15 01:06:53 2013 +0000 summary: Merge files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 02:10:49 2013 From: python-checkins at python.org (guido.van.rossum) Date: Tue, 15 Jan 2013 02:10:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Updates_for_cancelling_handle?= =?utf-8?q?rs_and_remove=5Freader/writer/connector=2E?= Message-ID: <3YlYQK5cPZzS0r@mail.python.org> http://hg.python.org/peps/rev/a2d0814e9b9c changeset: 4675:a2d0814e9b9c user: Guido van Rossum date: Mon Jan 14 17:10:45 2013 -0800 summary: Updates for cancelling handlers and remove_reader/writer/connector. files: pep-3156.txt | 11 ++++++----- 1 files changed, 6 insertions(+), 5 deletions(-) diff --git a/pep-3156.txt b/pep-3156.txt --- a/pep-3156.txt +++ b/pep-3156.txt @@ -371,9 +371,10 @@ used to cancel the callback. Note that, unlike ``call_later()``, the callback may be called many times. Calling ``add_reader()`` again for the same file descriptor implicitly cancels the previous - callback for that file descriptor. (TBD: Returning a - ``Handler`` that can be cancelled seems awkward. Let's forget - about that.) (TBD: Change this to raise an exception if a handler + callback for that file descriptor. Note: cancelling the handler + may be delayed until the handler would be called. If you plan to + close ``fd``, you should use ``remove_reader(fd)`` instead. + (TBD: Change this to raise an exception if a handler is already set.) - ``add_writer(fd, callback, *args)``. Like ``add_reader()``, @@ -384,8 +385,8 @@ currently set for the file descriptor. (The reason for providing this alternate interface is that it is often more convenient to remember the file descriptor than to remember the ``Handler`` - object.) (TBD: Return ``True`` if a handler was removed, ``False`` - if not.) + object.) Returns ``True`` if a handler was removed, ``False`` + if not. - ``remove_writer(fd)``. This is to ``add_writer()`` as ``remove_reader()`` is to ``add_reader()``. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue Jan 15 03:44:51 2013 From: python-checkins at python.org (daniel.holth) Date: Tue, 15 Jan 2013 03:44:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_add_Obsoleted-By?= Message-ID: <3YlbVq1phQzRxn@mail.python.org> http://hg.python.org/peps/rev/2eede0bf8fcb changeset: 4676:2eede0bf8fcb user: Daniel Holth date: Mon Jan 14 21:44:40 2013 -0500 summary: add Obsoleted-By files: pep-0426.txt | 37 ++++++++++++++++++------------------- 1 files changed, 18 insertions(+), 19 deletions(-) diff --git a/pep-0426.txt b/pep-0426.txt --- a/pep-0426.txt +++ b/pep-0426.txt @@ -24,11 +24,10 @@ Version 1.2 is specified in PEP 345. Version 1.3 of the metadata format adds fields designed to make -third-party packaging of Python Software easier and defines a -formal extension mechanism. The fields are "Setup-Requires-Dist" -"Provides-Extra", and "Extension". This version also adds the `extra` -variable to the `environment markers` specification and allows the -description to be placed into a payload section. +third-party packaging of Python Software easier and defines a formal +extension mechanism. This version also adds the `extra` variable to the +`environment markers` specification and allows the description to be +placed into a payload section. Metadata Files ============== @@ -354,24 +353,23 @@ Provides-Dist: virtual_package -Obsoletes-Dist (multiple use) -::::::::::::::::::::::::::::: +Obsoleted-By (optional) +::::::::::::::::::::::: -Each entry contains a string describing a distutils project's distribution -which this distribution renders obsolete, meaning that the two projects -should not be installed at the same time. +Indicates that this project is no longer being developed. The named +project provides a substitute or replacement. -Version declarations can be supplied. Version numbers must be in the -format specified in `Version Specifiers`_. +A version declaration may be supplied and must follow the rules described +in `Version Specifiers`_. -The most common use of this field will be in case a project name changes, -e.g. Gorgon 2.3 gets renamed to Torqued Python 1.0. When you install -Torqued Python, the Gorgon distribution should be removed. +The most common use of this field will be in case a project name changes. Examples:: - Obsoletes-Dist: Gorgon - Obsoletes-Dist: OtherProject (<3.0) + Name: BadName + Obsoleted-By: AcceptableName + + Obsoleted-By: AcceptableName (>=4.0.0) Requires-Python (optional) @@ -554,7 +552,6 @@ Here are some example of fields using such markers:: Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' - Obsoletes-Dist: pywin31; sys.platform == 'win32' Requires-Dist: foo (1,!=1.3); platform.machine == 'i386' Requires-Dist: bar; python_version == '2.4' or python_version == '2.5' Requires-External: libxslt; 'linux' in sys.platform @@ -590,7 +587,6 @@ - Requires-Dist - Setup-Requires-Dist - Provides-Dist -- Obsoletes-Dist - Classifier (The `extra` variable is only meaningful for Requires-Dist.) @@ -608,6 +604,8 @@ * Most fields are now optional. +* Removed `Obsoletes-Dist` + * Changed fields: - Description @@ -619,6 +617,7 @@ - Extension - Provides-Extra - Setup-Requires-Dist + - Obsoleted-By References ========== -- Repository URL: http://hg.python.org/peps From solipsis at pitrou.net Tue Jan 15 05:58:14 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 15 Jan 2013 05:58:14 +0100 Subject: [Python-checkins] Daily reference leaks (de719ed20c74): sum=-2 Message-ID: results for de719ed20c74 on branch "default" -------------------------------------------- test_concurrent_futures leaked [0, 0, -2] memory blocks, sum=-2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogRsRnNa', '-x'] From python-checkins at python.org Tue Jan 15 10:09:34 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 15 Jan 2013 10:09:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_test=5Fsocket_broken_i?= =?utf-8?q?n_previous_commit_=28changeset_13e2e44db99d=29=2E?= Message-ID: <3Ylm2k75QczP9T@mail.python.org> http://hg.python.org/cpython/rev/525407d89277 changeset: 81510:525407d89277 user: Serhiy Storchaka date: Tue Jan 15 11:08:30 2013 +0200 summary: Fix test_socket broken in previous commit (changeset 13e2e44db99d). Added new checks for socket.setblocking(). files: Lib/test/test_socket.py | 12 +++++++----- 1 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py --- a/Lib/test/test_socket.py +++ b/Lib/test/test_socket.py @@ -3566,7 +3566,10 @@ def testSetBlocking(self): # Testing whether set blocking works - self.serv.setblocking(0) + self.serv.setblocking(True) + self.assertIsNone(self.serv.gettimeout()) + self.serv.setblocking(False) + self.assertEqual(self.serv.gettimeout(), 0.0) start = time.time() try: self.serv.accept() @@ -3575,10 +3578,9 @@ end = time.time() self.assertTrue((end - start) < 1.0, "Error setting non-blocking mode.") # Issue 15989 - self.assertRaises(OverflowError, self.serv.setblocking, - _testcapi.INT_MAX + 1) - self.assertRaises(OverflowError, self.serv.setblocking, - _testcapi.UINT_MAX + 1) + if _testcapi.UINT_MAX < _testcapi.ULONG_MAX: + self.serv.setblocking(_testcapi.UINT_MAX + 1) + self.assertIsNone(self.serv.gettimeout()) def _testSetBlocking(self): pass -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 12:33:35 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 15 Jan 2013 12:33:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Check_for_NULL?= =?utf-8?q?_before_the_pointer_aligning_in_fastsearch=5Fmemchr=5F1char=2E?= Message-ID: <3YlqDv24RkzN4d@mail.python.org> http://hg.python.org/cpython/rev/ad9b5c69b8b6 changeset: 81511:ad9b5c69b8b6 branch: 3.3 parent: 81508:27600ad15931 user: Serhiy Storchaka date: Tue Jan 15 13:27:28 2013 +0200 summary: Check for NULL before the pointer aligning in fastsearch_memchr_1char. There is no guarantee that NULL is aligned. files: Objects/stringlib/fastsearch.h | 25 ++++++++------------- 1 files changed, 10 insertions(+), 15 deletions(-) diff --git a/Objects/stringlib/fastsearch.h b/Objects/stringlib/fastsearch.h --- a/Objects/stringlib/fastsearch.h +++ b/Objects/stringlib/fastsearch.h @@ -38,25 +38,18 @@ STRINGLIB_CHAR ch, unsigned char needle, Py_ssize_t maxcount, int mode) { - void *candidate; - const STRINGLIB_CHAR *found; - -#define DO_MEMCHR(memchr, s, needle, nchars) do { \ - candidate = memchr((const void *) (s), (needle), (nchars) * sizeof(STRINGLIB_CHAR)); \ - found = (const STRINGLIB_CHAR *) _Py_ALIGN_DOWN(candidate, sizeof(STRINGLIB_CHAR)); \ - } while (0) - if (mode == FAST_SEARCH) { const STRINGLIB_CHAR *ptr = s; const STRINGLIB_CHAR *e = s + n; while (ptr < e) { - DO_MEMCHR(memchr, ptr, needle, e - ptr); - if (found == NULL) + void *candidate = memchr((const void *) ptr, needle, (e - ptr) * sizeof(STRINGLIB_CHAR)); + if (candidate == NULL) return -1; - if (sizeof(STRINGLIB_CHAR) == 1 || *found == ch) - return (found - s); + ptr = (const STRINGLIB_CHAR *) _Py_ALIGN_DOWN(candidate, sizeof(STRINGLIB_CHAR)); + if (sizeof(STRINGLIB_CHAR) == 1 || *ptr == ch) + return (ptr - s); /* False positive */ - ptr = found + 1; + ptr++; } return -1; } @@ -66,9 +59,11 @@ faster than our hand-written loop in FASTSEARCH below */ else if (mode == FAST_RSEARCH) { while (n > 0) { - DO_MEMCHR(memrchr, s, needle, n); - if (found == NULL) + const STRINGLIB_CHAR *found; + void *candidate = memrchr((const void *) s, needle, n * sizeof(STRINGLIB_CHAR)); + if (candidate == NULL) return -1; + found = (const STRINGLIB_CHAR *) _Py_ALIGN_DOWN(candidate, sizeof(STRINGLIB_CHAR)); n = found - s; if (sizeof(STRINGLIB_CHAR) == 1 || *found == ch) return n; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 12:33:36 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 15 Jan 2013 12:33:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Check_for_NULL_before_the_pointer_aligning_in_fastsearch?= =?utf-8?q?=5Fmemchr=5F1char=2E?= Message-ID: <3YlqDw53cXzP9m@mail.python.org> http://hg.python.org/cpython/rev/1f66fc397c8d changeset: 81512:1f66fc397c8d parent: 81510:525407d89277 parent: 81511:ad9b5c69b8b6 user: Serhiy Storchaka date: Tue Jan 15 13:32:41 2013 +0200 summary: Check for NULL before the pointer aligning in fastsearch_memchr_1char. There is no guarantee that NULL is aligned. files: Objects/stringlib/fastsearch.h | 25 ++++++++------------- 1 files changed, 10 insertions(+), 15 deletions(-) diff --git a/Objects/stringlib/fastsearch.h b/Objects/stringlib/fastsearch.h --- a/Objects/stringlib/fastsearch.h +++ b/Objects/stringlib/fastsearch.h @@ -38,25 +38,18 @@ STRINGLIB_CHAR ch, unsigned char needle, Py_ssize_t maxcount, int mode) { - void *candidate; - const STRINGLIB_CHAR *found; - -#define DO_MEMCHR(memchr, s, needle, nchars) do { \ - candidate = memchr((const void *) (s), (needle), (nchars) * sizeof(STRINGLIB_CHAR)); \ - found = (const STRINGLIB_CHAR *) _Py_ALIGN_DOWN(candidate, sizeof(STRINGLIB_CHAR)); \ - } while (0) - if (mode == FAST_SEARCH) { const STRINGLIB_CHAR *ptr = s; const STRINGLIB_CHAR *e = s + n; while (ptr < e) { - DO_MEMCHR(memchr, ptr, needle, e - ptr); - if (found == NULL) + void *candidate = memchr((const void *) ptr, needle, (e - ptr) * sizeof(STRINGLIB_CHAR)); + if (candidate == NULL) return -1; - if (sizeof(STRINGLIB_CHAR) == 1 || *found == ch) - return (found - s); + ptr = (const STRINGLIB_CHAR *) _Py_ALIGN_DOWN(candidate, sizeof(STRINGLIB_CHAR)); + if (sizeof(STRINGLIB_CHAR) == 1 || *ptr == ch) + return (ptr - s); /* False positive */ - ptr = found + 1; + ptr++; } return -1; } @@ -66,9 +59,11 @@ faster than our hand-written loop in FASTSEARCH below */ else if (mode == FAST_RSEARCH) { while (n > 0) { - DO_MEMCHR(memrchr, s, needle, n); - if (found == NULL) + const STRINGLIB_CHAR *found; + void *candidate = memrchr((const void *) s, needle, n * sizeof(STRINGLIB_CHAR)); + if (candidate == NULL) return -1; + found = (const STRINGLIB_CHAR *) _Py_ALIGN_DOWN(candidate, sizeof(STRINGLIB_CHAR)); n = found - s; if (sizeof(STRINGLIB_CHAR) == 1 || *found == ch) return n; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 13:29:23 2013 From: python-checkins at python.org (nick.coghlan) Date: Tue, 15 Jan 2013 13:29:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_432=3A_Move_the_descripti?= =?utf-8?q?on_of_the_status_quo_to_the_end?= Message-ID: <3YlrTH2sZXzPCZ@mail.python.org> http://hg.python.org/peps/rev/316e42e482d6 changeset: 4677:316e42e482d6 user: Nick Coghlan date: Tue Jan 15 21:50:35 2013 +1000 summary: PEP 432: Move the description of the status quo to the end files: pep-0432.txt | 490 +++++++++++++++++++------------------- 1 files changed, 245 insertions(+), 245 deletions(-) diff --git a/pep-0432.txt b/pep-0432.txt --- a/pep-0432.txt +++ b/pep-0432.txt @@ -242,250 +242,6 @@ deliberately avoids adding any new settings of its own. -The Status Quo -============== - -The current mechanisms for configuring the interpreter have accumulated in -a fairly ad hoc fashion over the past 20+ years, leading to a rather -inconsistent interface with varying levels of documentation. - -(Note: some of the info below could probably be cleaned up and added to the -C API documentation - it's all CPython specific, so it doesn't belong in -the language reference) - - -Ignoring Environment Variables ------------------------------- - -The ``-E`` command line option allows all environment variables to be -ignored when initializing the Python interpreter. An embedding application -can enable this behaviour by setting ``Py_IgnoreEnvironmentFlag`` before -calling ``Py_Initialize()``. - -In the CPython source code, the ``Py_GETENV`` macro implicitly checks this -flag, and always produces ``NULL`` if it is set. - - - - - -Randomised Hashing ------------------- - -The randomised hashing is controlled via the ``-R`` command line option (in -releases prior to 3.3), as well as the ``PYTHONHASHSEED`` environment -variable. - -In Python 3.3, only the environment variable remains relevant. It can be -used to disable randomised hashing (by using a seed value of 0) or else -to force a specific hash value (e.g. for repeatability of testing, or -to share hash values between processes) - -However, embedding applications must use the ``Py_HashRandomizationFlag`` -to explicitly request hash randomisation (CPython sets it in ``Py_Main()`` -rather than in ``Py_Initialize()``). - -The new configuration API should make it straightforward for an -embedding application to reuse the ``PYTHONHASHSEED`` processing with -a text based configuration setting provided by other means (e.g. a -config file or separate environment variable). - - -Locating Python and the standard library ----------------------------------------- - -The location of the Python binary and the standard library is influenced -by several elements. The algorithm used to perform the calculation is -not documented anywhere other than in the source code [3_,4_]. Even that -description is incomplete, as it failed to be updated for the virtual -environment support added in Python 3.3 (detailed in PEP 405). - -These calculations are affected by the following function calls (made -prior to calling ``Py_Initialize()``) and environment variables: - -* ``Py_SetProgramName()`` -* ``Py_SetPythonHome()`` -* ``PYTHONHOME`` - -The filesystem is also inspected for ``pyvenv.cfg`` files (see PEP 405) or, -failing that, a ``lib/os.py`` (Windows) or ``lib/python$VERSION/os.py`` -file. - -The build time settings for ``PREFIX`` and ``EXEC_PREFIX`` are also relevant, -as are some registry settings on Windows. The hardcoded fallbacks are -based on the layout of the CPython source tree and build output when -working in a source checkout. - - -Configuring ``sys.path`` ------------------------- - -An embedding application may call ``Py_SetPath()`` prior to -``Py_Initialize()`` to completely override the calculation of -``sys.path``. It is not straightforward to only allow *some* of the -calculations, as modifying ``sys.path`` after initialization is -already complete means those modifications will not be in effect -when standard library modules are imported during the startup sequence. - -If ``Py_SetPath()`` is not used prior to the first call to ``Py_GetPath()`` -(implicit in ``Py_Initialize()``), then it builds on the location data -calculations above to calculate suitable path entries, along with -the ``PYTHONPATH`` environment variable. - - - -The ``site`` module, which is implicitly imported at startup (unless -disabled via the ``-S`` option) adds additional paths to this initial -set of paths, as described in its documentation [5_]. - -The ``-s`` command line option can be used to exclude the user site -directory from the list of directories added. Embedding applications -can control this by setting the ``Py_NoUserSiteDirectory`` global variable. - -The following commands can be used to check the default path configurations -for a given Python executable on a given system: - -* ``./python -c "import sys, pprint; pprint.pprint(sys.path)"`` - - standard configuration -* ``./python -s -c "import sys, pprint; pprint.pprint(sys.path)"`` - - user site directory disabled -* ``./python -S -c "import sys, pprint; pprint.pprint(sys.path)"`` - - all site path modifications disabled - -(Note: you can see similar information using ``-m site`` instead of ``-c``, -but this is slightly misleading as it calls ``os.abspath`` on all of the -path entries, making relative path entries look absolute. Using the ``site`` -module also causes problems in the last case, as on Python versions prior to -3.3, explicitly importing site will carry out the path modifications ``-S`` -avoids, while on 3.3+ combining ``-m site`` with ``-S`` currently fails) - -The calculation of ``sys.path[0]`` is comparatively straightforward: - -* For an ordinary script (Python source or compiled bytecode), - ``sys.path[0]`` will be the directory containing the script. -* For a valid ``sys.path`` entry (typically a zipfile or directory), - ``sys.path[0]`` will be that path -* For an interactive session, running from stdin or when using the ``-c`` or - ``-m`` switches, ``sys.path[0]`` will be the empty string, which the import - system interprets as allowing imports from the current directory - - -Configuring ``sys.argv`` ------------------------- - -Unlike most other settings discussed in this PEP, ``sys.argv`` is not -set implicitly by ``Py_Initialize()``. Instead, it must be set via an -explicitly call to ``Py_SetArgv()``. - -CPython calls this in ``Py_Main()`` after calling ``Py_Initialize()``. The -calculation of ``sys.argv[1:]`` is straightforward: they're the command line -arguments passed after the script name or the argument to the ``-c`` or -``-m`` options. - -The calculation of ``sys.argv[0]`` is a little more complicated: - -* For an ordinary script (source or bytecode), it will be the script name -* For a ``sys.path`` entry (typically a zipfile or directory) it will - initially be the zipfile or directory name, but will later be changed by - the ``runpy`` module to the full path to the imported ``__main__`` module. -* For a module specified with the ``-m`` switch, it will initially be the - string ``"-m"``, but will later be changed by the ``runpy`` module to the - full path to the executed module. -* For a package specified with the ``-m`` switch, it will initially be the - string ``"-m"``, but will later be changed by the ``runpy`` module to the - full path to the executed ``__main__`` submodule of the package. -* For a command executed with ``-c``, it will be the string ``"-c"`` -* For explicitly requested input from stdin, it will be the string ``"-"`` -* Otherwise, it will be the empty string - -Embedding applications must call Py_SetArgv themselves. The CPython logic -for doing so is part of ``Py_Main()`` and is not exposed separately. -However, the ``runpy`` module does provide roughly equivalent logic in -``runpy.run_module`` and ``runpy.run_path``. - - - -Other configuration settings ----------------------------- - -TBD: Cover the initialization of the following in more detail: - -* Completely disabling the import system -* The initial warning system state: - * ``sys.warnoptions`` - * (-W option, PYTHONWARNINGS) -* Arbitrary extended options (e.g. to automatically enable ``faulthandler``): - * ``sys._xoptions`` - * (-X option) -* The filesystem encoding used by: - * ``sys.getfsencoding`` - * ``os.fsencode`` - * ``os.fsdecode`` -* The IO encoding and buffering used by: - * ``sys.stdin`` - * ``sys.stdout`` - * ``sys.stderr`` - * (-u option, PYTHONIOENCODING, PYTHONUNBUFFEREDIO) -* Whether or not to implicitly cache bytecode files: - * ``sys.dont_write_bytecode`` - * (-B option, PYTHONDONTWRITEBYTECODE) -* Whether or not to enforce correct case in filenames on case-insensitive - platforms - * ``os.environ["PYTHONCASEOK"]`` -* The other settings exposed to Python code in ``sys.flags``: - - * ``debug`` (Enable debugging output in the pgen parser) - * ``inspect`` (Enter interactive interpreter after __main__ terminates) - * ``interactive`` (Treat stdin as a tty) - * ``optimize`` (__debug__ status, write .pyc or .pyo, strip doc strings) - * ``no_user_site`` (don't add the user site directory to sys.path) - * ``no_site`` (don't implicitly import site during startup) - * ``ignore_environment`` (whether environment vars are used during config) - * ``verbose`` (enable all sorts of random output) - * ``bytes_warning`` (warnings/errors for implicit str/bytes interaction) - * ``quiet`` (disable banner output even if verbose is also enabled or - stdin is a tty and the interpreter is launched in interactive mode) - -* Whether or not CPython's signal handlers should be installed - -Much of the configuration of CPython is currently handled through C level -global variables:: - - Py_BytesWarningFlag (-b) - Py_DebugFlag (-d option) - Py_InspectFlag (-i option, PYTHONINSPECT) - Py_InteractiveFlag (property of stdin, cannot be overridden) - Py_OptimizeFlag (-O option, PYTHONOPTIMIZE) - Py_DontWriteBytecodeFlag (-B option, PYTHONDONTWRITEBYTECODE) - Py_NoUserSiteDirectory (-s option, PYTHONNOUSERSITE) - Py_NoSiteFlag (-S option) - Py_UnbufferedStdioFlag (-u, PYTHONUNBUFFEREDIO) - Py_VerboseFlag (-v option, PYTHONVERBOSE) - -For the above variables, the conversion of command line options and -environment variables to C global variables is handled by ``Py_Main``, -so each embedding application must set those appropriately in order to -change them from their defaults. - -Some configuration can only be provided as OS level environment variables:: - - PYTHONSTARTUP - PYTHONCASEOK - PYTHONIOENCODING - -The ``Py_InitializeEx()`` API also accepts a boolean flag to indicate -whether or not CPython's signal handlers should be installed. - -Finally, some interactive behaviour (such as printing the introductory -banner) is triggered only when standard input is reported as a terminal -connection by the operating system. - -TBD: Document how the "-x" option is handled (skips processing of the -first comment line in the main script) - -Also see detailed sequence of operations notes at [1_] - - Design Details ============== @@ -623,7 +379,7 @@ #define Py_CoreConfig_INIT {0, -1, 0, 0} The core configuration settings pointer may be ``NULL``, in which case the -default values are ``ignore_environment = 0`` and ``use_hash_seed = -1``. +default values are ``ignore_environment = -1`` and ``use_hash_seed = -1``. The ``Py_CoreConfig_INIT`` macro is designed to allow easy initialization of a struct instance with sensible defaults:: @@ -1189,6 +945,250 @@ directory. +The Status Quo +============== + +The current mechanisms for configuring the interpreter have accumulated in +a fairly ad hoc fashion over the past 20+ years, leading to a rather +inconsistent interface with varying levels of documentation. + +(Note: some of the info below could probably be cleaned up and added to the +C API documentation for at least 3.3. - it's all CPython specific, so it +doesn't belong in the language reference) + + +Ignoring Environment Variables +------------------------------ + +The ``-E`` command line option allows all environment variables to be +ignored when initializing the Python interpreter. An embedding application +can enable this behaviour by setting ``Py_IgnoreEnvironmentFlag`` before +calling ``Py_Initialize()``. + +In the CPython source code, the ``Py_GETENV`` macro implicitly checks this +flag, and always produces ``NULL`` if it is set. + + + + + +Randomised Hashing +------------------ + +The randomised hashing is controlled via the ``-R`` command line option (in +releases prior to 3.3), as well as the ``PYTHONHASHSEED`` environment +variable. + +In Python 3.3, only the environment variable remains relevant. It can be +used to disable randomised hashing (by using a seed value of 0) or else +to force a specific hash value (e.g. for repeatability of testing, or +to share hash values between processes) + +However, embedding applications must use the ``Py_HashRandomizationFlag`` +to explicitly request hash randomisation (CPython sets it in ``Py_Main()`` +rather than in ``Py_Initialize()``). + +The new configuration API should make it straightforward for an +embedding application to reuse the ``PYTHONHASHSEED`` processing with +a text based configuration setting provided by other means (e.g. a +config file or separate environment variable). + + +Locating Python and the standard library +---------------------------------------- + +The location of the Python binary and the standard library is influenced +by several elements. The algorithm used to perform the calculation is +not documented anywhere other than in the source code [3_,4_]. Even that +description is incomplete, as it failed to be updated for the virtual +environment support added in Python 3.3 (detailed in PEP 405). + +These calculations are affected by the following function calls (made +prior to calling ``Py_Initialize()``) and environment variables: + +* ``Py_SetProgramName()`` +* ``Py_SetPythonHome()`` +* ``PYTHONHOME`` + +The filesystem is also inspected for ``pyvenv.cfg`` files (see PEP 405) or, +failing that, a ``lib/os.py`` (Windows) or ``lib/python$VERSION/os.py`` +file. + +The build time settings for ``PREFIX`` and ``EXEC_PREFIX`` are also relevant, +as are some registry settings on Windows. The hardcoded fallbacks are +based on the layout of the CPython source tree and build output when +working in a source checkout. + + +Configuring ``sys.path`` +------------------------ + +An embedding application may call ``Py_SetPath()`` prior to +``Py_Initialize()`` to completely override the calculation of +``sys.path``. It is not straightforward to only allow *some* of the +calculations, as modifying ``sys.path`` after initialization is +already complete means those modifications will not be in effect +when standard library modules are imported during the startup sequence. + +If ``Py_SetPath()`` is not used prior to the first call to ``Py_GetPath()`` +(implicit in ``Py_Initialize()``), then it builds on the location data +calculations above to calculate suitable path entries, along with +the ``PYTHONPATH`` environment variable. + + + +The ``site`` module, which is implicitly imported at startup (unless +disabled via the ``-S`` option) adds additional paths to this initial +set of paths, as described in its documentation [5_]. + +The ``-s`` command line option can be used to exclude the user site +directory from the list of directories added. Embedding applications +can control this by setting the ``Py_NoUserSiteDirectory`` global variable. + +The following commands can be used to check the default path configurations +for a given Python executable on a given system: + +* ``./python -c "import sys, pprint; pprint.pprint(sys.path)"`` + - standard configuration +* ``./python -s -c "import sys, pprint; pprint.pprint(sys.path)"`` + - user site directory disabled +* ``./python -S -c "import sys, pprint; pprint.pprint(sys.path)"`` + - all site path modifications disabled + +(Note: you can see similar information using ``-m site`` instead of ``-c``, +but this is slightly misleading as it calls ``os.abspath`` on all of the +path entries, making relative path entries look absolute. Using the ``site`` +module also causes problems in the last case, as on Python versions prior to +3.3, explicitly importing site will carry out the path modifications ``-S`` +avoids, while on 3.3+ combining ``-m site`` with ``-S`` currently fails) + +The calculation of ``sys.path[0]`` is comparatively straightforward: + +* For an ordinary script (Python source or compiled bytecode), + ``sys.path[0]`` will be the directory containing the script. +* For a valid ``sys.path`` entry (typically a zipfile or directory), + ``sys.path[0]`` will be that path +* For an interactive session, running from stdin or when using the ``-c`` or + ``-m`` switches, ``sys.path[0]`` will be the empty string, which the import + system interprets as allowing imports from the current directory + + +Configuring ``sys.argv`` +------------------------ + +Unlike most other settings discussed in this PEP, ``sys.argv`` is not +set implicitly by ``Py_Initialize()``. Instead, it must be set via an +explicitly call to ``Py_SetArgv()``. + +CPython calls this in ``Py_Main()`` after calling ``Py_Initialize()``. The +calculation of ``sys.argv[1:]`` is straightforward: they're the command line +arguments passed after the script name or the argument to the ``-c`` or +``-m`` options. + +The calculation of ``sys.argv[0]`` is a little more complicated: + +* For an ordinary script (source or bytecode), it will be the script name +* For a ``sys.path`` entry (typically a zipfile or directory) it will + initially be the zipfile or directory name, but will later be changed by + the ``runpy`` module to the full path to the imported ``__main__`` module. +* For a module specified with the ``-m`` switch, it will initially be the + string ``"-m"``, but will later be changed by the ``runpy`` module to the + full path to the executed module. +* For a package specified with the ``-m`` switch, it will initially be the + string ``"-m"``, but will later be changed by the ``runpy`` module to the + full path to the executed ``__main__`` submodule of the package. +* For a command executed with ``-c``, it will be the string ``"-c"`` +* For explicitly requested input from stdin, it will be the string ``"-"`` +* Otherwise, it will be the empty string + +Embedding applications must call Py_SetArgv themselves. The CPython logic +for doing so is part of ``Py_Main()`` and is not exposed separately. +However, the ``runpy`` module does provide roughly equivalent logic in +``runpy.run_module`` and ``runpy.run_path``. + + + +Other configuration settings +---------------------------- + +TBD: Cover the initialization of the following in more detail: + +* Completely disabling the import system +* The initial warning system state: + * ``sys.warnoptions`` + * (-W option, PYTHONWARNINGS) +* Arbitrary extended options (e.g. to automatically enable ``faulthandler``): + * ``sys._xoptions`` + * (-X option) +* The filesystem encoding used by: + * ``sys.getfsencoding`` + * ``os.fsencode`` + * ``os.fsdecode`` +* The IO encoding and buffering used by: + * ``sys.stdin`` + * ``sys.stdout`` + * ``sys.stderr`` + * (-u option, PYTHONIOENCODING, PYTHONUNBUFFEREDIO) +* Whether or not to implicitly cache bytecode files: + * ``sys.dont_write_bytecode`` + * (-B option, PYTHONDONTWRITEBYTECODE) +* Whether or not to enforce correct case in filenames on case-insensitive + platforms + * ``os.environ["PYTHONCASEOK"]`` +* The other settings exposed to Python code in ``sys.flags``: + + * ``debug`` (Enable debugging output in the pgen parser) + * ``inspect`` (Enter interactive interpreter after __main__ terminates) + * ``interactive`` (Treat stdin as a tty) + * ``optimize`` (__debug__ status, write .pyc or .pyo, strip doc strings) + * ``no_user_site`` (don't add the user site directory to sys.path) + * ``no_site`` (don't implicitly import site during startup) + * ``ignore_environment`` (whether environment vars are used during config) + * ``verbose`` (enable all sorts of random output) + * ``bytes_warning`` (warnings/errors for implicit str/bytes interaction) + * ``quiet`` (disable banner output even if verbose is also enabled or + stdin is a tty and the interpreter is launched in interactive mode) + +* Whether or not CPython's signal handlers should be installed + +Much of the configuration of CPython is currently handled through C level +global variables:: + + Py_BytesWarningFlag (-b) + Py_DebugFlag (-d option) + Py_InspectFlag (-i option, PYTHONINSPECT) + Py_InteractiveFlag (property of stdin, cannot be overridden) + Py_OptimizeFlag (-O option, PYTHONOPTIMIZE) + Py_DontWriteBytecodeFlag (-B option, PYTHONDONTWRITEBYTECODE) + Py_NoUserSiteDirectory (-s option, PYTHONNOUSERSITE) + Py_NoSiteFlag (-S option) + Py_UnbufferedStdioFlag (-u, PYTHONUNBUFFEREDIO) + Py_VerboseFlag (-v option, PYTHONVERBOSE) + +For the above variables, the conversion of command line options and +environment variables to C global variables is handled by ``Py_Main``, +so each embedding application must set those appropriately in order to +change them from their defaults. + +Some configuration can only be provided as OS level environment variables:: + + PYTHONSTARTUP + PYTHONCASEOK + PYTHONIOENCODING + +The ``Py_InitializeEx()`` API also accepts a boolean flag to indicate +whether or not CPython's signal handlers should be installed. + +Finally, some interactive behaviour (such as printing the introductory +banner) is triggered only when standard input is reported as a terminal +connection by the operating system. + +TBD: Document how the "-x" option is handled (skips processing of the +first comment line in the main script) + +Also see detailed sequence of operations notes at [1_] + + References ========== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue Jan 15 13:29:24 2013 From: python-checkins at python.org (nick.coghlan) Date: Tue, 15 Jan 2013 13:29:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_432=3A_Reduce_typing_in_p?= =?utf-8?q?roposed_API?= Message-ID: <3YlrTJ6bGvzPCZ@mail.python.org> http://hg.python.org/peps/rev/ba6561f6a514 changeset: 4678:ba6561f6a514 user: Nick Coghlan date: Tue Jan 15 22:28:45 2013 +1000 summary: PEP 432: Reduce typing in proposed API - consistently abbreviate Config - follow the Py*Object naming convention for Py*Config structs by dropping the underscore after the Py/_Py prefix files: pep-0432.txt | 97 +++++++++++++++++++++------------------ 1 files changed, 51 insertions(+), 46 deletions(-) diff --git a/pep-0432.txt b/pep-0432.txt --- a/pep-0432.txt +++ b/pep-0432.txt @@ -61,7 +61,8 @@ To keep the implementation complexity under control, this PEP does *not* propose wholesale changes to the way the interpreter state is accessed at runtime, nor does it propose changes to the way subinterpreters are -created after the main interpreter has already been initialized. Changing +created after the main interpreter has already been initialized (beyond +any changes needed to make sure they continue working as expected). Changing the order in which the existing initialization steps occur in order to make the startup sequence easier to maintain is already a substantial change, and attempting to make those other changes at the same time will make the @@ -239,7 +240,9 @@ Note that this just covers settings that are currently configurable in some manner when using the main CPython executable. While this PEP aims to make adding additional configuration settings easier in the future, it -deliberately avoids adding any new settings of its own. +deliberately avoids adding any new settings of its own (except where such +additional settings arise naturally in the course of migrating existing +settings to the new structure). Design Details @@ -280,7 +283,7 @@ * ``Py_IsInitialized()`` returns ``0`` * The embedding application determines and applies the settings required to complete the initialization process by calling - ``Py_ReadConfiguration`` and ``Py_EndInitialization``. + ``Py_ReadConfig`` and ``Py_EndInitialization``. * Initialized: @@ -318,8 +321,8 @@ over the initialization process:: /* Phase 1: Pre-Initialization */ - Py_CoreConfig core_config = Py_CoreConfig_INIT; - Py_Config config = Py_Config_INIT; + PyCoreConfig core_config = PyCoreConfig_INIT; + PyConfig config = PyConfig_INIT; /* Easily control the core configuration */ core_config.ignore_environment = 1; /* Ignore environment variables */ core_config.use_hash_seed = 0; /* Full hash randomisation */ @@ -327,13 +330,13 @@ /* Phase 2: Initialization */ /* Optionally preconfigure some settings here - they will then be * used to derive other settings */ - Py_ReadConfiguration(&config); + Py_ReadConfig(&config); /* Can completely override derived settings here */ Py_EndInitialization(&config); /* Phase 3: Initialized */ /* If an embedding application has no real concept of a main module - * it can leave the interpreter in this state indefinitely. - * Otherwise, it can launch __main__ via the Py_Run*AsMain functions. + * it can just stop the initialization process here. + * Alternatively, it can launch __main__ via the PyRun_*Main functions. */ @@ -352,11 +355,13 @@ for the seed (a seed value of zero disables randomised hashing). In addition, due to the possible use of ``PYTHONHASHSEED`` in configuring the hash randomisation, the question of whether or not to consider environment -variables must also be addressed early. +variables must also be addressed early. Finally, to support the CPython +build process, an option is offered to completely disable the import +system. The proposed API for this step in the startup sequence is:: - void Py_BeginInitialization(const Py_CoreConfig *config); + void Py_BeginInitialization(const PyCoreConfig *config); Like Py_Initialize, this part of the new API treats initialization failures as fatal errors. While that's still not particularly embedding friendly, @@ -364,27 +369,27 @@ to return error codes instead of aborting would be an even larger task than the one already being proposed. -The new ``Py_CoreConfig`` struct holds the settings required for preliminary +The new ``PyCoreConfig`` struct holds the settings required for preliminary configuration:: - /* Note: if changing anything in Py_CoreConfig, also update - * Py_CoreConfig_INIT */ + /* Note: if changing anything in PyCoreConfig, also update + * PyCoreConfig_INIT */ typedef struct { int ignore_environment; /* -E switch */ int use_hash_seed; /* PYTHONHASHSEED */ unsigned long hash_seed; /* PYTHONHASHSEED */ int _disable_importlib; /* Needed by freeze_importlib */ - } Py_CoreConfig; + } PyCoreConfig; - #define Py_CoreConfig_INIT {0, -1, 0, 0} + #define PyCoreConfig_INIT {0, -1, 0, 0} The core configuration settings pointer may be ``NULL``, in which case the default values are ``ignore_environment = -1`` and ``use_hash_seed = -1``. -The ``Py_CoreConfig_INIT`` macro is designed to allow easy initialization +The ``PyCoreConfig_INIT`` macro is designed to allow easy initialization of a struct instance with sensible defaults:: - Py_CoreConfig core_config = Py_CoreConfig_INIT; + PyCoreConfig core_config = PyCoreConfig_INIT; ``ignore_environment`` controls the processing of all Python related environment variables. If the flag is zero, then environment variables are @@ -498,7 +503,7 @@ settings needed to complete the process. No changes are made to the interpreter state at this point. The core API for this step is:: - int Py_ReadConfiguration(PyConfig *config); + int Py_ReadConfig(PyConfig *config); The config argument should be a pointer to a config struct (which may be a temporary one stored on the C stack). For any already configured value @@ -534,11 +539,11 @@ Supported configuration settings -------------------------------- -The new ``Py_Config`` struct holds the settings required to complete the +The new ``PyConfig`` struct holds the settings required to complete the interpreter configuration. All fields are either pointers to Python data types (not set == ``NULL``) or numeric flags (not set == ``-1``):: - /* Note: if changing anything in Py_Config, also update Py_Config_INIT */ + /* Note: if changing anything in PyConfig, also update PyConfig_INIT */ typedef struct { /* Argument processing */ PyListObject *raw_argv; @@ -611,30 +616,30 @@ int show_banner; /* -q switch (inverted) */ int inspect_main; /* -i switch, PYTHONINSPECT */ - } Py_Config; + } PyConfig; /* Struct initialization is pretty ugly in C89. Avoiding this mess would * be the most attractive aspect of using a PyDictObject* instead... */ - #define _Py_ArgConfig_INIT NULL, NULL, NULL, NULL - #define _Py_LocationConfig_INIT NULL, NULL, NULL, NULL, NULL, NULL - #define _Py_SiteConfig_INIT -1, -1 - #define _Py_ImportConfig_INIT -1, -1, NULL - #define _Py_StreamConfig_INIT -1, NULL, NULL, NULL, NULL, NULL, NULL - #define _Py_FilesystemConfig_INIT NULL - #define _Py_DebuggingConfig_INIT -1, -1, -1 - #define _Py_CodeGenConfig_INIT -1, -1 - #define _Py_SignalConfig_INIT -1 - #define _Py_ImplicitConfig_INIT NULL - #define _Py_MainConfig_INIT -1, NULL, NULL, NULL, NULL, NULL, -1 - #define _Py_InteractiveConfig_INIT NULL, -1, -1 + #define _PyArgConfig_INIT NULL, NULL, NULL, NULL + #define _PyLocationConfig_INIT NULL, NULL, NULL, NULL, NULL, NULL + #define _PySiteConfig_INIT -1, -1 + #define _PyImportConfig_INIT -1, -1, NULL + #define _PyStreamConfig_INIT -1, NULL, NULL, NULL, NULL, NULL, NULL + #define _PyFilesystemConfig_INIT NULL + #define _PyDebuggingConfig_INIT -1, -1, -1 + #define _PyCodeGenConfig_INIT -1, -1 + #define _PySignalConfig_INIT -1 + #define _PyImplicitConfig_INIT NULL + #define _PyMainConfig_INIT -1, NULL, NULL, NULL, NULL, NULL, -1 + #define _PyInteractiveConfig_INIT NULL, -1, -1 - #define Py_Config_INIT {_Py_ArgConfig_INIT, _Py_LocationConfig_INIT, - _Py_SiteConfig_INIT, _Py_ImportConfig_INIT, - _Py_StreamConfig_INIT, _Py_FilesystemConfig_INIT, - _Py_DebuggingConfig_INIT, _Py_CodeGenConfig_INIT, - _Py_SignalConfig_INIT, _Py_ImplicitConfig_INIT, - _Py_MainConfig_INIT, _Py_InteractiveConfig_INIT} + #define PyConfig_INIT {_PyArgConfig_INIT, _PyLocationConfig_INIT, + _PySiteConfig_INIT, _PyImportConfig_INIT, + _PyStreamConfig_INIT, _PyFilesystemConfig_INIT, + _PyDebuggingConfig_INIT, _PyCodeGenConfig_INIT, + _PySignalConfig_INIT, _PyImplicitConfig_INIT, + _PyMainConfig_INIT, _PyInteractiveConfig_INIT} @@ -646,14 +651,14 @@ configuration settings into effect and finish bootstrapping the interpreter up to full operation:: - int Py_EndInitialization(const Py_Config *config); + int Py_EndInitialization(const PyConfig *config); Like Py_ReadConfiguration, this call will raise an exception and report an error return rather than exhibiting fatal errors if a problem is found with the config data. All configuration settings are required - the configuration struct -should always be passed through ``Py_ReadConfiguration()`` to ensure it +should always be passed through ``Py_ReadConfig()`` to ensure it is fully populated. After a successful call, ``Py_IsInitializing()`` will be false, while @@ -792,7 +797,7 @@ The interpreter state will be updated to include details of the configuration settings supplied during initialization by extending the interpreter state -object with an embedded copy of the ``Py_CoreConfig`` and ``Py_Config`` +object with an embedded copy of the ``PyCoreConfig`` and ``PyConfig`` structs. For debugging purposes, the configuration settings will be exposed as @@ -834,7 +839,7 @@ ----------------------- Backwards compatibility will be preserved primarily by ensuring that -``Py_ReadConfiguration()`` interrogates all the previously defined +``Py_ReadConfig()`` interrogates all the previously defined configuration settings stored in global variables and environment variables, and that ``Py_EndInitialization()`` writes affected settings back to the relevant locations. @@ -911,14 +916,14 @@ * Should there be ``Py_PreparingMain()`` and ``Py_RunningMain()`` query APIs? * Should the answer to ``Py_IsInitialized()`` be exposed via the ``sys`` module? -* Is initialisation of the ``Py_Config`` struct too unwieldy to be +* Is initialisation of the ``PyConfig`` struct too unwieldy to be maintainable? Would a Python dictionary be a better choice, despite being harder to work with from C code? -* Would it be better to manage the flag variables in ``Py_Config`` as +* Would it be better to manage the flag variables in ``PyConfig`` as Python integers or as "negative means false, positive means true, zero means not set" so the struct can be initialized with a simple ``memset(&config, 0, sizeof(*config))``, eliminating the need to update - both Py_Config and Py_Config_INIT when adding new fields? + both PyConfig and PyConfig_INIT when adding new fields? * The name of the new system Python executable is a bikeshed waiting to be painted. The 3 options considered so far are ``spython``, ``pysystem`` and ``python-minimal``. The PEP text reflects my current preferred choice -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue Jan 15 14:14:43 2013 From: python-checkins at python.org (richard.oudkerk) Date: Tue, 15 Jan 2013 14:14:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEwNTI3?= =?utf-8?q?=3A_Remove_dead_code?= Message-ID: <3YlsTb0WwZzP57@mail.python.org> http://hg.python.org/cpython/rev/f07435fa6736 changeset: 81513:f07435fa6736 branch: 2.7 parent: 81506:da5e520a7ba5 user: Richard Oudkerk date: Tue Jan 15 13:13:35 2013 +0000 summary: Issue #10527: Remove dead code files: Lib/multiprocessing/connection.py | 23 ------------------- 1 files changed, 0 insertions(+), 23 deletions(-) diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -41,7 +41,6 @@ import time import tempfile import itertools -import select import _multiprocessing from multiprocessing import current_process, AuthenticationError @@ -201,28 +200,6 @@ return c1, c2 else: - if hasattr(select, 'poll'): - def _poll(fds, timeout): - if timeout is not None: - timeout = int(timeout) * 1000 # timeout is in milliseconds - fd_map = {} - pollster = select.poll() - for fd in fds: - pollster.register(fd, select.POLLIN) - if hasattr(fd, 'fileno'): - fd_map[fd.fileno()] = fd - else: - fd_map[fd] = fd - ls = [] - for fd, event in pollster.poll(timeout): - if event & select.POLLNVAL: - raise ValueError('invalid file descriptor %i' % fd) - ls.append(fd_map[fd]) - return ls - else: - def _poll(fds, timeout): - return select.select(fds, [], [], timeout)[0] - from _multiprocessing import win32 def Pipe(duplex=True): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 14:24:38 2013 From: python-checkins at python.org (richard.oudkerk) Date: Tue, 15 Jan 2013 14:24:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEwNTI3?= =?utf-8?q?=3A_Remove_dead_code?= Message-ID: <3Ylsj24d1CzPCZ@mail.python.org> http://hg.python.org/cpython/rev/49d45151b9ed changeset: 81514:49d45151b9ed branch: 3.2 parent: 81507:abf111b9a464 user: Richard Oudkerk date: Tue Jan 15 13:19:24 2013 +0000 summary: Issue #10527: Remove dead code files: Lib/multiprocessing/connection.py | 23 ------------------- 1 files changed, 0 insertions(+), 23 deletions(-) diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -41,7 +41,6 @@ import time import tempfile import itertools -import select import _multiprocessing from multiprocessing import current_process, AuthenticationError @@ -214,28 +213,6 @@ return c1, c2 else: - if hasattr(select, 'poll'): - def _poll(fds, timeout): - if timeout is not None: - timeout = int(timeout) * 1000 # timeout is in milliseconds - fd_map = {} - pollster = select.poll() - for fd in fds: - pollster.register(fd, select.POLLIN) - if hasattr(fd, 'fileno'): - fd_map[fd.fileno()] = fd - else: - fd_map[fd] = fd - ls = [] - for fd, event in pollster.poll(timeout): - if event & select.POLLNVAL: - raise ValueError('invalid file descriptor %i' % fd) - ls.append(fd_map[fd]) - return ls - else: - def _poll(fds, timeout): - return select.select(fds, [], [], timeout)[0] - from _multiprocessing import win32 def Pipe(duplex=True): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 14:24:40 2013 From: python-checkins at python.org (richard.oudkerk) Date: Tue, 15 Jan 2013 14:24:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Merge?= Message-ID: <3Ylsj401MYzPCZ@mail.python.org> http://hg.python.org/cpython/rev/44d35f578f0c changeset: 81515:44d35f578f0c branch: 3.3 parent: 81511:ad9b5c69b8b6 parent: 81514:49d45151b9ed user: Richard Oudkerk date: Tue Jan 15 13:22:34 2013 +0000 summary: Merge files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 14:24:41 2013 From: python-checkins at python.org (richard.oudkerk) Date: Tue, 15 Jan 2013 14:24:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge?= Message-ID: <3Ylsj533mPzPNV@mail.python.org> http://hg.python.org/cpython/rev/2ba6fa21303d changeset: 81516:2ba6fa21303d parent: 81512:1f66fc397c8d parent: 81515:44d35f578f0c user: Richard Oudkerk date: Tue Jan 15 13:23:32 2013 +0000 summary: Merge files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 14:39:59 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 15 Jan 2013 14:39:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE0ODUw?= =?utf-8?q?=3A_Now_a_chamap_decoder_treates_U+FFFE_as_=22undefined_mapping?= =?utf-8?q?=22?= Message-ID: <3Ylt2l5dDdzNlw@mail.python.org> http://hg.python.org/cpython/rev/33a8ef498b1e changeset: 81517:33a8ef498b1e branch: 2.7 parent: 81506:da5e520a7ba5 user: Serhiy Storchaka date: Tue Jan 15 14:42:59 2013 +0200 summary: Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" in any mapping, not only in an unicode string. files: Lib/test/test_codecs.py | 56 ++++++++++++++++++++++++++-- Misc/NEWS | 3 + Objects/unicodeobject.c | 46 ++++++++++++---------- 3 files changed, 79 insertions(+), 26 deletions(-) diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -1551,6 +1551,14 @@ (u"abc", 3) ) + self.assertRaises(UnicodeDecodeError, + codecs.charmap_decode, b"\x00\x01\x02", "strict", u"ab" + ) + + self.assertRaises(UnicodeDecodeError, + codecs.charmap_decode, "\x00\x01\x02", "strict", u"ab\ufffe" + ) + self.assertEqual( codecs.charmap_decode("\x00\x01\x02", "replace", u"ab"), (u"ab\ufffd", 3) @@ -1566,10 +1574,6 @@ (u"ab", 3) ) - self.assertRaises(UnicodeDecodeError, - codecs.charmap_decode, b"\x00\x01\x02", "strict", u"ab" - ) - self.assertEqual( codecs.charmap_decode("\x00\x01\x02", "ignore", u"ab\ufffe"), (u"ab", 3) @@ -1611,6 +1615,17 @@ {0: u'a', 1: u'b'} ) + self.assertRaises(UnicodeDecodeError, + codecs.charmap_decode, "\x00\x01\x02", "strict", + {0: u'a', 1: u'b', 2: None} + ) + + # Issue #14850 + self.assertRaises(UnicodeDecodeError, + codecs.charmap_decode, "\x00\x01\x02", "strict", + {0: u'a', 1: u'b', 2: u'\ufffe'} + ) + self.assertEqual( codecs.charmap_decode("\x00\x01\x02", "replace", {0: u'a', 1: u'b'}), @@ -1623,6 +1638,13 @@ (u"ab\ufffd", 3) ) + # Issue #14850 + self.assertEqual( + codecs.charmap_decode("\x00\x01\x02", "replace", + {0: u'a', 1: u'b', 2: u'\ufffe'}), + (u"ab\ufffd", 3) + ) + self.assertEqual( codecs.charmap_decode("\x00\x01\x02", "ignore", {0: u'a', 1: u'b'}), @@ -1635,7 +1657,14 @@ (u"ab", 3) ) - allbytes = bytes(range(256)) + # Issue #14850 + self.assertEqual( + codecs.charmap_decode("\x00\x01\x02", "ignore", + {0: u'a', 1: u'b', 2: u'\ufffe'}), + (u"ab", 3) + ) + + allbytes = "".join(chr(i) for i in xrange(256)) self.assertEqual( codecs.charmap_decode(allbytes, "ignore", {}), (u"", len(allbytes)) @@ -1669,6 +1698,11 @@ {0: a, 1: b}, ) + self.assertRaises(UnicodeDecodeError, + codecs.charmap_decode, "\x00\x01\x02", "strict", + {0: a, 1: b, 2: 0xFFFE}, + ) + self.assertEqual( codecs.charmap_decode("\x00\x01\x02", "replace", {0: a, 1: b}), @@ -1676,11 +1710,23 @@ ) self.assertEqual( + codecs.charmap_decode("\x00\x01\x02", "replace", + {0: a, 1: b, 2: 0xFFFE}), + (u"ab\ufffd", 3) + ) + + self.assertEqual( codecs.charmap_decode("\x00\x01\x02", "ignore", {0: a, 1: b}), (u"ab", 3) ) + self.assertEqual( + codecs.charmap_decode("\x00\x01\x02", "ignore", + {0: a, 1: b, 2: 0xFFFE}), + (u"ab", 3) + ) + class WithStmtTest(unittest.TestCase): def test_encodedfile(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -9,6 +9,9 @@ Core and Builtins ----------------- +- Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" + in any mapping, not only in an unicode string. + - Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Amaury Forgeot d'Arc. diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -4121,15 +4121,18 @@ if (PyErr_ExceptionMatches(PyExc_LookupError)) { /* No mapping found means: mapping is undefined. */ PyErr_Clear(); - x = Py_None; - Py_INCREF(x); + goto Undefined; } else goto onError; } /* Apply mapping */ + if (x == Py_None) + goto Undefined; if (PyInt_Check(x)) { long value = PyInt_AS_LONG(x); + if (value == 0xFFFE) + goto Undefined; if (value < 0 || value > 0x10FFFF) { PyErr_SetString(PyExc_TypeError, "character mapping must be in range(0x110000)"); @@ -4162,29 +4165,16 @@ #endif *p++ = (Py_UNICODE)value; } - else if (x == Py_None) { - /* undefined mapping */ - outpos = p-PyUnicode_AS_UNICODE(v); - startinpos = s-starts; - endinpos = startinpos+1; - if (unicode_decode_call_errorhandler( - errors, &errorHandler, - "charmap", "character maps to ", - starts, size, &startinpos, &endinpos, &exc, &s, - &v, &outpos, &p)) { - Py_DECREF(x); - goto onError; - } - Py_DECREF(x); - continue; - } else if (PyUnicode_Check(x)) { Py_ssize_t targetsize = PyUnicode_GET_SIZE(x); - if (targetsize == 1) + if (targetsize == 1) { /* 1-1 mapping */ - *p++ = *PyUnicode_AS_UNICODE(x); - + Py_UNICODE value = *PyUnicode_AS_UNICODE(x); + if (value == 0xFFFE) + goto Undefined; + *p++ = value; + } else if (targetsize > 1) { /* 1-n mapping */ if (targetsize > extrachars) { @@ -4218,6 +4208,20 @@ } Py_DECREF(x); ++s; + continue; +Undefined: + /* undefined mapping */ + Py_XDECREF(x); + outpos = p-PyUnicode_AS_UNICODE(v); + startinpos = s-starts; + endinpos = startinpos+1; + if (unicode_decode_call_errorhandler( + errors, &errorHandler, + "charmap", "character maps to ", + starts, size, &startinpos, &endinpos, &exc, &s, + &v, &outpos, &p)) { + goto onError; + } } } if (p - PyUnicode_AS_UNICODE(v) < PyUnicode_GET_SIZE(v)) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 14:40:01 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 15 Jan 2013 14:40:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0ODUw?= =?utf-8?q?=3A_Now_a_chamap_decoder_treates_U+FFFE_as_=22undefined_mapping?= =?utf-8?q?=22?= Message-ID: <3Ylt2n35H6zPNG@mail.python.org> http://hg.python.org/cpython/rev/13cd78a2a17b changeset: 81518:13cd78a2a17b branch: 3.2 parent: 81507:abf111b9a464 user: Serhiy Storchaka date: Tue Jan 15 14:43:21 2013 +0200 summary: Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" in any mapping, not only in an unicode string. files: Lib/test/test_codecs.py | 46 +++++++++++++++++++++++++++++ Misc/NEWS | 3 + Objects/unicodeobject.c | 46 +++++++++++++++------------- 3 files changed, 74 insertions(+), 21 deletions(-) diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -1586,6 +1586,10 @@ codecs.charmap_decode, b"\x00\x01\x02", "strict", "ab" ) + self.assertRaises(UnicodeDecodeError, + codecs.charmap_decode, b"\x00\x01\x02", "strict", "ab\ufffe" + ) + self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "replace", "ab"), ("ab\ufffd", 3) @@ -1642,6 +1646,17 @@ {0: 'a', 1: 'b'} ) + self.assertRaises(UnicodeDecodeError, + codecs.charmap_decode, b"\x00\x01\x02", "strict", + {0: 'a', 1: 'b', 2: None} + ) + + # Issue #14850 + self.assertRaises(UnicodeDecodeError, + codecs.charmap_decode, b"\x00\x01\x02", "strict", + {0: 'a', 1: 'b', 2: '\ufffe'} + ) + self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "replace", {0: 'a', 1: 'b'}), @@ -1654,6 +1669,13 @@ ("ab\ufffd", 3) ) + # Issue #14850 + self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "replace", + {0: 'a', 1: 'b', 2: '\ufffe'}), + ("ab\ufffd", 3) + ) + self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "ignore", {0: 'a', 1: 'b'}), @@ -1666,6 +1688,13 @@ ("ab", 3) ) + # Issue #14850 + self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "ignore", + {0: 'a', 1: 'b', 2: '\ufffe'}), + ("ab", 3) + ) + allbytes = bytes(range(256)) self.assertEqual( codecs.charmap_decode(allbytes, "ignore", {}), @@ -1700,6 +1729,11 @@ {0: a, 1: b}, ) + self.assertRaises(UnicodeDecodeError, + codecs.charmap_decode, b"\x00\x01\x02", "strict", + {0: a, 1: b, 2: 0xFFFE}, + ) + self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "replace", {0: a, 1: b}), @@ -1707,11 +1741,23 @@ ) self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "replace", + {0: a, 1: b, 2: 0xFFFE}), + ("ab\ufffd", 3) + ) + + self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "ignore", {0: a, 1: b}), ("ab", 3) ) + self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "ignore", + {0: a, 1: b, 2: 0xFFFE}), + ("ab", 3) + ) + class WithStmtTest(unittest.TestCase): def test_encodedfile(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" + in any mapping, not only in a string. + - Issue #11461: Fix the incremental UTF-16 decoder. Original patch by Amaury Forgeot d'Arc. diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -5245,15 +5245,18 @@ if (PyErr_ExceptionMatches(PyExc_LookupError)) { /* No mapping found means: mapping is undefined. */ PyErr_Clear(); - x = Py_None; - Py_INCREF(x); + goto Undefined; } else goto onError; } /* Apply mapping */ + if (x == Py_None) + goto Undefined; if (PyLong_Check(x)) { long value = PyLong_AS_LONG(x); + if (value == 0xFFFE) + goto Undefined; if (value < 0 || value > 0x10FFFF) { PyErr_SetString(PyExc_TypeError, "character mapping must be in range(0x110000)"); @@ -5286,29 +5289,16 @@ #endif *p++ = (Py_UNICODE)value; } - else if (x == Py_None) { - /* undefined mapping */ - outpos = p-PyUnicode_AS_UNICODE(v); - startinpos = s-starts; - endinpos = startinpos+1; - if (unicode_decode_call_errorhandler( - errors, &errorHandler, - "charmap", "character maps to ", - &starts, &e, &startinpos, &endinpos, &exc, &s, - &v, &outpos, &p)) { - Py_DECREF(x); - goto onError; - } - Py_DECREF(x); - continue; - } else if (PyUnicode_Check(x)) { Py_ssize_t targetsize = PyUnicode_GET_SIZE(x); - if (targetsize == 1) + if (targetsize == 1) { /* 1-1 mapping */ - *p++ = *PyUnicode_AS_UNICODE(x); - + Py_UNICODE value = *PyUnicode_AS_UNICODE(x); + if (value == 0xFFFE) + goto Undefined; + *p++ = value; + } else if (targetsize > 1) { /* 1-n mapping */ if (targetsize > extrachars) { @@ -5342,6 +5332,20 @@ } Py_DECREF(x); ++s; + continue; +Undefined: + /* undefined mapping */ + Py_XDECREF(x); + outpos = p-PyUnicode_AS_UNICODE(v); + startinpos = s-starts; + endinpos = startinpos+1; + if (unicode_decode_call_errorhandler( + errors, &errorHandler, + "charmap", "character maps to ", + &starts, &e, &startinpos, &endinpos, &exc, &s, + &v, &outpos, &p)) { + goto onError; + } } } if (p - PyUnicode_AS_UNICODE(v) < PyUnicode_GET_SIZE(v)) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 14:40:03 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 15 Jan 2013 14:40:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=2314850=3A_Now_a_chamap_decoder_treates_U+FFFE_as_=22un?= =?utf-8?q?defined_mapping=22?= Message-ID: <3Ylt2q0RyhzPRW@mail.python.org> http://hg.python.org/cpython/rev/6ac4f1609847 changeset: 81519:6ac4f1609847 branch: 3.3 parent: 81511:ad9b5c69b8b6 parent: 81518:13cd78a2a17b user: Serhiy Storchaka date: Tue Jan 15 15:01:20 2013 +0200 summary: Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" in any mapping, not only in an unicode string. files: Lib/test/test_codecs.py | 46 +++++++++++++++++++++++++++++ Misc/NEWS | 3 + Objects/unicodeobject.c | 41 +++++++++++++----------- 3 files changed, 71 insertions(+), 19 deletions(-) diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -1737,6 +1737,10 @@ codecs.charmap_decode, b"\x00\x01\x02", "strict", "ab" ) + self.assertRaises(UnicodeDecodeError, + codecs.charmap_decode, b"\x00\x01\x02", "strict", "ab\ufffe" + ) + self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "replace", "ab"), ("ab\ufffd", 3) @@ -1793,6 +1797,17 @@ {0: 'a', 1: 'b'} ) + self.assertRaises(UnicodeDecodeError, + codecs.charmap_decode, b"\x00\x01\x02", "strict", + {0: 'a', 1: 'b', 2: None} + ) + + # Issue #14850 + self.assertRaises(UnicodeDecodeError, + codecs.charmap_decode, b"\x00\x01\x02", "strict", + {0: 'a', 1: 'b', 2: '\ufffe'} + ) + self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "replace", {0: 'a', 1: 'b'}), @@ -1805,6 +1820,13 @@ ("ab\ufffd", 3) ) + # Issue #14850 + self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "replace", + {0: 'a', 1: 'b', 2: '\ufffe'}), + ("ab\ufffd", 3) + ) + self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "ignore", {0: 'a', 1: 'b'}), @@ -1817,6 +1839,13 @@ ("ab", 3) ) + # Issue #14850 + self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "ignore", + {0: 'a', 1: 'b', 2: '\ufffe'}), + ("ab", 3) + ) + allbytes = bytes(range(256)) self.assertEqual( codecs.charmap_decode(allbytes, "ignore", {}), @@ -1857,6 +1886,11 @@ {0: a, 1: b}, ) + self.assertRaises(UnicodeDecodeError, + codecs.charmap_decode, b"\x00\x01\x02", "strict", + {0: a, 1: b, 2: 0xFFFE}, + ) + self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "replace", {0: a, 1: b}), @@ -1864,11 +1898,23 @@ ) self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "replace", + {0: a, 1: b, 2: 0xFFFE}), + ("ab\ufffd", 3) + ) + + self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "ignore", {0: a, 1: b}), ("ab", 3) ) + self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "ignore", + {0: a, 1: b, 2: 0xFFFE}), + ("ab", 3) + ) + class WithStmtTest(unittest.TestCase): def test_encodedfile(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -12,6 +12,9 @@ Core and Builtins ----------------- +- Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" + in any mapping, not only in a string. + - Issue #16730: importlib.machinery.FileFinder now no longers raises an exception when trying to populate its cache and it finds out the directory is unreadable or has turned into a file. Reported and diagnosed by diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -7511,15 +7511,18 @@ if (PyErr_ExceptionMatches(PyExc_LookupError)) { /* No mapping found means: mapping is undefined. */ PyErr_Clear(); - x = Py_None; - Py_INCREF(x); + goto Undefined; } else goto onError; } /* Apply mapping */ + if (x == Py_None) + goto Undefined; if (PyLong_Check(x)) { long value = PyLong_AS_LONG(x); + if (value == 0xFFFE) + goto Undefined; if (value < 0 || value > MAX_UNICODE) { PyErr_Format(PyExc_TypeError, "character mapping must be in range(0x%lx)", @@ -7530,21 +7533,6 @@ if (unicode_putchar(&v, &outpos, value) < 0) goto onError; } - else if (x == Py_None) { - /* undefined mapping */ - startinpos = s-starts; - endinpos = startinpos+1; - if (unicode_decode_call_errorhandler( - errors, &errorHandler, - "charmap", "character maps to ", - &starts, &e, &startinpos, &endinpos, &exc, &s, - &v, &outpos)) { - Py_DECREF(x); - goto onError; - } - Py_DECREF(x); - continue; - } else if (PyUnicode_Check(x)) { Py_ssize_t targetsize; @@ -7554,8 +7542,10 @@ if (targetsize == 1) { /* 1-1 mapping */ - if (unicode_putchar(&v, &outpos, - PyUnicode_READ_CHAR(x, 0)) < 0) + Py_UCS4 value = PyUnicode_READ_CHAR(x, 0); + if (value == 0xFFFE) + goto Undefined; + if (unicode_putchar(&v, &outpos, value) < 0) goto onError; } else if (targetsize > 1) { @@ -7590,6 +7580,19 @@ } Py_DECREF(x); ++s; + continue; +Undefined: + /* undefined mapping */ + Py_XDECREF(x); + startinpos = s-starts; + endinpos = startinpos+1; + if (unicode_decode_call_errorhandler( + errors, &errorHandler, + "charmap", "character maps to ", + &starts, &e, &startinpos, &endinpos, &exc, &s, + &v, &outpos)) { + goto onError; + } } } if (unicode_resize(&v, outpos) < 0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 14:40:04 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 15 Jan 2013 14:40:04 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2314850=3A_Now_a_chamap_decoder_treates_U+FFFE_as?= =?utf-8?q?_=22undefined_mapping=22?= Message-ID: <3Ylt2r4gZ6zPWC@mail.python.org> http://hg.python.org/cpython/rev/03e22cc9407a changeset: 81520:03e22cc9407a parent: 81512:1f66fc397c8d parent: 81519:6ac4f1609847 user: Serhiy Storchaka date: Tue Jan 15 15:30:04 2013 +0200 summary: Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" in any mapping, not only in an unicode string. files: Lib/test/test_codecs.py | 46 +++++++++++++++++++++++++ Misc/NEWS | 3 + Objects/unicodeobject.c | 52 ++++++++++++++++++---------- 3 files changed, 82 insertions(+), 19 deletions(-) diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -1737,6 +1737,10 @@ codecs.charmap_decode, b"\x00\x01\x02", "strict", "ab" ) + self.assertRaises(UnicodeDecodeError, + codecs.charmap_decode, b"\x00\x01\x02", "strict", "ab\ufffe" + ) + self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "replace", "ab"), ("ab\ufffd", 3) @@ -1793,6 +1797,17 @@ {0: 'a', 1: 'b'} ) + self.assertRaises(UnicodeDecodeError, + codecs.charmap_decode, b"\x00\x01\x02", "strict", + {0: 'a', 1: 'b', 2: None} + ) + + # Issue #14850 + self.assertRaises(UnicodeDecodeError, + codecs.charmap_decode, b"\x00\x01\x02", "strict", + {0: 'a', 1: 'b', 2: '\ufffe'} + ) + self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "replace", {0: 'a', 1: 'b'}), @@ -1805,6 +1820,13 @@ ("ab\ufffd", 3) ) + # Issue #14850 + self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "replace", + {0: 'a', 1: 'b', 2: '\ufffe'}), + ("ab\ufffd", 3) + ) + self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "ignore", {0: 'a', 1: 'b'}), @@ -1817,6 +1839,13 @@ ("ab", 3) ) + # Issue #14850 + self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "ignore", + {0: 'a', 1: 'b', 2: '\ufffe'}), + ("ab", 3) + ) + allbytes = bytes(range(256)) self.assertEqual( codecs.charmap_decode(allbytes, "ignore", {}), @@ -1857,6 +1886,11 @@ {0: a, 1: b}, ) + self.assertRaises(UnicodeDecodeError, + codecs.charmap_decode, b"\x00\x01\x02", "strict", + {0: a, 1: b, 2: 0xFFFE}, + ) + self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "replace", {0: a, 1: b}), @@ -1864,11 +1898,23 @@ ) self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "replace", + {0: a, 1: b, 2: 0xFFFE}), + ("ab\ufffd", 3) + ) + + self.assertEqual( codecs.charmap_decode(b"\x00\x01\x02", "ignore", {0: a, 1: b}), ("ab", 3) ) + self.assertEqual( + codecs.charmap_decode(b"\x00\x01\x02", "ignore", + {0: a, 1: b, 2: 0xFFFE}), + ("ab", 3) + ) + class WithStmtTest(unittest.TestCase): def test_encodedfile(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #14850: Now a chamap decoder treates U+FFFE as "undefined mapping" + in any mapping, not only in a string. + - Issue #16730: importlib.machinery.FileFinder now no longers raises an exception when trying to populate its cache and it finds out the directory is unreadable or has turned into a file. Reported and diagnosed by diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -7393,15 +7393,18 @@ if (PyErr_ExceptionMatches(PyExc_LookupError)) { /* No mapping found means: mapping is undefined. */ PyErr_Clear(); - x = Py_None; - Py_INCREF(x); + goto Undefined; } else goto onError; } /* Apply mapping */ + if (x == Py_None) + goto Undefined; if (PyLong_Check(x)) { long value = PyLong_AS_LONG(x); + if (value == 0xFFFE) + goto Undefined; if (value < 0 || value > MAX_UNICODE) { PyErr_Format(PyExc_TypeError, "character mapping must be in range(0x%lx)", @@ -7415,25 +7418,23 @@ PyUnicode_WRITE(writer.kind, writer.data, writer.pos, value); writer.pos++; } - else if (x == Py_None) { - /* undefined mapping */ - startinpos = s-starts; - endinpos = startinpos+1; - if (unicode_decode_call_errorhandler_writer( - errors, &errorHandler, - "charmap", "character maps to ", - &starts, &e, &startinpos, &endinpos, &exc, &s, - &writer)) { - Py_DECREF(x); + else if (PyUnicode_Check(x)) { + if (PyUnicode_READY(x) == -1) goto onError; + if (PyUnicode_GET_LENGTH(x) == 1) { + Py_UCS4 value = PyUnicode_READ_CHAR(x, 0); + if (value == 0xFFFE) + goto Undefined; + if (_PyUnicodeWriter_Prepare(&writer, 1, value) == -1) + goto onError; + PyUnicode_WRITE(writer.kind, writer.data, writer.pos, value); + writer.pos++; } - Py_DECREF(x); - continue; - } - else if (PyUnicode_Check(x)) { - writer.overallocate = 1; - if (_PyUnicodeWriter_WriteStr(&writer, x) == -1) - goto onError; + else { + writer.overallocate = 1; + if (_PyUnicodeWriter_WriteStr(&writer, x) == -1) + goto onError; + } } else { /* wrong return value */ @@ -7444,6 +7445,19 @@ } Py_DECREF(x); ++s; + continue; +Undefined: + /* undefined mapping */ + Py_XDECREF(x); + startinpos = s-starts; + endinpos = startinpos+1; + if (unicode_decode_call_errorhandler_writer( + errors, &errorHandler, + "charmap", "character maps to ", + &starts, &e, &startinpos, &endinpos, &exc, &s, + &writer)) { + goto onError; + } } } Py_XDECREF(errorHandler); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 14:40:06 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 15 Jan 2013 14:40:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf-8?q?_Merge_heads?= Message-ID: <3Ylt2t0bMPzQSc@mail.python.org> http://hg.python.org/cpython/rev/2e1f1b096f3d changeset: 81521:2e1f1b096f3d branch: 2.7 parent: 81517:33a8ef498b1e parent: 81513:f07435fa6736 user: Serhiy Storchaka date: Tue Jan 15 15:32:49 2013 +0200 summary: Merge heads files: Lib/multiprocessing/connection.py | 23 ------------------- 1 files changed, 0 insertions(+), 23 deletions(-) diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -41,7 +41,6 @@ import time import tempfile import itertools -import select import _multiprocessing from multiprocessing import current_process, AuthenticationError @@ -201,28 +200,6 @@ return c1, c2 else: - if hasattr(select, 'poll'): - def _poll(fds, timeout): - if timeout is not None: - timeout = int(timeout) * 1000 # timeout is in milliseconds - fd_map = {} - pollster = select.poll() - for fd in fds: - pollster.register(fd, select.POLLIN) - if hasattr(fd, 'fileno'): - fd_map[fd.fileno()] = fd - else: - fd_map[fd] = fd - ls = [] - for fd, event in pollster.poll(timeout): - if event & select.POLLNVAL: - raise ValueError('invalid file descriptor %i' % fd) - ls.append(fd_map[fd]) - return ls - else: - def _poll(fds, timeout): - return select.select(fds, [], [], timeout)[0] - from _multiprocessing import win32 def Pipe(duplex=True): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 14:40:07 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 15 Jan 2013 14:40:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMik6?= =?utf-8?q?_Merge_heads?= Message-ID: <3Ylt2v3HfqzPXW@mail.python.org> http://hg.python.org/cpython/rev/2ededf960837 changeset: 81522:2ededf960837 branch: 3.2 parent: 81518:13cd78a2a17b parent: 81514:49d45151b9ed user: Serhiy Storchaka date: Tue Jan 15 15:32:59 2013 +0200 summary: Merge heads files: Lib/multiprocessing/connection.py | 23 ------------------- 1 files changed, 0 insertions(+), 23 deletions(-) diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -41,7 +41,6 @@ import time import tempfile import itertools -import select import _multiprocessing from multiprocessing import current_process, AuthenticationError @@ -214,28 +213,6 @@ return c1, c2 else: - if hasattr(select, 'poll'): - def _poll(fds, timeout): - if timeout is not None: - timeout = int(timeout) * 1000 # timeout is in milliseconds - fd_map = {} - pollster = select.poll() - for fd in fds: - pollster.register(fd, select.POLLIN) - if hasattr(fd, 'fileno'): - fd_map[fd.fileno()] = fd - else: - fd_map[fd] = fd - ls = [] - for fd, event in pollster.poll(timeout): - if event & select.POLLNVAL: - raise ValueError('invalid file descriptor %i' % fd) - ls.append(fd_map[fd]) - return ls - else: - def _poll(fds, timeout): - return select.select(fds, [], [], timeout)[0] - from _multiprocessing import win32 def Pipe(duplex=True): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 14:40:08 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 15 Jan 2013 14:40:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuMyk6?= =?utf-8?q?_Merge_heads?= Message-ID: <3Ylt2w5shdzPSS@mail.python.org> http://hg.python.org/cpython/rev/eb45c92e415a changeset: 81523:eb45c92e415a branch: 3.3 parent: 81519:6ac4f1609847 parent: 81515:44d35f578f0c user: Serhiy Storchaka date: Tue Jan 15 15:33:09 2013 +0200 summary: Merge heads files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 14:40:10 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 15 Jan 2013 14:40:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?q?=29=3A_Merge_heads?= Message-ID: <3Ylt2y1JQ9zPqf@mail.python.org> http://hg.python.org/cpython/rev/29c60517fa97 changeset: 81524:29c60517fa97 parent: 81520:03e22cc9407a parent: 81516:2ba6fa21303d user: Serhiy Storchaka date: Tue Jan 15 15:33:18 2013 +0200 summary: Merge heads files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 14:40:11 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 15 Jan 2013 14:40:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Null_merge?= Message-ID: <3Ylt2z4yK6zPNV@mail.python.org> http://hg.python.org/cpython/rev/1fea46c6a012 changeset: 81525:1fea46c6a012 branch: 3.3 parent: 81523:eb45c92e415a parent: 81522:2ededf960837 user: Serhiy Storchaka date: Tue Jan 15 15:33:52 2013 +0200 summary: Null merge files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 14:40:13 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 15 Jan 2013 14:40:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Null_merge?= Message-ID: <3Ylt310TgrzPLs@mail.python.org> http://hg.python.org/cpython/rev/fb17969ace93 changeset: 81526:fb17969ace93 parent: 81524:29c60517fa97 parent: 81525:1fea46c6a012 user: Serhiy Storchaka date: Tue Jan 15 15:34:08 2013 +0200 summary: Null merge files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 17:04:36 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 15 Jan 2013 17:04:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE1ODYx?= =?utf-8?q?=3A_tkinter_now_correctly_works_with_lists_and_tuples_containin?= =?utf-8?q?g?= Message-ID: <3YlxFc62RxzP6n@mail.python.org> http://hg.python.org/cpython/rev/411bb75be5d1 changeset: 81527:411bb75be5d1 branch: 3.2 parent: 81522:2ededf960837 user: Serhiy Storchaka date: Tue Jan 15 17:56:08 2013 +0200 summary: Issue #15861: tkinter now correctly works with lists and tuples containing strings with whitespaces, backslashes or unbalanced braces. files: Lib/tkinter/__init__.py | 31 ++- Lib/tkinter/test/test_ttk/test_functions.py | 40 ++- Lib/tkinter/test/test_ttk/test_widgets.py | 8 + Lib/tkinter/ttk.py | 122 +++------ Misc/NEWS | 3 + 5 files changed, 123 insertions(+), 81 deletions(-) diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -39,6 +39,7 @@ import _tkinter # If this fails your Python may not be configured for Tk TclError = _tkinter.TclError from tkinter.constants import * +import re wantobjects = 1 @@ -50,6 +51,34 @@ EXCEPTION = _tkinter.EXCEPTION +_magic_re = re.compile(r'([\\{}])') +_space_re = re.compile(r'([\s])', re.ASCII) + +def _join(value): + """Internal function.""" + return ' '.join(map(_stringify, value)) + +def _stringify(value): + """Internal function.""" + if isinstance(value, (list, tuple)): + if len(value) == 1: + value = _stringify(value[0]) + if value[0] == '{': + value = '{%s}' % value + else: + value = '{%s}' % _join(value) + else: + value = str(value) + if not value: + value = '{}' + elif _magic_re.search(value): + # add '\' before special characters and spaces + value = _magic_re.sub(r'\\\1', value) + value = _space_re.sub(r'\\\1', value) + elif value[0] == '"' or _space_re.search(value): + value = '{%s}' % value + return value + def _flatten(seq): """Internal function.""" res = () @@ -1075,7 +1104,7 @@ if isinstance(item, int): nv.append(str(item)) elif isinstance(item, str): - nv.append(('{%s}' if ' ' in item else '%s') % item) + nv.append(_stringify(item)) else: break else: diff --git a/Lib/tkinter/test/test_ttk/test_functions.py b/Lib/tkinter/test/test_ttk/test_functions.py --- a/Lib/tkinter/test/test_ttk/test_functions.py +++ b/Lib/tkinter/test/test_ttk/test_functions.py @@ -49,13 +49,17 @@ ttk._format_optdict({'test': {'left': 'as is'}}), {'-test': {'left': 'as is'}}) - # check script formatting and untouched value(s) + # check script formatting check_against( ttk._format_optdict( - {'test': [1, -1, '', '2m', 0], 'nochange1': 3, - 'nochange2': 'abc def'}, script=True), - {'-test': '{1 -1 {} 2m 0}', '-nochange1': 3, - '-nochange2': 'abc def' }) + {'test': [1, -1, '', '2m', 0], 'test2': 3, + 'test3': '', 'test4': 'abc def', + 'test5': '"abc"', 'test6': '{}', + 'test7': '} -spam {'}, script=True), + {'-test': '{1 -1 {} 2m 0}', '-test2': '3', + '-test3': '{}', '-test4': '{abc def}', + '-test5': '{"abc"}', '-test6': r'\{\}', + '-test7': r'\}\ -spam\ \{'}) opts = {'???': True, '?': False} orig_opts = opts.copy() @@ -69,6 +73,32 @@ ttk._format_optdict( {'option': ('one two', 'three')}), {'-option': '{one two} three'}) + check_against( + ttk._format_optdict( + {'option': ('one\ttwo', 'three')}), + {'-option': '{one\ttwo} three'}) + + # passing empty strings inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('', 'one')}), + {'-option': '{} one'}) + + # passing values with braces inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('one} {two', 'three')}), + {'-option': r'one\}\ \{two three'}) + + # passing quoted strings inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('"one"', 'two')}), + {'-option': '{"one"} two'}) + check_against( + ttk._format_optdict( + {'option': ('{one}', 'two')}), + {'-option': r'\{one\} two'}) # ignore an option amount_opts = len(ttk._format_optdict(opts, ignore=('?'))) / 2 diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -189,6 +189,14 @@ self.combo.configure(values=[1, '', 2]) self.assertEqual(self.combo['values'], ('1', '', '2')) + # testing values with spaces + self.combo['values'] = ['a b', 'a\tb', 'a\nb'] + self.assertEqual(self.combo['values'], ('a b', 'a\tb', 'a\nb')) + + # testing values with special characters + self.combo['values'] = [r'a\tb', '"a"', '} {'] + self.assertEqual(self.combo['values'], (r'a\tb', '"a"', '} {')) + # out of range self.assertRaises(tkinter.TclError, self.combo.current, len(self.combo['values'])) diff --git a/Lib/tkinter/ttk.py b/Lib/tkinter/ttk.py --- a/Lib/tkinter/ttk.py +++ b/Lib/tkinter/ttk.py @@ -26,8 +26,7 @@ "tclobjs_to_py", "setup_master"] import tkinter - -_flatten = tkinter._flatten +from tkinter import _flatten, _join, _stringify # Verify if Tk is new enough to not need the Tile package _REQUIRE_TILE = True if tkinter.TkVersion < 8.5 else False @@ -47,39 +46,54 @@ master.tk.eval('package require tile') # TclError may be raised here master._tile_loaded = True +def _format_optvalue(value, script=False): + """Internal function.""" + if script: + # if caller passes a Tcl script to tk.call, all the values need to + # be grouped into words (arguments to a command in Tcl dialect) + value = _stringify(value) + elif isinstance(value, (list, tuple)): + value = _join(value) + return value + def _format_optdict(optdict, script=False, ignore=None): """Formats optdict to a tuple to pass it to tk.call. E.g. (script=False): {'foreground': 'blue', 'padding': [1, 2, 3, 4]} returns: ('-foreground', 'blue', '-padding', '1 2 3 4')""" - format = "%s" if not script else "{%s}" opts = [] for opt, value in optdict.items(): - if ignore and opt in ignore: - continue + if not ignore or opt not in ignore: + opts.append("-%s" % opt) + if value is not None: + opts.append(_format_optvalue(value, script)) - if isinstance(value, (list, tuple)): - v = [] - for val in value: - if isinstance(val, str): - v.append(str(val) if val else '{}') - else: - v.append(str(val)) + return _flatten(opts) - # format v according to the script option, but also check for - # space in any value in v in order to group them correctly - value = format % ' '.join( - ('{%s}' if ' ' in val else '%s') % val for val in v) - - if script and value == '': - value = '{}' # empty string in Python is equivalent to {} in Tcl - - opts.append(("-%s" % opt, value)) - - # Remember: _flatten skips over None - return _flatten(opts) +def _mapdict_values(items): + # each value in mapdict is expected to be a sequence, where each item + # is another sequence containing a state (or several) and a value + # E.g. (script=False): + # [('active', 'selected', 'grey'), ('focus', [1, 2, 3, 4])] + # returns: + # ['active selected', 'grey', 'focus', [1, 2, 3, 4]] + opt_val = [] + for *state, val in items: + # hacks for bakward compatibility + state[0] # raise IndexError if empty + if len(state) == 1: + # if it is empty (something that evaluates to False), then + # format it to Tcl code to denote the "normal" state + state = state[0] or '' + else: + # group multiple states + state = ' '.join(state) # raise TypeError if not str + opt_val.append(state) + if val is not None: + opt_val.append(val) + return opt_val def _format_mapdict(mapdict, script=False): """Formats mapdict to pass it to tk.call. @@ -90,32 +104,11 @@ returns: ('-expand', '{active selected} grey focus {1, 2, 3, 4}')""" - # if caller passes a Tcl script to tk.call, all the values need to - # be grouped into words (arguments to a command in Tcl dialect) - format = "%s" if not script else "{%s}" opts = [] for opt, value in mapdict.items(): - - opt_val = [] - # each value in mapdict is expected to be a sequence, where each item - # is another sequence containing a state (or several) and a value - for statespec in value: - state, val = statespec[:-1], statespec[-1] - - if len(state) > 1: # group multiple states - state = "{%s}" % ' '.join(state) - else: # single state - # if it is empty (something that evaluates to False), then - # format it to Tcl code to denote the "normal" state - state = state[0] or '{}' - - if isinstance(val, (list, tuple)): # val needs to be grouped - val = "{%s}" % ' '.join(map(str, val)) - - opt_val.append("%s %s" % (state, val)) - - opts.append(("-%s" % opt, format % ' '.join(opt_val))) + opts.extend(("-%s" % opt, + _format_optvalue(_mapdict_values(value), script))) return _flatten(opts) @@ -129,7 +122,7 @@ iname = args[0] # next args, if any, are statespec/value pairs which is almost # a mapdict, but we just need the value - imagespec = _format_mapdict({None: args[1:]})[1] + imagespec = _join(_mapdict_values(args[1:])) spec = "%s %s" % (iname, imagespec) else: @@ -138,7 +131,7 @@ # themed styles on Windows XP and Vista. # Availability: Tk 8.6, Windows XP and Vista. class_name, part_id = args[:2] - statemap = _format_mapdict({None: args[2:]})[1] + statemap = _join(_mapdict_values(args[2:])) spec = "%s %s %s" % (class_name, part_id, statemap) opts = _format_optdict(kw, script) @@ -148,11 +141,11 @@ # otherwise it will clone {} (empty element) spec = args[0] # theme name if len(args) > 1: # elementfrom specified - opts = (args[1], ) + opts = (_format_optvalue(args[1], script),) if script: spec = '{%s}' % spec - opts = ' '.join(map(str, opts)) + opts = ' '.join(opts) return spec, opts @@ -189,7 +182,7 @@ for layout_elem in layout: elem, opts = layout_elem opts = opts or {} - fopts = ' '.join(map(str, _format_optdict(opts, True, "children"))) + fopts = ' '.join(_format_optdict(opts, True, ("children",))) head = "%s%s%s" % (' ' * indent, elem, (" %s" % fopts) if fopts else '') if "children" in opts: @@ -215,11 +208,11 @@ for name, opts in settings.items(): # will format specific keys according to Tcl code if opts.get('configure'): # format 'configure' - s = ' '.join(map(str, _format_optdict(opts['configure'], True))) + s = ' '.join(_format_optdict(opts['configure'], True)) script.append("ttk::style configure %s %s;" % (name, s)) if opts.get('map'): # format 'map' - s = ' '.join(map(str, _format_mapdict(opts['map'], True))) + s = ' '.join(_format_mapdict(opts['map'], True)) script.append("ttk::style map %s %s;" % (name, s)) if 'layout' in opts: # format 'layout' which may be empty @@ -706,30 +699,9 @@ exportselection, justify, height, postcommand, state, textvariable, values, width """ - # The "values" option may need special formatting, so leave to - # _format_optdict the responsibility to format it - if "values" in kw: - kw["values"] = _format_optdict({'v': kw["values"]})[1] - Entry.__init__(self, master, "ttk::combobox", **kw) - def __setitem__(self, item, value): - if item == "values": - value = _format_optdict({item: value})[1] - - Entry.__setitem__(self, item, value) - - - def configure(self, cnf=None, **kw): - """Custom Combobox configure, created to properly format the values - option.""" - if "values" in kw: - kw["values"] = _format_optdict({'v': kw["values"]})[1] - - return Entry.configure(self, cnf, **kw) - - def current(self, newindex=None): """If newindex is supplied, sets the combobox value to the element at position newindex in the list of values. Otherwise, diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -202,6 +202,9 @@ Library ------- +- Issue #15861: tkinter now correctly works with lists and tuples containing + strings with whitespaces, backslashes or unbalanced braces. + - Issue #10527: Use poll() instead of select() for multiprocessing pipes. - Issue #9720: zipfile now writes correct local headers for files larger than -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 17:04:38 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 15 Jan 2013 17:04:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=2315861=3A_tkinter_now_correctly_works_with_lists_and_t?= =?utf-8?q?uples_containing?= Message-ID: <3YlxFf3g7KzR5B@mail.python.org> http://hg.python.org/cpython/rev/927352d7e994 changeset: 81528:927352d7e994 branch: 3.3 parent: 81525:1fea46c6a012 parent: 81527:411bb75be5d1 user: Serhiy Storchaka date: Tue Jan 15 17:58:14 2013 +0200 summary: Issue #15861: tkinter now correctly works with lists and tuples containing strings with whitespaces, backslashes or unbalanced braces. files: Lib/tkinter/__init__.py | 31 ++- Lib/tkinter/test/test_ttk/test_functions.py | 40 ++- Lib/tkinter/test/test_ttk/test_widgets.py | 8 + Lib/tkinter/ttk.py | 122 +++------ Misc/NEWS | 3 + 5 files changed, 123 insertions(+), 81 deletions(-) diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -40,6 +40,7 @@ import _tkinter # If this fails your Python may not be configured for Tk TclError = _tkinter.TclError from tkinter.constants import * +import re wantobjects = 1 @@ -52,6 +53,34 @@ EXCEPTION = _tkinter.EXCEPTION +_magic_re = re.compile(r'([\\{}])') +_space_re = re.compile(r'([\s])', re.ASCII) + +def _join(value): + """Internal function.""" + return ' '.join(map(_stringify, value)) + +def _stringify(value): + """Internal function.""" + if isinstance(value, (list, tuple)): + if len(value) == 1: + value = _stringify(value[0]) + if value[0] == '{': + value = '{%s}' % value + else: + value = '{%s}' % _join(value) + else: + value = str(value) + if not value: + value = '{}' + elif _magic_re.search(value): + # add '\' before special characters and spaces + value = _magic_re.sub(r'\\\1', value) + value = _space_re.sub(r'\\\1', value) + elif value[0] == '"' or _space_re.search(value): + value = '{%s}' % value + return value + def _flatten(seq): """Internal function.""" res = () @@ -1089,7 +1118,7 @@ if isinstance(item, int): nv.append(str(item)) elif isinstance(item, str): - nv.append(('{%s}' if ' ' in item else '%s') % item) + nv.append(_stringify(item)) else: break else: diff --git a/Lib/tkinter/test/test_ttk/test_functions.py b/Lib/tkinter/test/test_ttk/test_functions.py --- a/Lib/tkinter/test/test_ttk/test_functions.py +++ b/Lib/tkinter/test/test_ttk/test_functions.py @@ -49,13 +49,17 @@ ttk._format_optdict({'test': {'left': 'as is'}}), {'-test': {'left': 'as is'}}) - # check script formatting and untouched value(s) + # check script formatting check_against( ttk._format_optdict( - {'test': [1, -1, '', '2m', 0], 'nochange1': 3, - 'nochange2': 'abc def'}, script=True), - {'-test': '{1 -1 {} 2m 0}', '-nochange1': 3, - '-nochange2': 'abc def' }) + {'test': [1, -1, '', '2m', 0], 'test2': 3, + 'test3': '', 'test4': 'abc def', + 'test5': '"abc"', 'test6': '{}', + 'test7': '} -spam {'}, script=True), + {'-test': '{1 -1 {} 2m 0}', '-test2': '3', + '-test3': '{}', '-test4': '{abc def}', + '-test5': '{"abc"}', '-test6': r'\{\}', + '-test7': r'\}\ -spam\ \{'}) opts = {'???': True, '?': False} orig_opts = opts.copy() @@ -69,6 +73,32 @@ ttk._format_optdict( {'option': ('one two', 'three')}), {'-option': '{one two} three'}) + check_against( + ttk._format_optdict( + {'option': ('one\ttwo', 'three')}), + {'-option': '{one\ttwo} three'}) + + # passing empty strings inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('', 'one')}), + {'-option': '{} one'}) + + # passing values with braces inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('one} {two', 'three')}), + {'-option': r'one\}\ \{two three'}) + + # passing quoted strings inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('"one"', 'two')}), + {'-option': '{"one"} two'}) + check_against( + ttk._format_optdict( + {'option': ('{one}', 'two')}), + {'-option': r'\{one\} two'}) # ignore an option amount_opts = len(ttk._format_optdict(opts, ignore=('?'))) / 2 diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -189,6 +189,14 @@ self.combo.configure(values=[1, '', 2]) self.assertEqual(self.combo['values'], ('1', '', '2')) + # testing values with spaces + self.combo['values'] = ['a b', 'a\tb', 'a\nb'] + self.assertEqual(self.combo['values'], ('a b', 'a\tb', 'a\nb')) + + # testing values with special characters + self.combo['values'] = [r'a\tb', '"a"', '} {'] + self.assertEqual(self.combo['values'], (r'a\tb', '"a"', '} {')) + # out of range self.assertRaises(tkinter.TclError, self.combo.current, len(self.combo['values'])) diff --git a/Lib/tkinter/ttk.py b/Lib/tkinter/ttk.py --- a/Lib/tkinter/ttk.py +++ b/Lib/tkinter/ttk.py @@ -26,8 +26,7 @@ "tclobjs_to_py", "setup_master"] import tkinter - -_flatten = tkinter._flatten +from tkinter import _flatten, _join, _stringify # Verify if Tk is new enough to not need the Tile package _REQUIRE_TILE = True if tkinter.TkVersion < 8.5 else False @@ -47,39 +46,54 @@ master.tk.eval('package require tile') # TclError may be raised here master._tile_loaded = True +def _format_optvalue(value, script=False): + """Internal function.""" + if script: + # if caller passes a Tcl script to tk.call, all the values need to + # be grouped into words (arguments to a command in Tcl dialect) + value = _stringify(value) + elif isinstance(value, (list, tuple)): + value = _join(value) + return value + def _format_optdict(optdict, script=False, ignore=None): """Formats optdict to a tuple to pass it to tk.call. E.g. (script=False): {'foreground': 'blue', 'padding': [1, 2, 3, 4]} returns: ('-foreground', 'blue', '-padding', '1 2 3 4')""" - format = "%s" if not script else "{%s}" opts = [] for opt, value in optdict.items(): - if ignore and opt in ignore: - continue + if not ignore or opt not in ignore: + opts.append("-%s" % opt) + if value is not None: + opts.append(_format_optvalue(value, script)) - if isinstance(value, (list, tuple)): - v = [] - for val in value: - if isinstance(val, str): - v.append(str(val) if val else '{}') - else: - v.append(str(val)) + return _flatten(opts) - # format v according to the script option, but also check for - # space in any value in v in order to group them correctly - value = format % ' '.join( - ('{%s}' if ' ' in val else '%s') % val for val in v) - - if script and value == '': - value = '{}' # empty string in Python is equivalent to {} in Tcl - - opts.append(("-%s" % opt, value)) - - # Remember: _flatten skips over None - return _flatten(opts) +def _mapdict_values(items): + # each value in mapdict is expected to be a sequence, where each item + # is another sequence containing a state (or several) and a value + # E.g. (script=False): + # [('active', 'selected', 'grey'), ('focus', [1, 2, 3, 4])] + # returns: + # ['active selected', 'grey', 'focus', [1, 2, 3, 4]] + opt_val = [] + for *state, val in items: + # hacks for bakward compatibility + state[0] # raise IndexError if empty + if len(state) == 1: + # if it is empty (something that evaluates to False), then + # format it to Tcl code to denote the "normal" state + state = state[0] or '' + else: + # group multiple states + state = ' '.join(state) # raise TypeError if not str + opt_val.append(state) + if val is not None: + opt_val.append(val) + return opt_val def _format_mapdict(mapdict, script=False): """Formats mapdict to pass it to tk.call. @@ -90,32 +104,11 @@ returns: ('-expand', '{active selected} grey focus {1, 2, 3, 4}')""" - # if caller passes a Tcl script to tk.call, all the values need to - # be grouped into words (arguments to a command in Tcl dialect) - format = "%s" if not script else "{%s}" opts = [] for opt, value in mapdict.items(): - - opt_val = [] - # each value in mapdict is expected to be a sequence, where each item - # is another sequence containing a state (or several) and a value - for statespec in value: - state, val = statespec[:-1], statespec[-1] - - if len(state) > 1: # group multiple states - state = "{%s}" % ' '.join(state) - else: # single state - # if it is empty (something that evaluates to False), then - # format it to Tcl code to denote the "normal" state - state = state[0] or '{}' - - if isinstance(val, (list, tuple)): # val needs to be grouped - val = "{%s}" % ' '.join(map(str, val)) - - opt_val.append("%s %s" % (state, val)) - - opts.append(("-%s" % opt, format % ' '.join(opt_val))) + opts.extend(("-%s" % opt, + _format_optvalue(_mapdict_values(value), script))) return _flatten(opts) @@ -129,7 +122,7 @@ iname = args[0] # next args, if any, are statespec/value pairs which is almost # a mapdict, but we just need the value - imagespec = _format_mapdict({None: args[1:]})[1] + imagespec = _join(_mapdict_values(args[1:])) spec = "%s %s" % (iname, imagespec) else: @@ -138,7 +131,7 @@ # themed styles on Windows XP and Vista. # Availability: Tk 8.6, Windows XP and Vista. class_name, part_id = args[:2] - statemap = _format_mapdict({None: args[2:]})[1] + statemap = _join(_mapdict_values(args[2:])) spec = "%s %s %s" % (class_name, part_id, statemap) opts = _format_optdict(kw, script) @@ -148,11 +141,11 @@ # otherwise it will clone {} (empty element) spec = args[0] # theme name if len(args) > 1: # elementfrom specified - opts = (args[1], ) + opts = (_format_optvalue(args[1], script),) if script: spec = '{%s}' % spec - opts = ' '.join(map(str, opts)) + opts = ' '.join(opts) return spec, opts @@ -189,7 +182,7 @@ for layout_elem in layout: elem, opts = layout_elem opts = opts or {} - fopts = ' '.join(map(str, _format_optdict(opts, True, "children"))) + fopts = ' '.join(_format_optdict(opts, True, ("children",))) head = "%s%s%s" % (' ' * indent, elem, (" %s" % fopts) if fopts else '') if "children" in opts: @@ -215,11 +208,11 @@ for name, opts in settings.items(): # will format specific keys according to Tcl code if opts.get('configure'): # format 'configure' - s = ' '.join(map(str, _format_optdict(opts['configure'], True))) + s = ' '.join(_format_optdict(opts['configure'], True)) script.append("ttk::style configure %s %s;" % (name, s)) if opts.get('map'): # format 'map' - s = ' '.join(map(str, _format_mapdict(opts['map'], True))) + s = ' '.join(_format_mapdict(opts['map'], True)) script.append("ttk::style map %s %s;" % (name, s)) if 'layout' in opts: # format 'layout' which may be empty @@ -706,30 +699,9 @@ exportselection, justify, height, postcommand, state, textvariable, values, width """ - # The "values" option may need special formatting, so leave to - # _format_optdict the responsibility to format it - if "values" in kw: - kw["values"] = _format_optdict({'v': kw["values"]})[1] - Entry.__init__(self, master, "ttk::combobox", **kw) - def __setitem__(self, item, value): - if item == "values": - value = _format_optdict({item: value})[1] - - Entry.__setitem__(self, item, value) - - - def configure(self, cnf=None, **kw): - """Custom Combobox configure, created to properly format the values - option.""" - if "values" in kw: - kw["values"] = _format_optdict({'v': kw["values"]})[1] - - return Entry.configure(self, cnf, **kw) - - def current(self, newindex=None): """If newindex is supplied, sets the combobox value to the element at position newindex in the list of values. Otherwise, diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -150,6 +150,9 @@ Library ------- +- Issue #15861: tkinter now correctly works with lists and tuples containing + strings with whitespaces, backslashes or unbalanced braces. + - Issue #10527: Use poll() instead of select() for multiprocessing pipes. - Issue #9720: zipfile now writes correct local headers for files larger than -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 17:04:40 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 15 Jan 2013 17:04:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2315861=3A_tkinter_now_correctly_works_with_lists?= =?utf-8?q?_and_tuples_containing?= Message-ID: <3YlxFh2H0zzR76@mail.python.org> http://hg.python.org/cpython/rev/340e97ebe911 changeset: 81529:340e97ebe911 parent: 81526:fb17969ace93 parent: 81528:927352d7e994 user: Serhiy Storchaka date: Tue Jan 15 17:59:53 2013 +0200 summary: Issue #15861: tkinter now correctly works with lists and tuples containing strings with whitespaces, backslashes or unbalanced braces. files: Lib/tkinter/__init__.py | 31 ++- Lib/tkinter/test/test_ttk/test_functions.py | 40 ++- Lib/tkinter/test/test_ttk/test_widgets.py | 8 + Lib/tkinter/ttk.py | 122 +++------ Misc/NEWS | 3 + 5 files changed, 123 insertions(+), 81 deletions(-) diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -40,6 +40,7 @@ import _tkinter # If this fails your Python may not be configured for Tk TclError = _tkinter.TclError from tkinter.constants import * +import re wantobjects = 1 @@ -52,6 +53,34 @@ EXCEPTION = _tkinter.EXCEPTION +_magic_re = re.compile(r'([\\{}])') +_space_re = re.compile(r'([\s])', re.ASCII) + +def _join(value): + """Internal function.""" + return ' '.join(map(_stringify, value)) + +def _stringify(value): + """Internal function.""" + if isinstance(value, (list, tuple)): + if len(value) == 1: + value = _stringify(value[0]) + if value[0] == '{': + value = '{%s}' % value + else: + value = '{%s}' % _join(value) + else: + value = str(value) + if not value: + value = '{}' + elif _magic_re.search(value): + # add '\' before special characters and spaces + value = _magic_re.sub(r'\\\1', value) + value = _space_re.sub(r'\\\1', value) + elif value[0] == '"' or _space_re.search(value): + value = '{%s}' % value + return value + def _flatten(seq): """Internal function.""" res = () @@ -1089,7 +1118,7 @@ if isinstance(item, int): nv.append(str(item)) elif isinstance(item, str): - nv.append(('{%s}' if ' ' in item else '%s') % item) + nv.append(_stringify(item)) else: break else: diff --git a/Lib/tkinter/test/test_ttk/test_functions.py b/Lib/tkinter/test/test_ttk/test_functions.py --- a/Lib/tkinter/test/test_ttk/test_functions.py +++ b/Lib/tkinter/test/test_ttk/test_functions.py @@ -49,13 +49,17 @@ ttk._format_optdict({'test': {'left': 'as is'}}), {'-test': {'left': 'as is'}}) - # check script formatting and untouched value(s) + # check script formatting check_against( ttk._format_optdict( - {'test': [1, -1, '', '2m', 0], 'nochange1': 3, - 'nochange2': 'abc def'}, script=True), - {'-test': '{1 -1 {} 2m 0}', '-nochange1': 3, - '-nochange2': 'abc def' }) + {'test': [1, -1, '', '2m', 0], 'test2': 3, + 'test3': '', 'test4': 'abc def', + 'test5': '"abc"', 'test6': '{}', + 'test7': '} -spam {'}, script=True), + {'-test': '{1 -1 {} 2m 0}', '-test2': '3', + '-test3': '{}', '-test4': '{abc def}', + '-test5': '{"abc"}', '-test6': r'\{\}', + '-test7': r'\}\ -spam\ \{'}) opts = {'???': True, '?': False} orig_opts = opts.copy() @@ -69,6 +73,32 @@ ttk._format_optdict( {'option': ('one two', 'three')}), {'-option': '{one two} three'}) + check_against( + ttk._format_optdict( + {'option': ('one\ttwo', 'three')}), + {'-option': '{one\ttwo} three'}) + + # passing empty strings inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('', 'one')}), + {'-option': '{} one'}) + + # passing values with braces inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('one} {two', 'three')}), + {'-option': r'one\}\ \{two three'}) + + # passing quoted strings inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('"one"', 'two')}), + {'-option': '{"one"} two'}) + check_against( + ttk._format_optdict( + {'option': ('{one}', 'two')}), + {'-option': r'\{one\} two'}) # ignore an option amount_opts = len(ttk._format_optdict(opts, ignore=('?'))) / 2 diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -189,6 +189,14 @@ self.combo.configure(values=[1, '', 2]) self.assertEqual(self.combo['values'], ('1', '', '2')) + # testing values with spaces + self.combo['values'] = ['a b', 'a\tb', 'a\nb'] + self.assertEqual(self.combo['values'], ('a b', 'a\tb', 'a\nb')) + + # testing values with special characters + self.combo['values'] = [r'a\tb', '"a"', '} {'] + self.assertEqual(self.combo['values'], (r'a\tb', '"a"', '} {')) + # out of range self.assertRaises(tkinter.TclError, self.combo.current, len(self.combo['values'])) diff --git a/Lib/tkinter/ttk.py b/Lib/tkinter/ttk.py --- a/Lib/tkinter/ttk.py +++ b/Lib/tkinter/ttk.py @@ -26,8 +26,7 @@ "tclobjs_to_py", "setup_master"] import tkinter - -_flatten = tkinter._flatten +from tkinter import _flatten, _join, _stringify # Verify if Tk is new enough to not need the Tile package _REQUIRE_TILE = True if tkinter.TkVersion < 8.5 else False @@ -47,39 +46,54 @@ master.tk.eval('package require tile') # TclError may be raised here master._tile_loaded = True +def _format_optvalue(value, script=False): + """Internal function.""" + if script: + # if caller passes a Tcl script to tk.call, all the values need to + # be grouped into words (arguments to a command in Tcl dialect) + value = _stringify(value) + elif isinstance(value, (list, tuple)): + value = _join(value) + return value + def _format_optdict(optdict, script=False, ignore=None): """Formats optdict to a tuple to pass it to tk.call. E.g. (script=False): {'foreground': 'blue', 'padding': [1, 2, 3, 4]} returns: ('-foreground', 'blue', '-padding', '1 2 3 4')""" - format = "%s" if not script else "{%s}" opts = [] for opt, value in optdict.items(): - if ignore and opt in ignore: - continue + if not ignore or opt not in ignore: + opts.append("-%s" % opt) + if value is not None: + opts.append(_format_optvalue(value, script)) - if isinstance(value, (list, tuple)): - v = [] - for val in value: - if isinstance(val, str): - v.append(str(val) if val else '{}') - else: - v.append(str(val)) + return _flatten(opts) - # format v according to the script option, but also check for - # space in any value in v in order to group them correctly - value = format % ' '.join( - ('{%s}' if ' ' in val else '%s') % val for val in v) - - if script and value == '': - value = '{}' # empty string in Python is equivalent to {} in Tcl - - opts.append(("-%s" % opt, value)) - - # Remember: _flatten skips over None - return _flatten(opts) +def _mapdict_values(items): + # each value in mapdict is expected to be a sequence, where each item + # is another sequence containing a state (or several) and a value + # E.g. (script=False): + # [('active', 'selected', 'grey'), ('focus', [1, 2, 3, 4])] + # returns: + # ['active selected', 'grey', 'focus', [1, 2, 3, 4]] + opt_val = [] + for *state, val in items: + # hacks for bakward compatibility + state[0] # raise IndexError if empty + if len(state) == 1: + # if it is empty (something that evaluates to False), then + # format it to Tcl code to denote the "normal" state + state = state[0] or '' + else: + # group multiple states + state = ' '.join(state) # raise TypeError if not str + opt_val.append(state) + if val is not None: + opt_val.append(val) + return opt_val def _format_mapdict(mapdict, script=False): """Formats mapdict to pass it to tk.call. @@ -90,32 +104,11 @@ returns: ('-expand', '{active selected} grey focus {1, 2, 3, 4}')""" - # if caller passes a Tcl script to tk.call, all the values need to - # be grouped into words (arguments to a command in Tcl dialect) - format = "%s" if not script else "{%s}" opts = [] for opt, value in mapdict.items(): - - opt_val = [] - # each value in mapdict is expected to be a sequence, where each item - # is another sequence containing a state (or several) and a value - for statespec in value: - state, val = statespec[:-1], statespec[-1] - - if len(state) > 1: # group multiple states - state = "{%s}" % ' '.join(state) - else: # single state - # if it is empty (something that evaluates to False), then - # format it to Tcl code to denote the "normal" state - state = state[0] or '{}' - - if isinstance(val, (list, tuple)): # val needs to be grouped - val = "{%s}" % ' '.join(map(str, val)) - - opt_val.append("%s %s" % (state, val)) - - opts.append(("-%s" % opt, format % ' '.join(opt_val))) + opts.extend(("-%s" % opt, + _format_optvalue(_mapdict_values(value), script))) return _flatten(opts) @@ -129,7 +122,7 @@ iname = args[0] # next args, if any, are statespec/value pairs which is almost # a mapdict, but we just need the value - imagespec = _format_mapdict({None: args[1:]})[1] + imagespec = _join(_mapdict_values(args[1:])) spec = "%s %s" % (iname, imagespec) else: @@ -138,7 +131,7 @@ # themed styles on Windows XP and Vista. # Availability: Tk 8.6, Windows XP and Vista. class_name, part_id = args[:2] - statemap = _format_mapdict({None: args[2:]})[1] + statemap = _join(_mapdict_values(args[2:])) spec = "%s %s %s" % (class_name, part_id, statemap) opts = _format_optdict(kw, script) @@ -148,11 +141,11 @@ # otherwise it will clone {} (empty element) spec = args[0] # theme name if len(args) > 1: # elementfrom specified - opts = (args[1], ) + opts = (_format_optvalue(args[1], script),) if script: spec = '{%s}' % spec - opts = ' '.join(map(str, opts)) + opts = ' '.join(opts) return spec, opts @@ -189,7 +182,7 @@ for layout_elem in layout: elem, opts = layout_elem opts = opts or {} - fopts = ' '.join(map(str, _format_optdict(opts, True, "children"))) + fopts = ' '.join(_format_optdict(opts, True, ("children",))) head = "%s%s%s" % (' ' * indent, elem, (" %s" % fopts) if fopts else '') if "children" in opts: @@ -215,11 +208,11 @@ for name, opts in settings.items(): # will format specific keys according to Tcl code if opts.get('configure'): # format 'configure' - s = ' '.join(map(str, _format_optdict(opts['configure'], True))) + s = ' '.join(_format_optdict(opts['configure'], True)) script.append("ttk::style configure %s %s;" % (name, s)) if opts.get('map'): # format 'map' - s = ' '.join(map(str, _format_mapdict(opts['map'], True))) + s = ' '.join(_format_mapdict(opts['map'], True)) script.append("ttk::style map %s %s;" % (name, s)) if 'layout' in opts: # format 'layout' which may be empty @@ -706,30 +699,9 @@ exportselection, justify, height, postcommand, state, textvariable, values, width """ - # The "values" option may need special formatting, so leave to - # _format_optdict the responsibility to format it - if "values" in kw: - kw["values"] = _format_optdict({'v': kw["values"]})[1] - Entry.__init__(self, master, "ttk::combobox", **kw) - def __setitem__(self, item, value): - if item == "values": - value = _format_optdict({item: value})[1] - - Entry.__setitem__(self, item, value) - - - def configure(self, cnf=None, **kw): - """Custom Combobox configure, created to properly format the values - option.""" - if "values" in kw: - kw["values"] = _format_optdict({'v': kw["values"]})[1] - - return Entry.configure(self, cnf, **kw) - - def current(self, newindex=None): """If newindex is supplied, sets the combobox value to the element at position newindex in the list of values. Otherwise, diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -220,6 +220,9 @@ Library ------- +- Issue #15861: tkinter now correctly works with lists and tuples containing + strings with whitespaces, backslashes or unbalanced braces. + - Issue #9720: zipfile now writes correct local headers for files larger than 4 GiB. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 17:04:42 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 15 Jan 2013 17:04:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE1ODYx?= =?utf-8?q?=3A_tkinter_now_correctly_works_with_lists_and_tuples_containin?= =?utf-8?q?g?= Message-ID: <3YlxFk0rtWzQp7@mail.python.org> http://hg.python.org/cpython/rev/917ae14831ec changeset: 81530:917ae14831ec branch: 2.7 parent: 81521:2e1f1b096f3d user: Serhiy Storchaka date: Tue Jan 15 18:01:21 2013 +0200 summary: Issue #15861: tkinter now correctly works with lists and tuples containing strings with whitespaces, backslashes or unbalanced braces. files: Lib/lib-tk/Tkinter.py | 34 ++- Lib/lib-tk/test/test_ttk/test_functions.py | 40 ++- Lib/lib-tk/test/test_ttk/test_widgets.py | 8 + Lib/lib-tk/ttk.py | 124 +++------ Misc/NEWS | 3 + 5 files changed, 128 insertions(+), 81 deletions(-) diff --git a/Lib/lib-tk/Tkinter.py b/Lib/lib-tk/Tkinter.py --- a/Lib/lib-tk/Tkinter.py +++ b/Lib/lib-tk/Tkinter.py @@ -41,6 +41,7 @@ TclError = _tkinter.TclError from types import * from Tkconstants import * +import re wantobjects = 1 @@ -58,6 +59,37 @@ except AttributeError: _tkinter.deletefilehandler = None +_magic_re = re.compile(r'([\\{}])') +_space_re = re.compile(r'([\s])') + +def _join(value): + """Internal function.""" + return ' '.join(map(_stringify, value)) + +def _stringify(value): + """Internal function.""" + if isinstance(value, (list, tuple)): + if len(value) == 1: + value = _stringify(value[0]) + if value[0] == '{': + value = '{%s}' % value + else: + value = '{%s}' % _join(value) + else: + if isinstance(value, basestring): + value = unicode(value) + else: + value = str(value) + if not value: + value = '{}' + elif _magic_re.search(value): + # add '\' before special characters and spaces + value = _magic_re.sub(r'\\\1', value) + value = _space_re.sub(r'\\\1', value) + elif value[0] == '"' or _space_re.search(value): + value = '{%s}' % value + return value + def _flatten(tuple): """Internal function.""" res = () @@ -1086,7 +1118,7 @@ nv.append('%d' % item) else: # format it to proper Tcl code if it contains space - nv.append(('{%s}' if ' ' in item else '%s') % item) + nv.append(_stringify(item)) else: v = ' '.join(nv) res = res + ('-'+k, v) diff --git a/Lib/lib-tk/test/test_ttk/test_functions.py b/Lib/lib-tk/test/test_ttk/test_functions.py --- a/Lib/lib-tk/test/test_ttk/test_functions.py +++ b/Lib/lib-tk/test/test_ttk/test_functions.py @@ -50,13 +50,17 @@ ttk._format_optdict({'test': {'left': 'as is'}}), {'-test': {'left': 'as is'}}) - # check script formatting and untouched value(s) + # check script formatting check_against( ttk._format_optdict( - {'test': [1, -1, '', '2m', 0], 'nochange1': 3, - 'nochange2': 'abc def'}, script=True), - {'-test': '{1 -1 {} 2m 0}', '-nochange1': 3, - '-nochange2': 'abc def' }) + {'test': [1, -1, '', '2m', 0], 'test2': 3, + 'test3': '', 'test4': 'abc def', + 'test5': '"abc"', 'test6': '{}', + 'test7': '} -spam {'}, script=True), + {'-test': '{1 -1 {} 2m 0}', '-test2': '3', + '-test3': '{}', '-test4': '{abc def}', + '-test5': '{"abc"}', '-test6': r'\{\}', + '-test7': r'\}\ -spam\ \{'}) opts = {u'???': True, u'?': False} orig_opts = opts.copy() @@ -70,6 +74,32 @@ ttk._format_optdict( {'option': ('one two', 'three')}), {'-option': '{one two} three'}) + check_against( + ttk._format_optdict( + {'option': ('one\ttwo', 'three')}), + {'-option': '{one\ttwo} three'}) + + # passing empty strings inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('', 'one')}), + {'-option': '{} one'}) + + # passing values with braces inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('one} {two', 'three')}), + {'-option': r'one\}\ \{two three'}) + + # passing quoted strings inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('"one"', 'two')}), + {'-option': '{"one"} two'}) + check_against( + ttk._format_optdict( + {'option': ('{one}', 'two')}), + {'-option': r'\{one\} two'}) # ignore an option amount_opts = len(ttk._format_optdict(opts, ignore=(u'?'))) // 2 diff --git a/Lib/lib-tk/test/test_ttk/test_widgets.py b/Lib/lib-tk/test/test_ttk/test_widgets.py --- a/Lib/lib-tk/test/test_ttk/test_widgets.py +++ b/Lib/lib-tk/test/test_ttk/test_widgets.py @@ -188,6 +188,14 @@ self.combo.configure(values=[1, '', 2]) self.assertEqual(self.combo['values'], ('1', '', '2')) + # testing values with spaces + self.combo['values'] = ['a b', 'a\tb', 'a\nb'] + self.assertEqual(self.combo['values'], ('a b', 'a\tb', 'a\nb')) + + # testing values with special characters + self.combo['values'] = [r'a\tb', '"a"', '} {'] + self.assertEqual(self.combo['values'], (r'a\tb', '"a"', '} {')) + # out of range self.assertRaises(Tkinter.TclError, self.combo.current, len(self.combo['values'])) diff --git a/Lib/lib-tk/ttk.py b/Lib/lib-tk/ttk.py --- a/Lib/lib-tk/ttk.py +++ b/Lib/lib-tk/ttk.py @@ -26,8 +26,7 @@ "tclobjs_to_py", "setup_master"] import Tkinter - -_flatten = Tkinter._flatten +from Tkinter import _flatten, _join, _stringify # Verify if Tk is new enough to not need the Tile package _REQUIRE_TILE = True if Tkinter.TkVersion < 8.5 else False @@ -47,39 +46,56 @@ master.tk.eval('package require tile') # TclError may be raised here master._tile_loaded = True +def _format_optvalue(value, script=False): + """Internal function.""" + if script: + # if caller passes a Tcl script to tk.call, all the values need to + # be grouped into words (arguments to a command in Tcl dialect) + value = _stringify(value) + elif isinstance(value, (list, tuple)): + value = _join(value) + return value + def _format_optdict(optdict, script=False, ignore=None): """Formats optdict to a tuple to pass it to tk.call. E.g. (script=False): {'foreground': 'blue', 'padding': [1, 2, 3, 4]} returns: ('-foreground', 'blue', '-padding', '1 2 3 4')""" - format = "%s" if not script else "{%s}" opts = [] for opt, value in optdict.iteritems(): - if ignore and opt in ignore: - continue + if not ignore or opt not in ignore: + opts.append("-%s" % opt) + if value is not None: + opts.append(_format_optvalue(value, script)) - if isinstance(value, (list, tuple)): - v = [] - for val in value: - if isinstance(val, basestring): - v.append(unicode(val) if val else '{}') - else: - v.append(str(val)) + return _flatten(opts) - # format v according to the script option, but also check for - # space in any value in v in order to group them correctly - value = format % ' '.join( - ('{%s}' if ' ' in val else '%s') % val for val in v) - - if script and value == '': - value = '{}' # empty string in Python is equivalent to {} in Tcl - - opts.append(("-%s" % opt, value)) - - # Remember: _flatten skips over None - return _flatten(opts) +def _mapdict_values(items): + # each value in mapdict is expected to be a sequence, where each item + # is another sequence containing a state (or several) and a value + # E.g. (script=False): + # [('active', 'selected', 'grey'), ('focus', [1, 2, 3, 4])] + # returns: + # ['active selected', 'grey', 'focus', [1, 2, 3, 4]] + opt_val = [] + for item in items: + state = item[:-1] + val = item[-1] + # hacks for bakward compatibility + state[0] # raise IndexError if empty + if len(state) == 1: + # if it is empty (something that evaluates to False), then + # format it to Tcl code to denote the "normal" state + state = state[0] or '' + else: + # group multiple states + state = ' '.join(state) # raise TypeError if not str + opt_val.append(state) + if val is not None: + opt_val.append(val) + return opt_val def _format_mapdict(mapdict, script=False): """Formats mapdict to pass it to tk.call. @@ -90,32 +106,11 @@ returns: ('-expand', '{active selected} grey focus {1, 2, 3, 4}')""" - # if caller passes a Tcl script to tk.call, all the values need to - # be grouped into words (arguments to a command in Tcl dialect) - format = "%s" if not script else "{%s}" opts = [] for opt, value in mapdict.iteritems(): - - opt_val = [] - # each value in mapdict is expected to be a sequence, where each item - # is another sequence containing a state (or several) and a value - for statespec in value: - state, val = statespec[:-1], statespec[-1] - - if len(state) > 1: # group multiple states - state = "{%s}" % ' '.join(state) - else: # single state - # if it is empty (something that evaluates to False), then - # format it to Tcl code to denote the "normal" state - state = state[0] or '{}' - - if isinstance(val, (list, tuple)): # val needs to be grouped - val = "{%s}" % ' '.join(map(str, val)) - - opt_val.append("%s %s" % (state, val)) - - opts.append(("-%s" % opt, format % ' '.join(opt_val))) + opts.extend(("-%s" % opt, + _format_optvalue(_mapdict_values(value), script))) return _flatten(opts) @@ -129,7 +124,7 @@ iname = args[0] # next args, if any, are statespec/value pairs which is almost # a mapdict, but we just need the value - imagespec = _format_mapdict({None: args[1:]})[1] + imagespec = _join(_mapdict_values(args[1:])) spec = "%s %s" % (iname, imagespec) else: @@ -138,7 +133,7 @@ # themed styles on Windows XP and Vista. # Availability: Tk 8.6, Windows XP and Vista. class_name, part_id = args[:2] - statemap = _format_mapdict({None: args[2:]})[1] + statemap = _join(_mapdict_values(args[2:])) spec = "%s %s %s" % (class_name, part_id, statemap) opts = _format_optdict(kw, script) @@ -148,11 +143,11 @@ # otherwise it will clone {} (empty element) spec = args[0] # theme name if len(args) > 1: # elementfrom specified - opts = (args[1], ) + opts = (_format_optvalue(args[1], script),) if script: spec = '{%s}' % spec - opts = ' '.join(map(str, opts)) + opts = ' '.join(opts) return spec, opts @@ -189,7 +184,7 @@ for layout_elem in layout: elem, opts = layout_elem opts = opts or {} - fopts = ' '.join(map(str, _format_optdict(opts, True, "children"))) + fopts = ' '.join(_format_optdict(opts, True, ("children",))) head = "%s%s%s" % (' ' * indent, elem, (" %s" % fopts) if fopts else '') if "children" in opts: @@ -215,11 +210,11 @@ for name, opts in settings.iteritems(): # will format specific keys according to Tcl code if opts.get('configure'): # format 'configure' - s = ' '.join(map(unicode, _format_optdict(opts['configure'], True))) + s = ' '.join(_format_optdict(opts['configure'], True)) script.append("ttk::style configure %s %s;" % (name, s)) if opts.get('map'): # format 'map' - s = ' '.join(map(unicode, _format_mapdict(opts['map'], True))) + s = ' '.join(_format_mapdict(opts['map'], True)) script.append("ttk::style map %s %s;" % (name, s)) if 'layout' in opts: # format 'layout' which may be empty @@ -706,30 +701,9 @@ exportselection, justify, height, postcommand, state, textvariable, values, width """ - # The "values" option may need special formatting, so leave to - # _format_optdict the responsibility to format it - if "values" in kw: - kw["values"] = _format_optdict({'v': kw["values"]})[1] - Entry.__init__(self, master, "ttk::combobox", **kw) - def __setitem__(self, item, value): - if item == "values": - value = _format_optdict({item: value})[1] - - Entry.__setitem__(self, item, value) - - - def configure(self, cnf=None, **kw): - """Custom Combobox configure, created to properly format the values - option.""" - if "values" in kw: - kw["values"] = _format_optdict({'v': kw["values"]})[1] - - return Entry.configure(self, cnf, **kw) - - def current(self, newindex=None): """If newindex is supplied, sets the combobox value to the element at position newindex in the list of values. Otherwise, diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -189,6 +189,9 @@ Library ------- +- Issue #15861: tkinter now correctly works with lists and tuples containing + strings with whitespaces, backslashes or unbalanced braces. + - Issue #10527: Use poll() instead of select() for multiprocessing pipes. - Issue #9720: zipfile now writes correct local headers for files larger than -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 18:57:35 2013 From: python-checkins at python.org (vinay.sajip) Date: Tue, 15 Jan 2013 18:57:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzk1MDE6?= =?utf-8?q?_Improved_shutdown_handling_to_deal_with_module_attributes?= Message-ID: <3Ylzlz1sT2zNy8@mail.python.org> http://hg.python.org/cpython/rev/966887830011 changeset: 81531:966887830011 branch: 2.7 user: Vinay Sajip date: Tue Jan 15 17:55:13 2013 +0000 summary: Issue #9501: Improved shutdown handling to deal with module attributes correctly. files: Lib/logging/__init__.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -624,7 +624,8 @@ # This function can be called during module teardown, when globals are # set to None. If _acquireLock is None, assume this is the case and do # nothing. - if _acquireLock is not None: + if (_acquireLock is not None and _handlerList is not None and + _releaseLock is not None): _acquireLock() try: if wr in _handlerList: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 18:57:36 2013 From: python-checkins at python.org (vinay.sajip) Date: Tue, 15 Jan 2013 18:57:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzk1MDE6?= =?utf-8?q?_Improved_shutdown_handling_to_deal_with_module_attributes?= Message-ID: <3Ylzm059HqzRC1@mail.python.org> http://hg.python.org/cpython/rev/8eac88f49cc0 changeset: 81532:8eac88f49cc0 branch: 3.2 parent: 81527:411bb75be5d1 user: Vinay Sajip date: Tue Jan 15 17:55:57 2013 +0000 summary: Issue #9501: Improved shutdown handling to deal with module attributes correctly. files: Lib/logging/__init__.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -711,7 +711,8 @@ # This function can be called during module teardown, when globals are # set to None. If _acquireLock is None, assume this is the case and do # nothing. - if _acquireLock is not None: + if (_acquireLock is not None and _handlerList is not None and + _releaseLock is not None): _acquireLock() try: if wr in _handlerList: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 18:57:39 2013 From: python-checkins at python.org (vinay.sajip) Date: Tue, 15 Jan 2013 18:57:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=239501=3A_Merged_fix_from_3=2E2=2E?= Message-ID: <3Ylzm313PqzPQD@mail.python.org> http://hg.python.org/cpython/rev/3161a94ff73c changeset: 81533:3161a94ff73c branch: 3.3 parent: 81528:927352d7e994 parent: 81532:8eac88f49cc0 user: Vinay Sajip date: Tue Jan 15 17:56:43 2013 +0000 summary: Issue #9501: Merged fix from 3.2. files: Lib/logging/__init__.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -708,7 +708,8 @@ # This function can be called during module teardown, when globals are # set to None. If _acquireLock is None, assume this is the case and do # nothing. - if _acquireLock is not None: + if (_acquireLock is not None and _handlerList is not None and + _releaseLock is not None): _acquireLock() try: if wr in _handlerList: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 18:57:40 2013 From: python-checkins at python.org (vinay.sajip) Date: Tue, 15 Jan 2013 18:57:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=239501=3A_Merged_fix_from_3=2E3=2E?= Message-ID: <3Ylzm445bjzRC1@mail.python.org> http://hg.python.org/cpython/rev/4fd8c35a05b2 changeset: 81534:4fd8c35a05b2 parent: 81529:340e97ebe911 parent: 81533:3161a94ff73c user: Vinay Sajip date: Tue Jan 15 17:57:18 2013 +0000 summary: Issue #9501: Merged fix from 3.3. files: Lib/logging/__init__.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -708,7 +708,8 @@ # This function can be called during module teardown, when globals are # set to None. If _acquireLock is None, assume this is the case and do # nothing. - if _acquireLock is not None: + if (_acquireLock is not None and _handlerList is not None and + _releaseLock is not None): _acquireLock() try: if wr in _handlerList: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 22:25:12 2013 From: python-checkins at python.org (philip.jenvey) Date: Tue, 15 Jan 2013 22:25:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_thinko?= Message-ID: <3Ym4MX0fjgzS14@mail.python.org> http://hg.python.org/cpython/rev/c044029ede9d changeset: 81535:c044029ede9d user: Philip Jenvey date: Tue Jan 15 13:24:12 2013 -0800 summary: thinko files: Python/fileutils.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Python/fileutils.c b/Python/fileutils.c --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -60,7 +60,7 @@ workaround is also enabled on error, for example if getting the locale failed. - Values of locale_is_ascii: + Values of force_ascii: 1: the workaround is used: _Py_wchar2char() uses encode_ascii_surrogateescape() and _Py_char2wchar() uses -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 22:49:43 2013 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 15 Jan 2013 22:49:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_remove_warning?= =?utf-8?q?_about_tb_circular_reference_=28closes_=237340=29?= Message-ID: <3Ym4vq0DRzzRGV@mail.python.org> http://hg.python.org/cpython/rev/3fa3e7975724 changeset: 81536:3fa3e7975724 branch: 3.3 parent: 81533:3161a94ff73c user: Benjamin Peterson date: Tue Jan 15 16:49:22 2013 -0500 summary: remove warning about tb circular reference (closes #7340) files: Doc/library/sys.rst | 15 --------------- 1 files changed, 0 insertions(+), 15 deletions(-) diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -215,21 +215,6 @@ a traceback object (see the Reference Manual) which encapsulates the call stack at the point where the exception originally occurred. - .. warning:: - - Assigning the *traceback* return value to a local variable in a function - that is handling an exception will cause a circular reference. Since most - functions don't need access to the traceback, the best solution is to use - something like ``exctype, value = sys.exc_info()[:2]`` to extract only the - exception type and value. If you do need the traceback, make sure to - delete it after use (best done with a :keyword:`try` - ... :keyword:`finally` statement) or to call :func:`exc_info` in a - function that does not itself handle an exception. - - Such cycles are normally automatically reclaimed when garbage collection - is enabled and they become unreachable, but it remains more efficient to - avoid creating cycles. - .. data:: exec_prefix -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Jan 15 22:49:44 2013 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 15 Jan 2013 22:49:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4zICgjNzM0MCk=?= Message-ID: <3Ym4vr30C0zS0l@mail.python.org> http://hg.python.org/cpython/rev/d866bbdd68e8 changeset: 81537:d866bbdd68e8 parent: 81535:c044029ede9d parent: 81536:3fa3e7975724 user: Benjamin Peterson date: Tue Jan 15 16:49:35 2013 -0500 summary: merge 3.3 (#7340) files: Doc/library/sys.rst | 15 --------------- 1 files changed, 0 insertions(+), 15 deletions(-) diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -215,21 +215,6 @@ a traceback object (see the Reference Manual) which encapsulates the call stack at the point where the exception originally occurred. - .. warning:: - - Assigning the *traceback* return value to a local variable in a function - that is handling an exception will cause a circular reference. Since most - functions don't need access to the traceback, the best solution is to use - something like ``exctype, value = sys.exc_info()[:2]`` to extract only the - exception type and value. If you do need the traceback, make sure to - delete it after use (best done with a :keyword:`try` - ... :keyword:`finally` statement) or to call :func:`exc_info` in a - function that does not itself handle an exception. - - Such cycles are normally automatically reclaimed when garbage collection - is enabled and they become unreachable, but it remains more efficient to - avoid creating cycles. - .. data:: exec_prefix -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Wed Jan 16 05:58:30 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 16 Jan 2013 05:58:30 +0100 Subject: [Python-checkins] Daily reference leaks (d866bbdd68e8): sum=11 Message-ID: results for d866bbdd68e8 on branch "default" -------------------------------------------- test_concurrent_futures leaked [2, 1, 1] memory blocks, sum=4 test_site leaked [2, 0, 0] references, sum=2 test_site leaked [2, 2, 1] memory blocks, sum=5 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogFrR2N3', '-x'] From python-checkins at python.org Wed Jan 16 13:02:11 2013 From: python-checkins at python.org (stefan.krah) Date: Wed, 16 Jan 2013 13:02:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE2NDIy?= =?utf-8?q?=3A_Use_strings_for_rounding_mode_constants_for_better_readabil?= =?utf-8?q?ity?= Message-ID: <3YmRqR4pD7zQSc@mail.python.org> http://hg.python.org/cpython/rev/733bb4fd9888 changeset: 81538:733bb4fd9888 branch: 3.3 parent: 81536:3fa3e7975724 user: Stefan Krah date: Wed Jan 16 12:58:59 2013 +0100 summary: Issue #16422: Use strings for rounding mode constants for better readability and pickling compatibility. files: Lib/test/test_decimal.py | 119 +++++++--------- Misc/NEWS | 3 + Modules/_decimal/_decimal.c | 122 +++++++--------- Modules/_decimal/tests/deccheck.py | 36 +--- 4 files changed, 127 insertions(+), 153 deletions(-) diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -78,14 +78,20 @@ d = getattr(context, attr) cls.assertTrue(all(d[s] if s in expected else not d[s] for s in d)) -RoundingModes = { - C: (C.ROUND_UP, C.ROUND_DOWN, C.ROUND_CEILING, C.ROUND_FLOOR, - C.ROUND_HALF_UP, C.ROUND_HALF_DOWN, C.ROUND_HALF_EVEN, - C.ROUND_05UP) if C else None, - P: (P.ROUND_UP, P.ROUND_DOWN, P.ROUND_CEILING, P.ROUND_FLOOR, - P.ROUND_HALF_UP, P.ROUND_HALF_DOWN, P.ROUND_HALF_EVEN, - P.ROUND_05UP) -} +ROUND_UP = P.ROUND_UP +ROUND_DOWN = P.ROUND_DOWN +ROUND_CEILING = P.ROUND_CEILING +ROUND_FLOOR = P.ROUND_FLOOR +ROUND_HALF_UP = P.ROUND_HALF_UP +ROUND_HALF_DOWN = P.ROUND_HALF_DOWN +ROUND_HALF_EVEN = P.ROUND_HALF_EVEN +ROUND_05UP = P.ROUND_05UP + +RoundingModes = [ + ROUND_UP, ROUND_DOWN, ROUND_CEILING, ROUND_FLOOR, + ROUND_HALF_UP, ROUND_HALF_DOWN, ROUND_HALF_EVEN, + ROUND_05UP +] # Tests are built around these assumed context defaults. # test_main() restores the original context. @@ -96,7 +102,7 @@ def init(m): if not m: return DefaultTestContext = m.Context( - prec=9, rounding=m.ROUND_HALF_EVEN, traps=dict.fromkeys(Signals[m], 0) + prec=9, rounding=ROUND_HALF_EVEN, traps=dict.fromkeys(Signals[m], 0) ) m.setcontext(DefaultTestContext) @@ -229,14 +235,14 @@ 'xor':'logical_xor'} # Map test-case names to roundings. - self.RoundingDict = {'ceiling' : self.decimal.ROUND_CEILING, - 'down' : self.decimal.ROUND_DOWN, - 'floor' : self.decimal.ROUND_FLOOR, - 'half_down' : self.decimal.ROUND_HALF_DOWN, - 'half_even' : self.decimal.ROUND_HALF_EVEN, - 'half_up' : self.decimal.ROUND_HALF_UP, - 'up' : self.decimal.ROUND_UP, - '05up' : self.decimal.ROUND_05UP} + self.RoundingDict = {'ceiling' : ROUND_CEILING, + 'down' : ROUND_DOWN, + 'floor' : ROUND_FLOOR, + 'half_down' : ROUND_HALF_DOWN, + 'half_even' : ROUND_HALF_EVEN, + 'half_up' : ROUND_HALF_UP, + 'up' : ROUND_UP, + '05up' : ROUND_05UP} # Map the test cases' error names to the actual errors. self.ErrorNames = {'clamped' : self.decimal.Clamped, @@ -2101,9 +2107,6 @@ Inexact = self.decimal.Inexact Rounded = self.decimal.Rounded Clamped = self.decimal.Clamped - ROUND_HALF_EVEN = self.decimal.ROUND_HALF_EVEN - ROUND_DOWN = self.decimal.ROUND_DOWN - ROUND_UP = self.decimal.ROUND_UP with localcontext(Context()) as c: c.prec = 7 @@ -2430,7 +2433,6 @@ def test_int(self): Decimal = self.decimal.Decimal - ROUND_DOWN = self.decimal.ROUND_DOWN for x in range(-250, 250): s = '%0.2f' % (x / 100.0) @@ -2448,7 +2450,6 @@ def test_trunc(self): Decimal = self.decimal.Decimal - ROUND_DOWN = self.decimal.ROUND_DOWN for x in range(-250, 250): s = '%0.2f' % (x / 100.0) @@ -2491,8 +2492,6 @@ def test_create_decimal_from_float(self): Decimal = self.decimal.Decimal Context = self.decimal.Context - ROUND_DOWN = self.decimal.ROUND_DOWN - ROUND_UP = self.decimal.ROUND_UP Inexact = self.decimal.Inexact context = Context(prec=5, rounding=ROUND_DOWN) @@ -2522,7 +2521,6 @@ Decimal = self.decimal.Decimal Context = self.decimal.Context InvalidOperation = self.decimal.InvalidOperation - ROUND_DOWN = self.decimal.ROUND_DOWN c = Context(Emax=99999, Emin=-99999) self.assertEqual( @@ -2723,7 +2721,6 @@ InvalidOperation = self.decimal.InvalidOperation DivisionByZero = self.decimal.DivisionByZero Overflow = self.decimal.Overflow - ROUND_HALF_EVEN = self.decimal.ROUND_HALF_EVEN c1 = Context() c2 = Context(prec=None, rounding=None, Emax=None, Emin=None, @@ -2739,6 +2736,21 @@ assert_signals(self, c, 'traps', [InvalidOperation, DivisionByZero, Overflow]) + @cpython_only + def test_from_legacy_strings(self): + import _testcapi + c = self.decimal.Context() + + for rnd in RoundingModes: + c.rounding = _testcapi.unicode_legacy_string(rnd) + self.assertEqual(c.rounding, rnd) + + s = _testcapi.unicode_legacy_string('') + self.assertRaises(TypeError, setattr, c, 'rounding', s) + + s = _testcapi.unicode_legacy_string('ROUND_\x00UP') + self.assertRaises(TypeError, setattr, c, 'rounding', s) + def test_pickle(self): Context = self.decimal.Context @@ -2762,7 +2774,7 @@ # Test interchangeability combinations = [(C, P), (P, C)] if C else [(P, P)] for dumper, loader in combinations: - for ri, _ in enumerate(RoundingModes[dumper]): + for ri, _ in enumerate(RoundingModes): for fi, _ in enumerate(OrderedSignals[dumper]): for ti, _ in enumerate(OrderedSignals[dumper]): @@ -2776,7 +2788,7 @@ sys.modules['decimal'] = dumper c = dumper.Context( prec=prec, Emin=emin, Emax=emax, - rounding=RoundingModes[dumper][ri], + rounding=RoundingModes[ri], capitals=caps, clamp=clamp, flags=OrderedSignals[dumper][:fi], traps=OrderedSignals[dumper][:ti] @@ -2791,7 +2803,7 @@ self.assertEqual(d.prec, prec) self.assertEqual(d.Emin, emin) self.assertEqual(d.Emax, emax) - self.assertEqual(d.rounding, RoundingModes[loader][ri]) + self.assertEqual(d.rounding, RoundingModes[ri]) self.assertEqual(d.capitals, caps) self.assertEqual(d.clamp, clamp) assert_signals(self, d, 'flags', OrderedSignals[loader][:fi]) @@ -3593,7 +3605,6 @@ Underflow = self.decimal.Underflow Clamped = self.decimal.Clamped Subnormal = self.decimal.Subnormal - ROUND_HALF_EVEN = self.decimal.ROUND_HALF_EVEN def raise_error(context, flag): if self.decimal == C: @@ -3960,17 +3971,6 @@ self.assertRaises(ValueError, setattr, c, 'Emin', 1) self.assertRaises(TypeError, setattr, c, 'Emin', (1,2,3)) - # rounding: always raise TypeError in order to get consistent - # exceptions across implementations. In decimal, rounding - # modes are strings, in _decimal they are integers. The idea - # is to view rounding as an abstract type and not mind the - # implementation details. - # Hence, a user should view the rounding modes as if they - # had been defined in a language that supports abstract - # data types, e.g. ocaml: - # - # type rounding = ROUND_DOWN | ROUND_HALF_UP | ... ;; - # self.assertRaises(TypeError, setattr, c, 'rounding', -1) self.assertRaises(TypeError, setattr, c, 'rounding', 9) self.assertRaises(TypeError, setattr, c, 'rounding', 1.0) @@ -4023,8 +4023,6 @@ decimal = self.decimal Decimal = decimal.Decimal Context = decimal.Context - ROUND_HALF_EVEN = decimal.ROUND_HALF_EVEN - ROUND_DOWN = decimal.ROUND_DOWN Clamped = decimal.Clamped DivisionByZero = decimal.DivisionByZero Inexact = decimal.Inexact @@ -4192,7 +4190,7 @@ c.prec = 425000000 c.Emax = 425000000 c.Emin = -425000000 - c.rounding = self.decimal.ROUND_HALF_DOWN + c.rounding = ROUND_HALF_DOWN c.capitals = 0 c.clamp = 1 for sig in OrderedSignals[self.decimal]: @@ -4584,7 +4582,6 @@ def test_py_rescale(self): # Coverage Decimal = P.Decimal - ROUND_UP = P.ROUND_UP localcontext = P.localcontext with localcontext() as c: @@ -4594,7 +4591,6 @@ def test_py__round(self): # Coverage Decimal = P.Decimal - ROUND_UP = P.ROUND_UP self.assertRaises(ValueError, Decimal("3.1234")._round, 0, ROUND_UP) @@ -4663,11 +4659,6 @@ self.assertEqual(C.DECIMAL128, 128) self.assertEqual(C.IEEE_CONTEXT_MAX_BITS, 512) - # Rounding modes - for i, v in enumerate(RoundingModes[C]): - self.assertEqual(v, i) - self.assertEqual(C.ROUND_TRUNC, 8) - # Conditions for i, v in enumerate(cond): self.assertEqual(v, 1<val)); } + /* Init string constants */ + for (i = 0; i < _PY_DEC_ROUND_GUARD; i++) { + ASSIGN_PTR(round_map[i], PyUnicode_InternFromString(mpd_round_string[i])); + Py_INCREF(round_map[i]); + CHECK_INT(PyModule_AddObject(m, mpd_round_string[i], round_map[i])); + } + /* Add specification version number */ CHECK_INT(PyModule_AddStringConstant(m, "__version__", " 1.70")); diff --git a/Modules/_decimal/tests/deccheck.py b/Modules/_decimal/tests/deccheck.py --- a/Modules/_decimal/tests/deccheck.py +++ b/Modules/_decimal/tests/deccheck.py @@ -158,17 +158,9 @@ C.FloatOperation: P.FloatOperation, } -RoundMap = { - C.ROUND_UP: P.ROUND_UP, - C.ROUND_DOWN: P.ROUND_DOWN, - C.ROUND_CEILING: P.ROUND_CEILING, - C.ROUND_FLOOR: P.ROUND_FLOOR, - C.ROUND_HALF_UP: P.ROUND_HALF_UP, - C.ROUND_HALF_DOWN: P.ROUND_HALF_DOWN, - C.ROUND_HALF_EVEN: P.ROUND_HALF_EVEN, - C.ROUND_05UP: P.ROUND_05UP -} -RoundModes = RoundMap.items() +RoundModes = [C.ROUND_UP, C.ROUND_DOWN, C.ROUND_CEILING, C.ROUND_FLOOR, + C.ROUND_HALF_UP, C.ROUND_HALF_DOWN, C.ROUND_HALF_EVEN, + C.ROUND_05UP] class Context(object): @@ -183,7 +175,7 @@ self.p.prec = self.c.prec self.p.Emin = self.c.Emin self.p.Emax = self.c.Emax - self.p.rounding = RoundMap[self.c.rounding] + self.p.rounding = self.c.rounding self.p.capitals = self.c.capitals self.settraps([sig for sig in self.c.traps if self.c.traps[sig]]) self.setstatus([sig for sig in self.c.flags if self.c.flags[sig]]) @@ -217,12 +209,12 @@ self.p.Emax = val def getround(self): - assert(self.c.rounding == RoundMap[self.p.rounding]) + assert(self.c.rounding == self.p.rounding) return self.c.rounding def setround(self, val): self.c.rounding = val - self.p.rounding = RoundMap[val] + self.p.rounding = val def getcapitals(self): assert(self.c.capitals == self.p.capitals) @@ -627,8 +619,12 @@ context.clear_status() - if not t.contextfunc and i == 0 or \ - convstr and isinstance(op, str): + if op in RoundModes: + t.cop.append(op) + t.pop.append(op) + + elif not t.contextfunc and i == 0 or \ + convstr and isinstance(op, str): try: c = C.Decimal(op) cex = None @@ -662,10 +658,6 @@ t.cop.append(op.c) t.pop.append(op.p) - elif op in RoundModes: - t.cop.append(op[0]) - t.pop.append(op[1]) - else: t.cop.append(op) t.pop.append(op) @@ -809,7 +801,7 @@ log(" prec: %d emin: %d emax: %d", (context.prec, context.Emin, context.Emax)) restr_range = 9999 if context.Emax > 9999 else context.Emax+99 - for rounding in sorted(RoundMap): + for rounding in RoundModes: context.rounding = rounding context.capitals = random.randrange(2) if spec['clamp'] == 'rand': @@ -941,7 +933,7 @@ def test_from_float(method, prec, exprange, restricted_range, itr, stat): """Iterate the __float__ method through many test cases.""" - for rounding in sorted(RoundMap): + for rounding in RoundModes: context.rounding = rounding for i in range(1000): f = randfloat() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Jan 16 13:02:13 2013 From: python-checkins at python.org (stefan.krah) Date: Wed, 16 Jan 2013 13:02:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy4zLg==?= Message-ID: <3YmRqT3NHDzRBl@mail.python.org> http://hg.python.org/cpython/rev/f1cba54fcc84 changeset: 81539:f1cba54fcc84 parent: 81537:d866bbdd68e8 parent: 81538:733bb4fd9888 user: Stefan Krah date: Wed Jan 16 13:00:11 2013 +0100 summary: Merge 3.3. files: Lib/test/test_decimal.py | 119 +++++++--------- Misc/NEWS | 3 + Modules/_decimal/_decimal.c | 122 +++++++--------- Modules/_decimal/tests/deccheck.py | 36 +--- 4 files changed, 127 insertions(+), 153 deletions(-) diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -78,14 +78,20 @@ d = getattr(context, attr) cls.assertTrue(all(d[s] if s in expected else not d[s] for s in d)) -RoundingModes = { - C: (C.ROUND_UP, C.ROUND_DOWN, C.ROUND_CEILING, C.ROUND_FLOOR, - C.ROUND_HALF_UP, C.ROUND_HALF_DOWN, C.ROUND_HALF_EVEN, - C.ROUND_05UP) if C else None, - P: (P.ROUND_UP, P.ROUND_DOWN, P.ROUND_CEILING, P.ROUND_FLOOR, - P.ROUND_HALF_UP, P.ROUND_HALF_DOWN, P.ROUND_HALF_EVEN, - P.ROUND_05UP) -} +ROUND_UP = P.ROUND_UP +ROUND_DOWN = P.ROUND_DOWN +ROUND_CEILING = P.ROUND_CEILING +ROUND_FLOOR = P.ROUND_FLOOR +ROUND_HALF_UP = P.ROUND_HALF_UP +ROUND_HALF_DOWN = P.ROUND_HALF_DOWN +ROUND_HALF_EVEN = P.ROUND_HALF_EVEN +ROUND_05UP = P.ROUND_05UP + +RoundingModes = [ + ROUND_UP, ROUND_DOWN, ROUND_CEILING, ROUND_FLOOR, + ROUND_HALF_UP, ROUND_HALF_DOWN, ROUND_HALF_EVEN, + ROUND_05UP +] # Tests are built around these assumed context defaults. # test_main() restores the original context. @@ -96,7 +102,7 @@ def init(m): if not m: return DefaultTestContext = m.Context( - prec=9, rounding=m.ROUND_HALF_EVEN, traps=dict.fromkeys(Signals[m], 0) + prec=9, rounding=ROUND_HALF_EVEN, traps=dict.fromkeys(Signals[m], 0) ) m.setcontext(DefaultTestContext) @@ -229,14 +235,14 @@ 'xor':'logical_xor'} # Map test-case names to roundings. - self.RoundingDict = {'ceiling' : self.decimal.ROUND_CEILING, - 'down' : self.decimal.ROUND_DOWN, - 'floor' : self.decimal.ROUND_FLOOR, - 'half_down' : self.decimal.ROUND_HALF_DOWN, - 'half_even' : self.decimal.ROUND_HALF_EVEN, - 'half_up' : self.decimal.ROUND_HALF_UP, - 'up' : self.decimal.ROUND_UP, - '05up' : self.decimal.ROUND_05UP} + self.RoundingDict = {'ceiling' : ROUND_CEILING, + 'down' : ROUND_DOWN, + 'floor' : ROUND_FLOOR, + 'half_down' : ROUND_HALF_DOWN, + 'half_even' : ROUND_HALF_EVEN, + 'half_up' : ROUND_HALF_UP, + 'up' : ROUND_UP, + '05up' : ROUND_05UP} # Map the test cases' error names to the actual errors. self.ErrorNames = {'clamped' : self.decimal.Clamped, @@ -2101,9 +2107,6 @@ Inexact = self.decimal.Inexact Rounded = self.decimal.Rounded Clamped = self.decimal.Clamped - ROUND_HALF_EVEN = self.decimal.ROUND_HALF_EVEN - ROUND_DOWN = self.decimal.ROUND_DOWN - ROUND_UP = self.decimal.ROUND_UP with localcontext(Context()) as c: c.prec = 7 @@ -2430,7 +2433,6 @@ def test_int(self): Decimal = self.decimal.Decimal - ROUND_DOWN = self.decimal.ROUND_DOWN for x in range(-250, 250): s = '%0.2f' % (x / 100.0) @@ -2448,7 +2450,6 @@ def test_trunc(self): Decimal = self.decimal.Decimal - ROUND_DOWN = self.decimal.ROUND_DOWN for x in range(-250, 250): s = '%0.2f' % (x / 100.0) @@ -2491,8 +2492,6 @@ def test_create_decimal_from_float(self): Decimal = self.decimal.Decimal Context = self.decimal.Context - ROUND_DOWN = self.decimal.ROUND_DOWN - ROUND_UP = self.decimal.ROUND_UP Inexact = self.decimal.Inexact context = Context(prec=5, rounding=ROUND_DOWN) @@ -2522,7 +2521,6 @@ Decimal = self.decimal.Decimal Context = self.decimal.Context InvalidOperation = self.decimal.InvalidOperation - ROUND_DOWN = self.decimal.ROUND_DOWN c = Context(Emax=99999, Emin=-99999) self.assertEqual( @@ -2723,7 +2721,6 @@ InvalidOperation = self.decimal.InvalidOperation DivisionByZero = self.decimal.DivisionByZero Overflow = self.decimal.Overflow - ROUND_HALF_EVEN = self.decimal.ROUND_HALF_EVEN c1 = Context() c2 = Context(prec=None, rounding=None, Emax=None, Emin=None, @@ -2739,6 +2736,21 @@ assert_signals(self, c, 'traps', [InvalidOperation, DivisionByZero, Overflow]) + @cpython_only + def test_from_legacy_strings(self): + import _testcapi + c = self.decimal.Context() + + for rnd in RoundingModes: + c.rounding = _testcapi.unicode_legacy_string(rnd) + self.assertEqual(c.rounding, rnd) + + s = _testcapi.unicode_legacy_string('') + self.assertRaises(TypeError, setattr, c, 'rounding', s) + + s = _testcapi.unicode_legacy_string('ROUND_\x00UP') + self.assertRaises(TypeError, setattr, c, 'rounding', s) + def test_pickle(self): Context = self.decimal.Context @@ -2762,7 +2774,7 @@ # Test interchangeability combinations = [(C, P), (P, C)] if C else [(P, P)] for dumper, loader in combinations: - for ri, _ in enumerate(RoundingModes[dumper]): + for ri, _ in enumerate(RoundingModes): for fi, _ in enumerate(OrderedSignals[dumper]): for ti, _ in enumerate(OrderedSignals[dumper]): @@ -2776,7 +2788,7 @@ sys.modules['decimal'] = dumper c = dumper.Context( prec=prec, Emin=emin, Emax=emax, - rounding=RoundingModes[dumper][ri], + rounding=RoundingModes[ri], capitals=caps, clamp=clamp, flags=OrderedSignals[dumper][:fi], traps=OrderedSignals[dumper][:ti] @@ -2791,7 +2803,7 @@ self.assertEqual(d.prec, prec) self.assertEqual(d.Emin, emin) self.assertEqual(d.Emax, emax) - self.assertEqual(d.rounding, RoundingModes[loader][ri]) + self.assertEqual(d.rounding, RoundingModes[ri]) self.assertEqual(d.capitals, caps) self.assertEqual(d.clamp, clamp) assert_signals(self, d, 'flags', OrderedSignals[loader][:fi]) @@ -3593,7 +3605,6 @@ Underflow = self.decimal.Underflow Clamped = self.decimal.Clamped Subnormal = self.decimal.Subnormal - ROUND_HALF_EVEN = self.decimal.ROUND_HALF_EVEN def raise_error(context, flag): if self.decimal == C: @@ -3960,17 +3971,6 @@ self.assertRaises(ValueError, setattr, c, 'Emin', 1) self.assertRaises(TypeError, setattr, c, 'Emin', (1,2,3)) - # rounding: always raise TypeError in order to get consistent - # exceptions across implementations. In decimal, rounding - # modes are strings, in _decimal they are integers. The idea - # is to view rounding as an abstract type and not mind the - # implementation details. - # Hence, a user should view the rounding modes as if they - # had been defined in a language that supports abstract - # data types, e.g. ocaml: - # - # type rounding = ROUND_DOWN | ROUND_HALF_UP | ... ;; - # self.assertRaises(TypeError, setattr, c, 'rounding', -1) self.assertRaises(TypeError, setattr, c, 'rounding', 9) self.assertRaises(TypeError, setattr, c, 'rounding', 1.0) @@ -4023,8 +4023,6 @@ decimal = self.decimal Decimal = decimal.Decimal Context = decimal.Context - ROUND_HALF_EVEN = decimal.ROUND_HALF_EVEN - ROUND_DOWN = decimal.ROUND_DOWN Clamped = decimal.Clamped DivisionByZero = decimal.DivisionByZero Inexact = decimal.Inexact @@ -4192,7 +4190,7 @@ c.prec = 425000000 c.Emax = 425000000 c.Emin = -425000000 - c.rounding = self.decimal.ROUND_HALF_DOWN + c.rounding = ROUND_HALF_DOWN c.capitals = 0 c.clamp = 1 for sig in OrderedSignals[self.decimal]: @@ -4584,7 +4582,6 @@ def test_py_rescale(self): # Coverage Decimal = P.Decimal - ROUND_UP = P.ROUND_UP localcontext = P.localcontext with localcontext() as c: @@ -4594,7 +4591,6 @@ def test_py__round(self): # Coverage Decimal = P.Decimal - ROUND_UP = P.ROUND_UP self.assertRaises(ValueError, Decimal("3.1234")._round, 0, ROUND_UP) @@ -4663,11 +4659,6 @@ self.assertEqual(C.DECIMAL128, 128) self.assertEqual(C.IEEE_CONTEXT_MAX_BITS, 512) - # Rounding modes - for i, v in enumerate(RoundingModes[C]): - self.assertEqual(v, i) - self.assertEqual(C.ROUND_TRUNC, 8) - # Conditions for i, v in enumerate(cond): self.assertEqual(v, 1<val)); } + /* Init string constants */ + for (i = 0; i < _PY_DEC_ROUND_GUARD; i++) { + ASSIGN_PTR(round_map[i], PyUnicode_InternFromString(mpd_round_string[i])); + Py_INCREF(round_map[i]); + CHECK_INT(PyModule_AddObject(m, mpd_round_string[i], round_map[i])); + } + /* Add specification version number */ CHECK_INT(PyModule_AddStringConstant(m, "__version__", " 1.70")); diff --git a/Modules/_decimal/tests/deccheck.py b/Modules/_decimal/tests/deccheck.py --- a/Modules/_decimal/tests/deccheck.py +++ b/Modules/_decimal/tests/deccheck.py @@ -158,17 +158,9 @@ C.FloatOperation: P.FloatOperation, } -RoundMap = { - C.ROUND_UP: P.ROUND_UP, - C.ROUND_DOWN: P.ROUND_DOWN, - C.ROUND_CEILING: P.ROUND_CEILING, - C.ROUND_FLOOR: P.ROUND_FLOOR, - C.ROUND_HALF_UP: P.ROUND_HALF_UP, - C.ROUND_HALF_DOWN: P.ROUND_HALF_DOWN, - C.ROUND_HALF_EVEN: P.ROUND_HALF_EVEN, - C.ROUND_05UP: P.ROUND_05UP -} -RoundModes = RoundMap.items() +RoundModes = [C.ROUND_UP, C.ROUND_DOWN, C.ROUND_CEILING, C.ROUND_FLOOR, + C.ROUND_HALF_UP, C.ROUND_HALF_DOWN, C.ROUND_HALF_EVEN, + C.ROUND_05UP] class Context(object): @@ -183,7 +175,7 @@ self.p.prec = self.c.prec self.p.Emin = self.c.Emin self.p.Emax = self.c.Emax - self.p.rounding = RoundMap[self.c.rounding] + self.p.rounding = self.c.rounding self.p.capitals = self.c.capitals self.settraps([sig for sig in self.c.traps if self.c.traps[sig]]) self.setstatus([sig for sig in self.c.flags if self.c.flags[sig]]) @@ -217,12 +209,12 @@ self.p.Emax = val def getround(self): - assert(self.c.rounding == RoundMap[self.p.rounding]) + assert(self.c.rounding == self.p.rounding) return self.c.rounding def setround(self, val): self.c.rounding = val - self.p.rounding = RoundMap[val] + self.p.rounding = val def getcapitals(self): assert(self.c.capitals == self.p.capitals) @@ -627,8 +619,12 @@ context.clear_status() - if not t.contextfunc and i == 0 or \ - convstr and isinstance(op, str): + if op in RoundModes: + t.cop.append(op) + t.pop.append(op) + + elif not t.contextfunc and i == 0 or \ + convstr and isinstance(op, str): try: c = C.Decimal(op) cex = None @@ -662,10 +658,6 @@ t.cop.append(op.c) t.pop.append(op.p) - elif op in RoundModes: - t.cop.append(op[0]) - t.pop.append(op[1]) - else: t.cop.append(op) t.pop.append(op) @@ -809,7 +801,7 @@ log(" prec: %d emin: %d emax: %d", (context.prec, context.Emin, context.Emax)) restr_range = 9999 if context.Emax > 9999 else context.Emax+99 - for rounding in sorted(RoundMap): + for rounding in RoundModes: context.rounding = rounding context.capitals = random.randrange(2) if spec['clamp'] == 'rand': @@ -941,7 +933,7 @@ def test_from_float(method, prec, exprange, restricted_range, itr, stat): """Iterate the __float__ method through many test cases.""" - for rounding in sorted(RoundMap): + for rounding in RoundModes: context.rounding = rounding for i in range(1000): f = randfloat() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Jan 16 13:43:06 2013 From: python-checkins at python.org (stefan.krah) Date: Wed, 16 Jan 2013 13:43:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Change_NEWS_en?= =?utf-8?q?try_to_make_it_somewhat_intelligible=2E?= Message-ID: <3YmSkf389hzMqZ@mail.python.org> http://hg.python.org/cpython/rev/6bfa2cf27b0b changeset: 81540:6bfa2cf27b0b branch: 3.3 parent: 81538:733bb4fd9888 user: Stefan Krah date: Wed Jan 16 13:40:57 2013 +0100 summary: Change NEWS entry to make it somewhat intelligible. files: Misc/NEWS | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -150,8 +150,8 @@ Library ------- -- Issue #16422: Use strings for rounding mode constants for better readability - and pickling compatibility. +- Issue #16422: For compatibility with the Python version, the C version of + decimal now uses strings instead of integers for rounding mode constants. - Issue #15861: tkinter now correctly works with lists and tuples containing strings with whitespaces, backslashes or unbalanced braces. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Jan 16 13:43:07 2013 From: python-checkins at python.org (stefan.krah) Date: Wed, 16 Jan 2013 13:43:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy4zLg==?= Message-ID: <3YmSkg68c9zR7M@mail.python.org> http://hg.python.org/cpython/rev/7fd1830d0acd changeset: 81541:7fd1830d0acd parent: 81539:f1cba54fcc84 parent: 81540:6bfa2cf27b0b user: Stefan Krah date: Wed Jan 16 13:41:38 2013 +0100 summary: Merge 3.3. files: Misc/NEWS | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -220,8 +220,8 @@ Library ------- -- Issue #16422: Use strings for rounding mode constants for better readability - and pickling compatibility. +- Issue #16422: For compatibility with the Python version, the C version of + decimal now uses strings instead of integers for rounding mode constants. - Issue #15861: tkinter now correctly works with lists and tuples containing strings with whitespaces, backslashes or unbalanced braces. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Jan 16 13:54:09 2013 From: python-checkins at python.org (nick.coghlan) Date: Wed, 16 Jan 2013 13:54:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Handle_transfer_of_config_to_?= =?utf-8?q?subinterpreters?= Message-ID: <3YmSzP4HZCzMgn@mail.python.org> http://hg.python.org/peps/rev/90f1a0ce0aaa changeset: 4679:90f1a0ce0aaa user: Nick Coghlan date: Wed Jan 16 22:54:01 2013 +1000 summary: Handle transfer of config to subinterpreters files: pep-0432.txt | 39 +++++++++++++++++++++++++++++++++++---- 1 files changed, 35 insertions(+), 4 deletions(-) diff --git a/pep-0432.txt b/pep-0432.txt --- a/pep-0432.txt +++ b/pep-0432.txt @@ -60,10 +60,8 @@ To keep the implementation complexity under control, this PEP does *not* propose wholesale changes to the way the interpreter state is accessed at -runtime, nor does it propose changes to the way subinterpreters are -created after the main interpreter has already been initialized (beyond -any changes needed to make sure they continue working as expected). Changing -the order in which the existing initialization steps occur in order to make +runtime. Changing the order in which the existing initialization steps +occur in order to make the startup sequence easier to maintain is already a substantial change, and attempting to make those other changes at the same time will make the change significantly more invasive and much harder to review. However, such @@ -73,6 +71,7 @@ once this PEP has been implemented. + Background ========== @@ -816,6 +815,38 @@ modified by the interpreter during runtime (except as noted above). +Creating and Configuring Subinterpreters +---------------------------------------- + +As the new configuration settings are stored in the interpreter state, they +need to be initialised when a new subinterpreter is created. This turns out +to be trickier than one might think due to ``PyThreadState_Swap(NULL);`` +(which is fortunately exercised by CPython's own embedding tests, allowing +this problem to be detected during development). + +To provide a straightforward solution for this case, the PEP proposes to +add a new API:: + + Py_InterpreterState *Py_InterpreterState_Main(); + +This will be a counterpart to Py_InterpreterState_Head(), reporting the +oldest currently existing interpreter rather than the newest. If +``Py_NewInterpreter()`` is called from a thread with an existing thread +state, then the interpreter configuration for that thread will be +used when initialising the new subinterpreter. If there is no current +thread state, the configuration from ``Py_InterpreterState_Main()`` +will be used. + +While the existing ``Py_InterpreterState_Head()`` API could be used instead, +that reference changes as subinterpreters are created and destroyed, while +``PyInterpreterState_Main()`` will always refer to the initial interpreter +state created in ``Py_BeginInitialization()``. + +A new constraint is also added to the embedding API: attempting to delete +the main interpreter while subinterpreters still exist will now be a fatal +error. + + Stable ABI ---------- -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed Jan 16 14:50:15 2013 From: python-checkins at python.org (stefan.krah) Date: Wed, 16 Jan 2013 14:50:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Increase_warni?= =?utf-8?q?ng_level_for_=5Fdecimal=2Eso_and_libmpdec_build_when_using_gcc?= =?utf-8?q?=2E?= Message-ID: <3YmVD72sHkzR8C@mail.python.org> http://hg.python.org/cpython/rev/5a729c08872a changeset: 81542:5a729c08872a branch: 3.3 parent: 81540:6bfa2cf27b0b user: Stefan Krah date: Wed Jan 16 14:45:16 2013 +0100 summary: Increase warning level for _decimal.so and libmpdec build when using gcc. files: setup.py | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -1995,6 +1995,11 @@ if not sysconfig.get_config_var('WITH_THREAD'): define_macros.append(('WITHOUT_THREADS', 1)) + # Increase warning level for gcc: + if 'gcc' in cc: + extra_compile_args.extend(['-Wextra', + '-Wno-missing-field-initializers']) + # Uncomment for extra functionality: #define_macros.append(('EXTRA_FUNCTIONALITY', 1)) ext = Extension ( -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Jan 16 14:50:16 2013 From: python-checkins at python.org (stefan.krah) Date: Wed, 16 Jan 2013 14:50:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy4zLg==?= Message-ID: <3YmVD85yXKzR8C@mail.python.org> http://hg.python.org/cpython/rev/5c0c6de528dd changeset: 81543:5c0c6de528dd parent: 81541:7fd1830d0acd parent: 81542:5a729c08872a user: Stefan Krah date: Wed Jan 16 14:49:15 2013 +0100 summary: Merge 3.3. files: setup.py | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -2006,6 +2006,11 @@ if not sysconfig.get_config_var('WITH_THREAD'): define_macros.append(('WITHOUT_THREADS', 1)) + # Increase warning level for gcc: + if 'gcc' in cc: + extra_compile_args.extend(['-Wextra', + '-Wno-missing-field-initializers']) + # Uncomment for extra functionality: #define_macros.append(('EXTRA_FUNCTIONALITY', 1)) ext = Extension ( -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Jan 16 15:19:53 2013 From: python-checkins at python.org (stefan.krah) Date: Wed, 16 Jan 2013 15:19:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Remove_trailin?= =?utf-8?q?g_whitespace=2E?= Message-ID: <3YmVtK5qRBzMcy@mail.python.org> http://hg.python.org/cpython/rev/750f44e621aa changeset: 81544:750f44e621aa branch: 3.3 parent: 81542:5a729c08872a user: Stefan Krah date: Wed Jan 16 15:16:10 2013 +0100 summary: Remove trailing whitespace. files: Modules/_decimal/README.txt | 2 +- Modules/_decimal/libmpdec/README.txt | 2 +- Modules/_decimal/libmpdec/io.c | 4 +- Modules/_decimal/libmpdec/literature/bignum.txt | 2 +- Modules/_decimal/libmpdec/literature/matrix-transform.txt | 1 + Modules/_decimal/libmpdec/literature/mulmod-ppro.txt | 2 +- Modules/_decimal/libmpdec/literature/six-step.txt | 1 + Modules/_decimal/libmpdec/literature/umodarith.lisp | 2 +- Modules/_decimal/libmpdec/mpdecimal.c | 6 +- Modules/_decimal/libmpdec/umodarith.h | 2 +- Modules/_decimal/libmpdec/vcstdint.h | 20 +++++----- 11 files changed, 23 insertions(+), 21 deletions(-) diff --git a/Modules/_decimal/README.txt b/Modules/_decimal/README.txt --- a/Modules/_decimal/README.txt +++ b/Modules/_decimal/README.txt @@ -37,7 +37,7 @@ NOTE ==== - + decimal.so is not built from a static libmpdec.a since doing so led to failures on AIX (user report) and Windows (mixing static and dynamic CRTs causes locale problems and more). diff --git a/Modules/_decimal/libmpdec/README.txt b/Modules/_decimal/libmpdec/README.txt --- a/Modules/_decimal/libmpdec/README.txt +++ b/Modules/_decimal/libmpdec/README.txt @@ -79,7 +79,7 @@ mulmod-ppro.txt -> Proof for the x87 FPU modular multiplication from umodarith.h. umodarith.lisp -> ACL2 proofs for many functions from umodarith.h. - + Library Author ============== diff --git a/Modules/_decimal/libmpdec/io.c b/Modules/_decimal/libmpdec/io.c --- a/Modules/_decimal/libmpdec/io.c +++ b/Modules/_decimal/libmpdec/io.c @@ -352,7 +352,7 @@ case 2: EXTRACT_DIGIT(s, x, 10UL, dot); default: if (s == dot) *s++ = '.'; *s++ = '0' + (char)x; } - + *s = '\0'; return s; } @@ -1059,7 +1059,7 @@ * Not allowed, since _mpd_to_string() never returns this form: * * 3) [sign] '.' digits [exponent-part] - * + * * Input: result->data := original numeric string (ASCII) * result->bytes := strlen(result->data) * result->nchars := strlen(result->data) diff --git a/Modules/_decimal/libmpdec/literature/bignum.txt b/Modules/_decimal/libmpdec/literature/bignum.txt --- a/Modules/_decimal/libmpdec/literature/bignum.txt +++ b/Modules/_decimal/libmpdec/literature/bignum.txt @@ -23,7 +23,7 @@ y = fnt(b) # forward transform of b z = pairwise multiply x[i] and y[i] result = inv_fnt(z) # backward transform of z. - + Extending the maximum transform length (Chinese Remainder Theorem): ------------------------------------------------------------------- diff --git a/Modules/_decimal/libmpdec/literature/matrix-transform.txt b/Modules/_decimal/libmpdec/literature/matrix-transform.txt --- a/Modules/_decimal/libmpdec/literature/matrix-transform.txt +++ b/Modules/_decimal/libmpdec/literature/matrix-transform.txt @@ -245,7 +245,7 @@ which means that all A[k] = A[m * C + n] are in the correct order. --- +-- [1] Joerg Arndt: "Matters Computational" http://www.jjj.de/fxt/ diff --git a/Modules/_decimal/libmpdec/literature/mulmod-ppro.txt b/Modules/_decimal/libmpdec/literature/mulmod-ppro.txt --- a/Modules/_decimal/libmpdec/literature/mulmod-ppro.txt +++ b/Modules/_decimal/libmpdec/literature/mulmod-ppro.txt @@ -251,7 +251,7 @@ apply Qlt_le_trans with (y := 1). qreduce ((2 # 1) ^ (-62) + (2 # 1) ^ (-126)). - + rewrite Qlt_mult_by_z with (z := 85070591730234615865843651857942052864 # 18446744073709551617). ring_simplify. diff --git a/Modules/_decimal/libmpdec/literature/six-step.txt b/Modules/_decimal/libmpdec/literature/six-step.txt --- a/Modules/_decimal/libmpdec/literature/six-step.txt +++ b/Modules/_decimal/libmpdec/literature/six-step.txt @@ -55,7 +55,7 @@ --- +-- [1] David H. Bailey: FFTs in External or Hierarchical Memory http://crd.lbl.gov/~dhbailey/dhbpapers/ diff --git a/Modules/_decimal/libmpdec/literature/umodarith.lisp b/Modules/_decimal/libmpdec/literature/umodarith.lisp --- a/Modules/_decimal/libmpdec/literature/umodarith.lisp +++ b/Modules/_decimal/libmpdec/literature/umodarith.lisp @@ -321,7 +321,7 @@ (defthmd dw-submod-correct (implies (and (< 0 m) (< m base) - (natp a) (< a m) + (natp a) (< a m) (< hi base) (< lo base) (natp m) (natp base) (natp hi) (natp lo)) diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -3119,7 +3119,7 @@ * ----------XX1234 -> sdigits * ----------X1 -> tiny-digits * |- prec -| - * + * * OR * * bdigits > prec AND @@ -5983,7 +5983,7 @@ mpd_qfinalize(result, ctx, status); } -/* +/* * If the exponent is infinite and base equals one, the result is one * with a coefficient of length prec. Otherwise, result is undefined. * Return the value of the comparison against one. @@ -7129,7 +7129,7 @@ goto nanresult; } - /* Let x := _mpd_qreciprocal(b, prec) + /* Let x := _mpd_qreciprocal(b, prec) * Then x is bounded by: * 1) 1/b - 10**(-prec - bdigits) < x < 1/b + 10**(-prec - bdigits) * 2) 1/b - 10**(-adigits - 4) < x < 1/b + 10**(-adigits - 4) diff --git a/Modules/_decimal/libmpdec/umodarith.h b/Modules/_decimal/libmpdec/umodarith.h --- a/Modules/_decimal/libmpdec/umodarith.h +++ b/Modules/_decimal/libmpdec/umodarith.h @@ -93,7 +93,7 @@ return d; } -/* +/* * Reduce double word modulo m. * Restrictions: m != 0 * ACL2 proof: umodarith.lisp: section dw-reduce diff --git a/Modules/_decimal/libmpdec/vcstdint.h b/Modules/_decimal/libmpdec/vcstdint.h --- a/Modules/_decimal/libmpdec/vcstdint.h +++ b/Modules/_decimal/libmpdec/vcstdint.h @@ -1,32 +1,32 @@ // ISO C9x compliant stdint.h for Microsoft Visual Studio -// Based on ISO/IEC 9899:TC2 Committee draft (May 6, 2005) WG14/N1124 -// +// Based on ISO/IEC 9899:TC2 Committee draft (May 6, 2005) WG14/N1124 +// // Copyright (c) 2006-2008 Alexander Chemeris -// +// // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions are met: -// +// // 1. Redistributions of source code must retain the above copyright notice, // this list of conditions and the following disclaimer. -// +// // 2. Redistributions in binary form must reproduce the above copyright // notice, this list of conditions and the following disclaimer in the // documentation and/or other materials provided with the distribution. -// +// // 3. The name of the author may be used to endorse or promote products // derived from this software without specific prior written permission. -// +// // THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED // WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF // MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO // EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, // SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, // PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; -// OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, +// OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, // WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR // OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF // ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -// +// /////////////////////////////////////////////////////////////////////////////// #ifndef _MSC_VER // [ @@ -47,7 +47,7 @@ // error C2733: second C linkage of overloaded function 'wmemchr' not allowed #if (_MSC_VER < 1300) && defined(__cplusplus) extern "C++" { -#endif +#endif # include #if (_MSC_VER < 1300) && defined(__cplusplus) } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Jan 16 15:19:55 2013 From: python-checkins at python.org (stefan.krah) Date: Wed, 16 Jan 2013 15:19:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy4zLg==?= Message-ID: <3YmVtM3BwLzR7d@mail.python.org> http://hg.python.org/cpython/rev/f87d85f7596c changeset: 81545:f87d85f7596c parent: 81543:5c0c6de528dd parent: 81544:750f44e621aa user: Stefan Krah date: Wed Jan 16 15:19:16 2013 +0100 summary: Merge 3.3. files: Modules/_decimal/README.txt | 2 +- Modules/_decimal/libmpdec/README.txt | 2 +- Modules/_decimal/libmpdec/io.c | 4 +- Modules/_decimal/libmpdec/literature/bignum.txt | 2 +- Modules/_decimal/libmpdec/literature/matrix-transform.txt | 1 + Modules/_decimal/libmpdec/literature/mulmod-ppro.txt | 2 +- Modules/_decimal/libmpdec/literature/six-step.txt | 1 + Modules/_decimal/libmpdec/literature/umodarith.lisp | 2 +- Modules/_decimal/libmpdec/mpdecimal.c | 6 +- Modules/_decimal/libmpdec/umodarith.h | 2 +- Modules/_decimal/libmpdec/vcstdint.h | 20 +++++----- 11 files changed, 23 insertions(+), 21 deletions(-) diff --git a/Modules/_decimal/README.txt b/Modules/_decimal/README.txt --- a/Modules/_decimal/README.txt +++ b/Modules/_decimal/README.txt @@ -37,7 +37,7 @@ NOTE ==== - + decimal.so is not built from a static libmpdec.a since doing so led to failures on AIX (user report) and Windows (mixing static and dynamic CRTs causes locale problems and more). diff --git a/Modules/_decimal/libmpdec/README.txt b/Modules/_decimal/libmpdec/README.txt --- a/Modules/_decimal/libmpdec/README.txt +++ b/Modules/_decimal/libmpdec/README.txt @@ -79,7 +79,7 @@ mulmod-ppro.txt -> Proof for the x87 FPU modular multiplication from umodarith.h. umodarith.lisp -> ACL2 proofs for many functions from umodarith.h. - + Library Author ============== diff --git a/Modules/_decimal/libmpdec/io.c b/Modules/_decimal/libmpdec/io.c --- a/Modules/_decimal/libmpdec/io.c +++ b/Modules/_decimal/libmpdec/io.c @@ -352,7 +352,7 @@ case 2: EXTRACT_DIGIT(s, x, 10UL, dot); default: if (s == dot) *s++ = '.'; *s++ = '0' + (char)x; } - + *s = '\0'; return s; } @@ -1059,7 +1059,7 @@ * Not allowed, since _mpd_to_string() never returns this form: * * 3) [sign] '.' digits [exponent-part] - * + * * Input: result->data := original numeric string (ASCII) * result->bytes := strlen(result->data) * result->nchars := strlen(result->data) diff --git a/Modules/_decimal/libmpdec/literature/bignum.txt b/Modules/_decimal/libmpdec/literature/bignum.txt --- a/Modules/_decimal/libmpdec/literature/bignum.txt +++ b/Modules/_decimal/libmpdec/literature/bignum.txt @@ -23,7 +23,7 @@ y = fnt(b) # forward transform of b z = pairwise multiply x[i] and y[i] result = inv_fnt(z) # backward transform of z. - + Extending the maximum transform length (Chinese Remainder Theorem): ------------------------------------------------------------------- diff --git a/Modules/_decimal/libmpdec/literature/matrix-transform.txt b/Modules/_decimal/libmpdec/literature/matrix-transform.txt --- a/Modules/_decimal/libmpdec/literature/matrix-transform.txt +++ b/Modules/_decimal/libmpdec/literature/matrix-transform.txt @@ -245,7 +245,7 @@ which means that all A[k] = A[m * C + n] are in the correct order. --- +-- [1] Joerg Arndt: "Matters Computational" http://www.jjj.de/fxt/ diff --git a/Modules/_decimal/libmpdec/literature/mulmod-ppro.txt b/Modules/_decimal/libmpdec/literature/mulmod-ppro.txt --- a/Modules/_decimal/libmpdec/literature/mulmod-ppro.txt +++ b/Modules/_decimal/libmpdec/literature/mulmod-ppro.txt @@ -251,7 +251,7 @@ apply Qlt_le_trans with (y := 1). qreduce ((2 # 1) ^ (-62) + (2 # 1) ^ (-126)). - + rewrite Qlt_mult_by_z with (z := 85070591730234615865843651857942052864 # 18446744073709551617). ring_simplify. diff --git a/Modules/_decimal/libmpdec/literature/six-step.txt b/Modules/_decimal/libmpdec/literature/six-step.txt --- a/Modules/_decimal/libmpdec/literature/six-step.txt +++ b/Modules/_decimal/libmpdec/literature/six-step.txt @@ -55,7 +55,7 @@ --- +-- [1] David H. Bailey: FFTs in External or Hierarchical Memory http://crd.lbl.gov/~dhbailey/dhbpapers/ diff --git a/Modules/_decimal/libmpdec/literature/umodarith.lisp b/Modules/_decimal/libmpdec/literature/umodarith.lisp --- a/Modules/_decimal/libmpdec/literature/umodarith.lisp +++ b/Modules/_decimal/libmpdec/literature/umodarith.lisp @@ -321,7 +321,7 @@ (defthmd dw-submod-correct (implies (and (< 0 m) (< m base) - (natp a) (< a m) + (natp a) (< a m) (< hi base) (< lo base) (natp m) (natp base) (natp hi) (natp lo)) diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -3119,7 +3119,7 @@ * ----------XX1234 -> sdigits * ----------X1 -> tiny-digits * |- prec -| - * + * * OR * * bdigits > prec AND @@ -5983,7 +5983,7 @@ mpd_qfinalize(result, ctx, status); } -/* +/* * If the exponent is infinite and base equals one, the result is one * with a coefficient of length prec. Otherwise, result is undefined. * Return the value of the comparison against one. @@ -7129,7 +7129,7 @@ goto nanresult; } - /* Let x := _mpd_qreciprocal(b, prec) + /* Let x := _mpd_qreciprocal(b, prec) * Then x is bounded by: * 1) 1/b - 10**(-prec - bdigits) < x < 1/b + 10**(-prec - bdigits) * 2) 1/b - 10**(-adigits - 4) < x < 1/b + 10**(-adigits - 4) diff --git a/Modules/_decimal/libmpdec/umodarith.h b/Modules/_decimal/libmpdec/umodarith.h --- a/Modules/_decimal/libmpdec/umodarith.h +++ b/Modules/_decimal/libmpdec/umodarith.h @@ -93,7 +93,7 @@ return d; } -/* +/* * Reduce double word modulo m. * Restrictions: m != 0 * ACL2 proof: umodarith.lisp: section dw-reduce diff --git a/Modules/_decimal/libmpdec/vcstdint.h b/Modules/_decimal/libmpdec/vcstdint.h --- a/Modules/_decimal/libmpdec/vcstdint.h +++ b/Modules/_decimal/libmpdec/vcstdint.h @@ -1,32 +1,32 @@ // ISO C9x compliant stdint.h for Microsoft Visual Studio -// Based on ISO/IEC 9899:TC2 Committee draft (May 6, 2005) WG14/N1124 -// +// Based on ISO/IEC 9899:TC2 Committee draft (May 6, 2005) WG14/N1124 +// // Copyright (c) 2006-2008 Alexander Chemeris -// +// // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions are met: -// +// // 1. Redistributions of source code must retain the above copyright notice, // this list of conditions and the following disclaimer. -// +// // 2. Redistributions in binary form must reproduce the above copyright // notice, this list of conditions and the following disclaimer in the // documentation and/or other materials provided with the distribution. -// +// // 3. The name of the author may be used to endorse or promote products // derived from this software without specific prior written permission. -// +// // THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED // WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF // MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO // EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, // SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, // PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; -// OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, +// OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, // WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR // OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF // ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -// +// /////////////////////////////////////////////////////////////////////////////// #ifndef _MSC_VER // [ @@ -47,7 +47,7 @@ // error C2733: second C linkage of overloaded function 'wmemchr' not allowed #if (_MSC_VER < 1300) && defined(__cplusplus) extern "C++" { -#endif +#endif # include #if (_MSC_VER < 1300) && defined(__cplusplus) } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Jan 16 22:59:07 2013 From: python-checkins at python.org (frank.wierzbicki) Date: Wed, 16 Jan 2013 22:59:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogQ2xvc2VkICMxNjg4?= =?utf-8?q?6=3A_test=5Fdictcomps_no_longer_depends_on_dict_order?= Message-ID: <3Ymj4C0SR3zS6v@mail.python.org> http://hg.python.org/cpython/rev/6d06b223c664 changeset: 81546:6d06b223c664 branch: 2.7 parent: 81531:966887830011 user: Frank Wierzbicki date: Wed Jan 16 13:52:22 2013 -0800 summary: Closed #16886: test_dictcomps no longer depends on dict order files: Lib/test/test_dictcomps.py | 117 ++++++++++++++++-------- 1 files changed, 77 insertions(+), 40 deletions(-) diff --git a/Lib/test/test_dictcomps.py b/Lib/test/test_dictcomps.py --- a/Lib/test/test_dictcomps.py +++ b/Lib/test/test_dictcomps.py @@ -1,54 +1,91 @@ +import unittest -doctests = """ +from test import test_support as support - >>> k = "old value" - >>> { k: None for k in range(10) } - {0: None, 1: None, 2: None, 3: None, 4: None, 5: None, 6: None, 7: None, 8: None, 9: None} - >>> k - 'old value' +# For scope testing. +g = "Global variable" - >>> { k: k+10 for k in range(10) } - {0: 10, 1: 11, 2: 12, 3: 13, 4: 14, 5: 15, 6: 16, 7: 17, 8: 18, 9: 19} - >>> g = "Global variable" - >>> { k: g for k in range(10) } - {0: 'Global variable', 1: 'Global variable', 2: 'Global variable', 3: 'Global variable', 4: 'Global variable', 5: 'Global variable', 6: 'Global variable', 7: 'Global variable', 8: 'Global variable', 9: 'Global variable'} +class DictComprehensionTest(unittest.TestCase): - >>> { k: v for k in range(10) for v in range(10) if k == v } - {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9} + def test_basics(self): + expected = {0: 10, 1: 11, 2: 12, 3: 13, 4: 14, 5: 15, 6: 16, 7: 17, + 8: 18, 9: 19} + actual = {k: k + 10 for k in range(10)} + self.assertEqual(actual, expected) - >>> { k: v for v in range(10) for k in range(v*9, v*10) } - {9: 1, 18: 2, 19: 2, 27: 3, 28: 3, 29: 3, 36: 4, 37: 4, 38: 4, 39: 4, 45: 5, 46: 5, 47: 5, 48: 5, 49: 5, 54: 6, 55: 6, 56: 6, 57: 6, 58: 6, 59: 6, 63: 7, 64: 7, 65: 7, 66: 7, 67: 7, 68: 7, 69: 7, 72: 8, 73: 8, 74: 8, 75: 8, 76: 8, 77: 8, 78: 8, 79: 8, 81: 9, 82: 9, 83: 9, 84: 9, 85: 9, 86: 9, 87: 9, 88: 9, 89: 9} + expected = {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9} + actual = {k: v for k in range(10) for v in range(10) if k == v} + self.assertEqual(actual, expected) - >>> { x: y for y, x in ((1, 2), (3, 4)) } = 5 # doctest: +IGNORE_EXCEPTION_DETAIL - Traceback (most recent call last): - ... - SyntaxError: ... + def test_scope_isolation(self): + k = "Local Variable" - >>> { x: y for y, x in ((1, 2), (3, 4)) } += 5 # doctest: +IGNORE_EXCEPTION_DETAIL - Traceback (most recent call last): - ... - SyntaxError: ... + expected = {0: None, 1: None, 2: None, 3: None, 4: None, 5: None, + 6: None, 7: None, 8: None, 9: None} + actual = {k: None for k in range(10)} + self.assertEqual(actual, expected) + self.assertEqual(k, "Local Variable") -""" + expected = {9: 1, 18: 2, 19: 2, 27: 3, 28: 3, 29: 3, 36: 4, 37: 4, + 38: 4, 39: 4, 45: 5, 46: 5, 47: 5, 48: 5, 49: 5, 54: 6, + 55: 6, 56: 6, 57: 6, 58: 6, 59: 6, 63: 7, 64: 7, 65: 7, + 66: 7, 67: 7, 68: 7, 69: 7, 72: 8, 73: 8, 74: 8, 75: 8, + 76: 8, 77: 8, 78: 8, 79: 8, 81: 9, 82: 9, 83: 9, 84: 9, + 85: 9, 86: 9, 87: 9, 88: 9, 89: 9} + actual = {k: v for v in range(10) for k in range(v * 9, v * 10)} + self.assertEqual(k, "Local Variable") + self.assertEqual(actual, expected) -__test__ = {'doctests' : doctests} + def test_scope_isolation_from_global(self): + expected = {0: None, 1: None, 2: None, 3: None, 4: None, 5: None, + 6: None, 7: None, 8: None, 9: None} + actual = {g: None for g in range(10)} + self.assertEqual(actual, expected) + self.assertEqual(g, "Global variable") -def test_main(verbose=None): - import sys - from test import test_support - from test import test_dictcomps - test_support.run_doctest(test_dictcomps, verbose) + expected = {9: 1, 18: 2, 19: 2, 27: 3, 28: 3, 29: 3, 36: 4, 37: 4, + 38: 4, 39: 4, 45: 5, 46: 5, 47: 5, 48: 5, 49: 5, 54: 6, + 55: 6, 56: 6, 57: 6, 58: 6, 59: 6, 63: 7, 64: 7, 65: 7, + 66: 7, 67: 7, 68: 7, 69: 7, 72: 8, 73: 8, 74: 8, 75: 8, + 76: 8, 77: 8, 78: 8, 79: 8, 81: 9, 82: 9, 83: 9, 84: 9, + 85: 9, 86: 9, 87: 9, 88: 9, 89: 9} + actual = {g: v for v in range(10) for g in range(v * 9, v * 10)} + self.assertEqual(g, "Global variable") + self.assertEqual(actual, expected) - # verify reference counting - if verbose and hasattr(sys, "gettotalrefcount"): - import gc - counts = [None] * 5 - for i in range(len(counts)): - test_support.run_doctest(test_dictcomps, verbose) - gc.collect() - counts[i] = sys.gettotalrefcount() - print(counts) + def test_global_visibility(self): + expected = {0: 'Global variable', 1: 'Global variable', + 2: 'Global variable', 3: 'Global variable', + 4: 'Global variable', 5: 'Global variable', + 6: 'Global variable', 7: 'Global variable', + 8: 'Global variable', 9: 'Global variable'} + actual = {k: g for k in range(10)} + self.assertEqual(actual, expected) + + def test_local_visibility(self): + v = "Local variable" + expected = {0: 'Local variable', 1: 'Local variable', + 2: 'Local variable', 3: 'Local variable', + 4: 'Local variable', 5: 'Local variable', + 6: 'Local variable', 7: 'Local variable', + 8: 'Local variable', 9: 'Local variable'} + actual = {k: v for k in range(10)} + self.assertEqual(actual, expected) + self.assertEqual(v, "Local variable") + + def test_illegal_assignment(self): + with self.assertRaisesRegexp(SyntaxError, "can't assign"): + compile("{x: y for y, x in ((1, 2), (3, 4))} = 5", "", + "exec") + + with self.assertRaisesRegexp(SyntaxError, "can't assign"): + compile("{x: y for y, x in ((1, 2), (3, 4))} += 5", "", + "exec") + + +def test_main(): + support.run_unittest(__name__) if __name__ == "__main__": - test_main(verbose=True) + test_main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Jan 16 22:59:09 2013 From: python-checkins at python.org (frank.wierzbicki) Date: Wed, 16 Jan 2013 22:59:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogQ2xvc2VkICMxNjg4?= =?utf-8?q?6=3A_test=5Fdictcomps_no_longer_depends_on_dict_order?= Message-ID: <3Ymj4F2HvdzS7b@mail.python.org> http://hg.python.org/cpython/rev/c7dfc307b88e changeset: 81547:c7dfc307b88e branch: 3.2 parent: 81532:8eac88f49cc0 user: Frank Wierzbicki date: Wed Jan 16 13:55:12 2013 -0800 summary: Closed #16886: test_dictcomps no longer depends on dict order files: Lib/test/test_dictcomps.py | 117 ++++++++++++++++-------- 1 files changed, 77 insertions(+), 40 deletions(-) diff --git a/Lib/test/test_dictcomps.py b/Lib/test/test_dictcomps.py --- a/Lib/test/test_dictcomps.py +++ b/Lib/test/test_dictcomps.py @@ -1,54 +1,91 @@ +import unittest -doctests = """ +from test import support - >>> k = "old value" - >>> { k: None for k in range(10) } - {0: None, 1: None, 2: None, 3: None, 4: None, 5: None, 6: None, 7: None, 8: None, 9: None} - >>> k - 'old value' +# For scope testing. +g = "Global variable" - >>> { k: k+10 for k in range(10) } - {0: 10, 1: 11, 2: 12, 3: 13, 4: 14, 5: 15, 6: 16, 7: 17, 8: 18, 9: 19} - >>> g = "Global variable" - >>> { k: g for k in range(10) } - {0: 'Global variable', 1: 'Global variable', 2: 'Global variable', 3: 'Global variable', 4: 'Global variable', 5: 'Global variable', 6: 'Global variable', 7: 'Global variable', 8: 'Global variable', 9: 'Global variable'} +class DictComprehensionTest(unittest.TestCase): - >>> { k: v for k in range(10) for v in range(10) if k == v } - {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9} + def test_basics(self): + expected = {0: 10, 1: 11, 2: 12, 3: 13, 4: 14, 5: 15, 6: 16, 7: 17, + 8: 18, 9: 19} + actual = {k: k + 10 for k in range(10)} + self.assertEqual(actual, expected) - >>> { k: v for v in range(10) for k in range(v*9, v*10) } - {9: 1, 18: 2, 19: 2, 27: 3, 28: 3, 29: 3, 36: 4, 37: 4, 38: 4, 39: 4, 45: 5, 46: 5, 47: 5, 48: 5, 49: 5, 54: 6, 55: 6, 56: 6, 57: 6, 58: 6, 59: 6, 63: 7, 64: 7, 65: 7, 66: 7, 67: 7, 68: 7, 69: 7, 72: 8, 73: 8, 74: 8, 75: 8, 76: 8, 77: 8, 78: 8, 79: 8, 81: 9, 82: 9, 83: 9, 84: 9, 85: 9, 86: 9, 87: 9, 88: 9, 89: 9} + expected = {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9} + actual = {k: v for k in range(10) for v in range(10) if k == v} + self.assertEqual(actual, expected) - >>> { x: y for y, x in ((1, 2), (3, 4)) } = 5 # doctest: +IGNORE_EXCEPTION_DETAIL - Traceback (most recent call last): - ... - SyntaxError: ... + def test_scope_isolation(self): + k = "Local Variable" - >>> { x: y for y, x in ((1, 2), (3, 4)) } += 5 # doctest: +IGNORE_EXCEPTION_DETAIL - Traceback (most recent call last): - ... - SyntaxError: ... + expected = {0: None, 1: None, 2: None, 3: None, 4: None, 5: None, + 6: None, 7: None, 8: None, 9: None} + actual = {k: None for k in range(10)} + self.assertEqual(actual, expected) + self.assertEqual(k, "Local Variable") -""" + expected = {9: 1, 18: 2, 19: 2, 27: 3, 28: 3, 29: 3, 36: 4, 37: 4, + 38: 4, 39: 4, 45: 5, 46: 5, 47: 5, 48: 5, 49: 5, 54: 6, + 55: 6, 56: 6, 57: 6, 58: 6, 59: 6, 63: 7, 64: 7, 65: 7, + 66: 7, 67: 7, 68: 7, 69: 7, 72: 8, 73: 8, 74: 8, 75: 8, + 76: 8, 77: 8, 78: 8, 79: 8, 81: 9, 82: 9, 83: 9, 84: 9, + 85: 9, 86: 9, 87: 9, 88: 9, 89: 9} + actual = {k: v for v in range(10) for k in range(v * 9, v * 10)} + self.assertEqual(k, "Local Variable") + self.assertEqual(actual, expected) -__test__ = {'doctests' : doctests} + def test_scope_isolation_from_global(self): + expected = {0: None, 1: None, 2: None, 3: None, 4: None, 5: None, + 6: None, 7: None, 8: None, 9: None} + actual = {g: None for g in range(10)} + self.assertEqual(actual, expected) + self.assertEqual(g, "Global variable") -def test_main(verbose=None): - import sys - from test import support - from test import test_dictcomps - support.run_doctest(test_dictcomps, verbose) + expected = {9: 1, 18: 2, 19: 2, 27: 3, 28: 3, 29: 3, 36: 4, 37: 4, + 38: 4, 39: 4, 45: 5, 46: 5, 47: 5, 48: 5, 49: 5, 54: 6, + 55: 6, 56: 6, 57: 6, 58: 6, 59: 6, 63: 7, 64: 7, 65: 7, + 66: 7, 67: 7, 68: 7, 69: 7, 72: 8, 73: 8, 74: 8, 75: 8, + 76: 8, 77: 8, 78: 8, 79: 8, 81: 9, 82: 9, 83: 9, 84: 9, + 85: 9, 86: 9, 87: 9, 88: 9, 89: 9} + actual = {g: v for v in range(10) for g in range(v * 9, v * 10)} + self.assertEqual(g, "Global variable") + self.assertEqual(actual, expected) - # verify reference counting - if verbose and hasattr(sys, "gettotalrefcount"): - import gc - counts = [None] * 5 - for i in range(len(counts)): - support.run_doctest(test_dictcomps, verbose) - gc.collect() - counts[i] = sys.gettotalrefcount() - print(counts) + def test_global_visibility(self): + expected = {0: 'Global variable', 1: 'Global variable', + 2: 'Global variable', 3: 'Global variable', + 4: 'Global variable', 5: 'Global variable', + 6: 'Global variable', 7: 'Global variable', + 8: 'Global variable', 9: 'Global variable'} + actual = {k: g for k in range(10)} + self.assertEqual(actual, expected) + + def test_local_visibility(self): + v = "Local variable" + expected = {0: 'Local variable', 1: 'Local variable', + 2: 'Local variable', 3: 'Local variable', + 4: 'Local variable', 5: 'Local variable', + 6: 'Local variable', 7: 'Local variable', + 8: 'Local variable', 9: 'Local variable'} + actual = {k: v for k in range(10)} + self.assertEqual(actual, expected) + self.assertEqual(v, "Local variable") + + def test_illegal_assignment(self): + with self.assertRaisesRegex(SyntaxError, "can't assign"): + compile("{x: y for y, x in ((1, 2), (3, 4))} = 5", "", + "exec") + + with self.assertRaisesRegex(SyntaxError, "can't assign"): + compile("{x: y for y, x in ((1, 2), (3, 4))} += 5", "", + "exec") + + +def test_main(): + support.run_unittest(__name__) if __name__ == "__main__": - test_main(verbose=True) + test_main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Jan 16 22:59:10 2013 From: python-checkins at python.org (frank.wierzbicki) Date: Wed, 16 Jan 2013 22:59:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_=2316886=3A_merge_with_3=2E2?= Message-ID: <3Ymj4G735vzRv7@mail.python.org> http://hg.python.org/cpython/rev/8ab5bd5f95b3 changeset: 81548:8ab5bd5f95b3 branch: 3.3 parent: 81544:750f44e621aa parent: 81547:c7dfc307b88e user: Frank Wierzbicki date: Wed Jan 16 13:56:54 2013 -0800 summary: #16886: merge with 3.2 files: Lib/test/test_dictcomps.py | 117 ++++++++++++++++-------- 1 files changed, 77 insertions(+), 40 deletions(-) diff --git a/Lib/test/test_dictcomps.py b/Lib/test/test_dictcomps.py --- a/Lib/test/test_dictcomps.py +++ b/Lib/test/test_dictcomps.py @@ -1,54 +1,91 @@ +import unittest -doctests = """ +from test import support - >>> k = "old value" - >>> { k: None for k in range(10) } - {0: None, 1: None, 2: None, 3: None, 4: None, 5: None, 6: None, 7: None, 8: None, 9: None} - >>> k - 'old value' +# For scope testing. +g = "Global variable" - >>> { k: k+10 for k in range(10) } - {0: 10, 1: 11, 2: 12, 3: 13, 4: 14, 5: 15, 6: 16, 7: 17, 8: 18, 9: 19} - >>> g = "Global variable" - >>> { k: g for k in range(10) } - {0: 'Global variable', 1: 'Global variable', 2: 'Global variable', 3: 'Global variable', 4: 'Global variable', 5: 'Global variable', 6: 'Global variable', 7: 'Global variable', 8: 'Global variable', 9: 'Global variable'} +class DictComprehensionTest(unittest.TestCase): - >>> { k: v for k in range(10) for v in range(10) if k == v } - {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9} + def test_basics(self): + expected = {0: 10, 1: 11, 2: 12, 3: 13, 4: 14, 5: 15, 6: 16, 7: 17, + 8: 18, 9: 19} + actual = {k: k + 10 for k in range(10)} + self.assertEqual(actual, expected) - >>> { k: v for v in range(10) for k in range(v*9, v*10) } - {9: 1, 18: 2, 19: 2, 27: 3, 28: 3, 29: 3, 36: 4, 37: 4, 38: 4, 39: 4, 45: 5, 46: 5, 47: 5, 48: 5, 49: 5, 54: 6, 55: 6, 56: 6, 57: 6, 58: 6, 59: 6, 63: 7, 64: 7, 65: 7, 66: 7, 67: 7, 68: 7, 69: 7, 72: 8, 73: 8, 74: 8, 75: 8, 76: 8, 77: 8, 78: 8, 79: 8, 81: 9, 82: 9, 83: 9, 84: 9, 85: 9, 86: 9, 87: 9, 88: 9, 89: 9} + expected = {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9} + actual = {k: v for k in range(10) for v in range(10) if k == v} + self.assertEqual(actual, expected) - >>> { x: y for y, x in ((1, 2), (3, 4)) } = 5 # doctest: +IGNORE_EXCEPTION_DETAIL - Traceback (most recent call last): - ... - SyntaxError: ... + def test_scope_isolation(self): + k = "Local Variable" - >>> { x: y for y, x in ((1, 2), (3, 4)) } += 5 # doctest: +IGNORE_EXCEPTION_DETAIL - Traceback (most recent call last): - ... - SyntaxError: ... + expected = {0: None, 1: None, 2: None, 3: None, 4: None, 5: None, + 6: None, 7: None, 8: None, 9: None} + actual = {k: None for k in range(10)} + self.assertEqual(actual, expected) + self.assertEqual(k, "Local Variable") -""" + expected = {9: 1, 18: 2, 19: 2, 27: 3, 28: 3, 29: 3, 36: 4, 37: 4, + 38: 4, 39: 4, 45: 5, 46: 5, 47: 5, 48: 5, 49: 5, 54: 6, + 55: 6, 56: 6, 57: 6, 58: 6, 59: 6, 63: 7, 64: 7, 65: 7, + 66: 7, 67: 7, 68: 7, 69: 7, 72: 8, 73: 8, 74: 8, 75: 8, + 76: 8, 77: 8, 78: 8, 79: 8, 81: 9, 82: 9, 83: 9, 84: 9, + 85: 9, 86: 9, 87: 9, 88: 9, 89: 9} + actual = {k: v for v in range(10) for k in range(v * 9, v * 10)} + self.assertEqual(k, "Local Variable") + self.assertEqual(actual, expected) -__test__ = {'doctests' : doctests} + def test_scope_isolation_from_global(self): + expected = {0: None, 1: None, 2: None, 3: None, 4: None, 5: None, + 6: None, 7: None, 8: None, 9: None} + actual = {g: None for g in range(10)} + self.assertEqual(actual, expected) + self.assertEqual(g, "Global variable") -def test_main(verbose=None): - import sys - from test import support - from test import test_dictcomps - support.run_doctest(test_dictcomps, verbose) + expected = {9: 1, 18: 2, 19: 2, 27: 3, 28: 3, 29: 3, 36: 4, 37: 4, + 38: 4, 39: 4, 45: 5, 46: 5, 47: 5, 48: 5, 49: 5, 54: 6, + 55: 6, 56: 6, 57: 6, 58: 6, 59: 6, 63: 7, 64: 7, 65: 7, + 66: 7, 67: 7, 68: 7, 69: 7, 72: 8, 73: 8, 74: 8, 75: 8, + 76: 8, 77: 8, 78: 8, 79: 8, 81: 9, 82: 9, 83: 9, 84: 9, + 85: 9, 86: 9, 87: 9, 88: 9, 89: 9} + actual = {g: v for v in range(10) for g in range(v * 9, v * 10)} + self.assertEqual(g, "Global variable") + self.assertEqual(actual, expected) - # verify reference counting - if verbose and hasattr(sys, "gettotalrefcount"): - import gc - counts = [None] * 5 - for i in range(len(counts)): - support.run_doctest(test_dictcomps, verbose) - gc.collect() - counts[i] = sys.gettotalrefcount() - print(counts) + def test_global_visibility(self): + expected = {0: 'Global variable', 1: 'Global variable', + 2: 'Global variable', 3: 'Global variable', + 4: 'Global variable', 5: 'Global variable', + 6: 'Global variable', 7: 'Global variable', + 8: 'Global variable', 9: 'Global variable'} + actual = {k: g for k in range(10)} + self.assertEqual(actual, expected) + + def test_local_visibility(self): + v = "Local variable" + expected = {0: 'Local variable', 1: 'Local variable', + 2: 'Local variable', 3: 'Local variable', + 4: 'Local variable', 5: 'Local variable', + 6: 'Local variable', 7: 'Local variable', + 8: 'Local variable', 9: 'Local variable'} + actual = {k: v for k in range(10)} + self.assertEqual(actual, expected) + self.assertEqual(v, "Local variable") + + def test_illegal_assignment(self): + with self.assertRaisesRegex(SyntaxError, "can't assign"): + compile("{x: y for y, x in ((1, 2), (3, 4))} = 5", "", + "exec") + + with self.assertRaisesRegex(SyntaxError, "can't assign"): + compile("{x: y for y, x in ((1, 2), (3, 4))} += 5", "", + "exec") + + +def test_main(): + support.run_unittest(__name__) if __name__ == "__main__": - test_main(verbose=True) + test_main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Jan 16 22:59:12 2013 From: python-checkins at python.org (frank.wierzbicki) Date: Wed, 16 Jan 2013 22:59:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_=2316886=3A_merge_with_3=2E3?= Message-ID: <3Ymj4J5JwMzRnh@mail.python.org> http://hg.python.org/cpython/rev/d0cef6c90efc changeset: 81549:d0cef6c90efc parent: 81545:f87d85f7596c parent: 81548:8ab5bd5f95b3 user: Frank Wierzbicki date: Wed Jan 16 13:58:14 2013 -0800 summary: #16886: merge with 3.3 files: Lib/test/test_dictcomps.py | 117 ++++++++++++++++-------- 1 files changed, 77 insertions(+), 40 deletions(-) diff --git a/Lib/test/test_dictcomps.py b/Lib/test/test_dictcomps.py --- a/Lib/test/test_dictcomps.py +++ b/Lib/test/test_dictcomps.py @@ -1,54 +1,91 @@ +import unittest -doctests = """ +from test import support - >>> k = "old value" - >>> { k: None for k in range(10) } - {0: None, 1: None, 2: None, 3: None, 4: None, 5: None, 6: None, 7: None, 8: None, 9: None} - >>> k - 'old value' +# For scope testing. +g = "Global variable" - >>> { k: k+10 for k in range(10) } - {0: 10, 1: 11, 2: 12, 3: 13, 4: 14, 5: 15, 6: 16, 7: 17, 8: 18, 9: 19} - >>> g = "Global variable" - >>> { k: g for k in range(10) } - {0: 'Global variable', 1: 'Global variable', 2: 'Global variable', 3: 'Global variable', 4: 'Global variable', 5: 'Global variable', 6: 'Global variable', 7: 'Global variable', 8: 'Global variable', 9: 'Global variable'} +class DictComprehensionTest(unittest.TestCase): - >>> { k: v for k in range(10) for v in range(10) if k == v } - {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9} + def test_basics(self): + expected = {0: 10, 1: 11, 2: 12, 3: 13, 4: 14, 5: 15, 6: 16, 7: 17, + 8: 18, 9: 19} + actual = {k: k + 10 for k in range(10)} + self.assertEqual(actual, expected) - >>> { k: v for v in range(10) for k in range(v*9, v*10) } - {9: 1, 18: 2, 19: 2, 27: 3, 28: 3, 29: 3, 36: 4, 37: 4, 38: 4, 39: 4, 45: 5, 46: 5, 47: 5, 48: 5, 49: 5, 54: 6, 55: 6, 56: 6, 57: 6, 58: 6, 59: 6, 63: 7, 64: 7, 65: 7, 66: 7, 67: 7, 68: 7, 69: 7, 72: 8, 73: 8, 74: 8, 75: 8, 76: 8, 77: 8, 78: 8, 79: 8, 81: 9, 82: 9, 83: 9, 84: 9, 85: 9, 86: 9, 87: 9, 88: 9, 89: 9} + expected = {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9} + actual = {k: v for k in range(10) for v in range(10) if k == v} + self.assertEqual(actual, expected) - >>> { x: y for y, x in ((1, 2), (3, 4)) } = 5 # doctest: +IGNORE_EXCEPTION_DETAIL - Traceback (most recent call last): - ... - SyntaxError: ... + def test_scope_isolation(self): + k = "Local Variable" - >>> { x: y for y, x in ((1, 2), (3, 4)) } += 5 # doctest: +IGNORE_EXCEPTION_DETAIL - Traceback (most recent call last): - ... - SyntaxError: ... + expected = {0: None, 1: None, 2: None, 3: None, 4: None, 5: None, + 6: None, 7: None, 8: None, 9: None} + actual = {k: None for k in range(10)} + self.assertEqual(actual, expected) + self.assertEqual(k, "Local Variable") -""" + expected = {9: 1, 18: 2, 19: 2, 27: 3, 28: 3, 29: 3, 36: 4, 37: 4, + 38: 4, 39: 4, 45: 5, 46: 5, 47: 5, 48: 5, 49: 5, 54: 6, + 55: 6, 56: 6, 57: 6, 58: 6, 59: 6, 63: 7, 64: 7, 65: 7, + 66: 7, 67: 7, 68: 7, 69: 7, 72: 8, 73: 8, 74: 8, 75: 8, + 76: 8, 77: 8, 78: 8, 79: 8, 81: 9, 82: 9, 83: 9, 84: 9, + 85: 9, 86: 9, 87: 9, 88: 9, 89: 9} + actual = {k: v for v in range(10) for k in range(v * 9, v * 10)} + self.assertEqual(k, "Local Variable") + self.assertEqual(actual, expected) -__test__ = {'doctests' : doctests} + def test_scope_isolation_from_global(self): + expected = {0: None, 1: None, 2: None, 3: None, 4: None, 5: None, + 6: None, 7: None, 8: None, 9: None} + actual = {g: None for g in range(10)} + self.assertEqual(actual, expected) + self.assertEqual(g, "Global variable") -def test_main(verbose=None): - import sys - from test import support - from test import test_dictcomps - support.run_doctest(test_dictcomps, verbose) + expected = {9: 1, 18: 2, 19: 2, 27: 3, 28: 3, 29: 3, 36: 4, 37: 4, + 38: 4, 39: 4, 45: 5, 46: 5, 47: 5, 48: 5, 49: 5, 54: 6, + 55: 6, 56: 6, 57: 6, 58: 6, 59: 6, 63: 7, 64: 7, 65: 7, + 66: 7, 67: 7, 68: 7, 69: 7, 72: 8, 73: 8, 74: 8, 75: 8, + 76: 8, 77: 8, 78: 8, 79: 8, 81: 9, 82: 9, 83: 9, 84: 9, + 85: 9, 86: 9, 87: 9, 88: 9, 89: 9} + actual = {g: v for v in range(10) for g in range(v * 9, v * 10)} + self.assertEqual(g, "Global variable") + self.assertEqual(actual, expected) - # verify reference counting - if verbose and hasattr(sys, "gettotalrefcount"): - import gc - counts = [None] * 5 - for i in range(len(counts)): - support.run_doctest(test_dictcomps, verbose) - gc.collect() - counts[i] = sys.gettotalrefcount() - print(counts) + def test_global_visibility(self): + expected = {0: 'Global variable', 1: 'Global variable', + 2: 'Global variable', 3: 'Global variable', + 4: 'Global variable', 5: 'Global variable', + 6: 'Global variable', 7: 'Global variable', + 8: 'Global variable', 9: 'Global variable'} + actual = {k: g for k in range(10)} + self.assertEqual(actual, expected) + + def test_local_visibility(self): + v = "Local variable" + expected = {0: 'Local variable', 1: 'Local variable', + 2: 'Local variable', 3: 'Local variable', + 4: 'Local variable', 5: 'Local variable', + 6: 'Local variable', 7: 'Local variable', + 8: 'Local variable', 9: 'Local variable'} + actual = {k: v for k in range(10)} + self.assertEqual(actual, expected) + self.assertEqual(v, "Local variable") + + def test_illegal_assignment(self): + with self.assertRaisesRegex(SyntaxError, "can't assign"): + compile("{x: y for y, x in ((1, 2), (3, 4))} = 5", "", + "exec") + + with self.assertRaisesRegex(SyntaxError, "can't assign"): + compile("{x: y for y, x in ((1, 2), (3, 4))} += 5", "", + "exec") + + +def test_main(): + support.run_unittest(__name__) if __name__ == "__main__": - test_main(verbose=True) + test_main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Jan 16 23:11:40 2013 From: python-checkins at python.org (frank.wierzbicki) Date: Wed, 16 Jan 2013 23:11:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Use_unittest?= =?utf-8?q?=2Emain=28=29_in_3=2E3_and_greater?= Message-ID: <3YmjLh5gLfzRnh@mail.python.org> http://hg.python.org/cpython/rev/489f4e716b4b changeset: 81550:489f4e716b4b branch: 3.3 parent: 81548:8ab5bd5f95b3 user: Frank Wierzbicki date: Wed Jan 16 14:09:57 2013 -0800 summary: Use unittest.main() in 3.3 and greater files: Lib/test/test_dictcomps.py | 5 +---- 1 files changed, 1 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_dictcomps.py b/Lib/test/test_dictcomps.py --- a/Lib/test/test_dictcomps.py +++ b/Lib/test/test_dictcomps.py @@ -84,8 +84,5 @@ "exec") -def test_main(): - support.run_unittest(__name__) - if __name__ == "__main__": - test_main() + unittest.main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Jan 16 23:11:42 2013 From: python-checkins at python.org (frank.wierzbicki) Date: Wed, 16 Jan 2013 23:11:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_merge_with_3=2E3?= Message-ID: <3YmjLk1T1PzRwR@mail.python.org> http://hg.python.org/cpython/rev/0dbd8a357a99 changeset: 81551:0dbd8a357a99 parent: 81549:d0cef6c90efc parent: 81550:489f4e716b4b user: Frank Wierzbicki date: Wed Jan 16 14:10:36 2013 -0800 summary: merge with 3.3 files: Lib/test/test_dictcomps.py | 5 +---- 1 files changed, 1 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_dictcomps.py b/Lib/test/test_dictcomps.py --- a/Lib/test/test_dictcomps.py +++ b/Lib/test/test_dictcomps.py @@ -84,8 +84,5 @@ "exec") -def test_main(): - support.run_unittest(__name__) - if __name__ == "__main__": - test_main() + unittest.main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Jan 16 23:27:10 2013 From: python-checkins at python.org (stefan.krah) Date: Wed, 16 Jan 2013 23:27:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzExNzI5?= =?utf-8?q?=3A_Backport_commit_bff052d9_from_libffi_upstream_in_order_to_f?= =?utf-8?q?ix?= Message-ID: <3YmjhZ4RbCzS1s@mail.python.org> http://hg.python.org/cpython/rev/190a115b7748 changeset: 81552:190a115b7748 branch: 3.3 parent: 81550:489f4e716b4b user: Stefan Krah date: Wed Jan 16 23:18:34 2013 +0100 summary: Issue #11729: Backport commit bff052d9 from libffi upstream in order to fix a ctypes build failure with clang. files: Modules/_ctypes/libffi/configure | 6 +++--- Modules/_ctypes/libffi/configure.ac | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Modules/_ctypes/libffi/configure b/Modules/_ctypes/libffi/configure --- a/Modules/_ctypes/libffi/configure +++ b/Modules/_ctypes/libffi/configure @@ -14322,10 +14322,10 @@ $as_echo_n "(cached) " >&6 else - libffi_cv_as_x86_pcrel=yes + libffi_cv_as_x86_pcrel=no echo '.text; foo: nop; .data; .long foo-.; .text' > conftest.s - if $CC $CFLAGS -c conftest.s 2>&1 | $EGREP -i 'illegal|warning' > /dev/null; then - libffi_cv_as_x86_pcrel=no + if $CC $CFLAGS -c conftest.s > /dev/null 2>&1; then + libffi_cv_as_x86_pcrel=yes fi fi diff --git a/Modules/_ctypes/libffi/configure.ac b/Modules/_ctypes/libffi/configure.ac --- a/Modules/_ctypes/libffi/configure.ac +++ b/Modules/_ctypes/libffi/configure.ac @@ -303,10 +303,10 @@ if test x$TARGET = xX86 || test x$TARGET = xX86_WIN32 || test x$TARGET = xX86_64; then AC_CACHE_CHECK([assembler supports pc related relocs], libffi_cv_as_x86_pcrel, [ - libffi_cv_as_x86_pcrel=yes + libffi_cv_as_x86_pcrel=no echo '.text; foo: nop; .data; .long foo-.; .text' > conftest.s - if $CC $CFLAGS -c conftest.s 2>&1 | $EGREP -i 'illegal|warning' > /dev/null; then - libffi_cv_as_x86_pcrel=no + if $CC $CFLAGS -c conftest.s > /dev/null 2>&1; then + libffi_cv_as_x86_pcrel=yes fi ]) if test "x$libffi_cv_as_x86_pcrel" = xyes; then -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Jan 16 23:27:12 2013 From: python-checkins at python.org (stefan.krah) Date: Wed, 16 Jan 2013 23:27:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy4zLg==?= Message-ID: <3Ymjhc0Y4czS75@mail.python.org> http://hg.python.org/cpython/rev/f3e348ab08c6 changeset: 81553:f3e348ab08c6 parent: 81551:0dbd8a357a99 parent: 81552:190a115b7748 user: Stefan Krah date: Wed Jan 16 23:25:41 2013 +0100 summary: Merge 3.3. files: Modules/_ctypes/libffi/configure | 6 +++--- Modules/_ctypes/libffi/configure.ac | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Modules/_ctypes/libffi/configure b/Modules/_ctypes/libffi/configure --- a/Modules/_ctypes/libffi/configure +++ b/Modules/_ctypes/libffi/configure @@ -14322,10 +14322,10 @@ $as_echo_n "(cached) " >&6 else - libffi_cv_as_x86_pcrel=yes + libffi_cv_as_x86_pcrel=no echo '.text; foo: nop; .data; .long foo-.; .text' > conftest.s - if $CC $CFLAGS -c conftest.s 2>&1 | $EGREP -i 'illegal|warning' > /dev/null; then - libffi_cv_as_x86_pcrel=no + if $CC $CFLAGS -c conftest.s > /dev/null 2>&1; then + libffi_cv_as_x86_pcrel=yes fi fi diff --git a/Modules/_ctypes/libffi/configure.ac b/Modules/_ctypes/libffi/configure.ac --- a/Modules/_ctypes/libffi/configure.ac +++ b/Modules/_ctypes/libffi/configure.ac @@ -303,10 +303,10 @@ if test x$TARGET = xX86 || test x$TARGET = xX86_WIN32 || test x$TARGET = xX86_64; then AC_CACHE_CHECK([assembler supports pc related relocs], libffi_cv_as_x86_pcrel, [ - libffi_cv_as_x86_pcrel=yes + libffi_cv_as_x86_pcrel=no echo '.text; foo: nop; .data; .long foo-.; .text' > conftest.s - if $CC $CFLAGS -c conftest.s 2>&1 | $EGREP -i 'illegal|warning' > /dev/null; then - libffi_cv_as_x86_pcrel=no + if $CC $CFLAGS -c conftest.s > /dev/null 2>&1; then + libffi_cv_as_x86_pcrel=yes fi ]) if test "x$libffi_cv_as_x86_pcrel" = xyes; then -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Thu Jan 17 05:56:47 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 17 Jan 2013 05:56:47 +0100 Subject: [Python-checkins] Daily reference leaks (f3e348ab08c6): sum=0 Message-ID: results for f3e348ab08c6 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogIlOgAX', '-x'] From python-checkins at python.org Thu Jan 17 12:54:11 2013 From: python-checkins at python.org (stefan.krah) Date: Thu, 17 Jan 2013 12:54:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Fix_build_with?= =?utf-8?q?_3=2Ex_gcc_versions=2E?= Message-ID: <3Yn3bl06wczS8T@mail.python.org> http://hg.python.org/cpython/rev/d536f34759f5 changeset: 81554:d536f34759f5 branch: 3.3 parent: 81552:190a115b7748 user: Stefan Krah date: Thu Jan 17 12:49:34 2013 +0100 summary: Fix build with 3.x gcc versions. files: setup.py | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-) diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -1997,8 +1997,12 @@ # Increase warning level for gcc: if 'gcc' in cc: - extra_compile_args.extend(['-Wextra', - '-Wno-missing-field-initializers']) + cmd = ("echo '' | gcc -Wextra -Wno-missing-field-initializers -E - " + "> /dev/null 2>&1") + ret = os.system(cmd) + if ret >> 8 == 0: + extra_compile_args.extend(['-Wextra', + '-Wno-missing-field-initializers']) # Uncomment for extra functionality: #define_macros.append(('EXTRA_FUNCTIONALITY', 1)) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Jan 17 12:54:12 2013 From: python-checkins at python.org (stefan.krah) Date: Thu, 17 Jan 2013 12:54:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy4zLg==?= Message-ID: <3Yn3bm34NbzSB0@mail.python.org> http://hg.python.org/cpython/rev/9638fc96deb7 changeset: 81555:9638fc96deb7 parent: 81553:f3e348ab08c6 parent: 81554:d536f34759f5 user: Stefan Krah date: Thu Jan 17 12:52:42 2013 +0100 summary: Merge 3.3. files: setup.py | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-) diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -2008,8 +2008,12 @@ # Increase warning level for gcc: if 'gcc' in cc: - extra_compile_args.extend(['-Wextra', - '-Wno-missing-field-initializers']) + cmd = ("echo '' | gcc -Wextra -Wno-missing-field-initializers -E - " + "> /dev/null 2>&1") + ret = os.system(cmd) + if ret >> 8 == 0: + extra_compile_args.extend(['-Wextra', + '-Wno-missing-field-initializers']) # Uncomment for extra functionality: #define_macros.append(('EXTRA_FUNCTIONALITY', 1)) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Jan 17 15:36:02 2013 From: python-checkins at python.org (stefan.krah) Date: Thu, 17 Jan 2013 15:36:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE0MTEw?= =?utf-8?q?=3A_Fix_test_failures_on_FreeBSD_if_the_user_is_in_the_wheel_gr?= =?utf-8?q?oup=2E?= Message-ID: <3Yn7BV1rwrzS9Y@mail.python.org> http://hg.python.org/cpython/rev/a94752d75c74 changeset: 81556:a94752d75c74 branch: 3.3 parent: 81554:d536f34759f5 user: Stefan Krah date: Thu Jan 17 15:31:00 2013 +0100 summary: Issue #14110: Fix test failures on FreeBSD if the user is in the wheel group. files: Lib/test/test_os.py | 9 ++++++--- 1 files changed, 6 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -49,6 +49,9 @@ else: USING_LINUXTHREADS = False +# Issue #14110: Some tests fail on FreeBSD if the user is in the wheel group. +HAVE_WHEEL_GROUP = sys.platform.startswith('freebsd') and os.getgid() == 0 + # Tests creating TESTFN class FileTests(unittest.TestCase): def setUp(self): @@ -1240,7 +1243,7 @@ if hasattr(os, 'setgid'): def test_setgid(self): - if os.getuid() != 0: + if os.getuid() != 0 and not HAVE_WHEEL_GROUP: self.assertRaises(os.error, os.setgid, 0) self.assertRaises(OverflowError, os.setgid, 1<<32) @@ -1252,7 +1255,7 @@ if hasattr(os, 'setegid'): def test_setegid(self): - if os.getuid() != 0: + if os.getuid() != 0 and not HAVE_WHEEL_GROUP: self.assertRaises(os.error, os.setegid, 0) self.assertRaises(OverflowError, os.setegid, 1<<32) @@ -1272,7 +1275,7 @@ if hasattr(os, 'setregid'): def test_setregid(self): - if os.getuid() != 0: + if os.getuid() != 0 and not HAVE_WHEEL_GROUP: self.assertRaises(os.error, os.setregid, 0, 0) self.assertRaises(OverflowError, os.setregid, 1<<32, 0) self.assertRaises(OverflowError, os.setregid, 0, 1<<32) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Jan 17 15:36:03 2013 From: python-checkins at python.org (stefan.krah) Date: Thu, 17 Jan 2013 15:36:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy4zLg==?= Message-ID: <3Yn7BW4yLTzSD5@mail.python.org> http://hg.python.org/cpython/rev/59ff2d75c836 changeset: 81557:59ff2d75c836 parent: 81555:9638fc96deb7 parent: 81556:a94752d75c74 user: Stefan Krah date: Thu Jan 17 15:34:50 2013 +0100 summary: Merge 3.3. files: Lib/test/test_os.py | 9 ++++++--- 1 files changed, 6 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -49,6 +49,9 @@ else: USING_LINUXTHREADS = False +# Issue #14110: Some tests fail on FreeBSD if the user is in the wheel group. +HAVE_WHEEL_GROUP = sys.platform.startswith('freebsd') and os.getgid() == 0 + # Tests creating TESTFN class FileTests(unittest.TestCase): def setUp(self): @@ -1240,7 +1243,7 @@ if hasattr(os, 'setgid'): def test_setgid(self): - if os.getuid() != 0: + if os.getuid() != 0 and not HAVE_WHEEL_GROUP: self.assertRaises(OSError, os.setgid, 0) self.assertRaises(OverflowError, os.setgid, 1<<32) @@ -1252,7 +1255,7 @@ if hasattr(os, 'setegid'): def test_setegid(self): - if os.getuid() != 0: + if os.getuid() != 0 and not HAVE_WHEEL_GROUP: self.assertRaises(OSError, os.setegid, 0) self.assertRaises(OverflowError, os.setegid, 1<<32) @@ -1272,7 +1275,7 @@ if hasattr(os, 'setregid'): def test_setregid(self): - if os.getuid() != 0: + if os.getuid() != 0 and not HAVE_WHEEL_GROUP: self.assertRaises(OSError, os.setregid, 0, 0) self.assertRaises(OverflowError, os.setregid, 1<<32, 0) self.assertRaises(OverflowError, os.setregid, 0, 1<<32) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Jan 17 15:37:06 2013 From: python-checkins at python.org (eli.bendersky) Date: Thu, 17 Jan 2013 15:37:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_docstring_typo_in_conc?= =?utf-8?q?urrent=2Efutures=2EFuture?= Message-ID: <3Yn7Ck5GXgzRFH@mail.python.org> http://hg.python.org/cpython/rev/f43cc9987a1f changeset: 81558:f43cc9987a1f user: Eli Bendersky date: Thu Jan 17 06:36:30 2013 -0800 summary: Fix docstring typo in concurrent.futures.Future files: Lib/concurrent/futures/_base.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/concurrent/futures/_base.py b/Lib/concurrent/futures/_base.py --- a/Lib/concurrent/futures/_base.py +++ b/Lib/concurrent/futures/_base.py @@ -331,7 +331,7 @@ return True def cancelled(self): - """Return True if the future has cancelled.""" + """Return True if the future was cancelled.""" with self._condition: return self._state in [CANCELLED, CANCELLED_AND_NOTIFIED] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Jan 17 17:08:21 2013 From: python-checkins at python.org (stefan.krah) Date: Thu, 17 Jan 2013 17:08:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2316982=3A_Fix_--wi?= =?utf-8?q?thout-threads_build_failure=2E?= Message-ID: <3Yn9F15D51zLn4@mail.python.org> http://hg.python.org/cpython/rev/8630fa732cf6 changeset: 81559:8630fa732cf6 user: Stefan Krah date: Thu Jan 17 17:07:17 2013 +0100 summary: Issue #16982: Fix --without-threads build failure. files: Modules/_ssl.c | 12 +++++++++--- 1 files changed, 9 insertions(+), 3 deletions(-) diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -2392,15 +2392,17 @@ PyObject *result; /* The high-level ssl.SSLSocket object */ PyObject *ssl_socket; - PyGILState_STATE gstate; const char *servername = SSL_get_servername(s, TLSEXT_NAMETYPE_host_name); - - gstate = PyGILState_Ensure(); +#ifdef WITH_THREAD + PyGILState_STATE gstate = PyGILState_Ensure(); +#endif if (ssl_ctx->set_hostname == NULL) { /* remove race condition in this the call back while if removing the * callback is in progress */ +#ifdef WITH_THREAD PyGILState_Release(gstate); +#endif return SSL_TLSEXT_ERR_OK; } @@ -2449,14 +2451,18 @@ Py_DECREF(result); } +#ifdef WITH_THREAD PyGILState_Release(gstate); +#endif return ret; error: Py_DECREF(ssl_socket); *al = SSL_AD_INTERNAL_ERROR; ret = SSL_TLSEXT_ERR_ALERT_FATAL; +#ifdef WITH_THREAD PyGILState_Release(gstate); +#endif return ret; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Jan 17 20:45:09 2013 From: python-checkins at python.org (stefan.krah) Date: Thu, 17 Jan 2013 20:45:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE2OTg3?= =?utf-8?q?=3A_Fix_definition_of_SHLIB=5FEXT=2E?= Message-ID: <3YnG390DrqzSDp@mail.python.org> http://hg.python.org/cpython/rev/937e9df47ccd changeset: 81560:937e9df47ccd branch: 3.3 parent: 81556:a94752d75c74 user: Stefan Krah date: Thu Jan 17 20:41:56 2013 +0100 summary: Issue #16987: Fix definition of SHLIB_EXT. files: configure | 11 ++++++----- configure.ac | 3 ++- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/configure b/configure --- a/configure +++ b/configure @@ -8384,11 +8384,6 @@ - -cat >>confdefs.h <<_ACEOF -#define SHLIB_EXT "$SO" -_ACEOF - # LDSHARED is the ld *command* used to create shared library # -- "cc -G" on SunOS 5.x, "ld -shared" on IRIX 5 # (Shared libraries in this instance are shared modules to be loaded into @@ -13721,6 +13716,12 @@ { $as_echo "$as_me:${as_lineno-$LINENO}: result: $SO" >&5 $as_echo "$SO" >&6; } + +cat >>confdefs.h <<_ACEOF +#define SHLIB_EXT "$SO" +_ACEOF + + # Check whether right shifting a negative integer extends the sign bit # or fills with zeros (like the Cray J90, according to Tim Peters). { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether right shift extends the sign bit" >&5 diff --git a/configure.ac b/configure.ac --- a/configure.ac +++ b/configure.ac @@ -1898,7 +1898,6 @@ AC_SUBST(CCSHARED) AC_SUBST(LINKFORSHARED) -AC_DEFINE_UNQUOTED(SHLIB_EXT, "$SO", [Define this to be extension of shared libraries (including the dot!).]) # LDSHARED is the ld *command* used to create shared library # -- "cc -G" on SunOS 5.x, "ld -shared" on IRIX 5 # (Shared libraries in this instance are shared modules to be loaded into @@ -3956,6 +3955,8 @@ fi AC_MSG_RESULT($SO) +AC_DEFINE_UNQUOTED(SHLIB_EXT, "$SO", [Define this to be extension of shared libraries (including the dot!).]) + # Check whether right shifting a negative integer extends the sign bit # or fills with zeros (like the Cray J90, according to Tim Peters). AC_MSG_CHECKING(whether right shift extends the sign bit) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Jan 17 20:45:11 2013 From: python-checkins at python.org (stefan.krah) Date: Thu, 17 Jan 2013 20:45:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy4zLg==?= Message-ID: <3YnG3C1V1wzSFw@mail.python.org> http://hg.python.org/cpython/rev/144c82bebf17 changeset: 81561:144c82bebf17 parent: 81559:8630fa732cf6 parent: 81560:937e9df47ccd user: Stefan Krah date: Thu Jan 17 20:44:02 2013 +0100 summary: Merge 3.3. files: configure | 11 ++++++----- configure.ac | 3 ++- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/configure b/configure --- a/configure +++ b/configure @@ -8384,11 +8384,6 @@ - -cat >>confdefs.h <<_ACEOF -#define SHLIB_EXT "$SO" -_ACEOF - # LDSHARED is the ld *command* used to create shared library # -- "cc -G" on SunOS 5.x, "ld -shared" on IRIX 5 # (Shared libraries in this instance are shared modules to be loaded into @@ -13721,6 +13716,12 @@ { $as_echo "$as_me:${as_lineno-$LINENO}: result: $SO" >&5 $as_echo "$SO" >&6; } + +cat >>confdefs.h <<_ACEOF +#define SHLIB_EXT "$SO" +_ACEOF + + # Check whether right shifting a negative integer extends the sign bit # or fills with zeros (like the Cray J90, according to Tim Peters). { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether right shift extends the sign bit" >&5 diff --git a/configure.ac b/configure.ac --- a/configure.ac +++ b/configure.ac @@ -1898,7 +1898,6 @@ AC_SUBST(CCSHARED) AC_SUBST(LINKFORSHARED) -AC_DEFINE_UNQUOTED(SHLIB_EXT, "$SO", [Define this to be extension of shared libraries (including the dot!).]) # LDSHARED is the ld *command* used to create shared library # -- "cc -G" on SunOS 5.x, "ld -shared" on IRIX 5 # (Shared libraries in this instance are shared modules to be loaded into @@ -3956,6 +3955,8 @@ fi AC_MSG_RESULT($SO) +AC_DEFINE_UNQUOTED(SHLIB_EXT, "$SO", [Define this to be extension of shared libraries (including the dot!).]) + # Check whether right shifting a negative integer extends the sign bit # or fills with zeros (like the Cray J90, according to Tim Peters). AC_MSG_CHECKING(whether right shift extends the sign bit) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Jan 17 20:57:04 2013 From: python-checkins at python.org (guido.van.rossum) Date: Thu, 17 Jan 2013 20:57:04 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Rename_create=5Ftransport=28?= =?utf-8?q?=29_to_create=5Fconnection=28=29=2E_Remove_type_argument=2E?= Message-ID: <3YnGJw19bQzSFV@mail.python.org> http://hg.python.org/peps/rev/a68dfdcf7879 changeset: 4680:a68dfdcf7879 user: Guido van Rossum date: Thu Jan 17 11:57:00 2013 -0800 summary: Rename create_transport() to create_connection(). Remove type argument. files: pep-3156.txt | 50 +++++++++++++++++++++++---------------- 1 files changed, 29 insertions(+), 21 deletions(-) diff --git a/pep-3156.txt b/pep-3156.txt --- a/pep-3156.txt +++ b/pep-3156.txt @@ -295,49 +295,57 @@ on success will be a tuple ``(host, port)``. Same implementation remarks as for ``getaddrinfo()``. -- ``create_transport(protocol_factory, host, port, **kwargs)``. - Creates a transport and a protocol and ties them together. Returns - a Future whose result on success is a (transport, protocol) pair. - The protocol is created by calling ``protocol_factory()`` without - arguments(*). Note that when the Future completes, the protocol's - ``connection_made()`` method has not yet been called; that will - happen when the connection handshake is complete. When it is - impossible to connect to the given host and port, the Future will - raise an exception instead. +- ``create_connection(protocol_factory, host, port, **kwargs)``. + Creates a stream connection to a given host and port. This creates + an implementation-dependent Transport to represent the connection, + then calls ``protocol_factory()`` to instantiate (or retrieve) the + user's Protocol implementation, and finally ties the two together. + (See below for the definitions of Transport and Protocol.) The + user's Protocol implementation is created or retrieved by calling + ``protocol_factory()`` without arguments(*). The return value is a + Future whose result on success is the ``(transport, protocol)`` + pair; if a failure prevents the creation of a successful connection, + the Future will have an appropriate exception set. Note that when + the Future completes, the protocol's ``connection_made()`` method + has not yet been called; that will happen when the connection + handshake is complete. (*) There is no requirement that ``protocol_factory`` is a class. If your protocol class needs to have specific arguments passed to its constructor, you can use ``lambda`` or ``functools.partial()``. + You can also pass a trivial ``lambda`` that returns a previously + constructed Protocol instance. Optional keyword arguments: - - ``family``, ``type``, ``proto``, ``flags``: Address familty, - socket type, protcol, and miscellaneous flags to be passed through - to ``getaddrinfo()``. These all default to ``0`` except ``type`` - which defaults to ``socket.SOCK_STREAM``. + - ``family``, ``proto``, ``flags``: Address familty, + protcol, and miscellaneous flags to be passed through + to ``getaddrinfo()``. These all default to ``0``. + (The socket type is always ``SOCK_STREAM``.) - ``ssl``: Pass ``True`` to create an SSL transport (by default a plain TCP is created). Or pass an ``ssl.SSLContext`` object to override the default SSL context object to be used. - TBD: Should this be called create_connection()? - - ``start_serving(protocol_factory, host, port, **kwds)``. Enters a loop that accepts connections. Returns a Future that completes once the loop is set up to serve; its return value is None. Each time a connection is accepted, ``protocol_factory`` is called without - arguments(*) to create a protocol, a transport is created to represent + arguments(*) to create a Protocol, a Transport is created to represent the network side of the connection, and the two are tied together by calling ``protocol.connection_made(transport)``. - (*) See footnote above for ``create_transport()``. + (*) See footnote above for ``create_connection()``. However, since + ``protocol_factory()`` is called once for each new incoming + connection, it is recommended that it return a new Protocol object + each time it is called. Optional keyword arguments: - - ``family``, ``type``, ``proto``, ``flags``: Address familty, - socket type, protcol, and miscellaneous flags to be passed through - to ``getaddrinfo()``. These all default to ``0`` except ``type`` - which defaults to ``socket.SOCK_STREAM``. + - ``family``, ``proto``, ``flags``: Address familty, + protcol, and miscellaneous flags to be passed through + to ``getaddrinfo()``. These all default to ``0``. + (The socket type is always ``SOCK_STREAM``.) TBD: Support SSL? I don't even know how to do that synchronously, and I suppose it needs a certificate. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu Jan 17 23:11:51 2013 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 17 Jan 2013 23:11:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E2=29=3A_Remove_outdate?= =?utf-8?q?d_statement?= Message-ID: <3YnKJR31TQzSDc@mail.python.org> http://hg.python.org/cpython/rev/ce6dc780e69e changeset: 81562:ce6dc780e69e branch: 3.2 parent: 81547:c7dfc307b88e user: Antoine Pitrou date: Thu Jan 17 23:08:03 2013 +0100 summary: Remove outdated statement files: Lib/test/test_capi.py | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -170,7 +170,6 @@ self.pendingcalls_wait(l, n) def test_subinterps(self): - # XXX this test leaks in refleak runs import builtins r, w = os.pipe() code = """if 1: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Jan 17 23:11:52 2013 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 17 Jan 2013 23:11:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Remove_outdated_statement?= Message-ID: <3YnKJS5gvrzSDc@mail.python.org> http://hg.python.org/cpython/rev/c6276dea445c changeset: 81563:c6276dea445c branch: 3.3 parent: 81560:937e9df47ccd parent: 81562:ce6dc780e69e user: Antoine Pitrou date: Thu Jan 17 23:08:26 2013 +0100 summary: Remove outdated statement files: Lib/test/test_capi.py | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -193,7 +193,6 @@ self.pendingcalls_wait(l, n) def test_subinterps(self): - # XXX this test leaks in refleak runs import builtins r, w = os.pipe() code = """if 1: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Jan 17 23:11:54 2013 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 17 Jan 2013 23:11:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Remove_outdated_statement?= Message-ID: <3YnKJV1twJzSDp@mail.python.org> http://hg.python.org/cpython/rev/0e191a9d9a13 changeset: 81564:0e191a9d9a13 parent: 81561:144c82bebf17 parent: 81563:c6276dea445c user: Antoine Pitrou date: Thu Jan 17 23:09:07 2013 +0100 summary: Remove outdated statement files: Lib/test/test_capi.py | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -193,7 +193,6 @@ self.pendingcalls_wait(l, n) def test_subinterps(self): - # XXX this test leaks in refleak runs import builtins r, w = os.pipe() code = """if 1: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Jan 17 23:13:30 2013 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 17 Jan 2013 23:13:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Remove_pointle?= =?utf-8?q?ss_discussion_of_the_lack_of_a_ternary_operator?= Message-ID: <3YnKLL2GZPzSDc@mail.python.org> http://hg.python.org/cpython/rev/af5feb869304 changeset: 81565:af5feb869304 branch: 2.7 parent: 81546:6d06b223c664 user: Antoine Pitrou