From solipsis at pitrou.net  Thu May  1 09:24:19 2014
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Thu, 01 May 2014 09:24:19 +0200
Subject: [Python-checkins] Daily reference leaks (2a56d3896d50): sum=7
Message-ID: <E1WflLv-0005F2-6o@vds2544.sivit.org>

results for 2a56d3896d50 on branch "default"
--------------------------------------------

test_asyncio leaked [4, 0, 0] memory blocks, sum=4
test_functools leaked [0, 0, 3] memory blocks, sum=3
test_site leaked [0, 2, -2] references, sum=0
test_site leaked [0, 2, -2] memory blocks, sum=0


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/refloguholWQ', '-x']

From python-checkins at python.org  Thu May  1 14:36:29 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Thu,  1 May 2014 14:36:29 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321377=3A_PyBytes?=
 =?utf-8?q?=5FConcat=28=29_now_tries_to_concatenate_in-place_when_the_firs?=
 =?utf-8?q?t?=
Message-ID: <3gKGL56wFLz7LjM@mail.python.org>

http://hg.python.org/cpython/rev/4ed1b6c7e2f3
changeset:   90530:4ed1b6c7e2f3
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu May 01 14:36:20 2014 +0200
summary:
  Issue #21377: PyBytes_Concat() now tries to concatenate in-place when the first argument has a reference count of 1.
Patch by Nikolaus Rath.

files:
  Misc/NEWS             |   3 ++
  Objects/bytesobject.c |  43 ++++++++++++++++++++++++++++--
  2 files changed, 42 insertions(+), 4 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #21377: PyBytes_Concat() now tries to concatenate in-place when the
+  first argument has a reference count of 1.  Patch by Nikolaus Rath.
+
 - Issue #20355: -W command line options now have higher priority than the
   PYTHONWARNINGS environment variable.  Patch by Arfrever.
 
diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -2781,7 +2781,6 @@
 void
 PyBytes_Concat(PyObject **pv, PyObject *w)
 {
-    PyObject *v;
     assert(pv != NULL);
     if (*pv == NULL)
         return;
@@ -2789,9 +2788,45 @@
         Py_CLEAR(*pv);
         return;
     }
-    v = bytes_concat(*pv, w);
-    Py_DECREF(*pv);
-    *pv = v;
+
+    if (Py_REFCNT(*pv) == 1 && PyBytes_CheckExact(*pv)) {
+        /* Only one reference, so we can resize in place */
+        size_t oldsize;
+        Py_buffer wb;
+        
+        wb.len = -1;
+        if (_getbuffer(w, &wb) < 0) {
+            PyErr_Format(PyExc_TypeError, "can't concat %.100s to %.100s",
+                         Py_TYPE(w)->tp_name, Py_TYPE(*pv)->tp_name);
+            Py_CLEAR(*pv);
+            return;
+        }
+
+        oldsize = PyBytes_GET_SIZE(*pv);
+        if (oldsize > PY_SSIZE_T_MAX - wb.len) {
+            PyErr_NoMemory();
+            goto error;
+        }
+        if (_PyBytes_Resize(pv, oldsize + wb.len) < 0)
+            goto error;
+
+        memcpy(PyBytes_AS_STRING(*pv) + oldsize, wb.buf, wb.len);
+        PyBuffer_Release(&wb);
+        return;
+
+      error:
+        PyBuffer_Release(&wb);
+        Py_CLEAR(*pv);
+        return;
+    }
+
+    else {
+        /* Multiple references, need to create new object */
+        PyObject *v;
+        v = bytes_concat(*pv, w);
+        Py_DECREF(*pv);
+        *pv = v;
+    }
 }
 
 void

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu May  1 15:19:45 2014
From: python-checkins at python.org (martin.v.loewis)
Date: Thu,  1 May 2014 15:19:45 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogVXBkYXRlIHRvIDEu?=
	=?utf-8?q?0=2E1g?=
Message-ID: <3gKHJ12DZ0z7LjV@mail.python.org>

http://hg.python.org/cpython/rev/94f4cad7d541
changeset:   90531:94f4cad7d541
branch:      3.4
parent:      90524:182b869283a5
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Thu May 01 14:28:48 2014 +0200
summary:
  Update to 1.0.1g

files:
  Misc/NEWS                          |  2 ++
  PC/VS9.0/pyproject.vsprops         |  2 +-
  PCbuild/pyproject.props            |  2 +-
  PCbuild/readme.txt                 |  2 +-
  Tools/buildbot/external-common.bat |  8 ++++----
  5 files changed, 9 insertions(+), 7 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -218,6 +218,8 @@
 Build
 -----
 
+- The Windows build now includes OpenSSL 1.0.1g
+
 - Issue #21285: Refactor and fix curses configure check to always search
   in a ncursesw directory.
 
diff --git a/PC/VS9.0/pyproject.vsprops b/PC/VS9.0/pyproject.vsprops
--- a/PC/VS9.0/pyproject.vsprops
+++ b/PC/VS9.0/pyproject.vsprops
@@ -62,7 +62,7 @@
 	/>
 	<UserMacro
 		Name="opensslDir"
-		Value="$(externalsDir)\openssl-1.0.1e"
+		Value="$(externalsDir)\openssl-1.0.1g"
 	/>
 	<UserMacro
 		Name="tcltkDir"
diff --git a/PCbuild/pyproject.props b/PCbuild/pyproject.props
--- a/PCbuild/pyproject.props
+++ b/PCbuild/pyproject.props
@@ -20,7 +20,7 @@
     <sqlite3Dir>$(externalsDir)\sqlite-3.8.3.1</sqlite3Dir>
     <bz2Dir>$(externalsDir)\bzip2-1.0.6</bz2Dir>
     <lzmaDir>$(externalsDir)\xz-5.0.5</lzmaDir>
-    <opensslDir>$(externalsDir)\openssl-1.0.1e</opensslDir>
+    <opensslDir>$(externalsDir)\openssl-1.0.1g</opensslDir>
     <tcltkDir>$(externalsDir)\tcltk</tcltkDir>
     <tcltk64Dir>$(externalsDir)\tcltk64</tcltk64Dir>
     <tcltkLib>$(tcltkDir)\lib\tcl86t.lib;$(tcltkDir)\lib\tk86t.lib</tcltkLib>
diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt
--- a/PCbuild/readme.txt
+++ b/PCbuild/readme.txt
@@ -168,7 +168,7 @@
     Homepage:
         http://tukaani.org/xz/
 _ssl
-    Python wrapper for version 1.0.1e of the OpenSSL secure sockets
+    Python wrapper for version 1.0.1g of the OpenSSL secure sockets
     library, which is built by ssl.vcxproj
     Homepage:
         http://www.openssl.org/
diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat
--- a/Tools/buildbot/external-common.bat
+++ b/Tools/buildbot/external-common.bat
@@ -14,7 +14,7 @@
 @rem if exist tk8.4.16 rd /s/q tk8.4.16
 @rem if exist tk-8.4.18.1 rd /s/q tk-8.4.18.1
 @rem if exist db-4.4.20 rd /s/q db-4.4.20
- at rem if exist openssl-1.0.1e rd /s/q openssl-1.0.1e
+ at rem if exist openssl-1.0.1e rd /s/q openssl-1.0.1g
 @rem if exist sqlite-3.7.12 rd /s/q sqlite-3.7.12    
 
 @rem bzip
@@ -24,9 +24,9 @@
 )
 
 @rem OpenSSL
-if not exist openssl-1.0.1e (
-    rd /s/q openssl-1.0.1d
-    svn export http://svn.python.org/projects/external/openssl-1.0.1e
+if not exist openssl-1.0.1g (
+    rd /s/q openssl-1.0.1e
+    svn export http://svn.python.org/projects/external/openssl-1.0.1g
 )
 
 @rem tcl/tk

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu May  1 15:19:46 2014
From: python-checkins at python.org (martin.v.loewis)
Date: Thu,  1 May 2014 15:19:46 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?q?=29=3A_Merge_with_3=2E4?=
Message-ID: <3gKHJ23cl5z7LjV@mail.python.org>

http://hg.python.org/cpython/rev/3cf9bd0a25ce
changeset:   90532:3cf9bd0a25ce
parent:      90530:4ed1b6c7e2f3
parent:      90531:94f4cad7d541
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Thu May 01 15:18:43 2014 +0200
summary:
  Merge with 3.4

files:
  Misc/NEWS                          |  2 ++
  PC/VS9.0/pyproject.vsprops         |  2 +-
  PCbuild/pyproject.props            |  2 +-
  PCbuild/readme.txt                 |  2 +-
  Tools/buildbot/external-common.bat |  8 ++++----
  5 files changed, 9 insertions(+), 7 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -325,6 +325,8 @@
 Build
 -----
 
+- The Windows build now includes OpenSSL 1.0.1g
+
 - Issue #19962: The Windows build process now creates "python.bat" in the
   root of the source tree, which passes all arguments through to the most
   recently built interpreter.
diff --git a/PC/VS9.0/pyproject.vsprops b/PC/VS9.0/pyproject.vsprops
--- a/PC/VS9.0/pyproject.vsprops
+++ b/PC/VS9.0/pyproject.vsprops
@@ -62,7 +62,7 @@
 	/>
 	<UserMacro
 		Name="opensslDir"
-		Value="$(externalsDir)\openssl-1.0.1e"
+		Value="$(externalsDir)\openssl-1.0.1g"
 	/>
 	<UserMacro
 		Name="tcltkDir"
diff --git a/PCbuild/pyproject.props b/PCbuild/pyproject.props
--- a/PCbuild/pyproject.props
+++ b/PCbuild/pyproject.props
@@ -20,7 +20,7 @@
     <sqlite3Dir>$(externalsDir)\sqlite-3.8.3.1</sqlite3Dir>
     <bz2Dir>$(externalsDir)\bzip2-1.0.6</bz2Dir>
     <lzmaDir>$(externalsDir)\xz-5.0.5</lzmaDir>
-    <opensslDir>$(externalsDir)\openssl-1.0.1e</opensslDir>
+    <opensslDir>$(externalsDir)\openssl-1.0.1g</opensslDir>
     <tclDir>$(externalsDir)\tcl-8.6.1.0</tclDir>
     <tkDir>$(externalsDir)\tk-8.6.1.0</tkDir>
     <tixDir>$(externalsDir)\tix-8.4.3.4</tixDir>
diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt
--- a/PCbuild/readme.txt
+++ b/PCbuild/readme.txt
@@ -168,7 +168,7 @@
     Homepage:
         http://tukaani.org/xz/
 _ssl
-    Python wrapper for version 1.0.1e of the OpenSSL secure sockets
+    Python wrapper for version 1.0.1g of the OpenSSL secure sockets
     library, which is built by ssl.vcxproj
     Homepage:
         http://www.openssl.org/
diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat
--- a/Tools/buildbot/external-common.bat
+++ b/Tools/buildbot/external-common.bat
@@ -14,7 +14,7 @@
 @rem if exist tk8.4.16 rd /s/q tk8.4.16
 @rem if exist tk-8.4.18.1 rd /s/q tk-8.4.18.1
 @rem if exist db-4.4.20 rd /s/q db-4.4.20
- at rem if exist openssl-1.0.1e rd /s/q openssl-1.0.1e
+ at rem if exist openssl-1.0.1e rd /s/q openssl-1.0.1g
 @rem if exist sqlite-3.7.12 rd /s/q sqlite-3.7.12
 
 @rem bzip
@@ -24,9 +24,9 @@
 )
 
 @rem OpenSSL
-if not exist openssl-1.0.1e (
-    rd /s/q openssl-1.0.1d
-    svn export http://svn.python.org/projects/external/openssl-1.0.1e
+if not exist openssl-1.0.1g (
+    rd /s/q openssl-1.0.1e
+    svn export http://svn.python.org/projects/external/openssl-1.0.1g
 )
 
 @rem tcl/tk/tix

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu May  1 15:54:25 2014
From: python-checkins at python.org (stefan.krah)
Date: Thu,  1 May 2014 15:54:25 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321407=3A_=5Fdecim?=
 =?utf-8?q?al_now_supports_function_signatures=2E?=
Message-ID: <3gKJ4133GBz7LmK@mail.python.org>

http://hg.python.org/cpython/rev/40b06a75d1c6
changeset:   90533:40b06a75d1c6
user:        Stefan Krah <skrah at bytereef.org>
date:        Thu May 01 15:53:42 2014 +0200
summary:
  Issue #21407: _decimal now supports function signatures.

files:
  Lib/test/test_decimal.py      |  138 +++
  Misc/NEWS                     |    2 +
  Modules/_decimal/docstrings.h |  862 ++++++++++++---------
  3 files changed, 623 insertions(+), 379 deletions(-)


diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py
--- a/Lib/test/test_decimal.py
+++ b/Lib/test/test_decimal.py
@@ -39,6 +39,7 @@
 import random
 import time
 import warnings
+import inspect
 try:
     import threading
 except ImportError:
@@ -5390,6 +5391,142 @@
             y = Decimal(10**(9*25)).__sizeof__()
             self.assertEqual(y, x+4)
 
+unittest.skipUnless(C, "test requires C version")
+class SignatureTest(unittest.TestCase):
+    """Function signatures"""
+
+    def test_inspect_module(self):
+        for attr in dir(P):
+            if attr.startswith('_'):
+                continue
+            p_func = getattr(P, attr)
+            c_func = getattr(C, attr)
+            if (attr == 'Decimal' or attr == 'Context' or
+                inspect.isfunction(p_func)):
+                p_sig = inspect.signature(p_func)
+                c_sig = inspect.signature(c_func)
+
+                # parameter names:
+                c_names = list(c_sig.parameters.keys())
+                p_names = [x for x in p_sig.parameters.keys() if not
+                           x.startswith('_')]
+
+                self.assertEqual(c_names, p_names,
+                                 msg="parameter name mismatch in %s" % p_func)
+
+                c_kind = [x.kind for x in c_sig.parameters.values()]
+                p_kind = [x[1].kind for x in p_sig.parameters.items() if not
+                          x[0].startswith('_')]
+
+                # parameters:
+                if attr != 'setcontext':
+                    self.assertEqual(c_kind, p_kind,
+                                     msg="parameter kind mismatch in %s" % p_func)
+
+    def test_inspect_types(self):
+
+        POS = inspect._ParameterKind.POSITIONAL_ONLY
+        POS_KWD = inspect._ParameterKind.POSITIONAL_OR_KEYWORD
+
+        # Type heuristic (type annotations would help!):
+        pdict = {C: {'other': C.Decimal(1),
+                     'third': C.Decimal(1),
+                     'x': C.Decimal(1),
+                     'y': C.Decimal(1),
+                     'z': C.Decimal(1),
+                     'a': C.Decimal(1),
+                     'b': C.Decimal(1),
+                     'c': C.Decimal(1),
+                     'exp': C.Decimal(1),
+                     'modulo': C.Decimal(1),
+                     'num': "1",
+                     'f': 1.0,
+                     'rounding': C.ROUND_HALF_UP,
+                     'context': C.getcontext()},
+                 P: {'other': P.Decimal(1),
+                     'third': P.Decimal(1),
+                     'a': P.Decimal(1),
+                     'b': P.Decimal(1),
+                     'c': P.Decimal(1),
+                     'exp': P.Decimal(1),
+                     'modulo': P.Decimal(1),
+                     'num': "1",
+                     'f': 1.0,
+                     'rounding': P.ROUND_HALF_UP,
+                     'context': P.getcontext()}}
+
+        def mkargs(module, sig):
+            args = []
+            kwargs = {}
+            for name, param in sig.parameters.items():
+                if name == 'self': continue
+                if param.kind == POS:
+                    args.append(pdict[module][name])
+                elif param.kind == POS_KWD:
+                    kwargs[name] = pdict[module][name]
+                else:
+                    raise TestFailed("unexpected parameter kind")
+            return args, kwargs
+
+        def tr(s):
+            """The C Context docstrings use 'x' in order to prevent confusion
+               with the article 'a' in the descriptions."""
+            if s == 'x': return 'a'
+            if s == 'y': return 'b'
+            if s == 'z': return 'c'
+            return s
+
+        def doit(ty):
+            p_type = getattr(P, ty)
+            c_type = getattr(C, ty)
+            for attr in dir(p_type):
+                if attr.startswith('_'):
+                    continue
+                p_func = getattr(p_type, attr)
+                c_func = getattr(c_type, attr)
+                if inspect.isfunction(p_func):
+                    p_sig = inspect.signature(p_func)
+                    c_sig = inspect.signature(c_func)
+
+                    # parameter names:
+                    p_names = list(p_sig.parameters.keys())
+                    c_names = [tr(x) for x in c_sig.parameters.keys()]
+
+                    self.assertEqual(c_names, p_names,
+                                     msg="parameter name mismatch in %s" % p_func)
+
+                    p_kind = [x.kind for x in p_sig.parameters.values()]
+                    c_kind = [x.kind for x in c_sig.parameters.values()]
+
+                    # 'self' parameter:
+                    self.assertIs(p_kind[0], POS_KWD)
+                    self.assertIs(c_kind[0], POS)
+
+                    # remaining parameters:
+                    if ty == 'Decimal':
+                        self.assertEqual(c_kind[1:], p_kind[1:],
+                                         msg="parameter kind mismatch in %s" % p_func)
+                    else: # Context methods are positional only in the C version.
+                        self.assertEqual(len(c_kind), len(p_kind),
+                                         msg="parameter kind mismatch in %s" % p_func)
+
+                    # Run the function:
+                    args, kwds = mkargs(C, c_sig)
+                    try:
+                        getattr(c_type(9), attr)(*args, **kwds)
+                    except Exception as err:
+                        raise TestFailed("invalid signature for %s: %s %s" % (c_func, args, kwds))
+
+                    args, kwds = mkargs(P, p_sig)
+                    try:
+                        getattr(p_type(9), attr)(*args, **kwds)
+                    except Exception as err:
+                        raise TestFailed("invalid signature for %s: %s %s" % (p_func, args, kwds))
+
+        doit('Decimal')
+        doit('Context')
+
+
 all_tests = [
   CExplicitConstructionTest, PyExplicitConstructionTest,
   CImplicitConstructionTest, PyImplicitConstructionTest,
@@ -5415,6 +5552,7 @@
     all_tests = all_tests[1::2]
 else:
     all_tests.insert(0, CheckAttributes)
+    all_tests.insert(1, SignatureTest)
 
 
 def test_main(arith=False, verbose=None, todo_tests=None, debug=None):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -310,6 +310,8 @@
 Extension Modules
 -----------------
 
+- Issue #21407: _decimal: The module now supports function signatures.
+
 - Issue #21276: posixmodule: Don't define USE_XATTRS on KFreeBSD and the Hurd.
 
 IDLE
diff --git a/Modules/_decimal/docstrings.h b/Modules/_decimal/docstrings.h
--- a/Modules/_decimal/docstrings.h
+++ b/Modules/_decimal/docstrings.h
@@ -19,26 +19,30 @@
 PyDoc_STRVAR(doc__decimal,
 "C decimal arithmetic module");
 
-PyDoc_STRVAR(doc_getcontext,"\n\
-getcontext() - Get the current default context.\n\
+PyDoc_STRVAR(doc_getcontext,
+"getcontext($module, /)\n--\n\n\
+Get the current default context.\n\
 \n");
 
-PyDoc_STRVAR(doc_setcontext,"\n\
-setcontext(c) - Set a new default context.\n\
+PyDoc_STRVAR(doc_setcontext,
+"setcontext($module, context, /)\n--\n\n\
+Set a new default context.\n\
 \n");
 
-PyDoc_STRVAR(doc_localcontext,"\n\
-localcontext(ctx=None) - Return a context manager that will set the default\n\
-context to a copy of ctx on entry to the with-statement and restore the\n\
-previous default context when exiting the with-statement. If no context is\n\
-specified, a copy of the current default context is used.\n\
+PyDoc_STRVAR(doc_localcontext,
+"localcontext($module, /, ctx=None)\n--\n\n\
+Return a context manager that will set the default context to a copy of ctx\n\
+on entry to the with-statement and restore the previous default context when\n\
+exiting the with-statement. If no context is specified, a copy of the current\n\
+default context is used.\n\
 \n");
 
 #ifdef EXTRA_FUNCTIONALITY
-PyDoc_STRVAR(doc_ieee_context,"\n\
-IEEEContext(bits) - Return a context object initialized to the proper values for\n\
-one of the IEEE interchange formats. The argument must be a multiple of 32 and\n\
-less than IEEE_CONTEXT_MAX_BITS. For the most common values, the constants\n\
+PyDoc_STRVAR(doc_ieee_context,
+"IEEEContext($module, bits, /)\n--\n\n\
+Return a context object initialized to the proper values for one of the\n\
+IEEE interchange formats.  The argument must be a multiple of 32 and less\n\
+than IEEE_CONTEXT_MAX_BITS.  For the most common values, the constants\n\
 DECIMAL32, DECIMAL64 and DECIMAL128 are provided.\n\
 \n");
 #endif
@@ -48,32 +52,34 @@
 /*                       Decimal Object and Methods                           */
 /******************************************************************************/
 
-PyDoc_STRVAR(doc_decimal,"\n\
-Decimal(value=\"0\", context=None): Construct a new Decimal object.\n\
-value can be an integer, string, tuple, or another Decimal object.\n\
-If no value is given, return Decimal('0'). The context does not affect\n\
-the conversion and is only passed to determine if the InvalidOperation\n\
-trap is active.\n\
+PyDoc_STRVAR(doc_decimal,
+"Decimal(value=\"0\", context=None)\n--\n\n\
+Construct a new Decimal object. 'value' can be an integer, string, tuple,\n\
+or another Decimal object. If no value is given, return Decimal('0'). The\n\
+context does not affect the conversion and is only passed to determine if\n\
+the InvalidOperation trap is active.\n\
 \n");
 
-PyDoc_STRVAR(doc_adjusted,"\n\
-adjusted() - Return the adjusted exponent of the number.\n\
-\n\
-Defined as exp + digits - 1.\n\
+PyDoc_STRVAR(doc_adjusted,
+"adjusted($self, /)\n--\n\n\
+Return the adjusted exponent of the number.  Defined as exp + digits - 1.\n\
 \n");
 
-PyDoc_STRVAR(doc_as_tuple,"\n\
-as_tuple() - Return a tuple representation of the number.\n\
+PyDoc_STRVAR(doc_as_tuple,
+"as_tuple($self, /)\n--\n\n\
+Return a tuple representation of the number.\n\
 \n");
 
-PyDoc_STRVAR(doc_canonical,"\n\
-canonical() - Return the canonical encoding of the argument. Currently,\n\
-the encoding of a Decimal instance is always canonical, so this operation\n\
-returns its argument unchanged.\n\
+PyDoc_STRVAR(doc_canonical,
+"canonical($self, /)\n--\n\n\
+Return the canonical encoding of the argument.  Currently, the encoding\n\
+of a Decimal instance is always canonical, so this operation returns its\n\
+argument unchanged.\n\
 \n");
 
-PyDoc_STRVAR(doc_compare,"\n\
-compare(other, context=None) - Compare self to other. Return a decimal value:\n\
+PyDoc_STRVAR(doc_compare,
+"compare($self, /, other, context=None)\n--\n\n\
+Compare self to other.  Return a decimal value:\n\
 \n\
     a or b is a NaN ==> Decimal('NaN')\n\
     a < b           ==> Decimal('-1')\n\
@@ -81,17 +87,18 @@
     a > b           ==> Decimal('1')\n\
 \n");
 
-PyDoc_STRVAR(doc_compare_signal,"\n\
-compare_signal(other, context=None) - Identical to compare, except that\n\
-all NaNs signal.\n\
+PyDoc_STRVAR(doc_compare_signal,
+"compare_signal($self, /, other, context=None)\n--\n\n\
+Identical to compare, except that all NaNs signal.\n\
 \n");
 
-PyDoc_STRVAR(doc_compare_total,"\n\
-compare_total(other, context=None) - Compare two operands using their\n\
-abstract representation rather than their numerical value. Similar to the\n\
-compare() method, but the result gives a total ordering on Decimal instances.\n\
-Two Decimal instances with the same numeric value but different representations\n\
-compare unequal in this ordering:\n\
+PyDoc_STRVAR(doc_compare_total,
+"compare_total($self, /, other, context=None)\n--\n\n\
+Compare two operands using their abstract representation rather than\n\
+their numerical value.  Similar to the compare() method, but the result\n\
+gives a total ordering on Decimal instances.  Two Decimal instances with\n\
+the same numeric value but different representations compare unequal\n\
+in this ordering:\n\
 \n\
     >>> Decimal('12.0').compare_total(Decimal('12'))\n\
     Decimal('-1')\n\
@@ -107,36 +114,39 @@
 InvalidOperation if the second operand cannot be converted exactly.\n\
 \n");
 
-PyDoc_STRVAR(doc_compare_total_mag,"\n\
-compare_total_mag(other, context=None) - Compare two operands using their\n\
-abstract representation rather than their value as in compare_total(), but\n\
-ignoring the sign of each operand. x.compare_total_mag(y) is equivalent to\n\
-x.copy_abs().compare_total(y.copy_abs()).\n\
+PyDoc_STRVAR(doc_compare_total_mag,
+"compare_total_mag($self, /, other, context=None)\n--\n\n\
+Compare two operands using their abstract representation rather than their\n\
+value as in compare_total(), but ignoring the sign of each operand.\n\
+\n\
+x.compare_total_mag(y) is equivalent to x.copy_abs().compare_total(y.copy_abs()).\n\
 \n\
 This operation is unaffected by context and is quiet: no flags are changed\n\
 and no rounding is performed. As an exception, the C version may raise\n\
 InvalidOperation if the second operand cannot be converted exactly.\n\
 \n");
 
-PyDoc_STRVAR(doc_conjugate,"\n\
-conjugate() - Return self.\n\
+PyDoc_STRVAR(doc_conjugate,
+"conjugate($self, /)\n--\n\n\
+Return self.\n\
 \n");
 
-PyDoc_STRVAR(doc_copy_abs,"\n\
-copy_abs() - Return the absolute value of the argument. This operation\n\
-is unaffected by context and is quiet: no flags are changed and no rounding\n\
-is performed.\n\
+PyDoc_STRVAR(doc_copy_abs,
+"copy_abs($self, /)\n--\n\n\
+Return the absolute value of the argument.  This operation is unaffected by\n\
+context and is quiet: no flags are changed and no rounding is performed.\n\
 \n");
 
-PyDoc_STRVAR(doc_copy_negate,"\n\
-copy_negate() - Return the negation of the argument. This operation is\n\
-unaffected by context and is quiet: no flags are changed and no rounding\n\
-is performed.\n\
+PyDoc_STRVAR(doc_copy_negate,
+"copy_negate($self, /)\n--\n\n\
+Return the negation of the argument.  This operation is unaffected by context\n\
+and is quiet: no flags are changed and no rounding is performed.\n\
 \n");
 
-PyDoc_STRVAR(doc_copy_sign,"\n\
-copy_sign(other, context=None) - Return a copy of the first operand with\n\
-the sign set to be the same as the sign of the second operand. For example:\n\
+PyDoc_STRVAR(doc_copy_sign,
+"copy_sign($self, /, other, context=None)\n--\n\n\
+Return a copy of the first operand with the sign set to be the same as the\n\
+sign of the second operand. For example:\n\
 \n\
     >>> Decimal('2.3').copy_sign(Decimal('-1.5'))\n\
     Decimal('-2.3')\n\
@@ -146,14 +156,16 @@
 InvalidOperation if the second operand cannot be converted exactly.\n\
 \n");
 
-PyDoc_STRVAR(doc_exp,"\n\
-exp(context=None) - Return the value of the (natural) exponential function\n\
-e**x at the given number. The function always uses the ROUND_HALF_EVEN mode\n\
-and the result is correctly rounded.\n\
+PyDoc_STRVAR(doc_exp,
+"exp($self, /, context=None)\n--\n\n\
+Return the value of the (natural) exponential function e**x at the given\n\
+number.  The function always uses the ROUND_HALF_EVEN mode and the result\n\
+is correctly rounded.\n\
 \n");
 
-PyDoc_STRVAR(doc_from_float,"\n\
-from_float(f) - Class method that converts a float to a decimal number, exactly.\n\
+PyDoc_STRVAR(doc_from_float,
+"from_float($cls, f, /)\n--\n\n\
+Class method that converts a float to a decimal number, exactly.\n\
 Since 0.1 is not exactly representable in binary floating point,\n\
 Decimal.from_float(0.1) is not the same as Decimal('0.1').\n\
 \n\
@@ -168,155 +180,176 @@
 \n\
 \n");
 
-PyDoc_STRVAR(doc_fma,"\n\
-fma(other, third, context=None) - Fused multiply-add. Return self*other+third\n\
-with no rounding of the intermediate product self*other.\n\
+PyDoc_STRVAR(doc_fma,
+"fma($self, /, other, third, context=None)\n--\n\n\
+Fused multiply-add.  Return self*other+third with no rounding of the\n\
+intermediate product self*other.\n\
 \n\
     >>> Decimal(2).fma(3, 5)\n\
     Decimal('11')\n\
 \n\
 \n");
 
-PyDoc_STRVAR(doc_is_canonical,"\n\
-is_canonical() - Return True if the argument is canonical and False otherwise.\n\
-Currently, a Decimal instance is always canonical, so this operation always\n\
-returns True.\n\
+PyDoc_STRVAR(doc_is_canonical,
+"is_canonical($self, /)\n--\n\n\
+Return True if the argument is canonical and False otherwise.  Currently,\n\
+a Decimal instance is always canonical, so this operation always returns\n\
+True.\n\
 \n");
 
-PyDoc_STRVAR(doc_is_finite,"\n\
-is_finite() - Return True if the argument is a finite number, and False if the\n\
-argument is infinite or a NaN.\n\
+PyDoc_STRVAR(doc_is_finite,
+"is_finite($self, /)\n--\n\n\
+Return True if the argument is a finite number, and False if the argument\n\
+is infinite or a NaN.\n\
 \n");
 
-PyDoc_STRVAR(doc_is_infinite,"\n\
-is_infinite() - Return True if the argument is either positive or negative\n\
-infinity and False otherwise.\n\
-\n");
-
-PyDoc_STRVAR(doc_is_nan,"\n\
-is_nan() - Return True if the argument is a (quiet or signaling) NaN and\n\
+PyDoc_STRVAR(doc_is_infinite,
+"is_infinite($self, /)\n--\n\n\
+Return True if the argument is either positive or negative infinity and\n\
 False otherwise.\n\
 \n");
 
-PyDoc_STRVAR(doc_is_normal,"\n\
-is_normal(context=None) - Return True if the argument is a normal finite\n\
-non-zero number with an adjusted exponent greater than or equal to Emin.\n\
-Return False if the argument is zero, subnormal, infinite or a NaN.\n\
+PyDoc_STRVAR(doc_is_nan,
+"is_nan($self, /)\n--\n\n\
+Return True if the argument is a (quiet or signaling) NaN and False\n\
+otherwise.\n\
 \n");
 
-PyDoc_STRVAR(doc_is_qnan,"\n\
-is_qnan() - Return True if the argument is a quiet NaN, and False otherwise.\n\
+PyDoc_STRVAR(doc_is_normal,
+"is_normal($self, /, context=None)\n--\n\n\
+Return True if the argument is a normal finite non-zero number with an\n\
+adjusted exponent greater than or equal to Emin. Return False if the\n\
+argument is zero, subnormal, infinite or a NaN.\n\
 \n");
 
-PyDoc_STRVAR(doc_is_signed,"\n\
-is_signed() - Return True if the argument has a negative sign and\n\
-False otherwise. Note that both zeros and NaNs can carry signs.\n\
+PyDoc_STRVAR(doc_is_qnan,
+"is_qnan($self, /)\n--\n\n\
+Return True if the argument is a quiet NaN, and False otherwise.\n\
 \n");
 
-PyDoc_STRVAR(doc_is_snan,"\n\
-is_snan() - Return True if the argument is a signaling NaN and False otherwise.\n\
+PyDoc_STRVAR(doc_is_signed,
+"is_signed($self, /)\n--\n\n\
+Return True if the argument has a negative sign and False otherwise.\n\
+Note that both zeros and NaNs can carry signs.\n\
 \n");
 
-PyDoc_STRVAR(doc_is_subnormal,"\n\
-is_subnormal(context=None) - Return True if the argument is subnormal, and\n\
-False otherwise. A number is subnormal if it is non-zero, finite, and has an\n\
-adjusted exponent less than Emin.\n\
+PyDoc_STRVAR(doc_is_snan,
+"is_snan($self, /)\n--\n\n\
+Return True if the argument is a signaling NaN and False otherwise.\n\
 \n");
 
-PyDoc_STRVAR(doc_is_zero,"\n\
-is_zero() - Return True if the argument is a (positive or negative) zero and\n\
-False otherwise.\n\
+PyDoc_STRVAR(doc_is_subnormal,
+"is_subnormal($self, /, context=None)\n--\n\n\
+Return True if the argument is subnormal, and False otherwise. A number is\n\
+subnormal if it is non-zero, finite, and has an adjusted exponent less\n\
+than Emin.\n\
 \n");
 
-PyDoc_STRVAR(doc_ln,"\n\
-ln(context=None) - Return the natural (base e) logarithm of the operand.\n\
-The function always uses the ROUND_HALF_EVEN mode and the result is\n\
-correctly rounded.\n\
+PyDoc_STRVAR(doc_is_zero,
+"is_zero($self, /)\n--\n\n\
+Return True if the argument is a (positive or negative) zero and False\n\
+otherwise.\n\
 \n");
 
-PyDoc_STRVAR(doc_log10,"\n\
-log10(context=None) - Return the base ten logarithm of the operand.\n\
-The function always uses the ROUND_HALF_EVEN mode and the result is\n\
-correctly rounded.\n\
+PyDoc_STRVAR(doc_ln,
+"ln($self, /, context=None)\n--\n\n\
+Return the natural (base e) logarithm of the operand. The function always\n\
+uses the ROUND_HALF_EVEN mode and the result is correctly rounded.\n\
 \n");
 
-PyDoc_STRVAR(doc_logb,"\n\
-logb(context=None) - For a non-zero number, return the adjusted exponent\n\
-of the operand as a Decimal instance. If the operand is a zero, then\n\
-Decimal('-Infinity') is returned and the DivisionByZero condition is\n\
-raised. If the operand is an infinity then Decimal('Infinity') is returned.\n\
+PyDoc_STRVAR(doc_log10,
+"log10($self, /, context=None)\n--\n\n\
+Return the base ten logarithm of the operand. The function always uses the\n\
+ROUND_HALF_EVEN mode and the result is correctly rounded.\n\
 \n");
 
-PyDoc_STRVAR(doc_logical_and,"\n\
-logical_and(other, context=None) - Return the digit-wise and of the two\n\
-(logical) operands.\n\
+PyDoc_STRVAR(doc_logb,
+"logb($self, /, context=None)\n--\n\n\
+For a non-zero number, return the adjusted exponent of the operand as a\n\
+Decimal instance.  If the operand is a zero, then Decimal('-Infinity') is\n\
+returned and the DivisionByZero condition is raised. If the operand is\n\
+an infinity then Decimal('Infinity') is returned.\n\
 \n");
 
-PyDoc_STRVAR(doc_logical_invert,"\n\
-logical_invert(context=None) - Return the digit-wise inversion of the\n\
-(logical) operand.\n\
+PyDoc_STRVAR(doc_logical_and,
+"logical_and($self, /, other, context=None)\n--\n\n\
+Return the digit-wise 'and' of the two (logical) operands.\n\
 \n");
 
-PyDoc_STRVAR(doc_logical_or,"\n\
-logical_or(other, context=None) - Return the digit-wise or of the two\n\
-(logical) operands.\n\
+PyDoc_STRVAR(doc_logical_invert,
+"logical_invert($self, /, context=None)\n--\n\n\
+Return the digit-wise inversion of the (logical) operand.\n\
 \n");
 
-PyDoc_STRVAR(doc_logical_xor,"\n\
-logical_xor(other, context=None) - Return the digit-wise exclusive or of the\n\
-two (logical) operands.\n\
+PyDoc_STRVAR(doc_logical_or,
+"logical_or($self, /, other, context=None)\n--\n\n\
+Return the digit-wise 'or' of the two (logical) operands.\n\
 \n");
 
-PyDoc_STRVAR(doc_max,"\n\
-max(other, context=None) - Maximum of self and other. If one operand is a\n\
-quiet NaN and the other is numeric, the numeric operand is returned.\n\
+PyDoc_STRVAR(doc_logical_xor,
+"logical_xor($self, /, other, context=None)\n--\n\n\
+Return the digit-wise 'exclusive or' of the two (logical) operands.\n\
 \n");
 
-PyDoc_STRVAR(doc_max_mag,"\n\
-max_mag(other, context=None) - Similar to the max() method, but the\n\
-comparison is done using the absolute values of the operands.\n\
+PyDoc_STRVAR(doc_max,
+"max($self, /, other, context=None)\n--\n\n\
+Maximum of self and other.  If one operand is a quiet NaN and the other is\n\
+numeric, the numeric operand is returned.\n\
 \n");
 
-PyDoc_STRVAR(doc_min,"\n\
-min(other, context=None) - Minimum of self and other. If one operand is a\n\
-quiet NaN and the other is numeric, the numeric operand is returned.\n\
+PyDoc_STRVAR(doc_max_mag,
+"max_mag($self, /, other, context=None)\n--\n\n\
+Similar to the max() method, but the comparison is done using the absolute\n\
+values of the operands.\n\
 \n");
 
-PyDoc_STRVAR(doc_min_mag,"\n\
-min_mag(other, context=None) - Similar to the min() method, but the\n\
-comparison is done using the absolute values of the operands.\n\
+PyDoc_STRVAR(doc_min,
+"min($self, /, other, context=None)\n--\n\n\
+Minimum of self and other. If one operand is a quiet NaN and the other is\n\
+numeric, the numeric operand is returned.\n\
 \n");
 
-PyDoc_STRVAR(doc_next_minus,"\n\
-next_minus(context=None) - Return the largest number representable in the\n\
-given context (or in the current default context if no context is given) that\n\
-is smaller than the given operand.\n\
+PyDoc_STRVAR(doc_min_mag,
+"min_mag($self, /, other, context=None)\n--\n\n\
+Similar to the min() method, but the comparison is done using the absolute\n\
+values of the operands.\n\
 \n");
 
-PyDoc_STRVAR(doc_next_plus,"\n\
-next_plus(context=None) - Return the smallest number representable in the\n\
-given context (or in the current default context if no context is given) that\n\
-is larger than the given operand.\n\
+PyDoc_STRVAR(doc_next_minus,
+"next_minus($self, /, context=None)\n--\n\n\
+Return the largest number representable in the given context (or in the\n\
+current default context if no context is given) that is smaller than the\n\
+given operand.\n\
 \n");
 
-PyDoc_STRVAR(doc_next_toward,"\n\
-next_toward(other, context=None) - If the two operands are unequal, return\n\
-the number closest to the first operand in the direction of the second operand.\n\
-If both operands are numerically equal, return a copy of the first operand\n\
-with the sign set to be the same as the sign of the second operand.\n\
+PyDoc_STRVAR(doc_next_plus,
+"next_plus($self, /, context=None)\n--\n\n\
+Return the smallest number representable in the given context (or in the\n\
+current default context if no context is given) that is larger than the\n\
+given operand.\n\
 \n");
 
-PyDoc_STRVAR(doc_normalize,"\n\
-normalize(context=None) - Normalize the number by stripping the rightmost\n\
-trailing zeros and converting any result equal to Decimal('0') to Decimal('0e0').\n\
-Used for producing canonical values for members of an equivalence class. For\n\
-example, Decimal('32.100') and Decimal('0.321000e+2') both normalize to the\n\
-equivalent value Decimal('32.1').\n\
+PyDoc_STRVAR(doc_next_toward,
+"next_toward($self, /, other, context=None)\n--\n\n\
+If the two operands are unequal, return the number closest to the first\n\
+operand in the direction of the second operand.  If both operands are\n\
+numerically equal, return a copy of the first operand with the sign set\n\
+to be the same as the sign of the second operand.\n\
 \n");
 
-PyDoc_STRVAR(doc_number_class,"\n\
-number_class(context=None) - Return a string describing the class of the\n\
-operand. The returned value is one of the following ten strings:\n\
+PyDoc_STRVAR(doc_normalize,
+"normalize($self, /, context=None)\n--\n\n\
+Normalize the number by stripping the rightmost trailing zeros and\n\
+converting any result equal to Decimal('0') to Decimal('0e0').  Used\n\
+for producing canonical values for members of an equivalence class.\n\
+For example, Decimal('32.100') and Decimal('0.321000e+2') both normalize\n\
+to the equivalent value Decimal('32.1').\n\
+\n");
+
+PyDoc_STRVAR(doc_number_class,
+"number_class($self, /, context=None)\n--\n\n\
+Return a string describing the class of the operand.  The returned value\n\
+is one of the following ten strings:\n\
 \n\
     * '-Infinity', indicating that the operand is negative infinity.\n\
     * '-Normal', indicating that the operand is a negative normal number.\n\
@@ -331,9 +364,10 @@
 \n\
 \n");
 
-PyDoc_STRVAR(doc_quantize,"\n\
-quantize(exp, rounding=None, context=None) - Return a value equal to the\n\
-first operand after rounding and having the exponent of the second operand.\n\
+PyDoc_STRVAR(doc_quantize,
+"quantize($self, /, exp, rounding=None, context=None)\n--\n\n\
+Return a value equal to the first operand after rounding and having the\n\
+exponent of the second operand.\n\
 \n\
     >>> Decimal('1.41421356').quantize(Decimal('1.000'))\n\
     Decimal('1.414')\n\
@@ -352,93 +386,98 @@
 argument is given, the rounding mode of the current thread's context is used.\n\
 \n");
 
-PyDoc_STRVAR(doc_radix,"\n\
-radix() - Return Decimal(10), the radix (base) in which the Decimal class does\n\
+PyDoc_STRVAR(doc_radix,
+"radix($self, /)\n--\n\n\
+Return Decimal(10), the radix (base) in which the Decimal class does\n\
 all its arithmetic. Included for compatibility with the specification.\n\
 \n");
 
-PyDoc_STRVAR(doc_remainder_near,"\n\
-remainder_near(other, context=None) - Return the remainder from dividing\n\
-self by other. This differs from self % other in that the sign of the\n\
-remainder is chosen so as to minimize its absolute value. More precisely, the\n\
-return value is self - n * other where n is the integer nearest to the exact\n\
-value of self / other, and if two integers are equally near then the even one\n\
-is chosen.\n\
+PyDoc_STRVAR(doc_remainder_near,
+"remainder_near($self, /, other, context=None)\n--\n\n\
+Return the remainder from dividing self by other.  This differs from\n\
+self % other in that the sign of the remainder is chosen so as to minimize\n\
+its absolute value. More precisely, the return value is self - n * other\n\
+where n is the integer nearest to the exact value of self / other, and\n\
+if two integers are equally near then the even one is chosen.\n\
 \n\
 If the result is zero then its sign will be the sign of self.\n\
 \n");
 
-PyDoc_STRVAR(doc_rotate,"\n\
-rotate(other, context=None) - Return the result of rotating the digits of the\n\
-first operand by an amount specified by the second operand. The second operand\n\
-must be an integer in the range -precision through precision. The absolute\n\
-value of the second operand gives the number of places to rotate. If the second\n\
-operand is positive then rotation is to the left; otherwise rotation is to the\n\
-right. The coefficient of the first operand is padded on the left with zeros to\n\
+PyDoc_STRVAR(doc_rotate,
+"rotate($self, /, other, context=None)\n--\n\n\
+Return the result of rotating the digits of the first operand by an amount\n\
+specified by the second operand.  The second operand must be an integer in\n\
+the range -precision through precision. The absolute value of the second\n\
+operand gives the number of places to rotate. If the second operand is\n\
+positive then rotation is to the left; otherwise rotation is to the right.\n\
+The coefficient of the first operand is padded on the left with zeros to\n\
 length precision if necessary. The sign and exponent of the first operand are\n\
 unchanged.\n\
 \n");
 
-PyDoc_STRVAR(doc_same_quantum,"\n\
-same_quantum(other, context=None) - Test whether self and other have the\n\
-same exponent or whether both are NaN.\n\
+PyDoc_STRVAR(doc_same_quantum,
+"same_quantum($self, /, other, context=None)\n--\n\n\
+Test whether self and other have the same exponent or whether both are NaN.\n\
 \n\
 This operation is unaffected by context and is quiet: no flags are changed\n\
 and no rounding is performed. As an exception, the C version may raise\n\
 InvalidOperation if the second operand cannot be converted exactly.\n\
 \n");
 
-PyDoc_STRVAR(doc_scaleb,"\n\
-scaleb(other, context=None) - Return the first operand with the exponent\n\
-adjusted the second. Equivalently, return the first operand multiplied by\n\
-10**other. The second operand must be an integer.\n\
+PyDoc_STRVAR(doc_scaleb,
+"scaleb($self, /, other, context=None)\n--\n\n\
+Return the first operand with the exponent adjusted the second.  Equivalently,\n\
+return the first operand multiplied by 10**other. The second operand must be\n\
+an integer.\n\
 \n");
 
-PyDoc_STRVAR(doc_shift,"\n\
-shift(other, context=None) - Return the result of shifting the digits of\n\
-the first operand by an amount specified by the second operand. The second\n\
-operand must be an integer in the range -precision through precision. The\n\
-absolute value of the second operand gives the number of places to shift.\n\
-If the second operand is positive, then the shift is to the left; otherwise\n\
-the shift is to the right. Digits shifted into the coefficient are zeros.\n\
-The sign and exponent of the first operand are unchanged.\n\
+PyDoc_STRVAR(doc_shift,
+"shift($self, /, other, context=None)\n--\n\n\
+Return the result of shifting the digits of the first operand by an amount\n\
+specified by the second operand.  The second operand must be an integer in\n\
+the range -precision through precision. The absolute value of the second\n\
+operand gives the number of places to shift. If the second operand is\n\
+positive, then the shift is to the left; otherwise the shift is to the\n\
+right. Digits shifted into the coefficient are zeros. The sign and exponent\n\
+of the first operand are unchanged.\n\
 \n");
 
-PyDoc_STRVAR(doc_sqrt,"\n\
-sqrt(context=None) - Return the square root of the argument to full precision.\n\
-The result is correctly rounded using the ROUND_HALF_EVEN rounding mode.\n\
+PyDoc_STRVAR(doc_sqrt,
+"sqrt($self, /, context=None)\n--\n\n\
+Return the square root of the argument to full precision. The result is\n\
+correctly rounded using the ROUND_HALF_EVEN rounding mode.\n\
 \n");
 
-PyDoc_STRVAR(doc_to_eng_string,"\n\
-to_eng_string(context=None) - Convert to an engineering-type string.\n\
-Engineering notation has an exponent which is a multiple of 3, so there\n\
-are up to 3 digits left of the decimal place. For example, Decimal('123E+1')\n\
-is converted to Decimal('1.23E+3').\n\
+PyDoc_STRVAR(doc_to_eng_string,
+"to_eng_string($self, /, context=None)\n--\n\n\
+Convert to an engineering-type string.  Engineering notation has an exponent\n\
+which is a multiple of 3, so there are up to 3 digits left of the decimal\n\
+place. For example, Decimal('123E+1') is converted to Decimal('1.23E+3').\n\
 \n\
 The value of context.capitals determines whether the exponent sign is lower\n\
 or upper case. Otherwise, the context does not affect the operation.\n\
 \n");
 
-PyDoc_STRVAR(doc_to_integral,"\n\
-to_integral(rounding=None, context=None) - Identical to the\n\
-to_integral_value() method. The to_integral() name has been kept\n\
-for compatibility with older versions.\n\
+PyDoc_STRVAR(doc_to_integral,
+"to_integral($self, /, rounding=None, context=None)\n--\n\n\
+Identical to the to_integral_value() method.  The to_integral() name has been\n\
+kept for compatibility with older versions.\n\
 \n");
 
-PyDoc_STRVAR(doc_to_integral_exact,"\n\
-to_integral_exact(rounding=None, context=None) - Round to the nearest\n\
-integer, signaling Inexact or Rounded as appropriate if rounding occurs.\n\
-The rounding mode is determined by the rounding parameter if given, else\n\
-by the given context. If neither parameter is given, then the rounding mode\n\
-of the current default context is used.\n\
+PyDoc_STRVAR(doc_to_integral_exact,
+"to_integral_exact($self, /, rounding=None, context=None)\n--\n\n\
+Round to the nearest integer, signaling Inexact or Rounded as appropriate if\n\
+rounding occurs.  The rounding mode is determined by the rounding parameter\n\
+if given, else by the given context. If neither parameter is given, then the\n\
+rounding mode of the current default context is used.\n\
 \n");
 
-PyDoc_STRVAR(doc_to_integral_value,"\n\
-to_integral_value(rounding=None, context=None) - Round to the nearest\n\
-integer without signaling Inexact or Rounded. The rounding mode is determined\n\
-by the rounding parameter if given, else by the given context. If neither\n\
-parameter is given, then the rounding mode of the current default context is\n\
-used.\n\
+PyDoc_STRVAR(doc_to_integral_value,
+"to_integral_value($self, /, rounding=None, context=None)\n--\n\n\
+Round to the nearest integer without signaling Inexact or Rounded.  The\n\
+rounding mode is determined by the rounding parameter if given, else by\n\
+the given context. If neither parameter is given, then the rounding mode\n\
+of the current default context is used.\n\
 \n");
 
 
@@ -446,9 +485,10 @@
 /*                       Context Object and Methods                           */
 /******************************************************************************/
 
-PyDoc_STRVAR(doc_context,"\n\
+PyDoc_STRVAR(doc_context,
+"Context(prec=None, rounding=None, Emin=None, Emax=None, capitals=None, clamp=None, flags=None, traps=None)\n--\n\n\
 The context affects almost all operations and controls rounding,\n\
-Over/Underflow, raising of exceptions and much more. A new context\n\
+Over/Underflow, raising of exceptions and much more.  A new context\n\
 can be constructed as follows:\n\
 \n\
     >>> c = Context(prec=28, Emin=-425000000, Emax=425000000,\n\
@@ -460,308 +500,372 @@
 \n");
 
 #ifdef EXTRA_FUNCTIONALITY
-PyDoc_STRVAR(doc_ctx_apply,"\n\
-apply(x) - Apply self to Decimal x.\n\
+PyDoc_STRVAR(doc_ctx_apply,
+"apply($self, x, /)\n--\n\n\
+Apply self to Decimal x.\n\
 \n");
 #endif
 
-PyDoc_STRVAR(doc_ctx_clear_flags,"\n\
-clear_flags() - Reset all flags to False.\n\
+PyDoc_STRVAR(doc_ctx_clear_flags,
+"clear_flags($self, /)\n--\n\n\
+Reset all flags to False.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_clear_traps,"\n\
-clear_traps() - Set all traps to False.\n\
+PyDoc_STRVAR(doc_ctx_clear_traps,
+"clear_traps($self, /)\n--\n\n\
+Set all traps to False.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_copy,"\n\
-copy() - Return a duplicate of the context with all flags cleared.\n\
+PyDoc_STRVAR(doc_ctx_copy,
+"copy($self, /)\n--\n\n\
+Return a duplicate of the context with all flags cleared.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_copy_decimal,"\n\
-copy_decimal(x) - Return a copy of Decimal x.\n\
+PyDoc_STRVAR(doc_ctx_copy_decimal,
+"copy_decimal($self, x, /)\n--\n\n\
+Return a copy of Decimal x.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_create_decimal,"\n\
-create_decimal(x) - Create a new Decimal instance from x, using self as the\n\
-context. Unlike the Decimal constructor, this function observes the context\n\
-limits.\n\
+PyDoc_STRVAR(doc_ctx_create_decimal,
+"create_decimal($self, num=\"0\", /)\n--\n\n\
+Create a new Decimal instance from num, using self as the context. Unlike the\n\
+Decimal constructor, this function observes the context limits.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_create_decimal_from_float,"\n\
-create_decimal_from_float(f) - Create a new Decimal instance from float f.\n\
-Unlike the Decimal.from_float() class method, this function observes the\n\
-context limits.\n\
+PyDoc_STRVAR(doc_ctx_create_decimal_from_float,
+"create_decimal_from_float($self, f, /)\n--\n\n\
+Create a new Decimal instance from float f.  Unlike the Decimal.from_float()\n\
+class method, this function observes the context limits.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_Etiny,"\n\
-Etiny() - Return a value equal to Emin - prec + 1, which is the minimum\n\
-exponent value for subnormal results. When underflow occurs, the exponent\n\
-is set to Etiny.\n\
+PyDoc_STRVAR(doc_ctx_Etiny,
+"Etiny($self, /)\n--\n\n\
+Return a value equal to Emin - prec + 1, which is the minimum exponent value\n\
+for subnormal results.  When underflow occurs, the exponent is set to Etiny.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_Etop,"\n\
-Etop() - Return a value equal to Emax - prec + 1. This is the maximum exponent\n\
-if the _clamp field of the context is set to 1 (IEEE clamp mode). Etop() must\n\
-not be negative.\n\
+PyDoc_STRVAR(doc_ctx_Etop,
+"Etop($self, /)\n--\n\n\
+Return a value equal to Emax - prec + 1.  This is the maximum exponent\n\
+if the _clamp field of the context is set to 1 (IEEE clamp mode).  Etop()\n\
+must not be negative.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_abs,"\n\
-abs(x) - Return the absolute value of x.\n\
+PyDoc_STRVAR(doc_ctx_abs,
+"abs($self, x, /)\n--\n\n\
+Return the absolute value of x.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_add,"\n\
-add(x, y) - Return the sum of x and y.\n\
+PyDoc_STRVAR(doc_ctx_add,
+"add($self, x, y, /)\n--\n\n\
+Return the sum of x and y.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_canonical,"\n\
-canonical(x) - Return a new instance of x.\n\
+PyDoc_STRVAR(doc_ctx_canonical,
+"canonical($self, x, /)\n--\n\n\
+Return a new instance of x.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_compare,"\n\
-compare(x, y) - Compare x and y numerically.\n\
+PyDoc_STRVAR(doc_ctx_compare,
+"compare($self, x, y, /)\n--\n\n\
+Compare x and y numerically.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_compare_signal,"\n\
-compare_signal(x, y) - Compare x and y numerically. All NaNs signal.\n\
+PyDoc_STRVAR(doc_ctx_compare_signal,
+"compare_signal($self, x, y, /)\n--\n\n\
+Compare x and y numerically.  All NaNs signal.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_compare_total,"\n\
-compare_total(x, y) - Compare x and y using their abstract representation.\n\
+PyDoc_STRVAR(doc_ctx_compare_total,
+"compare_total($self, x, y, /)\n--\n\n\
+Compare x and y using their abstract representation.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_compare_total_mag,"\n\
-compare_total_mag(x, y) - Compare x and y using their abstract representation,\n\
-ignoring sign.\n\
+PyDoc_STRVAR(doc_ctx_compare_total_mag,
+"compare_total_mag($self, x, y, /)\n--\n\n\
+Compare x and y using their abstract representation, ignoring sign.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_copy_abs,"\n\
-copy_abs(x) - Return a copy of x with the sign set to 0.\n\
+PyDoc_STRVAR(doc_ctx_copy_abs,
+"copy_abs($self, x, /)\n--\n\n\
+Return a copy of x with the sign set to 0.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_copy_negate,"\n\
-copy_negate(x) - Return a copy of x with the sign inverted.\n\
+PyDoc_STRVAR(doc_ctx_copy_negate,
+"copy_negate($self, x, /)\n--\n\n\
+Return a copy of x with the sign inverted.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_copy_sign,"\n\
-copy_sign(x, y) - Copy the sign from y to x.\n\
+PyDoc_STRVAR(doc_ctx_copy_sign,
+"copy_sign($self, x, y, /)\n--\n\n\
+Copy the sign from y to x.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_divide,"\n\
-divide(x, y) - Return x divided by y.\n\
+PyDoc_STRVAR(doc_ctx_divide,
+"divide($self, x, y, /)\n--\n\n\
+Return x divided by y.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_divide_int,"\n\
-divide_int(x, y) - Return x divided by y, truncated to an integer.\n\
+PyDoc_STRVAR(doc_ctx_divide_int,
+"divide_int($self, x, y, /)\n--\n\n\
+Return x divided by y, truncated to an integer.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_divmod,"\n\
-divmod(x, y) - Return quotient and remainder of the division x / y.\n\
+PyDoc_STRVAR(doc_ctx_divmod,
+"divmod($self, x, y, /)\n--\n\n\
+Return quotient and remainder of the division x / y.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_exp,"\n\
-exp(x) - Return e ** x.\n\
+PyDoc_STRVAR(doc_ctx_exp,
+"exp($self, x, /)\n--\n\n\
+Return e ** x.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_fma,"\n\
-fma(x, y, z) - Return x multiplied by y, plus z.\n\
+PyDoc_STRVAR(doc_ctx_fma,
+"fma($self, x, y, z, /)\n--\n\n\
+Return x multiplied by y, plus z.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_is_canonical,"\n\
-is_canonical(x) - Return True if x is canonical, False otherwise.\n\
+PyDoc_STRVAR(doc_ctx_is_canonical,
+"is_canonical($self, x, /)\n--\n\n\
+Return True if x is canonical, False otherwise.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_is_finite,"\n\
-is_finite(x) - Return True if x is finite, False otherwise.\n\
+PyDoc_STRVAR(doc_ctx_is_finite,
+"is_finite($self, x, /)\n--\n\n\
+Return True if x is finite, False otherwise.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_is_infinite,"\n\
-is_infinite(x) - Return True if x is infinite, False otherwise.\n\
+PyDoc_STRVAR(doc_ctx_is_infinite,
+"is_infinite($self, x, /)\n--\n\n\
+Return True if x is infinite, False otherwise.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_is_nan,"\n\
-is_nan(x) - Return True if x is a qNaN or sNaN, False otherwise.\n\
+PyDoc_STRVAR(doc_ctx_is_nan,
+"is_nan($self, x, /)\n--\n\n\
+Return True if x is a qNaN or sNaN, False otherwise.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_is_normal,"\n\
-is_normal(x) - Return True if x is a normal number, False otherwise.\n\
+PyDoc_STRVAR(doc_ctx_is_normal,
+"is_normal($self, x, /)\n--\n\n\
+Return True if x is a normal number, False otherwise.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_is_qnan,"\n\
-is_qnan(x) - Return True if x is a quiet NaN, False otherwise.\n\
+PyDoc_STRVAR(doc_ctx_is_qnan,
+"is_qnan($self, x, /)\n--\n\n\
+Return True if x is a quiet NaN, False otherwise.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_is_signed,"\n\
-is_signed(x) - Return True if x is negative, False otherwise.\n\
+PyDoc_STRVAR(doc_ctx_is_signed,
+"is_signed($self, x, /)\n--\n\n\
+Return True if x is negative, False otherwise.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_is_snan,"\n\
-is_snan() - Return True if x is a signaling NaN, False otherwise.\n\
+PyDoc_STRVAR(doc_ctx_is_snan,
+"is_snan($self, x, /)\n--\n\n\
+Return True if x is a signaling NaN, False otherwise.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_is_subnormal,"\n\
-is_subnormal(x) - Return True if x is subnormal, False otherwise.\n\
+PyDoc_STRVAR(doc_ctx_is_subnormal,
+"is_subnormal($self, x, /)\n--\n\n\
+Return True if x is subnormal, False otherwise.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_is_zero,"\n\
-is_zero(x) - Return True if x is a zero, False otherwise.\n\
+PyDoc_STRVAR(doc_ctx_is_zero,
+"is_zero($self, x, /)\n--\n\n\
+Return True if x is a zero, False otherwise.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_ln,"\n\
-ln(x) - Return the natural (base e) logarithm of x.\n\
+PyDoc_STRVAR(doc_ctx_ln,
+"ln($self, x, /)\n--\n\n\
+Return the natural (base e) logarithm of x.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_log10,"\n\
-log10(x) - Return the base 10 logarithm of x.\n\
+PyDoc_STRVAR(doc_ctx_log10,
+"log10($self, x, /)\n--\n\n\
+Return the base 10 logarithm of x.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_logb,"\n\
-logb(x) - Return the exponent of the magnitude of the operand's MSD.\n\
+PyDoc_STRVAR(doc_ctx_logb,
+"logb($self, x, /)\n--\n\n\
+Return the exponent of the magnitude of the operand's MSD.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_logical_and,"\n\
-logical_and(x, y) - Digit-wise and of x and y.\n\
+PyDoc_STRVAR(doc_ctx_logical_and,
+"logical_and($self, x, y, /)\n--\n\n\
+Digit-wise and of x and y.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_logical_invert,"\n\
-logical_invert(x) - Invert all digits of x.\n\
+PyDoc_STRVAR(doc_ctx_logical_invert,
+"logical_invert($self, x, /)\n--\n\n\
+Invert all digits of x.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_logical_or,"\n\
-logical_or(x, y) - Digit-wise or of x and y.\n\
+PyDoc_STRVAR(doc_ctx_logical_or,
+"logical_or($self, x, y, /)\n--\n\n\
+Digit-wise or of x and y.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_logical_xor,"\n\
-logical_xor(x, y) - Digit-wise xor of x and y.\n\
+PyDoc_STRVAR(doc_ctx_logical_xor,
+"logical_xor($self, x, y, /)\n--\n\n\
+Digit-wise xor of x and y.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_max,"\n\
-max(x, y) - Compare the values numerically and return the maximum.\n\
+PyDoc_STRVAR(doc_ctx_max,
+"max($self, x, y, /)\n--\n\n\
+Compare the values numerically and return the maximum.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_max_mag,"\n\
-max_mag(x, y) - Compare the values numerically with their sign ignored.\n\
+PyDoc_STRVAR(doc_ctx_max_mag,
+"max_mag($self, x, y, /)\n--\n\n\
+Compare the values numerically with their sign ignored.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_min,"\n\
-min(x, y) - Compare the values numerically and return the minimum.\n\
+PyDoc_STRVAR(doc_ctx_min,
+"min($self, x, y, /)\n--\n\n\
+Compare the values numerically and return the minimum.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_min_mag,"\n\
-min_mag(x, y) - Compare the values numerically with their sign ignored.\n\
+PyDoc_STRVAR(doc_ctx_min_mag,
+"min_mag($self, x, y, /)\n--\n\n\
+Compare the values numerically with their sign ignored.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_minus,"\n\
-minus(x) - Minus corresponds to the unary prefix minus operator in Python,\n\
-but applies the context to the result.\n\
+PyDoc_STRVAR(doc_ctx_minus,
+"minus($self, x, /)\n--\n\n\
+Minus corresponds to the unary prefix minus operator in Python, but applies\n\
+the context to the result.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_multiply,"\n\
-multiply(x, y) - Return the product of x and y.\n\
+PyDoc_STRVAR(doc_ctx_multiply,
+"multiply($self, x, y, /)\n--\n\n\
+Return the product of x and y.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_next_minus,"\n\
-next_minus(x) - Return the largest representable number smaller than x.\n\
+PyDoc_STRVAR(doc_ctx_next_minus,
+"next_minus($self, x, /)\n--\n\n\
+Return the largest representable number smaller than x.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_next_plus,"\n\
-next_plus(x) - Return the smallest representable number larger than x.\n\
+PyDoc_STRVAR(doc_ctx_next_plus,
+"next_plus($self, x, /)\n--\n\n\
+Return the smallest representable number larger than x.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_next_toward,"\n\
-next_toward(x) - Return the number closest to x, in the direction towards y.\n\
+PyDoc_STRVAR(doc_ctx_next_toward,
+"next_toward($self, x, y, /)\n--\n\n\
+Return the number closest to x, in the direction towards y.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_normalize,"\n\
-normalize(x) - Reduce x to its simplest form. Alias for reduce(x).\n\
+PyDoc_STRVAR(doc_ctx_normalize,
+"normalize($self, x, /)\n--\n\n\
+Reduce x to its simplest form. Alias for reduce(x).\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_number_class,"\n\
-number_class(x) - Return an indication of the class of x.\n\
+PyDoc_STRVAR(doc_ctx_number_class,
+"number_class($self, x, /)\n--\n\n\
+Return an indication of the class of x.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_plus,"\n\
-plus(x) - Plus corresponds to the unary prefix plus operator in Python,\n\
-but applies the context to the result.\n\
+PyDoc_STRVAR(doc_ctx_plus,
+"plus($self, x, /)\n--\n\n\
+Plus corresponds to the unary prefix plus operator in Python, but applies\n\
+the context to the result.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_power,"\n\
-power(x, y) - Compute x**y. If x is negative, then y must be integral.\n\
-The result will be inexact unless y is integral and the result is finite\n\
-and can be expressed exactly in 'precision' digits. In the Python version\n\
-the result is always correctly rounded, in the C version the result is\n\
-almost always correctly rounded.\n\
+PyDoc_STRVAR(doc_ctx_power,
+"power($self, /, a, b, modulo=None)\n--\n\n\
+Compute a**b. If 'a' is negative, then 'b' must be integral. The result\n\
+will be inexact unless 'a' is integral and the result is finite and can\n\
+be expressed exactly in 'precision' digits.  In the Python version the\n\
+result is always correctly rounded, in the C version the result is almost\n\
+always correctly rounded.\n\
 \n\
-power(x, y, m) - Compute (x**y) % m. The following restrictions hold:\n\
+If modulo is given, compute (a**b) % modulo. The following restrictions\n\
+hold:\n\
 \n\
     * all three arguments must be integral\n\
-    * y must be nonnegative\n\
-    * at least one of x or y must be nonzero\n\
-    * m must be nonzero and less than 10**prec in absolute value\n\
+    * 'b' must be nonnegative\n\
+    * at least one of 'a' or 'b' must be nonzero\n\
+    * modulo must be nonzero and less than 10**prec in absolute value\n\
 \n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_quantize,"\n\
-quantize(x, y) - Return a value equal to x (rounded), having the exponent of y.\n\
+PyDoc_STRVAR(doc_ctx_quantize,
+"quantize($self, x, y, /)\n--\n\n\
+Return a value equal to x (rounded), having the exponent of y.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_radix,"\n\
-radix() - Return 10.\n\
+PyDoc_STRVAR(doc_ctx_radix,
+"radix($self, /)\n--\n\n\
+Return 10.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_remainder,"\n\
-remainder(x, y) - Return the remainder from integer division. The sign of\n\
-the result, if non-zero, is the same as that of the original dividend.\n\
+PyDoc_STRVAR(doc_ctx_remainder,
+"remainder($self, x, y, /)\n--\n\n\
+Return the remainder from integer division.  The sign of the result,\n\
+if non-zero, is the same as that of the original dividend.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_remainder_near,"\n\
-remainder_near(x, y) - Return x - y * n, where n is the integer nearest the\n\
-exact value of x / y (if the result is 0 then its sign will be the sign of x).\n\
+PyDoc_STRVAR(doc_ctx_remainder_near,
+"remainder_near($self, x, y, /)\n--\n\n\
+Return x - y * n, where n is the integer nearest the exact value of x / y\n\
+(if the result is 0 then its sign will be the sign of x).\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_rotate,"\n\
-rotate(x, y) - Return a copy of x, rotated by y places.\n\
+PyDoc_STRVAR(doc_ctx_rotate,
+"rotate($self, x, y, /)\n--\n\n\
+Return a copy of x, rotated by y places.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_same_quantum,"\n\
-same_quantum(x, y) - Return True if the two operands have the same exponent.\n\
+PyDoc_STRVAR(doc_ctx_same_quantum,
+"same_quantum($self, x, y, /)\n--\n\n\
+Return True if the two operands have the same exponent.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_scaleb,"\n\
-scaleb(x, y) - Return the first operand after adding the second value\n\
-to its exp.\n\
+PyDoc_STRVAR(doc_ctx_scaleb,
+"scaleb($self, x, y, /)\n--\n\n\
+Return the first operand after adding the second value to its exp.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_shift,"\n\
-shift(x, y) - Return a copy of x, shifted by y places.\n\
+PyDoc_STRVAR(doc_ctx_shift,
+"shift($self, x, y, /)\n--\n\n\
+Return a copy of x, shifted by y places.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_sqrt,"\n\
-sqrt(x) - Square root of a non-negative number to context precision.\n\
+PyDoc_STRVAR(doc_ctx_sqrt,
+"sqrt($self, x, /)\n--\n\n\
+Square root of a non-negative number to context precision.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_subtract,"\n\
-subtract(x, y) - Return the difference between x and y.\n\
+PyDoc_STRVAR(doc_ctx_subtract,
+"subtract($self, x, y, /)\n--\n\n\
+Return the difference between x and y.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_to_eng_string,"\n\
-to_eng_string(x) - Convert a number to a string, using engineering notation.\n\
+PyDoc_STRVAR(doc_ctx_to_eng_string,
+"to_eng_string($self, x, /)\n--\n\n\
+Convert a number to a string, using engineering notation.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_to_integral,"\n\
-to_integral(x) - Identical to to_integral_value(x).\n\
+PyDoc_STRVAR(doc_ctx_to_integral,
+"to_integral($self, x, /)\n--\n\n\
+Identical to to_integral_value(x).\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_to_integral_exact,"\n\
-to_integral_exact(x) - Round to an integer. Signal if the result is\n\
-rounded or inexact.\n\
+PyDoc_STRVAR(doc_ctx_to_integral_exact,
+"to_integral_exact($self, x, /)\n--\n\n\
+Round to an integer. Signal if the result is rounded or inexact.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_to_integral_value,"\n\
-to_integral_value(x) - Round to an integer.\n\
+PyDoc_STRVAR(doc_ctx_to_integral_value,
+"to_integral_value($self, x, /)\n--\n\n\
+Round to an integer.\n\
 \n");
 
-PyDoc_STRVAR(doc_ctx_to_sci_string,"\n\
-to_sci_string(x) - Convert a number to a string using scientific notation.\n\
+PyDoc_STRVAR(doc_ctx_to_sci_string,
+"to_sci_string($self, x, /)\n--\n\n\
+Convert a number to a string using scientific notation.\n\
 \n");
 
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu May  1 17:43:10 2014
From: python-checkins at python.org (stefan.krah)
Date: Thu,  1 May 2014 17:43:10 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_typo_and_tests_without?=
	=?utf-8?q?_docstrings=2E?=
Message-ID: <3gKLTV1Fqjz7LjV@mail.python.org>

http://hg.python.org/cpython/rev/d8b527c7785f
changeset:   90534:d8b527c7785f
user:        Stefan Krah <skrah at bytereef.org>
date:        Thu May 01 17:42:33 2014 +0200
summary:
  Fix typo and tests without docstrings.

files:
  Lib/test/test_decimal.py |  5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py
--- a/Lib/test/test_decimal.py
+++ b/Lib/test/test_decimal.py
@@ -33,7 +33,7 @@
 import numbers
 import locale
 from test.support import (run_unittest, run_doctest, is_resource_enabled,
-                          requires_IEEE_754)
+                          requires_IEEE_754, requires_docstrings)
 from test.support import (check_warnings, import_fresh_module, TestFailed,
                           run_with_locale, cpython_only)
 import random
@@ -5391,7 +5391,8 @@
             y = Decimal(10**(9*25)).__sizeof__()
             self.assertEqual(y, x+4)
 
-unittest.skipUnless(C, "test requires C version")
+ at requires_docstrings
+ at unittest.skipUnless(C, "test requires C version")
 class SignatureTest(unittest.TestCase):
     """Function signatures"""
 

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Fri May  2 09:59:35 2014
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Fri, 02 May 2014 09:59:35 +0200
Subject: [Python-checkins] Daily reference leaks (d8b527c7785f): sum=7
Message-ID: <E1Wg8Nb-00005j-7U@vds2544.sivit.org>

results for d8b527c7785f on branch "default"
--------------------------------------------

test_asyncio leaked [0, 0, 4] memory blocks, sum=4
test_functools leaked [0, 0, 3] memory blocks, sum=3


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogzjeCvR', '-x']

From python-checkins at python.org  Fri May  2 14:34:53 2014
From: python-checkins at python.org (stefan.krah)
Date: Fri,  2 May 2014 14:34:53 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Use_=24type_instead_of_=24?=
 =?utf-8?q?cls_in_the_signature_specification=2E?=
Message-ID: <3gKtFn3vMvz7Ljp@mail.python.org>

http://hg.python.org/cpython/rev/663b54eb0184
changeset:   90535:663b54eb0184
user:        Stefan Krah <skrah at bytereef.org>
date:        Fri May 02 14:34:11 2014 +0200
summary:
  Use $type instead of $cls in the signature specification.

files:
  Modules/_decimal/docstrings.h |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Modules/_decimal/docstrings.h b/Modules/_decimal/docstrings.h
--- a/Modules/_decimal/docstrings.h
+++ b/Modules/_decimal/docstrings.h
@@ -164,7 +164,7 @@
 \n");
 
 PyDoc_STRVAR(doc_from_float,
-"from_float($cls, f, /)\n--\n\n\
+"from_float($type, f, /)\n--\n\n\
 Class method that converts a float to a decimal number, exactly.\n\
 Since 0.1 is not exactly representable in binary floating point,\n\
 Decimal.from_float(0.1) is not the same as Decimal('0.1').\n\

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  2 17:52:43 2014
From: python-checkins at python.org (zach.ware)
Date: Fri,  2 May 2014 17:52:43 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE4NjA0?=
 =?utf-8?q?=3A_Consolidated_checks_for_GUI_availability=2E?=
Message-ID: <3gKyf36mLBz7LjM@mail.python.org>

http://hg.python.org/cpython/rev/5f75eadecff1
changeset:   90536:5f75eadecff1
branch:      2.7
parent:      90517:0f6bdc2b0e38
user:        Zachary Ware <zachary.ware at gmail.com>
date:        Fri May 02 10:33:49 2014 -0500
summary:
  Issue #18604: Consolidated checks for GUI availability.

test_support._is_gui_available is now defined the same way on every
platform, and now includes the Windows-specific check that had been in the
Windows version of _is_gui_available and the OSX-specific check that was
in runtktests.check_tk_availability.  Also, every platform checks whether
Tk can be instantiated (if the platform-specific checks passed).

files:
  Lib/lib-tk/test/runtktests.py |  45 +---------------
  Lib/test/test_idle.py         |  14 +----
  Lib/test/test_support.py      |  67 ++++++++++++++++++----
  Lib/test/test_tk.py           |   6 +-
  Lib/test/test_ttk_guionly.py  |   6 +-
  Misc/NEWS                     |   4 +
  6 files changed, 65 insertions(+), 77 deletions(-)


diff --git a/Lib/lib-tk/test/runtktests.py b/Lib/lib-tk/test/runtktests.py
--- a/Lib/lib-tk/test/runtktests.py
+++ b/Lib/lib-tk/test/runtktests.py
@@ -14,49 +14,6 @@
 
 this_dir_path = os.path.abspath(os.path.dirname(__file__))
 
-_tk_unavailable = None
-
-def check_tk_availability():
-    """Check that Tk is installed and available."""
-    global _tk_unavailable
-
-    if _tk_unavailable is None:
-        _tk_unavailable = False
-        if sys.platform == 'darwin':
-            # The Aqua Tk implementations on OS X can abort the process if
-            # being called in an environment where a window server connection
-            # cannot be made, for instance when invoked by a buildbot or ssh
-            # process not running under the same user id as the current console
-            # user.  To avoid that, raise an exception if the window manager
-            # connection is not available.
-            from ctypes import cdll, c_int, pointer, Structure
-            from ctypes.util import find_library
-
-            app_services = cdll.LoadLibrary(find_library("ApplicationServices"))
-
-            if app_services.CGMainDisplayID() == 0:
-                _tk_unavailable = "cannot run without OS X window manager"
-            else:
-                class ProcessSerialNumber(Structure):
-                    _fields_ = [("highLongOfPSN", c_int),
-                                ("lowLongOfPSN", c_int)]
-                psn = ProcessSerialNumber()
-                psn_p = pointer(psn)
-                if (  (app_services.GetCurrentProcess(psn_p) < 0) or
-                      (app_services.SetFrontProcess(psn_p) < 0) ):
-                    _tk_unavailable = "cannot run without OS X gui process"
-        else:   # not OS X
-            import Tkinter
-            try:
-                Tkinter.Button()
-            except Tkinter.TclError as msg:
-                # assuming tk is not available
-                _tk_unavailable = "tk not available: %s" % msg
-
-    if _tk_unavailable:
-        raise unittest.SkipTest(_tk_unavailable)
-    return
-
 def is_package(path):
     for name in os.listdir(path):
         if name in ('__init__.py', '__init__.pyc', '__init.pyo'):
@@ -68,7 +25,7 @@
     and are inside packages found in the path starting at basepath.
 
     If packages is specified it should contain package names that want
-    their tests colleted.
+    their tests collected.
     """
     py_ext = '.py'
 
diff --git a/Lib/test/test_idle.py b/Lib/test/test_idle.py
--- a/Lib/test/test_idle.py
+++ b/Lib/test/test_idle.py
@@ -1,24 +1,12 @@
 import unittest
 from test import test_support as support
-from test.test_support import import_module, use_resources
+from test.test_support import import_module
 
 # Skip test if _thread or _tkinter wasn't built or idlelib was deleted.
 import_module('threading')  # imported by idlelib.PyShell, imports _thread
 tk = import_module('Tkinter')  # imports _tkinter
 idletest = import_module('idlelib.idle_test')
 
-# If buildbot improperly sets gui resource (#18365, #18441), remove it
-# so requires('gui') tests are skipped while non-gui tests still run.
-# If there is a problem with Macs, see #18441, msg 193805
-if use_resources and 'gui' in use_resources:
-    try:
-        root = tk.Tk()
-        root.destroy()
-        del root
-    except tk.TclError:
-        while 'gui' in use_resources:
-            use_resources.remove('gui')
-
 # Without test_main present, regrtest.runtest_inner (line1219) calls
 # unittest.TestLoader().loadTestsFromModule(this_module) which calls
 # load_tests() if it finds it. (Unittest.main does the same.)
diff --git a/Lib/test/test_support.py b/Lib/test/test_support.py
--- a/Lib/test/test_support.py
+++ b/Lib/test/test_support.py
@@ -270,12 +270,16 @@
         # is exited) but there is a .pyo file.
         unlink(os.path.join(dirname, modname + os.extsep + 'pyo'))
 
-# On some platforms, should not run gui test even if it is allowed
-# in `use_resources'.
-if sys.platform.startswith('win'):
-    import ctypes
-    import ctypes.wintypes
-    def _is_gui_available():
+# Check whether a gui is actually available
+def _is_gui_available():
+    if hasattr(_is_gui_available, 'result'):
+        return _is_gui_available.result
+    reason = None
+    if sys.platform.startswith('win'):
+        # if Python is running as a service (such as the buildbot service),
+        # gui interaction may be disallowed
+        import ctypes
+        import ctypes.wintypes
         UOI_FLAGS = 1
         WSF_VISIBLE = 0x0001
         class USEROBJECTFLAGS(ctypes.Structure):
@@ -295,10 +299,49 @@
             ctypes.byref(needed))
         if not res:
             raise ctypes.WinError()
-        return bool(uof.dwFlags & WSF_VISIBLE)
-else:
-    def _is_gui_available():
-        return True
+        if not bool(uof.dwFlags & WSF_VISIBLE):
+            reason = "gui not available (WSF_VISIBLE flag not set)"
+    elif sys.platform == 'darwin':
+        # The Aqua Tk implementations on OS X can abort the process if
+        # being called in an environment where a window server connection
+        # cannot be made, for instance when invoked by a buildbot or ssh
+        # process not running under the same user id as the current console
+        # user.  To avoid that, raise an exception if the window manager
+        # connection is not available.
+        from ctypes import cdll, c_int, pointer, Structure
+        from ctypes.util import find_library
+
+        app_services = cdll.LoadLibrary(find_library("ApplicationServices"))
+
+        if app_services.CGMainDisplayID() == 0:
+            reason = "gui tests cannot run without OS X window manager"
+        else:
+            class ProcessSerialNumber(Structure):
+                _fields_ = [("highLongOfPSN", c_int),
+                            ("lowLongOfPSN", c_int)]
+            psn = ProcessSerialNumber()
+            psn_p = pointer(psn)
+            if (  (app_services.GetCurrentProcess(psn_p) < 0) or
+                  (app_services.SetFrontProcess(psn_p) < 0) ):
+                reason = "cannot run without OS X gui process"
+
+    # check on every platform whether tkinter can actually do anything
+    if not reason:
+        try:
+            from Tkinter import Tk
+            root = Tk()
+            root.destroy()
+        except Exception as e:
+            err_string = str(e)
+            if len(err_string) > 50:
+                err_string = err_string[:50] + ' [...]'
+            reason = 'Tk unavailable due to {}: {}'.format(type(e).__name__,
+                                                           err_string)
+
+    _is_gui_available.reason = reason
+    _is_gui_available.result = not reason
+
+    return _is_gui_available.result
 
 def is_resource_enabled(resource):
     """Test whether a resource is enabled.  Known resources are set by
@@ -311,7 +354,7 @@
     If the caller's module is __main__ then automatically return True.  The
     possibility of False being returned occurs when regrtest.py is executing."""
     if resource == 'gui' and not _is_gui_available():
-        raise unittest.SkipTest("Cannot use the 'gui' resource")
+        raise ResourceDenied(_is_gui_available.reason)
     # see if the caller's module is __main__ - if so, treat as if
     # the resource was set
     if sys._getframe(1).f_globals.get("__name__") == "__main__":
@@ -1213,7 +1256,7 @@
 
 def requires_resource(resource):
     if resource == 'gui' and not _is_gui_available():
-        return unittest.skip("resource 'gui' is not available")
+        return unittest.skip(_is_gui_available.reason)
     if is_resource_enabled(resource):
         return _id
     else:
diff --git a/Lib/test/test_tk.py b/Lib/test/test_tk.py
--- a/Lib/test/test_tk.py
+++ b/Lib/test/test_tk.py
@@ -1,8 +1,9 @@
 import os
 from test import test_support
 
-# Skip test if _tkinter wasn't built.
+# Skip test if _tkinter wasn't built or gui resource is not available.
 test_support.import_module('_tkinter')
+test_support.requires('gui')
 
 this_dir = os.path.dirname(os.path.abspath(__file__))
 lib_tk_test = os.path.abspath(os.path.join(this_dir, os.path.pardir,
@@ -11,9 +12,6 @@
 with test_support.DirsOnSysPath(lib_tk_test):
     import runtktests
 
-# Skip test if tk cannot be initialized.
-runtktests.check_tk_availability()
-
 def test_main(enable_gui=False):
     if enable_gui:
         if test_support.use_resources is None:
diff --git a/Lib/test/test_ttk_guionly.py b/Lib/test/test_ttk_guionly.py
--- a/Lib/test/test_ttk_guionly.py
+++ b/Lib/test/test_ttk_guionly.py
@@ -2,8 +2,9 @@
 import unittest
 from test import test_support
 
-# Skip this test if _tkinter wasn't built.
+# Skip this test if _tkinter wasn't built or gui resource is not available.
 test_support.import_module('_tkinter')
+test_support.requires('gui')
 
 this_dir = os.path.dirname(os.path.abspath(__file__))
 lib_tk_test = os.path.abspath(os.path.join(this_dir, os.path.pardir,
@@ -12,9 +13,6 @@
 with test_support.DirsOnSysPath(lib_tk_test):
     import runtktests
 
-# Skip test if tk cannot be initialized.
-runtktests.check_tk_availability()
-
 import ttk
 from _tkinter import TclError
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -329,6 +329,10 @@
 Tests
 -----
 
+- Issue #18604: Consolidated checks for GUI availability.  All platforms now
+  at least check whether Tk can be instantiated when the GUI resource is
+  requested.
+
 - Issue #20946: Correct alignment assumptions of some ctypes tests.
 
 - Issue #20743: Fix a reference leak in test_tcl.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  2 17:52:45 2014
From: python-checkins at python.org (zach.ware)
Date: Fri,  2 May 2014 17:52:45 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE4NjA0?=
 =?utf-8?q?=3A_Consolidated_checks_for_GUI_availability=2E?=
Message-ID: <3gKyf52NlKz7LjM@mail.python.org>

http://hg.python.org/cpython/rev/eb361f69ddd1
changeset:   90537:eb361f69ddd1
branch:      3.4
parent:      90531:94f4cad7d541
user:        Zachary Ware <zachary.ware at gmail.com>
date:        Fri May 02 10:51:07 2014 -0500
summary:
  Issue #18604: Consolidated checks for GUI availability.

test_support._is_gui_available is now defined the same way on every
platform, and now includes the Windows-specific check that had been in the
Windows version of _is_gui_available and the OSX-specific check that was
in tkinter.test.support.check_tk_availability.  Also, every platform
checks whether Tk can be instantiated (if the platform-specific checks
passed).

files:
  Lib/test/support/__init__.py |  67 +++++++++++++++++++----
  Lib/test/test_idle.py        |  14 +----
  Lib/test/test_tk.py          |   3 +-
  Lib/test/test_ttk_guionly.py |   3 +-
  Lib/tkinter/test/support.py  |  46 +---------------
  Misc/NEWS                    |   4 +
  6 files changed, 64 insertions(+), 73 deletions(-)


diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py
--- a/Lib/test/support/__init__.py
+++ b/Lib/test/support/__init__.py
@@ -378,12 +378,16 @@
         unlink(importlib.util.cache_from_source(source, debug_override=True))
         unlink(importlib.util.cache_from_source(source, debug_override=False))
 
-# On some platforms, should not run gui test even if it is allowed
-# in `use_resources'.
-if sys.platform.startswith('win'):
-    import ctypes
-    import ctypes.wintypes
-    def _is_gui_available():
+# Check whether a gui is actually available
+def _is_gui_available():
+    if hasattr(_is_gui_available, 'result'):
+        return _is_gui_available.result
+    reason = None
+    if sys.platform.startswith('win'):
+        # if Python is running as a service (such as the buildbot service),
+        # gui interaction may be disallowed
+        import ctypes
+        import ctypes.wintypes
         UOI_FLAGS = 1
         WSF_VISIBLE = 0x0001
         class USEROBJECTFLAGS(ctypes.Structure):
@@ -403,10 +407,49 @@
             ctypes.byref(needed))
         if not res:
             raise ctypes.WinError()
-        return bool(uof.dwFlags & WSF_VISIBLE)
-else:
-    def _is_gui_available():
-        return True
+        if not bool(uof.dwFlags & WSF_VISIBLE):
+            reason = "gui not available (WSF_VISIBLE flag not set)"
+    elif sys.platform == 'darwin':
+        # The Aqua Tk implementations on OS X can abort the process if
+        # being called in an environment where a window server connection
+        # cannot be made, for instance when invoked by a buildbot or ssh
+        # process not running under the same user id as the current console
+        # user.  To avoid that, raise an exception if the window manager
+        # connection is not available.
+        from ctypes import cdll, c_int, pointer, Structure
+        from ctypes.util import find_library
+
+        app_services = cdll.LoadLibrary(find_library("ApplicationServices"))
+
+        if app_services.CGMainDisplayID() == 0:
+            reason = "gui tests cannot run without OS X window manager"
+        else:
+            class ProcessSerialNumber(Structure):
+                _fields_ = [("highLongOfPSN", c_int),
+                            ("lowLongOfPSN", c_int)]
+            psn = ProcessSerialNumber()
+            psn_p = pointer(psn)
+            if (  (app_services.GetCurrentProcess(psn_p) < 0) or
+                  (app_services.SetFrontProcess(psn_p) < 0) ):
+                reason = "cannot run without OS X gui process"
+
+    # check on every platform whether tkinter can actually do anything
+    if not reason:
+        try:
+            from tkinter import Tk
+            root = Tk()
+            root.destroy()
+        except Exception as e:
+            err_string = str(e)
+            if len(err_string) > 50:
+                err_string = err_string[:50] + ' [...]'
+            reason = 'Tk unavailable due to {}: {}'.format(type(e).__name__,
+                                                           err_string)
+
+    _is_gui_available.reason = reason
+    _is_gui_available.result = not reason
+
+    return _is_gui_available.result
 
 def is_resource_enabled(resource):
     """Test whether a resource is enabled.  Known resources are set by
@@ -421,7 +464,7 @@
     executing.
     """
     if resource == 'gui' and not _is_gui_available():
-        raise unittest.SkipTest("Cannot use the 'gui' resource")
+        raise ResourceDenied(_is_gui_available.reason)
     # see if the caller's module is __main__ - if so, treat as if
     # the resource was set
     if sys._getframe(1).f_globals.get("__name__") == "__main__":
@@ -1589,7 +1632,7 @@
 
 def requires_resource(resource):
     if resource == 'gui' and not _is_gui_available():
-        return unittest.skip("resource 'gui' is not available")
+        return unittest.skip(_is_gui_available.reason)
     if is_resource_enabled(resource):
         return _id
     else:
diff --git a/Lib/test/test_idle.py b/Lib/test/test_idle.py
--- a/Lib/test/test_idle.py
+++ b/Lib/test/test_idle.py
@@ -1,24 +1,12 @@
 import unittest
 from test import support
-from test.support import import_module, use_resources
+from test.support import import_module
 
 # Skip test if _thread or _tkinter wasn't built or idlelib was deleted.
 import_module('threading')  # imported by PyShell, imports _thread
 tk = import_module('tkinter')  # imports _tkinter
 idletest = import_module('idlelib.idle_test')
 
-# If buildbot improperly sets gui resource (#18365, #18441), remove it
-# so requires('gui') tests are skipped while non-gui tests still run.
-# If there is a problem with Macs, see #18441, msg 193805
-if use_resources and 'gui' in use_resources:
-    try:
-        root = tk.Tk()
-        root.destroy()
-        del root
-    except tk.TclError:
-        while 'gui' in use_resources:
-            use_resources.remove('gui')
-
 # Without test_main present, regrtest.runtest_inner (line1219) calls
 # unittest.TestLoader().loadTestsFromModule(this_module) which calls
 # load_tests() if it finds it. (Unittest.main does the same.)
diff --git a/Lib/test/test_tk.py b/Lib/test/test_tk.py
--- a/Lib/test/test_tk.py
+++ b/Lib/test/test_tk.py
@@ -6,8 +6,7 @@
 support.import_fresh_module('tkinter')
 
 # Skip test if tk cannot be initialized.
-from tkinter.test.support import check_tk_availability
-check_tk_availability()
+support.requires('gui')
 
 from tkinter.test import runtktests
 
diff --git a/Lib/test/test_ttk_guionly.py b/Lib/test/test_ttk_guionly.py
--- a/Lib/test/test_ttk_guionly.py
+++ b/Lib/test/test_ttk_guionly.py
@@ -9,8 +9,7 @@
 support.import_fresh_module('tkinter')
 
 # Skip test if tk cannot be initialized.
-from tkinter.test.support import check_tk_availability
-check_tk_availability()
+support.requires('gui')
 
 from _tkinter import TclError
 from tkinter import ttk
diff --git a/Lib/tkinter/test/support.py b/Lib/tkinter/test/support.py
--- a/Lib/tkinter/test/support.py
+++ b/Lib/tkinter/test/support.py
@@ -1,52 +1,10 @@
 import sys
 import tkinter
 import unittest
-
-_tk_unavailable = None
-
-def check_tk_availability():
-    """Check that Tk is installed and available."""
-    global _tk_unavailable
-
-    if _tk_unavailable is None:
-        _tk_unavailable = False
-        if sys.platform == 'darwin':
-            # The Aqua Tk implementations on OS X can abort the process if
-            # being called in an environment where a window server connection
-            # cannot be made, for instance when invoked by a buildbot or ssh
-            # process not running under the same user id as the current console
-            # user.  To avoid that, raise an exception if the window manager
-            # connection is not available.
-            from ctypes import cdll, c_int, pointer, Structure
-            from ctypes.util import find_library
-
-            app_services = cdll.LoadLibrary(find_library("ApplicationServices"))
-
-            if app_services.CGMainDisplayID() == 0:
-                _tk_unavailable = "cannot run without OS X window manager"
-            else:
-                class ProcessSerialNumber(Structure):
-                    _fields_ = [("highLongOfPSN", c_int),
-                                ("lowLongOfPSN", c_int)]
-                psn = ProcessSerialNumber()
-                psn_p = pointer(psn)
-                if (  (app_services.GetCurrentProcess(psn_p) < 0) or
-                      (app_services.SetFrontProcess(psn_p) < 0) ):
-                    _tk_unavailable = "cannot run without OS X gui process"
-        else:   # not OS X
-            import tkinter
-            try:
-                tkinter.Button()
-            except tkinter.TclError as msg:
-                # assuming tk is not available
-                _tk_unavailable = "tk not available: %s" % msg
-
-    if _tk_unavailable:
-        raise unittest.SkipTest(_tk_unavailable)
-    return
+from test.support import requires
 
 def get_tk_root():
-    check_tk_availability()     # raise exception if tk unavailable
+    requires('gui')             # raise exception if tk unavailable
     try:
         root = tkinter._default_root
     except AttributeError:
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -259,6 +259,10 @@
 Tests
 -----
 
+- Issue #18604: Consolidated checks for GUI availability.  All platforms now
+  at least check whether Tk can be instantiated when the GUI resource is
+  requested.
+
 - Issue #21275: Fix a socket test on KFreeBSD.
 
 - Issue #21223: Pass test_site/test_startup_imports when some of the extensions

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  2 17:52:46 2014
From: python-checkins at python.org (zach.ware)
Date: Fri,  2 May 2014 17:52:46 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Closes_=2318604=3A_Merge_with_3=2E4?=
Message-ID: <3gKyf64yW0z7LkV@mail.python.org>

http://hg.python.org/cpython/rev/82caec3865e3
changeset:   90538:82caec3865e3
parent:      90535:663b54eb0184
parent:      90537:eb361f69ddd1
user:        Zachary Ware <zachary.ware at gmail.com>
date:        Fri May 02 10:52:12 2014 -0500
summary:
  Closes #18604: Merge with 3.4

files:
  Lib/test/support/__init__.py |  67 +++++++++++++++++++----
  Lib/test/test_idle.py        |  14 +----
  Lib/test/test_tk.py          |   3 +-
  Lib/test/test_ttk_guionly.py |   3 +-
  Lib/tkinter/test/support.py  |  46 +---------------
  Misc/NEWS                    |   4 +
  6 files changed, 64 insertions(+), 73 deletions(-)


diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py
--- a/Lib/test/support/__init__.py
+++ b/Lib/test/support/__init__.py
@@ -378,12 +378,16 @@
         unlink(importlib.util.cache_from_source(source, debug_override=True))
         unlink(importlib.util.cache_from_source(source, debug_override=False))
 
-# On some platforms, should not run gui test even if it is allowed
-# in `use_resources'.
-if sys.platform.startswith('win'):
-    import ctypes
-    import ctypes.wintypes
-    def _is_gui_available():
+# Check whether a gui is actually available
+def _is_gui_available():
+    if hasattr(_is_gui_available, 'result'):
+        return _is_gui_available.result
+    reason = None
+    if sys.platform.startswith('win'):
+        # if Python is running as a service (such as the buildbot service),
+        # gui interaction may be disallowed
+        import ctypes
+        import ctypes.wintypes
         UOI_FLAGS = 1
         WSF_VISIBLE = 0x0001
         class USEROBJECTFLAGS(ctypes.Structure):
@@ -403,10 +407,49 @@
             ctypes.byref(needed))
         if not res:
             raise ctypes.WinError()
-        return bool(uof.dwFlags & WSF_VISIBLE)
-else:
-    def _is_gui_available():
-        return True
+        if not bool(uof.dwFlags & WSF_VISIBLE):
+            reason = "gui not available (WSF_VISIBLE flag not set)"
+    elif sys.platform == 'darwin':
+        # The Aqua Tk implementations on OS X can abort the process if
+        # being called in an environment where a window server connection
+        # cannot be made, for instance when invoked by a buildbot or ssh
+        # process not running under the same user id as the current console
+        # user.  To avoid that, raise an exception if the window manager
+        # connection is not available.
+        from ctypes import cdll, c_int, pointer, Structure
+        from ctypes.util import find_library
+
+        app_services = cdll.LoadLibrary(find_library("ApplicationServices"))
+
+        if app_services.CGMainDisplayID() == 0:
+            reason = "gui tests cannot run without OS X window manager"
+        else:
+            class ProcessSerialNumber(Structure):
+                _fields_ = [("highLongOfPSN", c_int),
+                            ("lowLongOfPSN", c_int)]
+            psn = ProcessSerialNumber()
+            psn_p = pointer(psn)
+            if (  (app_services.GetCurrentProcess(psn_p) < 0) or
+                  (app_services.SetFrontProcess(psn_p) < 0) ):
+                reason = "cannot run without OS X gui process"
+
+    # check on every platform whether tkinter can actually do anything
+    if not reason:
+        try:
+            from tkinter import Tk
+            root = Tk()
+            root.destroy()
+        except Exception as e:
+            err_string = str(e)
+            if len(err_string) > 50:
+                err_string = err_string[:50] + ' [...]'
+            reason = 'Tk unavailable due to {}: {}'.format(type(e).__name__,
+                                                           err_string)
+
+    _is_gui_available.reason = reason
+    _is_gui_available.result = not reason
+
+    return _is_gui_available.result
 
 def is_resource_enabled(resource):
     """Test whether a resource is enabled.  Known resources are set by
@@ -421,7 +464,7 @@
     executing.
     """
     if resource == 'gui' and not _is_gui_available():
-        raise unittest.SkipTest("Cannot use the 'gui' resource")
+        raise ResourceDenied(_is_gui_available.reason)
     # see if the caller's module is __main__ - if so, treat as if
     # the resource was set
     if sys._getframe(1).f_globals.get("__name__") == "__main__":
@@ -1589,7 +1632,7 @@
 
 def requires_resource(resource):
     if resource == 'gui' and not _is_gui_available():
-        return unittest.skip("resource 'gui' is not available")
+        return unittest.skip(_is_gui_available.reason)
     if is_resource_enabled(resource):
         return _id
     else:
diff --git a/Lib/test/test_idle.py b/Lib/test/test_idle.py
--- a/Lib/test/test_idle.py
+++ b/Lib/test/test_idle.py
@@ -1,24 +1,12 @@
 import unittest
 from test import support
-from test.support import import_module, use_resources
+from test.support import import_module
 
 # Skip test if _thread or _tkinter wasn't built or idlelib was deleted.
 import_module('threading')  # imported by PyShell, imports _thread
 tk = import_module('tkinter')  # imports _tkinter
 idletest = import_module('idlelib.idle_test')
 
-# If buildbot improperly sets gui resource (#18365, #18441), remove it
-# so requires('gui') tests are skipped while non-gui tests still run.
-# If there is a problem with Macs, see #18441, msg 193805
-if use_resources and 'gui' in use_resources:
-    try:
-        root = tk.Tk()
-        root.destroy()
-        del root
-    except tk.TclError:
-        while 'gui' in use_resources:
-            use_resources.remove('gui')
-
 # Without test_main present, regrtest.runtest_inner (line1219) calls
 # unittest.TestLoader().loadTestsFromModule(this_module) which calls
 # load_tests() if it finds it. (Unittest.main does the same.)
diff --git a/Lib/test/test_tk.py b/Lib/test/test_tk.py
--- a/Lib/test/test_tk.py
+++ b/Lib/test/test_tk.py
@@ -6,8 +6,7 @@
 support.import_fresh_module('tkinter')
 
 # Skip test if tk cannot be initialized.
-from tkinter.test.support import check_tk_availability
-check_tk_availability()
+support.requires('gui')
 
 from tkinter.test import runtktests
 
diff --git a/Lib/test/test_ttk_guionly.py b/Lib/test/test_ttk_guionly.py
--- a/Lib/test/test_ttk_guionly.py
+++ b/Lib/test/test_ttk_guionly.py
@@ -9,8 +9,7 @@
 support.import_fresh_module('tkinter')
 
 # Skip test if tk cannot be initialized.
-from tkinter.test.support import check_tk_availability
-check_tk_availability()
+support.requires('gui')
 
 from _tkinter import TclError
 from tkinter import ttk
diff --git a/Lib/tkinter/test/support.py b/Lib/tkinter/test/support.py
--- a/Lib/tkinter/test/support.py
+++ b/Lib/tkinter/test/support.py
@@ -1,52 +1,10 @@
 import sys
 import tkinter
 import unittest
-
-_tk_unavailable = None
-
-def check_tk_availability():
-    """Check that Tk is installed and available."""
-    global _tk_unavailable
-
-    if _tk_unavailable is None:
-        _tk_unavailable = False
-        if sys.platform == 'darwin':
-            # The Aqua Tk implementations on OS X can abort the process if
-            # being called in an environment where a window server connection
-            # cannot be made, for instance when invoked by a buildbot or ssh
-            # process not running under the same user id as the current console
-            # user.  To avoid that, raise an exception if the window manager
-            # connection is not available.
-            from ctypes import cdll, c_int, pointer, Structure
-            from ctypes.util import find_library
-
-            app_services = cdll.LoadLibrary(find_library("ApplicationServices"))
-
-            if app_services.CGMainDisplayID() == 0:
-                _tk_unavailable = "cannot run without OS X window manager"
-            else:
-                class ProcessSerialNumber(Structure):
-                    _fields_ = [("highLongOfPSN", c_int),
-                                ("lowLongOfPSN", c_int)]
-                psn = ProcessSerialNumber()
-                psn_p = pointer(psn)
-                if (  (app_services.GetCurrentProcess(psn_p) < 0) or
-                      (app_services.SetFrontProcess(psn_p) < 0) ):
-                    _tk_unavailable = "cannot run without OS X gui process"
-        else:   # not OS X
-            import tkinter
-            try:
-                tkinter.Button()
-            except tkinter.TclError as msg:
-                # assuming tk is not available
-                _tk_unavailable = "tk not available: %s" % msg
-
-    if _tk_unavailable:
-        raise unittest.SkipTest(_tk_unavailable)
-    return
+from test.support import requires
 
 def get_tk_root():
-    check_tk_availability()     # raise exception if tk unavailable
+    requires('gui')             # raise exception if tk unavailable
     try:
         root = tkinter._default_root
     except AttributeError:
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -383,6 +383,10 @@
 Tests
 -----
 
+- Issue #18604: Consolidated checks for GUI availability.  All platforms now
+  at least check whether Tk can be instantiated when the GUI resource is
+  requested.
+
 - Issue #21275: Fix a socket test on KFreeBSD.
 
 - Issue #21223: Pass test_site/test_startup_imports when some of the extensions

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  2 18:13:05 2014
From: python-checkins at python.org (zach.ware)
Date: Fri,  2 May 2014 18:13:05 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?devguide=3A_Add_myself_to_the_develop?=
 =?utf-8?q?er_log_with_what_I_believe_to_be_correct_information?=
Message-ID: <3gKz5Y6Kclz7Ljp@mail.python.org>

http://hg.python.org/devguide/rev/a957ee1372d0
changeset:   692:a957ee1372d0
user:        Zachary Ware <zachary.ware at gmail.com>
date:        Fri May 02 11:08:59 2014 -0500
summary:
  Add myself to the developer log with what I believe to be correct information

files:
  developers.rst |  3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)


diff --git a/developers.rst b/developers.rst
--- a/developers.rst
+++ b/developers.rst
@@ -34,6 +34,9 @@
 - Yury Selivanov was given push privileges on Jan 23 2014 by GFB, for "inspect"
   module and general contributions, on recommendation by Nick Coghlan.
 
+- Zachary Ware was given push privileges on Nov 02 2013 by BAC, on the
+  recommendation of Brian Curtin.
+
 - Donald Stufft was given push privileges on Aug 14 2013 by BAC, for PEP
   editing, on the recommendation of Nick Coghlan.
 

-- 
Repository URL: http://hg.python.org/devguide

From python-checkins at python.org  Fri May  2 18:13:07 2014
From: python-checkins at python.org (zach.ware)
Date: Fri,  2 May 2014 18:13:07 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?devguide=3A_Add_myself_as_a_Windows_?=
	=?utf-8?q?=22expert=22?=
Message-ID: <3gKz5b0td3z7Lln@mail.python.org>

http://hg.python.org/devguide/rev/46f9c5adc151
changeset:   693:46f9c5adc151
user:        Zachary Ware <zachary.ware at gmail.com>
date:        Fri May 02 11:12:42 2014 -0500
summary:
  Add myself as a Windows "expert"

files:
  experts.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/experts.rst b/experts.rst
--- a/experts.rst
+++ b/experts.rst
@@ -289,7 +289,7 @@
 NetBSD1
 OS2/EMX               aimacintyre
 Solaris/OpenIndiana   jcea
-Windows               tim.golden
+Windows               tim.golden, zach.ware
 JVM/Java              frank.wierzbicki
 ===================   ===========
 

-- 
Repository URL: http://hg.python.org/devguide

From python-checkins at python.org  Fri May  2 22:02:03 2014
From: python-checkins at python.org (zach.ware)
Date: Fri,  2 May 2014 22:02:03 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?devguide=3A_Remove_=27Distutils2=27_c?=
 =?utf-8?q?omponent=2C_fix_broken_link=2E?=
Message-ID: <3gL49l75J1z7Lll@mail.python.org>

http://hg.python.org/devguide/rev/bbd224986df3
changeset:   694:bbd224986df3
user:        Zachary Ware <zachary.ware at gmail.com>
date:        Fri May 02 11:48:24 2014 -0500
summary:
  Remove 'Distutils2' component, fix broken link.

files:
  triaging.rst |  7 ++-----
  1 files changed, 2 insertions(+), 5 deletions(-)


diff --git a/triaging.rst b/triaging.rst
--- a/triaging.rst
+++ b/triaging.rst
@@ -86,8 +86,6 @@
     The `Developer's guide`_.
 Distutils
     The distutils package in `Lib/distutils`_.
-Distutils2
-    The packaging module in `Lib/packaging`_.
 Documentation
     The documentation in Doc_ (used to build the HTML doc at http://docs.python.org/).
 email
@@ -114,7 +112,7 @@
     `Lib/doctest.py`_.
 
     The CPython tests in `Lib/test`_, the test runner in `Lib/test/regrtest.py`_
-    and the `Lib/test/support.py`_ module.
+    and the `Lib/test/support`_ package.
 Tkinter
     The `Lib/tkinter`_ package.
 Unicode
@@ -303,11 +301,10 @@
 .. _Lib/doctest.py: http://hg.python.org/cpython/file/default/Lib/doctest.py
 .. _Lib/idlelib: http://hg.python.org/cpython/file/default/Lib/idlelib/
 .. _Lib/io.py: http://hg.python.org/cpython/file/default/Lib/io.py
-.. _Lib/packaging: http://hg.python.org/cpython/file/default/Lib/packaging/
 .. _Lib/re.py: http://hg.python.org/cpython/file/default/Lib/re.py
 .. _Lib/test: http://hg.python.org/cpython/file/default/Lib/test/
 .. _Lib/test/regrtest.py: http://hg.python.org/cpython/file/default/Lib/test/regrtest.py
-.. _Lib/test/support.py: http://hg.python.org/cpython/file/default/Lib/test/support.py
+.. _Lib/test/support: http://hg.python.org/cpython/file/default/Lib/test/support/
 .. _Lib/tkinter: http://hg.python.org/cpython/file/default/Lib/tkinter/
 .. _Lib/unittest: http://hg.python.org/cpython/file/default/Lib/unittest/
 .. _Lib/xml: http://hg.python.org/cpython/file/default/Lib/xml/

-- 
Repository URL: http://hg.python.org/devguide

From python-checkins at python.org  Fri May  2 22:02:05 2014
From: python-checkins at python.org (zach.ware)
Date: Fri,  2 May 2014 22:02:05 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?devguide=3A_Fix_broken_link?=
Message-ID: <3gL49n1TpKz7Llh@mail.python.org>

http://hg.python.org/devguide/rev/8453b54f88ed
changeset:   695:8453b54f88ed
user:        Zachary Ware <zachary.ware at gmail.com>
date:        Fri May 02 14:36:08 2014 -0500
summary:
  Fix broken link

files:
  buildbots.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/buildbots.rst b/buildbots.rst
--- a/buildbots.rst
+++ b/buildbots.rst
@@ -184,7 +184,7 @@
 work on one or several buildbots.  Since your work is hosted in a distinct
 repository, you can't trigger builds on the regular builders.  Instead,
 you have to use one of the `custom builders
-<http://www.python.org/dev/buildbot/all/waterfall?category=custom.stable&category=custom.unstable>`_.
+<http://buildbot.python.org/all/waterfall?category=custom.stable&category=custom.unstable>`_.
 
 When creating ("forcing") a build on a custom builder, you have to provide
 at least two parameters:

-- 
Repository URL: http://hg.python.org/devguide

From python-checkins at python.org  Fri May  2 22:02:06 2014
From: python-checkins at python.org (zach.ware)
Date: Fri,  2 May 2014 22:02:06 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?devguide=3A_Fix_broken_link_to_Skip?=
 =?utf-8?q?=27s_optimizer_paper=2C_update_bug_link?=
Message-ID: <3gL49p33dHz7Ll6@mail.python.org>

http://hg.python.org/devguide/rev/375b0b0b186b
changeset:   696:375b0b0b186b
user:        Zachary Ware <zachary.ware at gmail.com>
date:        Fri May 02 14:44:20 2014 -0500
summary:
  Fix broken link to Skip's optimizer paper, update bug link

files:
  compiler.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/compiler.rst b/compiler.rst
--- a/compiler.rst
+++ b/compiler.rst
@@ -553,10 +553,10 @@
 .. _SPARK: http://pages.cpsc.ucalgary.ca/~aycock/spark/
 
 .. [#skip-peephole] Skip Montanaro's Peephole Optimizer Paper
-   (http://www.python.org/workshops/1998-11/proceedings/papers/montanaro/montanaro.html)
+   (http://www.smontanaro.net/python/spam7/optimizer.html)
 
 .. [#Bytecodehacks] Bytecodehacks Project
    (http://bytecodehacks.sourceforge.net/bch-docs/bch/index.html)
 
 .. [#CALL_ATTR] CALL_ATTR opcode
-   (http://www.python.org/sf/709744)
+   (http://bugs.python.org/issue709744)

-- 
Repository URL: http://hg.python.org/devguide

From python-checkins at python.org  Fri May  2 22:02:07 2014
From: python-checkins at python.org (zach.ware)
Date: Fri,  2 May 2014 22:02:07 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?devguide=3A_Fix_broken_links?=
Message-ID: <3gL49q4m5tz7LmD@mail.python.org>

http://hg.python.org/devguide/rev/d83d55f2c4f5
changeset:   697:d83d55f2c4f5
user:        Zachary Ware <zachary.ware at gmail.com>
date:        Fri May 02 14:58:53 2014 -0500
summary:
  Fix broken links

files:
  faq.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/faq.rst b/faq.rst
--- a/faq.rst
+++ b/faq.rst
@@ -56,7 +56,7 @@
 
 .. _python-ideas: http://mail.python.org/mailman/listinfo/python-ideas
 .. _issue tracker: http://bugs.python.org
-.. _PEP Index: http://www.python.org/dev/peps
+.. _PEP Index: http://www.python.org/dev/peps/
 .. _draft PEP: http://www.python.org/dev/peps/pep-0001/
 .. _Status Quo Wins a Stalemate: http://www.curiousefficiency.org/posts/2011/02/status-quo-wins-stalemate.html
 .. _Justifying Python Language Changes: http://www.curiousefficiency.org/posts/2011/02/justifying-python-language-changes.html
@@ -209,7 +209,7 @@
    Host hg.python.org
      Compression yes
 
-.. _download Mercurial: http://mercurial.selenic.com/downloads/
+.. _download Mercurial: http://mercurial.selenic.com/downloads
 .. _OpenSSH: http://www.openssh.org/
 
 

-- 
Repository URL: http://hg.python.org/devguide

From python-checkins at python.org  Fri May  2 22:07:29 2014
From: python-checkins at python.org (victor.stinner)
Date: Fri,  2 May 2014 22:07:29 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321393=3A_random?=
 =?utf-8?q?=2Ec=3A_on_Windows=2C_close_the_hCryptProv_handle_at_exit?=
Message-ID: <3gL4J15bJGz7Ll5@mail.python.org>

http://hg.python.org/cpython/rev/8704198680ba
changeset:   90539:8704198680ba
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Fri May 02 22:06:44 2014 +0200
summary:
  Issue #21393: random.c: on Windows, close the hCryptProv handle at exit

files:
  Python/random.c |  9 ++++++---
  1 files changed, 6 insertions(+), 3 deletions(-)


diff --git a/Python/random.c b/Python/random.c
--- a/Python/random.c
+++ b/Python/random.c
@@ -15,8 +15,6 @@
 #endif
 
 #ifdef MS_WINDOWS
-/* This handle is never explicitly released. Instead, the operating
-   system will release it when the process terminates. */
 static HCRYPTPROV hCryptProv = 0;
 
 static int
@@ -298,7 +296,12 @@
 void
 _PyRandom_Fini(void)
 {
-#ifndef MS_WINDOWS
+#ifdef MS_WINDOWS
+    if (hCryptProv) {
+        CloseHandle(hCryptProv);
+        hCryptProv = 0;
+    }
+#else
     dev_urandom_close();
 #endif
 }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  2 22:31:30 2014
From: python-checkins at python.org (victor.stinner)
Date: Fri,  2 May 2014 22:31:30 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321233=3A_Add_new_?=
 =?utf-8?q?C_functions=3A_PyMem=5FRawCalloc=28=29=2C_PyMem=5FCalloc=28=29?=
 =?utf-8?q?=2C?=
Message-ID: <3gL4qk3N7GzRwm@mail.python.org>

http://hg.python.org/cpython/rev/5b0fda8f5718
changeset:   90540:5b0fda8f5718
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Fri May 02 22:31:14 2014 +0200
summary:
  Issue #21233: Add new C functions: PyMem_RawCalloc(), PyMem_Calloc(),
PyObject_Calloc(), _PyObject_GC_Calloc(). bytes(int) and bytearray(int) are now
using ``calloc()`` instead of ``malloc()`` for large objects which is faster
and use less memory (until the bytearray buffer is filled with data).

files:
  Doc/c-api/memory.rst      |   36 ++++++-
  Doc/whatsnew/3.5.rst      |   20 +++-
  Include/objimpl.h         |    4 +-
  Include/pymem.h           |    5 +
  Misc/NEWS                 |    6 +
  Modules/_testcapimodule.c |   73 ++++++++++++++-
  Modules/_tracemalloc.c    |   57 +++++++++--
  Modules/gcmodule.c        |   24 ++++-
  Objects/bytearrayobject.c |   16 ++-
  Objects/bytesobject.c     |   76 +++++++++-----
  Objects/obmalloc.c        |  126 ++++++++++++++++++++++---
  11 files changed, 369 insertions(+), 74 deletions(-)


diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst
--- a/Doc/c-api/memory.rst
+++ b/Doc/c-api/memory.rst
@@ -92,8 +92,8 @@
 need to be held.
 
 The default raw memory block allocator uses the following functions:
-:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when
-requesting zero bytes.
+:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`; call
+``malloc(1)`` (or ``calloc(1, 1)``) when requesting zero bytes.
 
 .. versionadded:: 3.4
 
@@ -106,6 +106,17 @@
    been initialized in any way.
 
 
+.. c:function:: void* PyMem_RawCalloc(size_t nelem, size_t elsize)
+
+   Allocates *nelem* elements each whose size in bytes is *elsize* and returns
+   a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the
+   request fails. The memory is initialized to zeros. Requesting zero elements
+   or elements of size zero bytes returns a distinct non-*NULL* pointer if
+   possible, as if ``PyMem_RawCalloc(1, 1)`` had been called instead.
+
+   .. versionadded:: 3.5
+
+
 .. c:function:: void* PyMem_RawRealloc(void *p, size_t n)
 
    Resizes the memory block pointed to by *p* to *n* bytes. The contents will
@@ -136,8 +147,8 @@
 memory from the Python heap.
 
 The default memory block allocator uses the following functions:
-:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when
-requesting zero bytes.
+:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`; call
+``malloc(1)`` (or ``calloc(1, 1)``) when requesting zero bytes.
 
 .. warning::
 
@@ -152,6 +163,17 @@
    been called instead. The memory will not have been initialized in any way.
 
 
+.. c:function:: void* PyMem_Calloc(size_t nelem, size_t elsize)
+
+   Allocates *nelem* elements each whose size in bytes is *elsize* and returns
+   a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the
+   request fails. The memory is initialized to zeros. Requesting zero elements
+   or elements of size zero bytes returns a distinct non-*NULL* pointer if
+   possible, as if ``PyMem_Calloc(1, 1)`` had been called instead.
+
+   .. versionadded:: 3.5
+
+
 .. c:function:: void* PyMem_Realloc(void *p, size_t n)
 
    Resizes the memory block pointed to by *p* to *n* bytes. The contents will be
@@ -222,11 +244,17 @@
    +----------------------------------------------------------+---------------------------------------+
    | ``void* malloc(void *ctx, size_t size)``                 | allocate a memory block               |
    +----------------------------------------------------------+---------------------------------------+
+   | ``void* calloc(void *ctx, size_t nelem, size_t elsize)`` | allocate a memory block initialized   |
+   |                                                          | with zeros                            |
+   +----------------------------------------------------------+---------------------------------------+
    | ``void* realloc(void *ctx, void *ptr, size_t new_size)`` | allocate or resize a memory block     |
    +----------------------------------------------------------+---------------------------------------+
    | ``void free(void *ctx, void *ptr)``                      | free a memory block                   |
    +----------------------------------------------------------+---------------------------------------+
 
+   .. versionchanged:: 3.5
+      Add a new field ``calloc``.
+
 .. c:type:: PyMemAllocatorDomain
 
    Enum used to identify an allocator domain. Domains:
diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst
--- a/Doc/whatsnew/3.5.rst
+++ b/Doc/whatsnew/3.5.rst
@@ -164,7 +164,10 @@
 
 Major performance enhancements have been added:
 
-* None yet.
+* Construction of ``bytes(int)`` and ``bytearray(int)`` (filled by zero bytes)
+  is faster and use less memory (until the bytearray buffer is filled with
+  data) for large objects. ``calloc()`` is used instead of ``malloc()`` to
+  allocate memory for these objects.
 
 
 Build and C API Changes
@@ -172,7 +175,12 @@
 
 Changes to Python's build process and to the C API include:
 
-* None yet.
+* New ``calloc`` functions:
+
+  * :c:func:`PyMem_RawCalloc`
+  * :c:func:`PyMem_Calloc`
+  * :c:func:`PyObject_Calloc`
+  * :c:func:`_PyObject_GC_Calloc`
 
 
 Deprecated
@@ -209,6 +217,9 @@
 This section lists previously described changes and other bugfixes
 that may require changes to your code.
 
+Changes in the Python API
+-------------------------
+
 * Before Python 3.5, a :class:`datetime.time` object was considered to be false
   if it represented midnight in UTC.  This behavior was considered obscure and
   error-prone and has been removed in Python 3.5.  See :issue:`13936` for full
@@ -217,3 +228,8 @@
 * :meth:`ssl.SSLSocket.send()` now raises either :exc:`ssl.SSLWantReadError`
   or :exc:`ssl.SSLWantWriteError` on a non-blocking socket if the operation
   would block. Previously, it would return 0.  See :issue:`20951`.
+
+Changes in the C API
+--------------------
+
+* The :c:type:`PyMemAllocator` structure has a new ``calloc`` field.
diff --git a/Include/objimpl.h b/Include/objimpl.h
--- a/Include/objimpl.h
+++ b/Include/objimpl.h
@@ -95,6 +95,7 @@
    the raw memory.
 */
 PyAPI_FUNC(void *) PyObject_Malloc(size_t size);
+PyAPI_FUNC(void *) PyObject_Calloc(size_t nelem, size_t elsize);
 PyAPI_FUNC(void *) PyObject_Realloc(void *ptr, size_t new_size);
 PyAPI_FUNC(void) PyObject_Free(void *ptr);
 
@@ -321,7 +322,8 @@
         (!PyTuple_CheckExact(obj) || _PyObject_GC_IS_TRACKED(obj)))
 #endif /* Py_LIMITED_API */
 
-PyAPI_FUNC(PyObject *) _PyObject_GC_Malloc(size_t);
+PyAPI_FUNC(PyObject *) _PyObject_GC_Malloc(size_t size);
+PyAPI_FUNC(PyObject *) _PyObject_GC_Calloc(size_t size);
 PyAPI_FUNC(PyObject *) _PyObject_GC_New(PyTypeObject *);
 PyAPI_FUNC(PyVarObject *) _PyObject_GC_NewVar(PyTypeObject *, Py_ssize_t);
 PyAPI_FUNC(void) PyObject_GC_Track(void *);
diff --git a/Include/pymem.h b/Include/pymem.h
--- a/Include/pymem.h
+++ b/Include/pymem.h
@@ -13,6 +13,7 @@
 
 #ifndef Py_LIMITED_API
 PyAPI_FUNC(void *) PyMem_RawMalloc(size_t size);
+PyAPI_FUNC(void *) PyMem_RawCalloc(size_t nelem, size_t elsize);
 PyAPI_FUNC(void *) PyMem_RawRealloc(void *ptr, size_t new_size);
 PyAPI_FUNC(void) PyMem_RawFree(void *ptr);
 #endif
@@ -57,6 +58,7 @@
 */
 
 PyAPI_FUNC(void *) PyMem_Malloc(size_t size);
+PyAPI_FUNC(void *) PyMem_Calloc(size_t nelem, size_t elsize);
 PyAPI_FUNC(void *) PyMem_Realloc(void *ptr, size_t new_size);
 PyAPI_FUNC(void) PyMem_Free(void *ptr);
 
@@ -132,6 +134,9 @@
     /* allocate a memory block */
     void* (*malloc) (void *ctx, size_t size);
 
+    /* allocate a memory block initialized by zeros */
+    void* (*calloc) (void *ctx, size_t nelem, size_t elsize);
+
     /* allocate or resize a memory block */
     void* (*realloc) (void *ctx, void *ptr, size_t new_size);
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,12 @@
 Core and Builtins
 -----------------
 
+- Issue #21233: Add new C functions: PyMem_RawCalloc(), PyMem_Calloc(),
+  PyObject_Calloc(), _PyObject_GC_Calloc(). bytes(int) and bytearray(int)
+  are now using ``calloc()`` instead of ``malloc()`` for large objects which
+  is faster and use less memory (until the bytearray buffer is filled with
+  data).
+
 - Issue #21377: PyBytes_Concat() now tries to concatenate in-place when the
   first argument has a reference count of 1.  Patch by Nikolaus Rath.
 
diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -2710,6 +2710,20 @@
 {
     void *ptr;
 
+    ptr = PyMem_RawMalloc(0);
+    if (ptr == NULL) {
+        PyErr_SetString(PyExc_RuntimeError, "PyMem_RawMalloc(0) returns NULL");
+        return NULL;
+    }
+    PyMem_RawFree(ptr);
+
+    ptr = PyMem_RawCalloc(0, 0);
+    if (ptr == NULL) {
+        PyErr_SetString(PyExc_RuntimeError, "PyMem_RawCalloc(0, 0) returns NULL");
+        return NULL;
+    }
+    PyMem_RawFree(ptr);
+
     ptr = PyMem_Malloc(0);
     if (ptr == NULL) {
         PyErr_SetString(PyExc_RuntimeError, "PyMem_Malloc(0) returns NULL");
@@ -2717,6 +2731,13 @@
     }
     PyMem_Free(ptr);
 
+    ptr = PyMem_Calloc(0, 0);
+    if (ptr == NULL) {
+        PyErr_SetString(PyExc_RuntimeError, "PyMem_Calloc(0, 0) returns NULL");
+        return NULL;
+    }
+    PyMem_Free(ptr);
+
     ptr = PyObject_Malloc(0);
     if (ptr == NULL) {
         PyErr_SetString(PyExc_RuntimeError, "PyObject_Malloc(0) returns NULL");
@@ -2724,6 +2745,13 @@
     }
     PyObject_Free(ptr);
 
+    ptr = PyObject_Calloc(0, 0);
+    if (ptr == NULL) {
+        PyErr_SetString(PyExc_RuntimeError, "PyObject_Calloc(0, 0) returns NULL");
+        return NULL;
+    }
+    PyObject_Free(ptr);
+
     Py_RETURN_NONE;
 }
 
@@ -2731,6 +2759,8 @@
     PyMemAllocator alloc;
 
     size_t malloc_size;
+    size_t calloc_nelem;
+    size_t calloc_elsize;
     void *realloc_ptr;
     size_t realloc_new_size;
     void *free_ptr;
@@ -2743,6 +2773,14 @@
     return hook->alloc.malloc(hook->alloc.ctx, size);
 }
 
+static void* hook_calloc (void* ctx, size_t nelem, size_t elsize)
+{
+    alloc_hook_t *hook = (alloc_hook_t *)ctx;
+    hook->calloc_nelem = nelem;
+    hook->calloc_elsize = elsize;
+    return hook->alloc.calloc(hook->alloc.ctx, nelem, elsize);
+}
+
 static void* hook_realloc (void* ctx, void* ptr, size_t new_size)
 {
     alloc_hook_t *hook = (alloc_hook_t *)ctx;
@@ -2765,16 +2803,14 @@
     const char *error_msg;
     alloc_hook_t hook;
     PyMemAllocator alloc;
-    size_t size, size2;
+    size_t size, size2, nelem, elsize;
     void *ptr, *ptr2;
 
-    hook.malloc_size = 0;
-    hook.realloc_ptr = NULL;
-    hook.realloc_new_size = 0;
-    hook.free_ptr = NULL;
+    memset(&hook, 0, sizeof(hook));
 
     alloc.ctx = &hook;
     alloc.malloc = &hook_malloc;
+    alloc.calloc = &hook_calloc;
     alloc.realloc = &hook_realloc;
     alloc.free = &hook_free;
     PyMem_GetAllocator(domain, &hook.alloc);
@@ -2831,6 +2867,33 @@
         goto fail;
     }
 
+    nelem = 2;
+    elsize = 5;
+    switch(domain)
+    {
+    case PYMEM_DOMAIN_RAW: ptr = PyMem_RawCalloc(nelem, elsize); break;
+    case PYMEM_DOMAIN_MEM: ptr = PyMem_Calloc(nelem, elsize); break;
+    case PYMEM_DOMAIN_OBJ: ptr = PyObject_Calloc(nelem, elsize); break;
+    default: ptr = NULL; break;
+    }
+
+    if (ptr == NULL) {
+        error_msg = "calloc failed";
+        goto fail;
+    }
+
+    if (hook.calloc_nelem != nelem || hook.calloc_elsize != elsize) {
+        error_msg = "calloc invalid nelem or elsize";
+        goto fail;
+    }
+
+    switch(domain)
+    {
+    case PYMEM_DOMAIN_RAW: PyMem_RawFree(ptr); break;
+    case PYMEM_DOMAIN_MEM: PyMem_Free(ptr); break;
+    case PYMEM_DOMAIN_OBJ: PyObject_Free(ptr); break;
+    }
+
     Py_INCREF(Py_None);
     res = Py_None;
     goto finally;
diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c
--- a/Modules/_tracemalloc.c
+++ b/Modules/_tracemalloc.c
@@ -476,17 +476,22 @@
 }
 
 static void*
-tracemalloc_malloc(void *ctx, size_t size)
+tracemalloc_alloc(int use_calloc, void *ctx, size_t nelem, size_t elsize)
 {
     PyMemAllocator *alloc = (PyMemAllocator *)ctx;
     void *ptr;
 
-    ptr = alloc->malloc(alloc->ctx, size);
+    assert(nelem <= PY_SIZE_MAX / elsize);
+
+    if (use_calloc)
+        ptr = alloc->calloc(alloc->ctx, nelem, elsize);
+    else
+        ptr = alloc->malloc(alloc->ctx, nelem * elsize);
     if (ptr == NULL)
         return NULL;
 
     TABLES_LOCK();
-    if (tracemalloc_add_trace(ptr, size) < 0) {
+    if (tracemalloc_add_trace(ptr, nelem * elsize) < 0) {
         /* Failed to allocate a trace for the new memory block */
         TABLES_UNLOCK();
         alloc->free(alloc->ctx, ptr);
@@ -560,13 +565,16 @@
 }
 
 static void*
-tracemalloc_malloc_gil(void *ctx, size_t size)
+tracemalloc_alloc_gil(int use_calloc, void *ctx, size_t nelem, size_t elsize)
 {
     void *ptr;
 
     if (get_reentrant()) {
         PyMemAllocator *alloc = (PyMemAllocator *)ctx;
-        return alloc->malloc(alloc->ctx, size);
+        if (use_calloc)
+            return alloc->calloc(alloc->ctx, nelem, elsize);
+        else
+            return alloc->malloc(alloc->ctx, nelem * elsize);
     }
 
     /* Ignore reentrant call. PyObjet_Malloc() calls PyMem_Malloc() for
@@ -574,13 +582,25 @@
        allocation twice. */
     set_reentrant(1);
 
-    ptr = tracemalloc_malloc(ctx, size);
+    ptr = tracemalloc_alloc(use_calloc, ctx, nelem, elsize);
 
     set_reentrant(0);
     return ptr;
 }
 
 static void*
+tracemalloc_malloc_gil(void *ctx, size_t size)
+{
+    return tracemalloc_alloc_gil(0, ctx, 1, size);
+}
+
+static void*
+tracemalloc_calloc_gil(void *ctx, size_t nelem, size_t elsize)
+{
+    return tracemalloc_alloc_gil(1, ctx, nelem, elsize);
+}
+
+static void*
 tracemalloc_realloc_gil(void *ctx, void *ptr, size_t new_size)
 {
     void *ptr2;
@@ -614,7 +634,7 @@
 
 #ifdef TRACE_RAW_MALLOC
 static void*
-tracemalloc_raw_malloc(void *ctx, size_t size)
+tracemalloc_raw_alloc(int use_calloc, void *ctx, size_t nelem, size_t elsize)
 {
 #ifdef WITH_THREAD
     PyGILState_STATE gil_state;
@@ -623,7 +643,10 @@
 
     if (get_reentrant()) {
         PyMemAllocator *alloc = (PyMemAllocator *)ctx;
-        return alloc->malloc(alloc->ctx, size);
+        if (use_calloc)
+            return alloc->calloc(alloc->ctx, nelem, elsize);
+        else
+            return alloc->malloc(alloc->ctx, nelem * elsize);
     }
 
     /* Ignore reentrant call. PyGILState_Ensure() may call PyMem_RawMalloc()
@@ -633,10 +656,10 @@
 
 #ifdef WITH_THREAD
     gil_state = PyGILState_Ensure();
-    ptr = tracemalloc_malloc(ctx, size);
+    ptr = tracemalloc_alloc(use_calloc, ctx, nelem, elsize);
     PyGILState_Release(gil_state);
 #else
-    ptr = tracemalloc_malloc(ctx, size);
+    ptr = tracemalloc_alloc(use_calloc, ctx, nelem, elsize);
 #endif
 
     set_reentrant(0);
@@ -644,6 +667,18 @@
 }
 
 static void*
+tracemalloc_raw_malloc(void *ctx, size_t size)
+{
+    return tracemalloc_raw_alloc(0, ctx, 1, size);
+}
+
+static void*
+tracemalloc_raw_calloc(void *ctx, size_t nelem, size_t elsize)
+{
+    return tracemalloc_raw_alloc(1, ctx, nelem, elsize);
+}
+
+static void*
 tracemalloc_raw_realloc(void *ctx, void *ptr, size_t new_size)
 {
 #ifdef WITH_THREAD
@@ -856,6 +891,7 @@
 
 #ifdef TRACE_RAW_MALLOC
     alloc.malloc = tracemalloc_raw_malloc;
+    alloc.calloc = tracemalloc_raw_calloc;
     alloc.realloc = tracemalloc_raw_realloc;
     alloc.free = tracemalloc_free;
 
@@ -865,6 +901,7 @@
 #endif
 
     alloc.malloc = tracemalloc_malloc_gil;
+    alloc.calloc = tracemalloc_calloc_gil;
     alloc.realloc = tracemalloc_realloc_gil;
     alloc.free = tracemalloc_free;
 
diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -1703,15 +1703,19 @@
         _PyObject_GC_UNTRACK(op);
 }
 
-PyObject *
-_PyObject_GC_Malloc(size_t basicsize)
+static PyObject *
+_PyObject_GC_Alloc(int use_calloc, size_t basicsize)
 {
     PyObject *op;
     PyGC_Head *g;
+    size_t size;
     if (basicsize > PY_SSIZE_T_MAX - sizeof(PyGC_Head))
         return PyErr_NoMemory();
-    g = (PyGC_Head *)PyObject_MALLOC(
-        sizeof(PyGC_Head) + basicsize);
+    size = sizeof(PyGC_Head) + basicsize;
+    if (use_calloc)
+        g = (PyGC_Head *)PyObject_Calloc(1, size);
+    else
+        g = (PyGC_Head *)PyObject_Malloc(size);
     if (g == NULL)
         return PyErr_NoMemory();
     g->gc.gc_refs = 0;
@@ -1731,6 +1735,18 @@
 }
 
 PyObject *
+_PyObject_GC_Malloc(size_t basicsize)
+{
+    return _PyObject_GC_Alloc(0, basicsize);
+}
+
+PyObject *
+_PyObject_GC_Calloc(size_t basicsize)
+{
+    return _PyObject_GC_Alloc(1, basicsize);
+}
+
+PyObject *
 _PyObject_GC_New(PyTypeObject *tp)
 {
     PyObject *op = _PyObject_GC_Malloc(_PyObject_SIZE(tp));
diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c
--- a/Objects/bytearrayobject.c
+++ b/Objects/bytearrayobject.c
@@ -813,9 +813,21 @@
     }
     else {
         if (count > 0) {
-            if (PyByteArray_Resize((PyObject *)self, count))
+            void *sval;
+            Py_ssize_t alloc;
+
+            assert (Py_SIZE(self) == 0);
+
+            alloc = count + 1;
+            sval = PyObject_Calloc(1, alloc);
+            if (sval == NULL)
                 return -1;
-            memset(PyByteArray_AS_STRING(self), 0, count);
+
+            PyObject_Free(self->ob_bytes);
+
+            self->ob_bytes = self->ob_start = sval;
+            Py_SIZE(self) = count;
+            self->ob_alloc = alloc;
         }
         return 0;
     }
diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -71,6 +71,44 @@
    PyBytes_FromStringAndSize()) or the length of the string in the `str'
    parameter (for PyBytes_FromString()).
 */
+static PyObject *
+_PyBytes_FromSize(Py_ssize_t size, int use_calloc)
+{
+    PyBytesObject *op;
+    assert(size >= 0);
+    if (size == 0 && (op = nullstring) != NULL) {
+#ifdef COUNT_ALLOCS
+        null_strings++;
+#endif
+        Py_INCREF(op);
+        return (PyObject *)op;
+    }
+
+    if (size > PY_SSIZE_T_MAX - PyBytesObject_SIZE) {
+        PyErr_SetString(PyExc_OverflowError,
+                        "byte string is too large");
+        return NULL;
+    }
+
+    /* Inline PyObject_NewVar */
+    if (use_calloc)
+        op = (PyBytesObject *)PyObject_Calloc(1, PyBytesObject_SIZE + size);
+    else
+        op = (PyBytesObject *)PyObject_Malloc(PyBytesObject_SIZE + size);
+    if (op == NULL)
+        return PyErr_NoMemory();
+    (void)PyObject_INIT_VAR(op, &PyBytes_Type, size);
+    op->ob_shash = -1;
+    if (!use_calloc)
+        op->ob_sval[size] = '\0';
+    /* empty byte string singleton */
+    if (size == 0) {
+        nullstring = op;
+        Py_INCREF(op);
+    }
+    return (PyObject *) op;
+}
+
 PyObject *
 PyBytes_FromStringAndSize(const char *str, Py_ssize_t size)
 {
@@ -80,13 +118,6 @@
             "Negative size passed to PyBytes_FromStringAndSize");
         return NULL;
     }
-    if (size == 0 && (op = nullstring) != NULL) {
-#ifdef COUNT_ALLOCS
-        null_strings++;
-#endif
-        Py_INCREF(op);
-        return (PyObject *)op;
-    }
     if (size == 1 && str != NULL &&
         (op = characters[*str & UCHAR_MAX]) != NULL)
     {
@@ -97,26 +128,15 @@
         return (PyObject *)op;
     }
 
-    if (size > PY_SSIZE_T_MAX - PyBytesObject_SIZE) {
-        PyErr_SetString(PyExc_OverflowError,
-                        "byte string is too large");
+    op = (PyBytesObject *)_PyBytes_FromSize(size, 0);
+    if (op == NULL)
         return NULL;
-    }
-
-    /* Inline PyObject_NewVar */
-    op = (PyBytesObject *)PyObject_MALLOC(PyBytesObject_SIZE + size);
-    if (op == NULL)
-        return PyErr_NoMemory();
-    (void)PyObject_INIT_VAR(op, &PyBytes_Type, size);
-    op->ob_shash = -1;
-    if (str != NULL)
-        Py_MEMCPY(op->ob_sval, str, size);
-    op->ob_sval[size] = '\0';
+    if (str == NULL)
+        return (PyObject *) op;
+
+    Py_MEMCPY(op->ob_sval, str, size);
     /* share short strings */
-    if (size == 0) {
-        nullstring = op;
-        Py_INCREF(op);
-    } else if (size == 1 && str != NULL) {
+    if (size == 1) {
         characters[*str & UCHAR_MAX] = op;
         Py_INCREF(op);
     }
@@ -2482,7 +2502,7 @@
                             "argument");
             return NULL;
         }
-        return PyBytes_FromString("");
+        return PyBytes_FromStringAndSize(NULL, 0);
     }
 
     if (PyUnicode_Check(x)) {
@@ -2532,11 +2552,9 @@
         return NULL;
     }
     else {
-        new = PyBytes_FromStringAndSize(NULL, size);
+        new = _PyBytes_FromSize(size, 1);
         if (new == NULL)
             return NULL;
-        if (size > 0)
-            memset(((PyBytesObject*)new)->ob_sval, 0, size);
         return new;
     }
 
diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c
--- a/Objects/obmalloc.c
+++ b/Objects/obmalloc.c
@@ -5,6 +5,7 @@
 #ifdef PYMALLOC_DEBUG   /* WITH_PYMALLOC && PYMALLOC_DEBUG */
 /* Forward declaration */
 static void* _PyMem_DebugMalloc(void *ctx, size_t size);
+static void* _PyMem_DebugCalloc(void *ctx, size_t nelem, size_t elsize);
 static void _PyMem_DebugFree(void *ctx, void *p);
 static void* _PyMem_DebugRealloc(void *ctx, void *ptr, size_t size);
 
@@ -43,6 +44,7 @@
 
 /* Forward declaration */
 static void* _PyObject_Malloc(void *ctx, size_t size);
+static void* _PyObject_Calloc(void *ctx, size_t nelem, size_t elsize);
 static void _PyObject_Free(void *ctx, void *p);
 static void* _PyObject_Realloc(void *ctx, void *ptr, size_t size);
 #endif
@@ -51,7 +53,7 @@
 static void *
 _PyMem_RawMalloc(void *ctx, size_t size)
 {
-    /* PyMem_Malloc(0) means malloc(1). Some systems would return NULL
+    /* PyMem_RawMalloc(0) means malloc(1). Some systems would return NULL
        for malloc(0), which would be treated as an error. Some platforms would
        return a pointer with no memory behind it, which would break pymalloc.
        To solve these problems, allocate an extra byte. */
@@ -61,6 +63,20 @@
 }
 
 static void *
+_PyMem_RawCalloc(void *ctx, size_t nelem, size_t elsize)
+{
+    /* PyMem_RawCalloc(0, 0) means calloc(1, 1). Some systems would return NULL
+       for calloc(0, 0), which would be treated as an error. Some platforms
+       would return a pointer with no memory behind it, which would break
+       pymalloc.  To solve these problems, allocate an extra byte. */
+    if (nelem == 0 || elsize == 0) {
+        nelem = 1;
+        elsize = 1;
+    }
+    return calloc(nelem, elsize);
+}
+
+static void *
 _PyMem_RawRealloc(void *ctx, void *ptr, size_t size)
 {
     if (size == 0)
@@ -123,9 +139,9 @@
 #endif
 
 
-#define PYRAW_FUNCS _PyMem_RawMalloc, _PyMem_RawRealloc, _PyMem_RawFree
+#define PYRAW_FUNCS _PyMem_RawMalloc, _PyMem_RawCalloc, _PyMem_RawRealloc, _PyMem_RawFree
 #ifdef WITH_PYMALLOC
-#  define PYOBJ_FUNCS _PyObject_Malloc, _PyObject_Realloc, _PyObject_Free
+#  define PYOBJ_FUNCS _PyObject_Malloc, _PyObject_Calloc, _PyObject_Realloc, _PyObject_Free
 #else
 #  define PYOBJ_FUNCS PYRAW_FUNCS
 #endif
@@ -147,7 +163,7 @@
     {'o', {NULL, PYOBJ_FUNCS}}
     };
 
-#define PYDBG_FUNCS _PyMem_DebugMalloc, _PyMem_DebugRealloc, _PyMem_DebugFree
+#define PYDBG_FUNCS _PyMem_DebugMalloc, _PyMem_DebugCalloc, _PyMem_DebugRealloc, _PyMem_DebugFree
 #endif
 
 static PyMemAllocator _PyMem_Raw = {
@@ -196,6 +212,7 @@
     PyMemAllocator alloc;
 
     alloc.malloc = _PyMem_DebugMalloc;
+    alloc.calloc = _PyMem_DebugCalloc;
     alloc.realloc = _PyMem_DebugRealloc;
     alloc.free = _PyMem_DebugFree;
 
@@ -228,9 +245,10 @@
     case PYMEM_DOMAIN_MEM: *allocator = _PyMem; break;
     case PYMEM_DOMAIN_OBJ: *allocator = _PyObject; break;
     default:
-        /* unknown domain */
+        /* unknown domain: set all attributes to NULL */
         allocator->ctx = NULL;
         allocator->malloc = NULL;
+        allocator->calloc = NULL;
         allocator->realloc = NULL;
         allocator->free = NULL;
     }
@@ -272,8 +290,16 @@
      */
     if (size > (size_t)PY_SSIZE_T_MAX)
         return NULL;
+    return _PyMem_Raw.malloc(_PyMem_Raw.ctx, size);
+}
 
-    return _PyMem_Raw.malloc(_PyMem_Raw.ctx, size);
+void *
+PyMem_RawCalloc(size_t nelem, size_t elsize)
+{
+    /* see PyMem_RawMalloc() */
+    if (elsize != 0 && nelem > (size_t)PY_SSIZE_T_MAX / elsize)
+        return NULL;
+    return _PyMem_Raw.calloc(_PyMem_Raw.ctx, nelem, elsize);
 }
 
 void*
@@ -300,6 +326,15 @@
 }
 
 void *
+PyMem_Calloc(size_t nelem, size_t elsize)
+{
+    /* see PyMem_RawMalloc() */
+    if (elsize != 0 && nelem > (size_t)PY_SSIZE_T_MAX / elsize)
+        return NULL;
+    return _PyMem.calloc(_PyMem.ctx, nelem, elsize);
+}
+
+void *
 PyMem_Realloc(void *ptr, size_t new_size)
 {
     /* see PyMem_RawMalloc() */
@@ -352,6 +387,15 @@
 }
 
 void *
+PyObject_Calloc(size_t nelem, size_t elsize)
+{
+    /* see PyMem_RawMalloc() */
+    if (elsize != 0 && nelem > (size_t)PY_SSIZE_T_MAX / elsize)
+        return NULL;
+    return _PyObject.calloc(_PyObject.ctx, nelem, elsize);
+}
+
+void *
 PyObject_Realloc(void *ptr, size_t new_size)
 {
     /* see PyMem_RawMalloc() */
@@ -1122,8 +1166,9 @@
  */
 
 static void *
-_PyObject_Malloc(void *ctx, size_t nbytes)
+_PyObject_Alloc(int use_calloc, void *ctx, size_t nelem, size_t elsize)
 {
+    size_t nbytes;
     block *bp;
     poolp pool;
     poolp next;
@@ -1138,9 +1183,12 @@
         goto redirect;
 #endif
 
-    /*
-     * This implicitly redirects malloc(0).
-     */
+    if (nelem == 0 || elsize == 0)
+        goto redirect;
+
+    assert(nelem <= PY_SSIZE_T_MAX / elsize);
+    nbytes = nelem * elsize;
+
     if ((nbytes - 1) < SMALL_REQUEST_THRESHOLD) {
         LOCK();
         /*
@@ -1158,6 +1206,8 @@
             assert(bp != NULL);
             if ((pool->freeblock = *(block **)bp) != NULL) {
                 UNLOCK();
+                if (use_calloc)
+                    memset(bp, 0, nbytes);
                 return (void *)bp;
             }
             /*
@@ -1170,6 +1220,8 @@
                 pool->nextoffset += INDEX2SIZE(size);
                 *(block **)(pool->freeblock) = NULL;
                 UNLOCK();
+                if (use_calloc)
+                    memset(bp, 0, nbytes);
                 return (void *)bp;
             }
             /* Pool is full, unlink from used pools. */
@@ -1178,6 +1230,8 @@
             next->prevpool = pool;
             pool->nextpool = next;
             UNLOCK();
+            if (use_calloc)
+                memset(bp, 0, nbytes);
             return (void *)bp;
         }
 
@@ -1257,6 +1311,8 @@
                 assert(bp != NULL);
                 pool->freeblock = *(block **)bp;
                 UNLOCK();
+                if (use_calloc)
+                    memset(bp, 0, nbytes);
                 return (void *)bp;
             }
             /*
@@ -1272,6 +1328,8 @@
             pool->freeblock = bp + size;
             *(block **)(pool->freeblock) = NULL;
             UNLOCK();
+            if (use_calloc)
+                memset(bp, 0, nbytes);
             return (void *)bp;
         }
 
@@ -1311,13 +1369,29 @@
      * has been reached.
      */
     {
-        void *result = PyMem_RawMalloc(nbytes);
+        void *result;
+        if (use_calloc)
+            result = PyMem_RawCalloc(nelem, elsize);
+        else
+            result = PyMem_RawMalloc(nbytes);
         if (!result)
             _Py_AllocatedBlocks--;
         return result;
     }
 }
 
+static void *
+_PyObject_Malloc(void *ctx, size_t nbytes)
+{
+    return _PyObject_Alloc(0, ctx, 1, nbytes);
+}
+
+static void *
+_PyObject_Calloc(void *ctx, size_t nelem, size_t elsize)
+{
+    return _PyObject_Alloc(1, ctx, nelem, elsize);
+}
+
 /* free */
 
 ATTRIBUTE_NO_ADDRESS_SAFETY_ANALYSIS
@@ -1561,7 +1635,7 @@
 #endif
 
     if (p == NULL)
-        return _PyObject_Malloc(ctx, nbytes);
+        return _PyObject_Alloc(0, ctx, 1, nbytes);
 
 #ifdef WITH_VALGRIND
     /* Treat running_on_valgrind == -1 the same as 0 */
@@ -1589,7 +1663,7 @@
             }
             size = nbytes;
         }
-        bp = _PyObject_Malloc(ctx, nbytes);
+        bp = _PyObject_Alloc(0, ctx, 1, nbytes);
         if (bp != NULL) {
             memcpy(bp, p, size);
             _PyObject_Free(ctx, p);
@@ -1745,7 +1819,7 @@
 */
 
 static void *
-_PyMem_DebugMalloc(void *ctx, size_t nbytes)
+_PyMem_DebugAlloc(int use_calloc, void *ctx, size_t nbytes)
 {
     debug_alloc_api_t *api = (debug_alloc_api_t *)ctx;
     uchar *p;           /* base address of malloc'ed block */
@@ -1758,7 +1832,10 @@
         /* overflow:  can't represent total as a size_t */
         return NULL;
 
-    p = (uchar *)api->alloc.malloc(api->alloc.ctx, total);
+    if (use_calloc)
+        p = (uchar *)api->alloc.calloc(api->alloc.ctx, 1, total);
+    else
+        p = (uchar *)api->alloc.malloc(api->alloc.ctx, total);
     if (p == NULL)
         return NULL;
 
@@ -1767,7 +1844,7 @@
     p[SST] = (uchar)api->api_id;
     memset(p + SST + 1, FORBIDDENBYTE, SST-1);
 
-    if (nbytes > 0)
+    if (nbytes > 0 && !use_calloc)
         memset(p + 2*SST, CLEANBYTE, nbytes);
 
     /* at tail, write pad (SST bytes) and serialno (SST bytes) */
@@ -1778,6 +1855,21 @@
     return p + 2*SST;
 }
 
+static void *
+_PyMem_DebugMalloc(void *ctx, size_t nbytes)
+{
+    return _PyMem_DebugAlloc(0, ctx, nbytes);
+}
+
+static void *
+_PyMem_DebugCalloc(void *ctx, size_t nelem, size_t elsize)
+{
+    size_t nbytes;
+    assert(elsize == 0 || nelem <= PY_SSIZE_T_MAX / elsize);
+    nbytes = nelem * elsize;
+    return _PyMem_DebugAlloc(1, ctx, nbytes);
+}
+
 /* The debug free first checks the 2*SST bytes on each end for sanity (in
    particular, that the FORBIDDENBYTEs with the api ID are still intact).
    Then fills the original bytes with DEADBYTE.
@@ -1811,7 +1903,7 @@
     int i;
 
     if (p == NULL)
-        return _PyMem_DebugMalloc(ctx, nbytes);
+        return _PyMem_DebugAlloc(0, ctx, nbytes);
 
     _PyMem_DebugCheckAddress(api->api_id, p);
     bumpserialno();

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  2 23:26:18 2014
From: python-checkins at python.org (victor.stinner)
Date: Fri,  2 May 2014 23:26:18 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321233=3A_Oops=2C_?=
 =?utf-8?q?Fix_=5FPyObject=5FAlloc=28=29=3A_initialize_nbytes_before_going?=
 =?utf-8?q?_to?=
Message-ID: <3gL62y0Xr0z7LjN@mail.python.org>

http://hg.python.org/cpython/rev/62438d1b11c7
changeset:   90541:62438d1b11c7
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Fri May 02 23:26:03 2014 +0200
summary:
  Issue #21233: Oops, Fix _PyObject_Alloc(): initialize nbytes before going to
redirect.

files:
  Objects/obmalloc.c |  6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c
--- a/Objects/obmalloc.c
+++ b/Objects/obmalloc.c
@@ -1183,12 +1183,12 @@
         goto redirect;
 #endif
 
+    assert(nelem <= PY_SSIZE_T_MAX / elsize);
+    nbytes = nelem * elsize;
+
     if (nelem == 0 || elsize == 0)
         goto redirect;
 
-    assert(nelem <= PY_SSIZE_T_MAX / elsize);
-    nbytes = nelem * elsize;
-
     if ((nbytes - 1) < SMALL_REQUEST_THRESHOLD) {
         LOCK();
         /*

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Sat May  3 09:52:34 2014
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sat, 03 May 2014 09:52:34 +0200
Subject: [Python-checkins] Daily reference leaks (62438d1b11c7): sum=3
Message-ID: <E1WgUkM-0002bZ-0l@vds2544.sivit.org>

results for 62438d1b11c7 on branch "default"
--------------------------------------------

test_functools leaked [0, 0, 3] memory blocks, sum=3


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogGqWH_X', '-x']

From python-checkins at python.org  Sun May  4 00:26:31 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Sun,  4 May 2014 00:26:31 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgMjEzNzU6?=
 =?utf-8?q?__Fix_possible_Py=5Fssizet_overflow_in_heapq=2E?=
Message-ID: <3gLlKz6Phsz7Ljr@mail.python.org>

http://hg.python.org/cpython/rev/725cb631699a
changeset:   90542:725cb631699a
branch:      3.4
parent:      90537:eb361f69ddd1
user:        Raymond Hettinger <python at rcn.com>
date:        Sat May 03 15:22:07 2014 -0700
summary:
  Issue 21375:  Fix possible Py_ssizet overflow in heapq.

files:
  Modules/_heapqmodule.c |  16 ++++++++--------
  1 files changed, 8 insertions(+), 8 deletions(-)


diff --git a/Modules/_heapqmodule.c b/Modules/_heapqmodule.c
--- a/Modules/_heapqmodule.c
+++ b/Modules/_heapqmodule.c
@@ -62,7 +62,7 @@
 static int
 _siftup(PyListObject *heap, Py_ssize_t pos)
 {
-    Py_ssize_t startpos, endpos, childpos, rightpos;
+    Py_ssize_t startpos, endpos, childpos, rightpos, limit;
     int cmp;
     PyObject *newitem, *tmp, *olditem;
     Py_ssize_t size;
@@ -79,9 +79,10 @@
     Py_INCREF(newitem);
 
     /* Bubble up the smaller child until hitting a leaf. */
-    childpos = 2*pos + 1;    /* leftmost child position  */
-    while (childpos < endpos) {
+    limit = endpos / 2;          /* smallest pos that has no child */
+    while (pos < limit) {
         /* Set childpos to index of smaller child.   */
+        childpos = 2*pos + 1;    /* leftmost child position  */
         rightpos = childpos + 1;
         if (rightpos < endpos) {
             cmp = PyObject_RichCompareBool(
@@ -108,7 +109,6 @@
         PyList_SET_ITEM(heap, pos, tmp);
         Py_DECREF(olditem);
         pos = childpos;
-        childpos = 2*pos + 1;
         if (size != PyList_GET_SIZE(heap)) {
             PyErr_SetString(PyExc_RuntimeError,
                             "list changed size during iteration");
@@ -419,7 +419,7 @@
 static int
 _siftupmax(PyListObject *heap, Py_ssize_t pos)
 {
-    Py_ssize_t startpos, endpos, childpos, rightpos;
+    Py_ssize_t startpos, endpos, childpos, rightpos, limit;
     int cmp;
     PyObject *newitem, *tmp;
 
@@ -434,9 +434,10 @@
     Py_INCREF(newitem);
 
     /* Bubble up the smaller child until hitting a leaf. */
-    childpos = 2*pos + 1;    /* leftmost child position  */
-    while (childpos < endpos) {
+    limit = endpos / 2;          /* smallest pos that has no child */
+    while (pos < limit) {
         /* Set childpos to index of smaller child.   */
+        childpos = 2*pos + 1;    /* leftmost child position  */
         rightpos = childpos + 1;
         if (rightpos < endpos) {
             cmp = PyObject_RichCompareBool(
@@ -456,7 +457,6 @@
         Py_DECREF(PyList_GET_ITEM(heap, pos));
         PyList_SET_ITEM(heap, pos, tmp);
         pos = childpos;
-        childpos = 2*pos + 1;
     }
 
     /* The leaf at pos is empty now.  Put newitem there, and bubble

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May  4 00:26:33 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Sun,  4 May 2014 00:26:33 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?q?=29=3A_merge?=
Message-ID: <3gLlL11V7Hz7Lkb@mail.python.org>

http://hg.python.org/cpython/rev/5d076506b3f5
changeset:   90543:5d076506b3f5
parent:      90541:62438d1b11c7
parent:      90542:725cb631699a
user:        Raymond Hettinger <python at rcn.com>
date:        Sat May 03 15:26:17 2014 -0700
summary:
  merge

files:
  Modules/_heapqmodule.c |  16 ++++++++--------
  1 files changed, 8 insertions(+), 8 deletions(-)


diff --git a/Modules/_heapqmodule.c b/Modules/_heapqmodule.c
--- a/Modules/_heapqmodule.c
+++ b/Modules/_heapqmodule.c
@@ -62,7 +62,7 @@
 static int
 _siftup(PyListObject *heap, Py_ssize_t pos)
 {
-    Py_ssize_t startpos, endpos, childpos, rightpos;
+    Py_ssize_t startpos, endpos, childpos, rightpos, limit;
     int cmp;
     PyObject *newitem, *tmp, *olditem;
     Py_ssize_t size;
@@ -79,9 +79,10 @@
     Py_INCREF(newitem);
 
     /* Bubble up the smaller child until hitting a leaf. */
-    childpos = 2*pos + 1;    /* leftmost child position  */
-    while (childpos < endpos) {
+    limit = endpos / 2;          /* smallest pos that has no child */
+    while (pos < limit) {
         /* Set childpos to index of smaller child.   */
+        childpos = 2*pos + 1;    /* leftmost child position  */
         rightpos = childpos + 1;
         if (rightpos < endpos) {
             cmp = PyObject_RichCompareBool(
@@ -108,7 +109,6 @@
         PyList_SET_ITEM(heap, pos, tmp);
         Py_DECREF(olditem);
         pos = childpos;
-        childpos = 2*pos + 1;
         if (size != PyList_GET_SIZE(heap)) {
             PyErr_SetString(PyExc_RuntimeError,
                             "list changed size during iteration");
@@ -419,7 +419,7 @@
 static int
 _siftupmax(PyListObject *heap, Py_ssize_t pos)
 {
-    Py_ssize_t startpos, endpos, childpos, rightpos;
+    Py_ssize_t startpos, endpos, childpos, rightpos, limit;
     int cmp;
     PyObject *newitem, *tmp;
 
@@ -434,9 +434,10 @@
     Py_INCREF(newitem);
 
     /* Bubble up the smaller child until hitting a leaf. */
-    childpos = 2*pos + 1;    /* leftmost child position  */
-    while (childpos < endpos) {
+    limit = endpos / 2;          /* smallest pos that has no child */
+    while (pos < limit) {
         /* Set childpos to index of smaller child.   */
+        childpos = 2*pos + 1;    /* leftmost child position  */
         rightpos = childpos + 1;
         if (rightpos < endpos) {
             cmp = PyObject_RichCompareBool(
@@ -456,7 +457,6 @@
         Py_DECREF(PyList_GET_ITEM(heap, pos));
         PyList_SET_ITEM(heap, pos, tmp);
         pos = childpos;
-        childpos = 2*pos + 1;
     }
 
     /* The leaf at pos is empty now.  Put newitem there, and bubble

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May  4 00:30:18 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Sun,  4 May 2014 00:30:18 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgMjEzNzU6?=
 =?utf-8?q?__Fix_possible_Py=5Fssizet_overflow_in_heapq=2E?=
Message-ID: <3gLlQL3C3Qz7LlD@mail.python.org>

http://hg.python.org/cpython/rev/b768d41dec0a
changeset:   90544:b768d41dec0a
branch:      2.7
parent:      90536:5f75eadecff1
user:        Raymond Hettinger <python at rcn.com>
date:        Sat May 03 15:27:14 2014 -0700
summary:
  Issue 21375:  Fix possible Py_ssizet overflow in heapq.

files:
  Modules/_heapqmodule.c |  16 ++++++++--------
  1 files changed, 8 insertions(+), 8 deletions(-)


diff --git a/Modules/_heapqmodule.c b/Modules/_heapqmodule.c
--- a/Modules/_heapqmodule.c
+++ b/Modules/_heapqmodule.c
@@ -86,7 +86,7 @@
 static int
 _siftup(PyListObject *heap, Py_ssize_t pos)
 {
-    Py_ssize_t startpos, endpos, childpos, rightpos;
+    Py_ssize_t startpos, endpos, childpos, rightpos, limit;
     int cmp;
     PyObject *newitem, *tmp, *olditem;
     Py_ssize_t size;
@@ -103,9 +103,10 @@
     Py_INCREF(newitem);
 
     /* Bubble up the smaller child until hitting a leaf. */
-    childpos = 2*pos + 1;    /* leftmost child position  */
-    while (childpos < endpos) {
+    limit = endpos / 2;          /* smallest pos that has no child */
+    while (pos < limit) {
         /* Set childpos to index of smaller child.   */
+        childpos = 2*pos + 1;    /* leftmost child position  */
         rightpos = childpos + 1;
         if (rightpos < endpos) {
             cmp = cmp_lt(
@@ -131,7 +132,6 @@
         PyList_SET_ITEM(heap, pos, tmp);
         Py_DECREF(olditem);
         pos = childpos;
-        childpos = 2*pos + 1;
         if (size != PyList_GET_SIZE(heap)) {
             PyErr_SetString(PyExc_RuntimeError,
                             "list changed size during iteration");
@@ -439,7 +439,7 @@
 static int
 _siftupmax(PyListObject *heap, Py_ssize_t pos)
 {
-    Py_ssize_t startpos, endpos, childpos, rightpos;
+    Py_ssize_t startpos, endpos, childpos, rightpos, limit;
     int cmp;
     PyObject *newitem, *tmp;
 
@@ -454,9 +454,10 @@
     Py_INCREF(newitem);
 
     /* Bubble up the smaller child until hitting a leaf. */
-    childpos = 2*pos + 1;    /* leftmost child position  */
-    while (childpos < endpos) {
+    limit = endpos / 2;          /* smallest pos that has no child */
+    while (pos < limit) {
         /* Set childpos to index of smaller child.   */
+        childpos = 2*pos + 1;    /* leftmost child position  */
         rightpos = childpos + 1;
         if (rightpos < endpos) {
             cmp = cmp_lt(
@@ -475,7 +476,6 @@
         Py_DECREF(PyList_GET_ITEM(heap, pos));
         PyList_SET_ITEM(heap, pos, tmp);
         pos = childpos;
-        childpos = 2*pos + 1;
     }
 
     /* The leaf at pos is empty now.  Put newitem there, and bubble

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May  4 01:32:21 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Sun,  4 May 2014 01:32:21 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_21101=3A__Internal_A?=
 =?utf-8?q?PI_for_dict_getitem_and_setitem_where_the_hash_value_is?=
Message-ID: <3gLmnx0GF9z7Ljd@mail.python.org>

http://hg.python.org/cpython/rev/39f475aa0163
changeset:   90545:39f475aa0163
parent:      90543:5d076506b3f5
user:        Raymond Hettinger <python at rcn.com>
date:        Sat May 03 16:32:11 2014 -0700
summary:
  Issue 21101:  Internal API for dict getitem and setitem where the hash value is known.

files:
  Include/dictobject.h |   4 ++
  Objects/dictobject.c |  56 ++++++++++++++++++++++++++++++++
  2 files changed, 60 insertions(+), 0 deletions(-)


diff --git a/Include/dictobject.h b/Include/dictobject.h
--- a/Include/dictobject.h
+++ b/Include/dictobject.h
@@ -50,6 +50,8 @@
 
 PyAPI_FUNC(PyObject *) PyDict_New(void);
 PyAPI_FUNC(PyObject *) PyDict_GetItem(PyObject *mp, PyObject *key);
+PyAPI_FUNC(PyObject *) _PyDict_GetItem_KnownHash(PyObject *mp, PyObject *key,
+                                       Py_hash_t hash);
 PyAPI_FUNC(PyObject *) PyDict_GetItemWithError(PyObject *mp, PyObject *key);
 PyAPI_FUNC(PyObject *) _PyDict_GetItemIdWithError(PyObject *dp,
                                                   struct _Py_Identifier *key);
@@ -58,6 +60,8 @@
     PyObject *mp, PyObject *key, PyObject *defaultobj);
 #endif
 PyAPI_FUNC(int) PyDict_SetItem(PyObject *mp, PyObject *key, PyObject *item);
+PyAPI_FUNC(int) _PyDict_SetItem_KnownHash(PyObject *mp, PyObject *key,
+                                          PyObject *item, Py_hash_t hash);
 PyAPI_FUNC(int) PyDict_DelItem(PyObject *mp, PyObject *key);
 PyAPI_FUNC(void) PyDict_Clear(PyObject *mp);
 PyAPI_FUNC(int) PyDict_Next(
diff --git a/Objects/dictobject.c b/Objects/dictobject.c
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -1101,6 +1101,44 @@
     return *value_addr;
 }
 
+PyObject *
+_PyDict_GetItem_KnownHash(PyObject *op, PyObject *key, Py_hash_t hash)
+{
+    PyDictObject *mp = (PyDictObject *)op;
+    PyDictKeyEntry *ep;
+    PyThreadState *tstate;
+    PyObject **value_addr;
+
+    if (!PyDict_Check(op))
+        return NULL;
+
+    /* We can arrive here with a NULL tstate during initialization: try
+       running "python -Wi" for an example related to string interning.
+       Let's just hope that no exception occurs then...  This must be
+       _PyThreadState_Current and not PyThreadState_GET() because in debug
+       mode, the latter complains if tstate is NULL. */
+    tstate = (PyThreadState*)_Py_atomic_load_relaxed(
+        &_PyThreadState_Current);
+    if (tstate != NULL && tstate->curexc_type != NULL) {
+        /* preserve the existing exception */
+        PyObject *err_type, *err_value, *err_tb;
+        PyErr_Fetch(&err_type, &err_value, &err_tb);
+        ep = (mp->ma_keys->dk_lookup)(mp, key, hash, &value_addr);
+        /* ignore errors */
+        PyErr_Restore(err_type, err_value, err_tb);
+        if (ep == NULL)
+            return NULL;
+    }
+    else {
+        ep = (mp->ma_keys->dk_lookup)(mp, key, hash, &value_addr);
+        if (ep == NULL) {
+            PyErr_Clear();
+            return NULL;
+        }
+    }
+    return *value_addr;
+}
+
 /* Variant of PyDict_GetItem() that doesn't suppress exceptions.
    This returns NULL *with* an exception set if an exception occurred.
    It returns NULL *without* an exception set if the key wasn't present.
@@ -1208,6 +1246,24 @@
 }
 
 int
+_PyDict_SetItem_KnownHash(PyObject *op, PyObject *key, PyObject *value,
+                         Py_hash_t hash)
+{
+    PyDictObject *mp;
+
+    if (!PyDict_Check(op)) {
+        PyErr_BadInternalCall();
+        return -1;
+    }
+    assert(key);
+    assert(value);
+    mp = (PyDictObject *)op;
+
+    /* insertdict() handles any resizing that might be necessary */
+    return insertdict(mp, key, hash, value);
+}
+
+int
 PyDict_DelItem(PyObject *op, PyObject *key)
 {
     PyDictObject *mp;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May  4 01:41:27 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Sun,  4 May 2014 01:41:27 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321101=3A__Elimina?=
 =?utf-8?q?te_double_hashing_in_the_C_code_for_collections=2ECounter=28=29?=
 =?utf-8?q?=2E?=
Message-ID: <3gLn0R23Krz7Ljd@mail.python.org>

http://hg.python.org/cpython/rev/592a57682ced
changeset:   90546:592a57682ced
user:        Raymond Hettinger <python at rcn.com>
date:        Sat May 03 16:41:19 2014 -0700
summary:
  Issue #21101:  Eliminate double hashing in the C code for collections.Counter().

files:
  Misc/NEWS                    |   3 +++
  Modules/_collectionsmodule.c |  17 ++++++++++++++---
  2 files changed, 17 insertions(+), 3 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -73,6 +73,9 @@
   Decimal.quantize() method in the Python version.  It had never been
   present in the C version.
 
+- Issue #21101: Eliminate double hashing in the C speed-up code for
+  collections.Counter().
+
 - Issue #21321: itertools.islice() now releases the reference to the source
   iterator when the slice is exhausted.  Patch by Anton Afanasyev.
 
diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c
--- a/Modules/_collectionsmodule.c
+++ b/Modules/_collectionsmodule.c
@@ -1831,18 +1831,29 @@
     if (mapping_get != NULL && mapping_get == dict_get &&
         mapping_setitem != NULL && mapping_setitem == dict_setitem) {
         while (1) {
+            Py_hash_t hash;
+
             key = PyIter_Next(it);
             if (key == NULL)
                 break;
-            oldval = PyDict_GetItem(mapping, key);
+
+            if (!PyUnicode_CheckExact(key) ||
+                (hash = ((PyASCIIObject *) key)->hash) == -1)
+            {
+                hash = PyObject_Hash(key);
+                if (hash == -1) 
+                    goto done;
+            }
+
+            oldval = _PyDict_GetItem_KnownHash(mapping, key, hash);
             if (oldval == NULL) {
-                if (PyDict_SetItem(mapping, key, one) == -1)
+                if (_PyDict_SetItem_KnownHash(mapping, key, one, hash) == -1)
                     break;
             } else {
                 newval = PyNumber_Add(oldval, one);
                 if (newval == NULL)
                     break;
-                if (PyDict_SetItem(mapping, key, newval) == -1)
+                if (_PyDict_SetItem_KnownHash(mapping, key, newval, hash) == -1)
                     break;
                 Py_CLEAR(newval);
             }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May  4 02:19:04 2014
From: python-checkins at python.org (benjamin.peterson)
Date: Sun,  4 May 2014 02:19:04 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_do_not_expose_known_hash_a?=
	=?utf-8?q?pi_in_stable_API?=
Message-ID: <3gLnqr0875z7LjZ@mail.python.org>

http://hg.python.org/cpython/rev/e947dc6a33d9
changeset:   90547:e947dc6a33d9
user:        Benjamin Peterson <benjamin at python.org>
date:        Sat May 03 19:39:15 2014 -0400
summary:
  do not expose known hash api in stable API

files:
  Include/dictobject.h |  4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Include/dictobject.h b/Include/dictobject.h
--- a/Include/dictobject.h
+++ b/Include/dictobject.h
@@ -50,8 +50,10 @@
 
 PyAPI_FUNC(PyObject *) PyDict_New(void);
 PyAPI_FUNC(PyObject *) PyDict_GetItem(PyObject *mp, PyObject *key);
+#ifndef Py_LIMITED_API
 PyAPI_FUNC(PyObject *) _PyDict_GetItem_KnownHash(PyObject *mp, PyObject *key,
                                        Py_hash_t hash);
+#endif
 PyAPI_FUNC(PyObject *) PyDict_GetItemWithError(PyObject *mp, PyObject *key);
 PyAPI_FUNC(PyObject *) _PyDict_GetItemIdWithError(PyObject *dp,
                                                   struct _Py_Identifier *key);
@@ -60,8 +62,10 @@
     PyObject *mp, PyObject *key, PyObject *defaultobj);
 #endif
 PyAPI_FUNC(int) PyDict_SetItem(PyObject *mp, PyObject *key, PyObject *item);
+#ifndef Py_LIMITED_API
 PyAPI_FUNC(int) _PyDict_SetItem_KnownHash(PyObject *mp, PyObject *key,
                                           PyObject *item, Py_hash_t hash);
+#endif
 PyAPI_FUNC(int) PyDict_DelItem(PyObject *mp, PyObject *key);
 PyAPI_FUNC(void) PyDict_Clear(PyObject *mp);
 PyAPI_FUNC(int) PyDict_Next(

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May  4 02:19:05 2014
From: python-checkins at python.org (benjamin.peterson)
Date: Sun,  4 May 2014 02:19:05 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_improve_test_c?=
 =?utf-8?q?overage_of_filecmp_=28closes_=2321357=29?=
Message-ID: <3gLnqs31qxz7LlH@mail.python.org>

http://hg.python.org/cpython/rev/57dacba9febf
changeset:   90548:57dacba9febf
branch:      3.4
parent:      90542:725cb631699a
user:        Benjamin Peterson <benjamin at python.org>
date:        Sat May 03 20:07:16 2014 -0400
summary:
  improve test coverage of filecmp (closes #21357)

Patch by Diana Clarke.

files:
  Lib/test/test_filecmp.py |  80 ++++++++++++++++++++++++++-
  1 files changed, 75 insertions(+), 5 deletions(-)


diff --git a/Lib/test/test_filecmp.py b/Lib/test/test_filecmp.py
--- a/Lib/test/test_filecmp.py
+++ b/Lib/test/test_filecmp.py
@@ -1,7 +1,12 @@
-import os, filecmp, shutil, tempfile
+import filecmp
+import os
+import shutil
+import tempfile
 import unittest
+
 from test import support
 
+
 class FileCompareTestCase(unittest.TestCase):
     def setUp(self):
         self.name = support.TESTFN
@@ -120,30 +125,95 @@
         else:
             self.assertEqual([d.left_list, d.right_list],[['file'], ['file']])
         self.assertEqual(d.common, ['file'])
-        self.assertTrue(d.left_only == d.right_only == [])
+        self.assertEqual(d.left_only, [])
+        self.assertEqual(d.right_only, [])
         self.assertEqual(d.same_files, ['file'])
         self.assertEqual(d.diff_files, [])
+        expected_report = [
+            "diff {} {}".format(self.dir, self.dir_same),
+            "Identical files : ['file']",
+        ]
+        self._assert_report(d.report, expected_report)
 
-        # Check attributes for comparison of two different directories
+        # Check attributes for comparison of two different directories (right)
         left_dir, right_dir = self.dir, self.dir_diff
         d = filecmp.dircmp(left_dir, right_dir)
         self.assertEqual(d.left, left_dir)
         self.assertEqual(d.right, right_dir)
         self.assertEqual(d.left_list, ['file'])
-        self.assertTrue(d.right_list == ['file', 'file2'])
+        self.assertEqual(d.right_list, ['file', 'file2'])
         self.assertEqual(d.common, ['file'])
         self.assertEqual(d.left_only, [])
         self.assertEqual(d.right_only, ['file2'])
         self.assertEqual(d.same_files, ['file'])
         self.assertEqual(d.diff_files, [])
+        expected_report = [
+            "diff {} {}".format(self.dir, self.dir_diff),
+            "Only in {} : ['file2']".format(self.dir_diff),
+            "Identical files : ['file']",
+        ]
+        self._assert_report(d.report, expected_report)
+
+        # Check attributes for comparison of two different directories (left)
+        left_dir, right_dir = self.dir, self.dir_diff
+        shutil.move(
+            os.path.join(self.dir_diff, 'file2'),
+            os.path.join(self.dir, 'file2')
+        )
+        d = filecmp.dircmp(left_dir, right_dir)
+        self.assertEqual(d.left, left_dir)
+        self.assertEqual(d.right, right_dir)
+        self.assertEqual(d.left_list, ['file', 'file2'])
+        self.assertEqual(d.right_list, ['file'])
+        self.assertEqual(d.common, ['file'])
+        self.assertEqual(d.left_only, ['file2'])
+        self.assertEqual(d.right_only, [])
+        self.assertEqual(d.same_files, ['file'])
+        self.assertEqual(d.diff_files, [])
+        expected_report = [
+            "diff {} {}".format(self.dir, self.dir_diff),
+            "Only in {} : ['file2']".format(self.dir),
+            "Identical files : ['file']",
+        ]
+        self._assert_report(d.report, expected_report)
 
         # Add different file2
-        output = open(os.path.join(self.dir, 'file2'), 'w')
+        output = open(os.path.join(self.dir_diff, 'file2'), 'w')
         output.write('Different contents.\n')
         output.close()
         d = filecmp.dircmp(self.dir, self.dir_diff)
         self.assertEqual(d.same_files, ['file'])
         self.assertEqual(d.diff_files, ['file2'])
+        expected_report = [
+            "diff {} {}".format(self.dir, self.dir_diff),
+            "Identical files : ['file']",
+            "Differing files : ['file2']",
+        ]
+        self._assert_report(d.report, expected_report)
+
+    def test_report_partial_closure(self):
+        left_dir, right_dir = self.dir, self.dir_same
+        d = filecmp.dircmp(left_dir, right_dir)
+        expected_report = [
+            "diff {} {}".format(self.dir, self.dir_same),
+            "Identical files : ['file']",
+        ]
+        self._assert_report(d.report_partial_closure, expected_report)
+
+    def test_report_full_closure(self):
+        left_dir, right_dir = self.dir, self.dir_same
+        d = filecmp.dircmp(left_dir, right_dir)
+        expected_report = [
+            "diff {} {}".format(self.dir, self.dir_same),
+            "Identical files : ['file']",
+        ]
+        self._assert_report(d.report_full_closure, expected_report)
+
+    def _assert_report(self, dircmp_report, expected_report_lines):
+        with support.captured_stdout() as stdout:
+            dircmp_report()
+            report_lines = stdout.getvalue().strip().split('\n')
+            self.assertEqual(report_lines, expected_report_lines)
 
 
 def test_main():

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May  4 02:19:06 2014
From: python-checkins at python.org (benjamin.peterson)
Date: Sun,  4 May 2014 02:19:06 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjEzNTcp?=
Message-ID: <3gLnqt5Nzbz7Llb@mail.python.org>

http://hg.python.org/cpython/rev/c597654a7fd8
changeset:   90549:c597654a7fd8
parent:      90547:e947dc6a33d9
parent:      90548:57dacba9febf
user:        Benjamin Peterson <benjamin at python.org>
date:        Sat May 03 20:16:59 2014 -0400
summary:
  merge 3.4 (#21357)

files:
  Lib/test/test_filecmp.py |  80 ++++++++++++++++++++++++++-
  1 files changed, 75 insertions(+), 5 deletions(-)


diff --git a/Lib/test/test_filecmp.py b/Lib/test/test_filecmp.py
--- a/Lib/test/test_filecmp.py
+++ b/Lib/test/test_filecmp.py
@@ -1,7 +1,12 @@
-import os, filecmp, shutil, tempfile
+import filecmp
+import os
+import shutil
+import tempfile
 import unittest
+
 from test import support
 
+
 class FileCompareTestCase(unittest.TestCase):
     def setUp(self):
         self.name = support.TESTFN
@@ -120,30 +125,95 @@
         else:
             self.assertEqual([d.left_list, d.right_list],[['file'], ['file']])
         self.assertEqual(d.common, ['file'])
-        self.assertTrue(d.left_only == d.right_only == [])
+        self.assertEqual(d.left_only, [])
+        self.assertEqual(d.right_only, [])
         self.assertEqual(d.same_files, ['file'])
         self.assertEqual(d.diff_files, [])
+        expected_report = [
+            "diff {} {}".format(self.dir, self.dir_same),
+            "Identical files : ['file']",
+        ]
+        self._assert_report(d.report, expected_report)
 
-        # Check attributes for comparison of two different directories
+        # Check attributes for comparison of two different directories (right)
         left_dir, right_dir = self.dir, self.dir_diff
         d = filecmp.dircmp(left_dir, right_dir)
         self.assertEqual(d.left, left_dir)
         self.assertEqual(d.right, right_dir)
         self.assertEqual(d.left_list, ['file'])
-        self.assertTrue(d.right_list == ['file', 'file2'])
+        self.assertEqual(d.right_list, ['file', 'file2'])
         self.assertEqual(d.common, ['file'])
         self.assertEqual(d.left_only, [])
         self.assertEqual(d.right_only, ['file2'])
         self.assertEqual(d.same_files, ['file'])
         self.assertEqual(d.diff_files, [])
+        expected_report = [
+            "diff {} {}".format(self.dir, self.dir_diff),
+            "Only in {} : ['file2']".format(self.dir_diff),
+            "Identical files : ['file']",
+        ]
+        self._assert_report(d.report, expected_report)
+
+        # Check attributes for comparison of two different directories (left)
+        left_dir, right_dir = self.dir, self.dir_diff
+        shutil.move(
+            os.path.join(self.dir_diff, 'file2'),
+            os.path.join(self.dir, 'file2')
+        )
+        d = filecmp.dircmp(left_dir, right_dir)
+        self.assertEqual(d.left, left_dir)
+        self.assertEqual(d.right, right_dir)
+        self.assertEqual(d.left_list, ['file', 'file2'])
+        self.assertEqual(d.right_list, ['file'])
+        self.assertEqual(d.common, ['file'])
+        self.assertEqual(d.left_only, ['file2'])
+        self.assertEqual(d.right_only, [])
+        self.assertEqual(d.same_files, ['file'])
+        self.assertEqual(d.diff_files, [])
+        expected_report = [
+            "diff {} {}".format(self.dir, self.dir_diff),
+            "Only in {} : ['file2']".format(self.dir),
+            "Identical files : ['file']",
+        ]
+        self._assert_report(d.report, expected_report)
 
         # Add different file2
-        output = open(os.path.join(self.dir, 'file2'), 'w')
+        output = open(os.path.join(self.dir_diff, 'file2'), 'w')
         output.write('Different contents.\n')
         output.close()
         d = filecmp.dircmp(self.dir, self.dir_diff)
         self.assertEqual(d.same_files, ['file'])
         self.assertEqual(d.diff_files, ['file2'])
+        expected_report = [
+            "diff {} {}".format(self.dir, self.dir_diff),
+            "Identical files : ['file']",
+            "Differing files : ['file2']",
+        ]
+        self._assert_report(d.report, expected_report)
+
+    def test_report_partial_closure(self):
+        left_dir, right_dir = self.dir, self.dir_same
+        d = filecmp.dircmp(left_dir, right_dir)
+        expected_report = [
+            "diff {} {}".format(self.dir, self.dir_same),
+            "Identical files : ['file']",
+        ]
+        self._assert_report(d.report_partial_closure, expected_report)
+
+    def test_report_full_closure(self):
+        left_dir, right_dir = self.dir, self.dir_same
+        d = filecmp.dircmp(left_dir, right_dir)
+        expected_report = [
+            "diff {} {}".format(self.dir, self.dir_same),
+            "Identical files : ['file']",
+        ]
+        self._assert_report(d.report_full_closure, expected_report)
+
+    def _assert_report(self, dircmp_report, expected_report_lines):
+        with support.captured_stdout() as stdout:
+            dircmp_report()
+            report_lines = stdout.getvalue().strip().split('\n')
+            self.assertEqual(report_lines, expected_report_lines)
 
 
 def test_main():

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May  4 02:19:08 2014
From: python-checkins at python.org (benjamin.peterson)
Date: Sun,  4 May 2014 02:19:08 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_use_with_block?=
 =?utf-8?q?s_to_make_sure_files_are_closed?=
Message-ID: <3gLnqw013vz7Llv@mail.python.org>

http://hg.python.org/cpython/rev/b8f5c8d07365
changeset:   90550:b8f5c8d07365
branch:      3.4
parent:      90548:57dacba9febf
user:        Benjamin Peterson <benjamin at python.org>
date:        Sat May 03 20:18:50 2014 -0400
summary:
  use with blocks to make sure files are closed

files:
  Lib/test/test_filecmp.py |  30 +++++++++++----------------
  1 files changed, 12 insertions(+), 18 deletions(-)


diff --git a/Lib/test/test_filecmp.py b/Lib/test/test_filecmp.py
--- a/Lib/test/test_filecmp.py
+++ b/Lib/test/test_filecmp.py
@@ -14,13 +14,11 @@
         self.name_diff = support.TESTFN + '-diff'
         data = 'Contents of file go here.\n'
         for name in [self.name, self.name_same, self.name_diff]:
-            output = open(name, 'w')
-            output.write(data)
-            output.close()
+            with open(name, 'w') as output:
+                output.write(data)
 
-        output = open(self.name_diff, 'a+')
-        output.write('An extra line.\n')
-        output.close()
+        with open(self.name_diff, 'a+') as output:
+            output.write('An extra line.\n')
         self.dir = tempfile.gettempdir()
 
     def tearDown(self):
@@ -71,13 +69,11 @@
                 fn = 'FiLe'     # Verify case-insensitive comparison
             else:
                 fn = 'file'
-            output = open(os.path.join(dir, fn), 'w')
-            output.write(data)
-            output.close()
+            with open(os.path.join(dir, fn), 'w') as output:
+                output.write(data)
 
-        output = open(os.path.join(self.dir_diff, 'file2'), 'w')
-        output.write('An extra file.\n')
-        output.close()
+        with open(os.path.join(self.dir_diff, 'file2'), 'w') as output:
+            output.write('An extra file.\n')
 
     def tearDown(self):
         for dir in (self.dir, self.dir_same, self.dir_diff):
@@ -104,9 +100,8 @@
                         "Comparing directory to same fails")
 
         # Add different file2
-        output = open(os.path.join(self.dir, 'file2'), 'w')
-        output.write('Different contents.\n')
-        output.close()
+        with open(os.path.join(self.dir, 'file2'), 'w') as output:
+            output.write('Different contents.\n')
 
         self.assertFalse(filecmp.cmpfiles(self.dir, self.dir_same,
                                      ['file', 'file2']) ==
@@ -178,9 +173,8 @@
         self._assert_report(d.report, expected_report)
 
         # Add different file2
-        output = open(os.path.join(self.dir_diff, 'file2'), 'w')
-        output.write('Different contents.\n')
-        output.close()
+        with open(os.path.join(self.dir_diff, 'file2'), 'w') as output:
+            output.write('Different contents.\n')
         d = filecmp.dircmp(self.dir, self.dir_diff)
         self.assertEqual(d.same_files, ['file'])
         self.assertEqual(d.diff_files, ['file2'])

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May  4 02:19:09 2014
From: python-checkins at python.org (benjamin.peterson)
Date: Sun,  4 May 2014 02:19:09 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?b?KTogbWVyZ2UgMy40?=
Message-ID: <3gLnqx1Mrlz7Lkc@mail.python.org>

http://hg.python.org/cpython/rev/3fe5798a6aee
changeset:   90551:3fe5798a6aee
parent:      90549:c597654a7fd8
parent:      90550:b8f5c8d07365
user:        Benjamin Peterson <benjamin at python.org>
date:        Sat May 03 20:18:56 2014 -0400
summary:
  merge 3.4

files:
  Lib/test/test_filecmp.py |  30 +++++++++++----------------
  1 files changed, 12 insertions(+), 18 deletions(-)


diff --git a/Lib/test/test_filecmp.py b/Lib/test/test_filecmp.py
--- a/Lib/test/test_filecmp.py
+++ b/Lib/test/test_filecmp.py
@@ -14,13 +14,11 @@
         self.name_diff = support.TESTFN + '-diff'
         data = 'Contents of file go here.\n'
         for name in [self.name, self.name_same, self.name_diff]:
-            output = open(name, 'w')
-            output.write(data)
-            output.close()
+            with open(name, 'w') as output:
+                output.write(data)
 
-        output = open(self.name_diff, 'a+')
-        output.write('An extra line.\n')
-        output.close()
+        with open(self.name_diff, 'a+') as output:
+            output.write('An extra line.\n')
         self.dir = tempfile.gettempdir()
 
     def tearDown(self):
@@ -71,13 +69,11 @@
                 fn = 'FiLe'     # Verify case-insensitive comparison
             else:
                 fn = 'file'
-            output = open(os.path.join(dir, fn), 'w')
-            output.write(data)
-            output.close()
+            with open(os.path.join(dir, fn), 'w') as output:
+                output.write(data)
 
-        output = open(os.path.join(self.dir_diff, 'file2'), 'w')
-        output.write('An extra file.\n')
-        output.close()
+        with open(os.path.join(self.dir_diff, 'file2'), 'w') as output:
+            output.write('An extra file.\n')
 
     def tearDown(self):
         for dir in (self.dir, self.dir_same, self.dir_diff):
@@ -104,9 +100,8 @@
                         "Comparing directory to same fails")
 
         # Add different file2
-        output = open(os.path.join(self.dir, 'file2'), 'w')
-        output.write('Different contents.\n')
-        output.close()
+        with open(os.path.join(self.dir, 'file2'), 'w') as output:
+            output.write('Different contents.\n')
 
         self.assertFalse(filecmp.cmpfiles(self.dir, self.dir_same,
                                      ['file', 'file2']) ==
@@ -178,9 +173,8 @@
         self._assert_report(d.report, expected_report)
 
         # Add different file2
-        output = open(os.path.join(self.dir_diff, 'file2'), 'w')
-        output.write('Different contents.\n')
-        output.close()
+        with open(os.path.join(self.dir_diff, 'file2'), 'w') as output:
+            output.write('Different contents.\n')
         d = filecmp.dircmp(self.dir, self.dir_diff)
         self.assertEqual(d.same_files, ['file'])
         self.assertEqual(d.diff_files, ['file2'])

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May  4 02:22:38 2014
From: python-checkins at python.org (benjamin.peterson)
Date: Sun,  4 May 2014 02:22:38 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_improve_idioms_=28closes_?=
	=?utf-8?q?=2320642=29?=
Message-ID: <3gLnvy0Mrsz7LjZ@mail.python.org>

http://hg.python.org/cpython/rev/0df3004581fe
changeset:   90552:0df3004581fe
user:        Benjamin Peterson <benjamin at python.org>
date:        Sat May 03 20:22:00 2014 -0400
summary:
  improve idioms (closes #20642)

Patch by Claudiu Popa.

files:
  Lib/copy.py |  8 +++-----
  1 files changed, 3 insertions(+), 5 deletions(-)


diff --git a/Lib/copy.py b/Lib/copy.py
--- a/Lib/copy.py
+++ b/Lib/copy.py
@@ -221,17 +221,15 @@
 d[list] = _deepcopy_list
 
 def _deepcopy_tuple(x, memo):
-    y = []
-    for a in x:
-        y.append(deepcopy(a, memo))
+    y = [deepcopy(a, memo) for a in x]
     # We're not going to put the tuple in the memo, but it's still important we
     # check for it, in case the tuple contains recursive mutable structures.
     try:
         return memo[id(x)]
     except KeyError:
         pass
-    for i in range(len(x)):
-        if x[i] is not y[i]:
+    for k, j in zip(x, y):
+        if k is not j:
             y = tuple(y)
             break
     else:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May  4 03:36:55 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Sun,  4 May 2014 03:36:55 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Simplify_and_speedup_the_i?=
 =?utf-8?q?nternals_of_the_heapq_module=2E?=
Message-ID: <3gLqYg3ZTmz7Ljr@mail.python.org>

http://hg.python.org/cpython/rev/95e903c17047
changeset:   90553:95e903c17047
user:        Raymond Hettinger <python at rcn.com>
date:        Sat May 03 18:36:48 2014 -0700
summary:
  Simplify and speedup the internals of the heapq module.

files:
  Modules/_heapqmodule.c |  111 +++++++++-------------------
  1 files changed, 38 insertions(+), 73 deletions(-)


diff --git a/Modules/_heapqmodule.c b/Modules/_heapqmodule.c
--- a/Modules/_heapqmodule.c
+++ b/Modules/_heapqmodule.c
@@ -11,7 +11,7 @@
 static int
 _siftdown(PyListObject *heap, Py_ssize_t startpos, Py_ssize_t pos)
 {
-    PyObject *newitem, *parent, *olditem;
+    PyObject *newitem, *parent;
     int cmp;
     Py_ssize_t parentpos;
     Py_ssize_t size;
@@ -23,39 +23,28 @@
         return -1;
     }
 
-    newitem = PyList_GET_ITEM(heap, pos);
-    Py_INCREF(newitem);
     /* Follow the path to the root, moving parents down until finding
        a place newitem fits. */
+    newitem = PyList_GET_ITEM(heap, pos);
     while (pos > startpos){
         parentpos = (pos - 1) >> 1;
         parent = PyList_GET_ITEM(heap, parentpos);
         cmp = PyObject_RichCompareBool(newitem, parent, Py_LT);
-        if (cmp == -1) {
-            Py_DECREF(newitem);
+        if (cmp == -1)
             return -1;
-        }
         if (size != PyList_GET_SIZE(heap)) {
-            Py_DECREF(newitem);
             PyErr_SetString(PyExc_RuntimeError,
                             "list changed size during iteration");
             return -1;
         }
         if (cmp == 0)
             break;
-        Py_INCREF(parent);
-        olditem = PyList_GET_ITEM(heap, pos);
+        parent = PyList_GET_ITEM(heap, parentpos);
+        newitem = PyList_GET_ITEM(heap, pos);
+        PyList_SET_ITEM(heap, parentpos, newitem);
         PyList_SET_ITEM(heap, pos, parent);
-        Py_DECREF(olditem);
         pos = parentpos;
-        if (size != PyList_GET_SIZE(heap)) {
-            PyErr_SetString(PyExc_RuntimeError,
-                            "list changed size during iteration");
-            return -1;
-        }
     }
-    Py_DECREF(PyList_GET_ITEM(heap, pos));
-    PyList_SET_ITEM(heap, pos, newitem);
     return 0;
 }
 
@@ -63,20 +52,16 @@
 _siftup(PyListObject *heap, Py_ssize_t pos)
 {
     Py_ssize_t startpos, endpos, childpos, rightpos, limit;
+    PyObject *tmp1, *tmp2;
     int cmp;
-    PyObject *newitem, *tmp, *olditem;
-    Py_ssize_t size;
 
     assert(PyList_Check(heap));
-    size = PyList_GET_SIZE(heap);
-    endpos = size;
+    endpos = PyList_GET_SIZE(heap);
     startpos = pos;
     if (pos >= endpos) {
         PyErr_SetString(PyExc_IndexError, "index out of range");
         return -1;
     }
-    newitem = PyList_GET_ITEM(heap, pos);
-    Py_INCREF(newitem);
 
     /* Bubble up the smaller child until hitting a leaf. */
     limit = endpos / 2;          /* smallest pos that has no child */
@@ -89,37 +74,24 @@
                 PyList_GET_ITEM(heap, childpos),
                 PyList_GET_ITEM(heap, rightpos),
                 Py_LT);
-            if (cmp == -1) {
-                Py_DECREF(newitem);
+            if (cmp == -1)
+                return -1;
+            if (cmp == 0)
+                childpos = rightpos;
+            if (endpos != PyList_GET_SIZE(heap)) {
+                PyErr_SetString(PyExc_RuntimeError,
+                                "list changed size during iteration");
                 return -1;
             }
-            if (cmp == 0)
-                childpos = rightpos;
-        }
-        if (size != PyList_GET_SIZE(heap)) {
-            Py_DECREF(newitem);
-            PyErr_SetString(PyExc_RuntimeError,
-                            "list changed size during iteration");
-            return -1;
         }
         /* Move the smaller child up. */
-        tmp = PyList_GET_ITEM(heap, childpos);
-        Py_INCREF(tmp);
-        olditem = PyList_GET_ITEM(heap, pos);
-        PyList_SET_ITEM(heap, pos, tmp);
-        Py_DECREF(olditem);
+        tmp1 = PyList_GET_ITEM(heap, childpos);
+        tmp2 = PyList_GET_ITEM(heap, pos);
+        PyList_SET_ITEM(heap, childpos, tmp2);
+        PyList_SET_ITEM(heap, pos, tmp1);
         pos = childpos;
-        if (size != PyList_GET_SIZE(heap)) {
-            PyErr_SetString(PyExc_RuntimeError,
-                            "list changed size during iteration");
-            return -1;
-        }
     }
-
-    /* The leaf at pos is empty now.  Put newitem there, and bubble
-       it up to its final resting place (by sifting its parents down). */
-    Py_DECREF(PyList_GET_ITEM(heap, pos));
-    PyList_SET_ITEM(heap, pos, newitem);
+    /* Bubble it up to its final resting place (by sifting its parents down). */
     return _siftdown(heap, startpos, pos);
 }
 
@@ -392,27 +364,23 @@
         return -1;
     }
 
-    newitem = PyList_GET_ITEM(heap, pos);
-    Py_INCREF(newitem);
     /* Follow the path to the root, moving parents down until finding
        a place newitem fits. */
+    newitem = PyList_GET_ITEM(heap, pos);
     while (pos > startpos){
         parentpos = (pos - 1) >> 1;
         parent = PyList_GET_ITEM(heap, parentpos);
         cmp = PyObject_RichCompareBool(parent, newitem, Py_LT);
-        if (cmp == -1) {
-            Py_DECREF(newitem);
+        if (cmp == -1)
             return -1;
-        }
         if (cmp == 0)
             break;
-        Py_INCREF(parent);
-        Py_DECREF(PyList_GET_ITEM(heap, pos));
+        parent = PyList_GET_ITEM(heap, parentpos);
+        newitem = PyList_GET_ITEM(heap, pos);
+        PyList_SET_ITEM(heap, parentpos, newitem);
         PyList_SET_ITEM(heap, pos, parent);
         pos = parentpos;
     }
-    Py_DECREF(PyList_GET_ITEM(heap, pos));
-    PyList_SET_ITEM(heap, pos, newitem);
     return 0;
 }
 
@@ -420,8 +388,8 @@
 _siftupmax(PyListObject *heap, Py_ssize_t pos)
 {
     Py_ssize_t startpos, endpos, childpos, rightpos, limit;
+    PyObject *tmp1, *tmp2;
     int cmp;
-    PyObject *newitem, *tmp;
 
     assert(PyList_Check(heap));
     endpos = PyList_GET_SIZE(heap);
@@ -430,8 +398,6 @@
         PyErr_SetString(PyExc_IndexError, "index out of range");
         return -1;
     }
-    newitem = PyList_GET_ITEM(heap, pos);
-    Py_INCREF(newitem);
 
     /* Bubble up the smaller child until hitting a leaf. */
     limit = endpos / 2;          /* smallest pos that has no child */
@@ -444,25 +410,24 @@
                 PyList_GET_ITEM(heap, rightpos),
                 PyList_GET_ITEM(heap, childpos),
                 Py_LT);
-            if (cmp == -1) {
-                Py_DECREF(newitem);
+            if (cmp == -1)
+                return -1;
+            if (cmp == 0)
+                childpos = rightpos;
+            if (endpos != PyList_GET_SIZE(heap)) {
+                PyErr_SetString(PyExc_RuntimeError,
+                                "list changed size during iteration");
                 return -1;
             }
-            if (cmp == 0)
-                childpos = rightpos;
         }
         /* Move the smaller child up. */
-        tmp = PyList_GET_ITEM(heap, childpos);
-        Py_INCREF(tmp);
-        Py_DECREF(PyList_GET_ITEM(heap, pos));
-        PyList_SET_ITEM(heap, pos, tmp);
+        tmp1 = PyList_GET_ITEM(heap, childpos);
+        tmp2 = PyList_GET_ITEM(heap, pos);
+        PyList_SET_ITEM(heap, childpos, tmp2);
+        PyList_SET_ITEM(heap, pos, tmp1);
         pos = childpos;
     }
-
-    /* The leaf at pos is empty now.  Put newitem there, and bubble
-       it up to its final resting place (by sifting its parents down). */
-    Py_DECREF(PyList_GET_ITEM(heap, pos));
-    PyList_SET_ITEM(heap, pos, newitem);
+    /* Bubble it up to its final resting place (by sifting its parents down). */
     return _siftdownmax(heap, startpos, pos);
 }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May  4 03:46:02 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Sun,  4 May 2014 03:46:02 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Neaten-up_a_bit_add_add_mi?=
	=?utf-8?q?ssing_size_change_check=2E?=
Message-ID: <3gLqmB4pr8z7Ljt@mail.python.org>

http://hg.python.org/cpython/rev/2b88baf64151
changeset:   90554:2b88baf64151
user:        Raymond Hettinger <python at rcn.com>
date:        Sat May 03 18:45:54 2014 -0700
summary:
  Neaten-up a bit add add missing size change check.

files:
  Modules/_heapqmodule.c |  17 +++++++++++------
  1 files changed, 11 insertions(+), 6 deletions(-)


diff --git a/Modules/_heapqmodule.c b/Modules/_heapqmodule.c
--- a/Modules/_heapqmodule.c
+++ b/Modules/_heapqmodule.c
@@ -12,9 +12,8 @@
 _siftdown(PyListObject *heap, Py_ssize_t startpos, Py_ssize_t pos)
 {
     PyObject *newitem, *parent;
+    Py_ssize_t parentpos, size;
     int cmp;
-    Py_ssize_t parentpos;
-    Py_ssize_t size;
 
     assert(PyList_Check(heap));
     size = PyList_GET_SIZE(heap);
@@ -26,7 +25,7 @@
     /* Follow the path to the root, moving parents down until finding
        a place newitem fits. */
     newitem = PyList_GET_ITEM(heap, pos);
-    while (pos > startpos){
+    while (pos > startpos) {
         parentpos = (pos - 1) >> 1;
         parent = PyList_GET_ITEM(heap, parentpos);
         cmp = PyObject_RichCompareBool(newitem, parent, Py_LT);
@@ -355,11 +354,12 @@
 _siftdownmax(PyListObject *heap, Py_ssize_t startpos, Py_ssize_t pos)
 {
     PyObject *newitem, *parent;
+    Py_ssize_t parentpos, size;
     int cmp;
-    Py_ssize_t parentpos;
 
     assert(PyList_Check(heap));
-    if (pos >= PyList_GET_SIZE(heap)) {
+    size = PyList_GET_SIZE(heap);
+    if (pos >= size) {
         PyErr_SetString(PyExc_IndexError, "index out of range");
         return -1;
     }
@@ -367,12 +367,17 @@
     /* Follow the path to the root, moving parents down until finding
        a place newitem fits. */
     newitem = PyList_GET_ITEM(heap, pos);
-    while (pos > startpos){
+    while (pos > startpos) {
         parentpos = (pos - 1) >> 1;
         parent = PyList_GET_ITEM(heap, parentpos);
         cmp = PyObject_RichCompareBool(parent, newitem, Py_LT);
         if (cmp == -1)
             return -1;
+        if (size != PyList_GET_SIZE(heap)) {
+            PyErr_SetString(PyExc_RuntimeError,
+                            "list changed size during iteration");
+            return -1;
+        }
         if (cmp == 0)
             break;
         parent = PyList_GET_ITEM(heap, parentpos);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May  4 04:06:40 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Sun,  4 May 2014 04:06:40 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321421=3A__Add_=5F?=
 =?utf-8?q?=5Fslots=5F=5F_to_the_MappingViews_ABCs=2E?=
Message-ID: <3gLrD03FKpz7Lkx@mail.python.org>

http://hg.python.org/cpython/rev/2c6a231e2c1e
changeset:   90555:2c6a231e2c1e
user:        Raymond Hettinger <python at rcn.com>
date:        Sat May 03 19:06:32 2014 -0700
summary:
  Issue #21421:  Add __slots__ to the MappingViews ABCs.

files:
  Lib/_collections_abc.py |  8 ++++++++
  Misc/NEWS               |  3 +++
  2 files changed, 11 insertions(+), 0 deletions(-)


diff --git a/Lib/_collections_abc.py b/Lib/_collections_abc.py
--- a/Lib/_collections_abc.py
+++ b/Lib/_collections_abc.py
@@ -440,6 +440,8 @@
 
 class MappingView(Sized):
 
+    __slots__ = '_mapping',
+
     def __init__(self, mapping):
         self._mapping = mapping
 
@@ -452,6 +454,8 @@
 
 class KeysView(MappingView, Set):
 
+    __slots__ = ()
+
     @classmethod
     def _from_iterable(self, it):
         return set(it)
@@ -467,6 +471,8 @@
 
 class ItemsView(MappingView, Set):
 
+    __slots__ = ()
+
     @classmethod
     def _from_iterable(self, it):
         return set(it)
@@ -489,6 +495,8 @@
 
 class ValuesView(MappingView):
 
+    __slots__ = ()
+
     def __contains__(self, value):
         for key in self._mapping:
             if value == self._mapping[key]:
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -73,6 +73,9 @@
   Decimal.quantize() method in the Python version.  It had never been
   present in the C version.
 
+- Issue #21421: Add __slots__ to the MappingViews ABC.
+  Patch by Josh Rosenberg.
+
 - Issue #21101: Eliminate double hashing in the C speed-up code for
   collections.Counter().
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May  4 04:25:58 2014
From: python-checkins at python.org (ned.deily)
Date: Sun,  4 May 2014 04:25:58 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE4NjA0?=
 =?utf-8?q?=3A_Skip_the_Tk_instantiation_test_on_OS_X_because_it_can?=
Message-ID: <3gLrfG5wMCz7Lkx@mail.python.org>

http://hg.python.org/cpython/rev/7ecb6e4b1077
changeset:   90556:7ecb6e4b1077
branch:      3.4
parent:      90550:b8f5c8d07365
user:        Ned Deily <nad at acm.org>
date:        Sat May 03 19:24:05 2014 -0700
summary:
  Issue #18604: Skip the Tk instantiation test on OS X because it can
cause GUI tests to segfault in Cocoa Tk when run under regrtest -j
(multiple threads running subprocesses).

files:
  Lib/test/support/__init__.py |  4 +++-
  1 files changed, 3 insertions(+), 1 deletions(-)


diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py
--- a/Lib/test/support/__init__.py
+++ b/Lib/test/support/__init__.py
@@ -434,7 +434,9 @@
                 reason = "cannot run without OS X gui process"
 
     # check on every platform whether tkinter can actually do anything
-    if not reason:
+    # but skip the test on OS X because it can cause segfaults in Cocoa Tk
+    # when running regrtest with the -j option (multiple threads/subprocesses)
+    if (not reason) and (sys.platform != 'darwin'):
         try:
             from tkinter import Tk
             root = Tk()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May  4 04:26:00 2014
From: python-checkins at python.org (ned.deily)
Date: Sun,  4 May 2014 04:26:00 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Issue_=2318604=3A_merge_from_3=2E4?=
Message-ID: <3gLrfJ0Qqxz7Lkx@mail.python.org>

http://hg.python.org/cpython/rev/7f6d9990a9b1
changeset:   90557:7f6d9990a9b1
parent:      90555:2c6a231e2c1e
parent:      90556:7ecb6e4b1077
user:        Ned Deily <nad at acm.org>
date:        Sat May 03 19:25:34 2014 -0700
summary:
  Issue #18604: merge from 3.4

files:
  Lib/test/support/__init__.py |  4 +++-
  1 files changed, 3 insertions(+), 1 deletions(-)


diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py
--- a/Lib/test/support/__init__.py
+++ b/Lib/test/support/__init__.py
@@ -434,7 +434,9 @@
                 reason = "cannot run without OS X gui process"
 
     # check on every platform whether tkinter can actually do anything
-    if not reason:
+    # but skip the test on OS X because it can cause segfaults in Cocoa Tk
+    # when running regrtest with the -j option (multiple threads/subprocesses)
+    if (not reason) and (sys.platform != 'darwin'):
         try:
             from tkinter import Tk
             root = Tk()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May  4 06:58:55 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Sun,  4 May 2014 06:58:55 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319414=3A_Have_the?=
 =?utf-8?q?_OrderedDict_mark_deleted_links_as_unusable=2E?=
Message-ID: <3gLw2l43VSz7Ljr@mail.python.org>

http://hg.python.org/cpython/rev/a3c345ba3563
changeset:   90558:a3c345ba3563
user:        Raymond Hettinger <python at rcn.com>
date:        Sat May 03 21:58:45 2014 -0700
summary:
  Issue #19414: Have the OrderedDict mark deleted links as unusable.
This gives an earlier and more visible failure if a link is deleted
during iteration.

files:
  Lib/collections/__init__.py  |   2 ++
  Lib/test/test_collections.py |  10 ++++++++++
  Misc/NEWS                    |   3 +++
  3 files changed, 15 insertions(+), 0 deletions(-)


diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py
--- a/Lib/collections/__init__.py
+++ b/Lib/collections/__init__.py
@@ -96,6 +96,8 @@
         link_next = link.next
         link_prev.next = link_next
         link_next.prev = link_prev
+        link.prev = None
+        link.next = None
 
     def __iter__(self):
         'od.__iter__() <==> iter(od)'
diff --git a/Lib/test/test_collections.py b/Lib/test/test_collections.py
--- a/Lib/test/test_collections.py
+++ b/Lib/test/test_collections.py
@@ -1193,6 +1193,16 @@
                          [t[1] for t in reversed(pairs)])
         self.assertEqual(list(reversed(od.items())), list(reversed(pairs)))
 
+    def test_detect_deletion_during_iteration(self):
+        od = OrderedDict.fromkeys('abc')
+        it = iter(od)
+        key = next(it)
+        del od[key]
+        with self.assertRaises(Exception):
+            # Note, the exact exception raised is not guaranteed
+            # The only guarantee that the next() will not succeed
+            next(it)
+
     def test_popitem(self):
         pairs = [('c', 1), ('b', 2), ('a', 3), ('d', 4), ('e', 5), ('f', 6)]
         shuffle(pairs)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -73,6 +73,9 @@
   Decimal.quantize() method in the Python version.  It had never been
   present in the C version.
 
+- Issue #19414: Have the OrderedDict mark deleted links as unusable.
+  This gives an early failure if the link is deleted during iteration.
+
 - Issue #21421: Add __slots__ to the MappingViews ABC.
   Patch by Josh Rosenberg.
 

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Sun May  4 10:03:37 2014
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sun, 04 May 2014 10:03:37 +0200
Subject: [Python-checkins] Daily reference leaks (0df3004581fe): sum=7
Message-ID: <E1WgrOb-0005TP-Ai@vds2544.sivit.org>

results for 0df3004581fe on branch "default"
--------------------------------------------

test_functools leaked [0, 0, 3] memory blocks, sum=3
test_site leaked [0, 2, 0] references, sum=2
test_site leaked [0, 2, 0] memory blocks, sum=2


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogP1JMoT', '-x']

From python-checkins at python.org  Sun May  4 13:41:32 2014
From: python-checkins at python.org (larry.hastings)
Date: Sun,  4 May 2014 13:41:32 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMDg4?=
 =?utf-8?q?=3A_Bugfix_for_curses=2Ewindow=2Eaddch=28=29_regression_in_3=2E?=
 =?utf-8?q?4=2E0=2E?=
Message-ID: <3gM4zJ1dXpz7LmW@mail.python.org>

http://hg.python.org/cpython/rev/4f26430b03fd
changeset:   90559:4f26430b03fd
branch:      3.4
parent:      90556:7ecb6e4b1077
user:        Larry Hastings <larry at hastings.org>
date:        Sun May 04 04:41:18 2014 -0700
summary:
  Issue #21088: Bugfix for curses.window.addch() regression in 3.4.0.
In porting to Argument Clinic, the first two arguments were reversed.

files:
  Lib/test/test_curses.py |  30 +++++++++++++++++++++++++++++
  Misc/NEWS               |   3 ++
  Modules/_cursesmodule.c |  24 +++++++++++-----------
  3 files changed, 45 insertions(+), 12 deletions(-)


diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py
--- a/Lib/test/test_curses.py
+++ b/Lib/test/test_curses.py
@@ -17,6 +17,7 @@
 
 import unittest
 from test.support import requires, import_module
+import inspect
 requires('curses')
 
 # If either of these don't exist, skip the tests.
@@ -331,6 +332,34 @@
     else:
         raise AssertionError("TypeError not raised")
 
+def test_issue21088(stdscr):
+    #
+    # http://bugs.python.org/issue21088
+    #
+    # the bug:
+    # when converting curses.window.addch to Argument Clinic
+    # the first two parameters were switched.
+
+    # if someday we can represent the signature of addch
+    # we will need to rewrite this test.
+    try:
+        signature = inspect.signature(stdscr.addch)
+        self.assertFalse(signature)
+    except ValueError:
+        # not generating a signature is fine.
+        pass
+
+    # So.  No signature for addch.
+    # But Argument Clinic gave us a human-readable equivalent
+    # as the first line of the docstring.  So we parse that,
+    # and ensure that the parameters appear in the correct order.
+    # Since this is parsing output from Argument Clinic, we can
+    # be reasonably certain the generated parsing code will be
+    # correct too.
+    human_readable_signature = stdscr.addch.__doc__.split("\n")[0]
+    offset = human_readable_signature.find("[y, x,]")
+    assert offset >= 0, ""
+
 def main(stdscr):
     curses.savetty()
     try:
@@ -344,6 +373,7 @@
         test_unget_wch(stdscr)
         test_issue10570()
         test_encoding(stdscr)
+        test_issue21088(stdscr)
     finally:
         curses.resetty()
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -39,6 +39,9 @@
 Library
 -------
 
+- Issue #21088: Bugfix for curses.window.addch() regression in 3.4.0.
+  In porting to Argument Clinic, the first two arguments were reversed.
+
 - Issue #21321: itertools.islice() now releases the reference to the source
   iterator when the slice is exhausted.  Patch by Anton Afanasyev.
 
diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c
--- a/Modules/_cursesmodule.c
+++ b/Modules/_cursesmodule.c
@@ -560,10 +560,10 @@
 curses.window.addch
 
     [
+    y: int
+      Y-coordinate.
     x: int
       X-coordinate.
-    y: int
-      Y-coordinate.
     ]
 
     ch: object
@@ -584,13 +584,13 @@
 [clinic start generated code]*/
 
 PyDoc_STRVAR(curses_window_addch__doc__,
-"addch([x, y,] ch, [attr])\n"
+"addch([y, x,] ch, [attr])\n"
 "Paint character ch at (y, x) with attributes attr.\n"
 "\n"
+"  y\n"
+"    Y-coordinate.\n"
 "  x\n"
 "    X-coordinate.\n"
-"  y\n"
-"    Y-coordinate.\n"
 "  ch\n"
 "    Character to add.\n"
 "  attr\n"
@@ -605,15 +605,15 @@
     {"addch", (PyCFunction)curses_window_addch, METH_VARARGS, curses_window_addch__doc__},
 
 static PyObject *
-curses_window_addch_impl(PyCursesWindowObject *self, int group_left_1, int x, int y, PyObject *ch, int group_right_1, long attr);
+curses_window_addch_impl(PyCursesWindowObject *self, int group_left_1, int y, int x, PyObject *ch, int group_right_1, long attr);
 
 static PyObject *
 curses_window_addch(PyCursesWindowObject *self, PyObject *args)
 {
     PyObject *return_value = NULL;
     int group_left_1 = 0;
+    int y = 0;
     int x = 0;
-    int y = 0;
     PyObject *ch;
     int group_right_1 = 0;
     long attr = 0;
@@ -629,12 +629,12 @@
             group_right_1 = 1;
             break;
         case 3:
-            if (!PyArg_ParseTuple(args, "iiO:addch", &x, &y, &ch))
+            if (!PyArg_ParseTuple(args, "iiO:addch", &y, &x, &ch))
                 goto exit;
             group_left_1 = 1;
             break;
         case 4:
-            if (!PyArg_ParseTuple(args, "iiOl:addch", &x, &y, &ch, &attr))
+            if (!PyArg_ParseTuple(args, "iiOl:addch", &y, &x, &ch, &attr))
                 goto exit;
             group_right_1 = 1;
             group_left_1 = 1;
@@ -643,15 +643,15 @@
             PyErr_SetString(PyExc_TypeError, "curses.window.addch requires 1 to 4 arguments");
             goto exit;
     }
-    return_value = curses_window_addch_impl(self, group_left_1, x, y, ch, group_right_1, attr);
+    return_value = curses_window_addch_impl(self, group_left_1, y, x, ch, group_right_1, attr);
 
 exit:
     return return_value;
 }
 
 static PyObject *
-curses_window_addch_impl(PyCursesWindowObject *self, int group_left_1, int x, int y, PyObject *ch, int group_right_1, long attr)
-/*[clinic end generated code: output=43acb91a5c98f615 input=fe7e3711d5bbf1f6]*/
+curses_window_addch_impl(PyCursesWindowObject *self, int group_left_1, int y, int x, PyObject *ch, int group_right_1, long attr)
+/*[clinic end generated code: output=d4b97cc287010c54 input=5a41efb34a2de338]*/
 {
     PyCursesWindowObject *cwself = (PyCursesWindowObject *)self;
     int coordinates_group = group_left_1;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May  4 13:48:29 2014
From: python-checkins at python.org (larry.hastings)
Date: Sun,  4 May 2014 13:48:29 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Issue_=2321088=3A_Merge_from_3=2E4=2E?=
Message-ID: <3gM57K1NDvz7LkK@mail.python.org>

http://hg.python.org/cpython/rev/3aa5fae8c313
changeset:   90560:3aa5fae8c313
parent:      90558:a3c345ba3563
parent:      90559:4f26430b03fd
user:        Larry Hastings <larry at hastings.org>
date:        Sun May 04 04:45:57 2014 -0700
summary:
  Issue #21088: Merge from 3.4.

files:
  Lib/test/test_curses.py |  30 +++++++++++++++++++++++++++++
  Misc/NEWS               |   3 ++
  Modules/_cursesmodule.c |  24 +++++++++++-----------
  3 files changed, 45 insertions(+), 12 deletions(-)


diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py
--- a/Lib/test/test_curses.py
+++ b/Lib/test/test_curses.py
@@ -17,6 +17,7 @@
 
 import unittest
 from test.support import requires, import_module
+import inspect
 requires('curses')
 
 # If either of these don't exist, skip the tests.
@@ -331,6 +332,34 @@
     else:
         raise AssertionError("TypeError not raised")
 
+def test_issue21088(stdscr):
+    #
+    # http://bugs.python.org/issue21088
+    #
+    # the bug:
+    # when converting curses.window.addch to Argument Clinic
+    # the first two parameters were switched.
+
+    # if someday we can represent the signature of addch
+    # we will need to rewrite this test.
+    try:
+        signature = inspect.signature(stdscr.addch)
+        self.assertFalse(signature)
+    except ValueError:
+        # not generating a signature is fine.
+        pass
+
+    # So.  No signature for addch.
+    # But Argument Clinic gave us a human-readable equivalent
+    # as the first line of the docstring.  So we parse that,
+    # and ensure that the parameters appear in the correct order.
+    # Since this is parsing output from Argument Clinic, we can
+    # be reasonably certain the generated parsing code will be
+    # correct too.
+    human_readable_signature = stdscr.addch.__doc__.split("\n")[0]
+    offset = human_readable_signature.find("[y, x,]")
+    assert offset >= 0, ""
+
 def main(stdscr):
     curses.savetty()
     try:
@@ -344,6 +373,7 @@
         test_unget_wch(stdscr)
         test_issue10570()
         test_encoding(stdscr)
+        test_issue21088(stdscr)
     finally:
         curses.resetty()
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -69,6 +69,9 @@
 Library
 -------
 
+- Issue #21088: Bugfix for curses.window.addch() regression in 3.4.0.
+  In porting to Argument Clinic, the first two arguments were reversed.
+
 - Issue #10650: Remove the non-standard 'watchexp' parameter from the
   Decimal.quantize() method in the Python version.  It had never been
   present in the C version.
diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c
--- a/Modules/_cursesmodule.c
+++ b/Modules/_cursesmodule.c
@@ -560,10 +560,10 @@
 curses.window.addch
 
     [
+    y: int
+      Y-coordinate.
     x: int
       X-coordinate.
-    y: int
-      Y-coordinate.
     ]
 
     ch: object
@@ -584,13 +584,13 @@
 [clinic start generated code]*/
 
 PyDoc_STRVAR(curses_window_addch__doc__,
-"addch([x, y,] ch, [attr])\n"
+"addch([y, x,] ch, [attr])\n"
 "Paint character ch at (y, x) with attributes attr.\n"
 "\n"
+"  y\n"
+"    Y-coordinate.\n"
 "  x\n"
 "    X-coordinate.\n"
-"  y\n"
-"    Y-coordinate.\n"
 "  ch\n"
 "    Character to add.\n"
 "  attr\n"
@@ -605,15 +605,15 @@
     {"addch", (PyCFunction)curses_window_addch, METH_VARARGS, curses_window_addch__doc__},
 
 static PyObject *
-curses_window_addch_impl(PyCursesWindowObject *self, int group_left_1, int x, int y, PyObject *ch, int group_right_1, long attr);
+curses_window_addch_impl(PyCursesWindowObject *self, int group_left_1, int y, int x, PyObject *ch, int group_right_1, long attr);
 
 static PyObject *
 curses_window_addch(PyCursesWindowObject *self, PyObject *args)
 {
     PyObject *return_value = NULL;
     int group_left_1 = 0;
+    int y = 0;
     int x = 0;
-    int y = 0;
     PyObject *ch;
     int group_right_1 = 0;
     long attr = 0;
@@ -629,12 +629,12 @@
             group_right_1 = 1;
             break;
         case 3:
-            if (!PyArg_ParseTuple(args, "iiO:addch", &x, &y, &ch))
+            if (!PyArg_ParseTuple(args, "iiO:addch", &y, &x, &ch))
                 goto exit;
             group_left_1 = 1;
             break;
         case 4:
-            if (!PyArg_ParseTuple(args, "iiOl:addch", &x, &y, &ch, &attr))
+            if (!PyArg_ParseTuple(args, "iiOl:addch", &y, &x, &ch, &attr))
                 goto exit;
             group_right_1 = 1;
             group_left_1 = 1;
@@ -643,15 +643,15 @@
             PyErr_SetString(PyExc_TypeError, "curses.window.addch requires 1 to 4 arguments");
             goto exit;
     }
-    return_value = curses_window_addch_impl(self, group_left_1, x, y, ch, group_right_1, attr);
+    return_value = curses_window_addch_impl(self, group_left_1, y, x, ch, group_right_1, attr);
 
 exit:
     return return_value;
 }
 
 static PyObject *
-curses_window_addch_impl(PyCursesWindowObject *self, int group_left_1, int x, int y, PyObject *ch, int group_right_1, long attr)
-/*[clinic end generated code: output=43acb91a5c98f615 input=fe7e3711d5bbf1f6]*/
+curses_window_addch_impl(PyCursesWindowObject *self, int group_left_1, int y, int x, PyObject *ch, int group_right_1, long attr)
+/*[clinic end generated code: output=d4b97cc287010c54 input=5a41efb34a2de338]*/
 {
     PyCursesWindowObject *cwself = (PyCursesWindowObject *)self;
     int coordinates_group = group_left_1;

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Mon May  5 09:41:04 2014
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Mon, 05 May 2014 09:41:04 +0200
Subject: [Python-checkins] Daily reference leaks (3aa5fae8c313): sum=-1
Message-ID: <E1WhDWK-0007jg-NI@vds2544.sivit.org>

results for 3aa5fae8c313 on branch "default"
--------------------------------------------

test_functools leaked [0, 0, 3] memory blocks, sum=3
test_site leaked [0, -2, 0] references, sum=-2
test_site leaked [0, -2, 0] memory blocks, sum=-2


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogHBBPC_', '-x']

From python-checkins at python.org  Mon May  5 16:35:35 2014
From: python-checkins at python.org (guido.van.rossum)
Date: Mon,  5 May 2014 16:35:35 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_asyncio_docs?=
 =?utf-8?q?=3A_ProactorEventLoop_does_not_support_SSL=2E?=
Message-ID: <3gMmng3yfmz7Ljs@mail.python.org>

http://hg.python.org/cpython/rev/048381e58162
changeset:   90561:048381e58162
branch:      3.4
parent:      90559:4f26430b03fd
user:        Guido van Rossum <guido at python.org>
date:        Mon May 05 07:34:56 2014 -0700
summary:
  asyncio docs: ProactorEventLoop does not support SSL.

files:
  Doc/library/asyncio-subprocess.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst
--- a/Doc/library/asyncio-subprocess.rst
+++ b/Doc/library/asyncio-subprocess.rst
@@ -8,7 +8,7 @@
 
 On Windows, the default event loop uses :class:`selectors.SelectSelector`
 which only supports sockets. The :class:`ProactorEventLoop` should be used to
-support subprocesses.
+support subprocesses. However, the latter does not support SSL.
 
 On Mac OS X older than 10.9 (Mavericks), :class:`selectors.KqueueSelector`
 does not support character devices like PTY, whereas it is used by the

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May  5 16:35:36 2014
From: python-checkins at python.org (guido.van.rossum)
Date: Mon,  5 May 2014 16:35:36 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge_3=2E4-=3Edefault=3A_asyncio_docs=3A_ProactorEventL?=
 =?utf-8?q?oop_does_not_support_SSL=2E?=
Message-ID: <3gMmnh6SWDz7LlR@mail.python.org>

http://hg.python.org/cpython/rev/5d0783376c88
changeset:   90562:5d0783376c88
parent:      90560:3aa5fae8c313
parent:      90561:048381e58162
user:        Guido van Rossum <guido at python.org>
date:        Mon May 05 07:35:29 2014 -0700
summary:
  Merge 3.4->default: asyncio docs: ProactorEventLoop does not support SSL.

files:
  Doc/library/asyncio-subprocess.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst
--- a/Doc/library/asyncio-subprocess.rst
+++ b/Doc/library/asyncio-subprocess.rst
@@ -8,7 +8,7 @@
 
 On Windows, the default event loop uses :class:`selectors.SelectSelector`
 which only supports sockets. The :class:`ProactorEventLoop` should be used to
-support subprocesses.
+support subprocesses. However, the latter does not support SSL.
 
 On Mac OS X older than 10.9 (Mavericks), :class:`selectors.KqueueSelector`
 does not support character devices like PTY, whereas it is used by the

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May  5 20:47:53 2014
From: python-checkins at python.org (tim.golden)
Date: Mon,  5 May 2014 20:47:53 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue18314_Allow_unlink_to?=
 =?utf-8?q?_remove_junctions=2E_Includes_support_for_creating?=
Message-ID: <3gMtNn2k5cz7Ljq@mail.python.org>

http://hg.python.org/cpython/rev/5c1c14ff1f13
changeset:   90563:5c1c14ff1f13
user:        Tim Golden <mail at timgolden.me.uk>
date:        Mon May 05 19:46:17 2014 +0100
summary:
  Issue18314 Allow unlink to remove junctions. Includes support for creating junctions. Patch by Kim Gr?sman

files:
  Lib/test/test_os.py   |   36 +++++++
  Modules/_winapi.c     |  137 ++++++++++++++++++++++++++++++
  Modules/posixmodule.c |   45 +-------
  Modules/winreparse.h  |   53 +++++++++++
  4 files changed, 235 insertions(+), 36 deletions(-)


diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -39,6 +39,10 @@
     import fcntl
 except ImportError:
     fcntl = None
+try:
+    import _winapi
+except ImportError:
+    _winapi = None
 
 from test.script_helper import assert_python_ok
 
@@ -1773,6 +1777,37 @@
             shutil.rmtree(level1)
 
 
+ at unittest.skipUnless(sys.platform == "win32", "Win32 specific tests")
+class Win32JunctionTests(unittest.TestCase):
+    junction = 'junctiontest'
+    junction_target = os.path.dirname(os.path.abspath(__file__))
+
+    def setUp(self):
+        assert os.path.exists(self.junction_target)
+        assert not os.path.exists(self.junction)
+
+    def tearDown(self):
+        if os.path.exists(self.junction):
+            # os.rmdir delegates to Windows' RemoveDirectoryW,
+            # which removes junction points safely.
+            os.rmdir(self.junction)
+
+    def test_create_junction(self):
+        _winapi.CreateJunction(self.junction_target, self.junction)
+        self.assertTrue(os.path.exists(self.junction))
+        self.assertTrue(os.path.isdir(self.junction))
+
+        # Junctions are not recognized as links.
+        self.assertFalse(os.path.islink(self.junction))
+
+    def test_unlink_removes_junction(self):
+        _winapi.CreateJunction(self.junction_target, self.junction)
+        self.assertTrue(os.path.exists(self.junction))
+
+        os.unlink(self.junction)
+        self.assertFalse(os.path.exists(self.junction))
+
+
 @support.skip_unless_symlink
 class NonLocalSymlinkTests(unittest.TestCase):
 
@@ -2544,6 +2579,7 @@
         RemoveDirsTests,
         CPUCountTests,
         FDInheritanceTests,
+        Win32JunctionTests,
     )
 
 if __name__ == "__main__":
diff --git a/Modules/_winapi.c b/Modules/_winapi.c
--- a/Modules/_winapi.c
+++ b/Modules/_winapi.c
@@ -40,6 +40,7 @@
 #define WINDOWS_LEAN_AND_MEAN
 #include "windows.h"
 #include <crtdbg.h>
+#include "winreparse.h"
 
 #if defined(MS_WIN32) && !defined(MS_WIN64)
 #define HANDLE_TO_PYNUM(handle) \
@@ -401,6 +402,140 @@
 }
 
 static PyObject *
+winapi_CreateJunction(PyObject *self, PyObject *args)
+{
+    /* Input arguments */
+    LPWSTR src_path = NULL;
+    LPWSTR dst_path = NULL;
+
+    /* Privilege adjustment */
+    HANDLE token = NULL;
+    TOKEN_PRIVILEGES tp;
+
+    /* Reparse data buffer */
+    const USHORT prefix_len = 4;
+    USHORT print_len = 0;
+    USHORT rdb_size = 0;
+    PREPARSE_DATA_BUFFER rdb = NULL;
+
+    /* Junction point creation */
+    HANDLE junction = NULL;
+    DWORD ret = 0;
+
+    if (!PyArg_ParseTuple(args, "uu", &src_path, &dst_path))
+        return NULL;
+
+    if (src_path == NULL || dst_path == NULL)
+        return PyErr_SetFromWindowsErr(ERROR_INVALID_PARAMETER);
+
+    if (wcsncmp(src_path, L"\\??\\", prefix_len) == 0)
+        return PyErr_SetFromWindowsErr(ERROR_INVALID_PARAMETER);
+
+    /* Adjust privileges to allow rewriting directory entry as a
+       junction point. */
+    if (!OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES, &token))
+        goto cleanup;
+
+    if (!LookupPrivilegeValue(NULL, SE_RESTORE_NAME, &tp.Privileges[0].Luid))
+        goto cleanup;
+
+    tp.PrivilegeCount = 1;
+    tp.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;
+    if (!AdjustTokenPrivileges(token, FALSE, &tp, sizeof(TOKEN_PRIVILEGES),
+                               NULL, NULL))
+        goto cleanup;
+
+    if (GetFileAttributesW(src_path) == INVALID_FILE_ATTRIBUTES)
+        goto cleanup;
+
+    /* Store the absolute link target path length in print_len. */
+    print_len = (USHORT)GetFullPathNameW(src_path, 0, NULL, NULL);
+    if (print_len == 0)
+        goto cleanup;
+
+    /* NUL terminator should not be part of print_len. */
+    --print_len;
+
+    /* REPARSE_DATA_BUFFER usage is heavily under-documented, especially for
+       junction points. Here's what I've learned along the way:
+       - A junction point has two components: a print name and a substitute
+         name. They both describe the link target, but the substitute name is
+         the physical target and the print name is shown in directory listings.
+       - The print name must be a native name, prefixed with "\??\".
+       - Both names are stored after each other in the same buffer (the
+         PathBuffer) and both must be NUL-terminated.
+       - There are four members defining their respective offset and length
+         inside PathBuffer: SubstituteNameOffset, SubstituteNameLength,
+         PrintNameOffset and PrintNameLength.
+       - The total size we need to allocate for the REPARSE_DATA_BUFFER, thus,
+         is the sum of:
+         - the fixed header size (REPARSE_DATA_BUFFER_HEADER_SIZE)
+         - the size of the MountPointReparseBuffer member without the PathBuffer
+         - the size of the prefix ("\??\") in bytes
+         - the size of the print name in bytes
+         - the size of the substitute name in bytes
+         - the size of two NUL terminators in bytes */
+    rdb_size = REPARSE_DATA_BUFFER_HEADER_SIZE +
+        sizeof(rdb->MountPointReparseBuffer) -
+        sizeof(rdb->MountPointReparseBuffer.PathBuffer) +
+        /* Two +1's for NUL terminators. */
+        (prefix_len + print_len + 1 + print_len + 1) * sizeof(WCHAR);
+    rdb = (PREPARSE_DATA_BUFFER)PyMem_RawMalloc(rdb_size);
+    if (rdb == NULL)
+        goto cleanup;
+
+    memset(rdb, 0, rdb_size);
+    rdb->ReparseTag = IO_REPARSE_TAG_MOUNT_POINT;
+    rdb->ReparseDataLength = rdb_size - REPARSE_DATA_BUFFER_HEADER_SIZE;
+    rdb->MountPointReparseBuffer.SubstituteNameOffset = 0;
+    rdb->MountPointReparseBuffer.SubstituteNameLength =
+        (prefix_len + print_len) * sizeof(WCHAR);
+    rdb->MountPointReparseBuffer.PrintNameOffset =
+        rdb->MountPointReparseBuffer.SubstituteNameLength + sizeof(WCHAR);
+    rdb->MountPointReparseBuffer.PrintNameLength = print_len * sizeof(WCHAR);
+
+    /* Store the full native path of link target at the substitute name
+       offset (0). */
+    wcscpy(rdb->MountPointReparseBuffer.PathBuffer, L"\\??\\");
+    if (GetFullPathNameW(src_path, print_len + 1,
+                         rdb->MountPointReparseBuffer.PathBuffer + prefix_len,
+                         NULL) == 0)
+        goto cleanup;
+
+    /* Copy everything but the native prefix to the print name offset. */
+    wcscpy(rdb->MountPointReparseBuffer.PathBuffer +
+             prefix_len + print_len + 1,
+             rdb->MountPointReparseBuffer.PathBuffer + prefix_len);
+
+    /* Create a directory for the junction point. */
+    if (!CreateDirectoryW(dst_path, NULL))
+        goto cleanup;
+
+    junction = CreateFileW(dst_path, GENERIC_READ | GENERIC_WRITE, 0, NULL,
+        OPEN_EXISTING,
+        FILE_FLAG_OPEN_REPARSE_POINT | FILE_FLAG_BACKUP_SEMANTICS, NULL);
+    if (junction == INVALID_HANDLE_VALUE)
+        goto cleanup;
+
+    /* Make the directory entry a junction point. */
+    if (!DeviceIoControl(junction, FSCTL_SET_REPARSE_POINT, rdb, rdb_size,
+                         NULL, 0, &ret, NULL))
+        goto cleanup;
+
+cleanup:
+    ret = GetLastError();
+
+    CloseHandle(token);
+    CloseHandle(junction);
+    PyMem_RawFree(rdb);
+
+    if (ret != 0)
+        return PyErr_SetFromWindowsErr(ret);
+
+    Py_RETURN_NONE;
+}
+
+static PyObject *
 winapi_CreateNamedPipe(PyObject *self, PyObject *args)
 {
     LPCTSTR lpName;
@@ -1225,6 +1360,8 @@
      METH_VARARGS | METH_KEYWORDS, ""},
     {"CreateFile", winapi_CreateFile, METH_VARARGS,
      ""},
+    {"CreateJunction", winapi_CreateJunction, METH_VARARGS,
+     ""},
     {"CreateNamedPipe", winapi_CreateNamedPipe, METH_VARARGS,
      ""},
     {"CreatePipe", winapi_CreatePipe, METH_VARARGS,
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -27,6 +27,8 @@
 #include "Python.h"
 #ifndef MS_WINDOWS
 #include "posixmodule.h"
+#else
+#include "winreparse.h"
 #endif
 
 #ifdef __cplusplus
@@ -301,6 +303,9 @@
 #ifndef IO_REPARSE_TAG_SYMLINK
 #define IO_REPARSE_TAG_SYMLINK (0xA000000CL)
 #endif
+#ifndef IO_REPARSE_TAG_MOUNT_POINT
+#define IO_REPARSE_TAG_MOUNT_POINT (0xA0000003L)
+#endif
 #include "osdefs.h"
 #include <malloc.h>
 #include <windows.h>
@@ -1109,41 +1114,6 @@
 #endif
 
 #ifdef MS_WINDOWS
-/* The following structure was copied from
-   http://msdn.microsoft.com/en-us/library/ms791514.aspx as the required
-   include doesn't seem to be present in the Windows SDK (at least as included
-   with Visual Studio Express). */
-typedef struct _REPARSE_DATA_BUFFER {
-    ULONG ReparseTag;
-    USHORT ReparseDataLength;
-    USHORT Reserved;
-    union {
-        struct {
-            USHORT SubstituteNameOffset;
-            USHORT SubstituteNameLength;
-            USHORT PrintNameOffset;
-            USHORT PrintNameLength;
-            ULONG Flags;
-            WCHAR PathBuffer[1];
-        } SymbolicLinkReparseBuffer;
-
-        struct {
-            USHORT SubstituteNameOffset;
-            USHORT  SubstituteNameLength;
-            USHORT  PrintNameOffset;
-            USHORT  PrintNameLength;
-            WCHAR  PathBuffer[1];
-        } MountPointReparseBuffer;
-
-        struct {
-            UCHAR  DataBuffer[1];
-        } GenericReparseBuffer;
-    };
-} REPARSE_DATA_BUFFER, *PREPARSE_DATA_BUFFER;
-
-#define REPARSE_DATA_BUFFER_HEADER_SIZE  FIELD_OFFSET(REPARSE_DATA_BUFFER,\
-                                                      GenericReparseBuffer)
-#define MAXIMUM_REPARSE_DATA_BUFFER_SIZE  ( 16 * 1024 )
 
 static int
 win32_get_reparse_tag(HANDLE reparse_point_handle, ULONG *reparse_tag)
@@ -4492,7 +4462,10 @@
             find_data_handle = FindFirstFileW(lpFileName, &find_data);
 
             if(find_data_handle != INVALID_HANDLE_VALUE) {
-                is_link = find_data.dwReserved0 == IO_REPARSE_TAG_SYMLINK;
+                /* IO_REPARSE_TAG_SYMLINK if it is a symlink and
+                   IO_REPARSE_TAG_MOUNT_POINT if it is a junction point. */
+                is_link = find_data.dwReserved0 == IO_REPARSE_TAG_SYMLINK ||
+                          find_data.dwReserved0 == IO_REPARSE_TAG_MOUNT_POINT;
                 FindClose(find_data_handle);
             }
         }
diff --git a/Modules/winreparse.h b/Modules/winreparse.h
new file mode 100644
--- /dev/null
+++ b/Modules/winreparse.h
@@ -0,0 +1,53 @@
+#ifndef Py_WINREPARSE_H
+#define Py_WINREPARSE_H
+
+#ifdef MS_WINDOWS
+#include <Windows.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* The following structure was copied from
+   http://msdn.microsoft.com/en-us/library/ff552012.aspx as the required
+   include doesn't seem to be present in the Windows SDK (at least as included
+   with Visual Studio Express). */
+typedef struct _REPARSE_DATA_BUFFER {
+    ULONG ReparseTag;
+    USHORT ReparseDataLength;
+    USHORT Reserved;
+    union {
+        struct {
+            USHORT SubstituteNameOffset;
+            USHORT SubstituteNameLength;
+            USHORT PrintNameOffset;
+            USHORT PrintNameLength;
+            ULONG Flags;
+            WCHAR PathBuffer[1];
+        } SymbolicLinkReparseBuffer;
+
+        struct {
+            USHORT SubstituteNameOffset;
+            USHORT  SubstituteNameLength;
+            USHORT  PrintNameOffset;
+            USHORT  PrintNameLength;
+            WCHAR  PathBuffer[1];
+        } MountPointReparseBuffer;
+
+        struct {
+            UCHAR  DataBuffer[1];
+        } GenericReparseBuffer;
+    };
+} REPARSE_DATA_BUFFER, *PREPARSE_DATA_BUFFER;
+
+#define REPARSE_DATA_BUFFER_HEADER_SIZE  FIELD_OFFSET(REPARSE_DATA_BUFFER,\
+                                                      GenericReparseBuffer)
+#define MAXIMUM_REPARSE_DATA_BUFFER_SIZE  ( 16 * 1024 )
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* MS_WINDOWS */
+
+#endif /* !Py_WINREPARSE_H */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May  5 22:00:03 2014
From: python-checkins at python.org (tim.golden)
Date: Mon,  5 May 2014 22:00:03 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue18314_ACKS_=26_NEWS?=
Message-ID: <3gMw032ZHVz7LjR@mail.python.org>

http://hg.python.org/cpython/rev/e791a36ab6a2
changeset:   90564:e791a36ab6a2
user:        Tim Golden <mail at timgolden.me.uk>
date:        Mon May 05 20:59:50 2014 +0100
summary:
  Issue18314 ACKS & NEWS

files:
  Misc/ACKS |  1 +
  Misc/NEWS |  2 ++
  2 files changed, 3 insertions(+), 0 deletions(-)


diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -470,6 +470,7 @@
 Shelley Gooch
 David Goodger
 Hans de Graaff
+Kim Gr?sman
 Nathaniel Gray
 Eddy De Greef
 Grant Griffin
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -69,6 +69,8 @@
 Library
 -------
 
+- Issue #18314: Unlink now removes junctions on Windows. Patch by Kim Gr?sman
+
 - Issue #21088: Bugfix for curses.window.addch() regression in 3.4.0.
   In porting to Argument Clinic, the first two arguments were reversed.
 

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Tue May  6 08:45:35 2014
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Tue, 06 May 2014 08:45:35 +0200
Subject: [Python-checkins] Daily reference leaks (e791a36ab6a2): sum=7
Message-ID: <E1WhZ8B-00014G-Cf@vds2544.sivit.org>

results for e791a36ab6a2 on branch "default"
--------------------------------------------

test_asyncio leaked [0, 4, 0] memory blocks, sum=4
test_functools leaked [0, 0, 3] memory blocks, sum=3
test_site leaked [0, 2, -2] references, sum=0
test_site leaked [0, 2, -2] memory blocks, sum=0


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogtSpFNU', '-x']

From python-checkins at python.org  Tue May  6 11:32:48 2014
From: python-checkins at python.org (victor.stinner)
Date: Tue,  6 May 2014 11:32:48 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321233=3A_Fix_=5FP?=
 =?utf-8?q?yObject=5FAlloc=28=29_when_compiled_with_WITH=5FVALGRIND_define?=
 =?utf-8?q?d?=
Message-ID: <3gNG1r53yVz7Lmh@mail.python.org>

http://hg.python.org/cpython/rev/358a12f4d4bc
changeset:   90565:358a12f4d4bc
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Tue May 06 11:32:29 2014 +0200
summary:
  Issue #21233: Fix _PyObject_Alloc() when compiled with WITH_VALGRIND defined

files:
  Objects/obmalloc.c |  6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c
--- a/Objects/obmalloc.c
+++ b/Objects/obmalloc.c
@@ -1176,6 +1176,9 @@
 
     _Py_AllocatedBlocks++;
 
+    assert(nelem <= PY_SSIZE_T_MAX / elsize);
+    nbytes = nelem * elsize;
+
 #ifdef WITH_VALGRIND
     if (UNLIKELY(running_on_valgrind == -1))
         running_on_valgrind = RUNNING_ON_VALGRIND;
@@ -1183,9 +1186,6 @@
         goto redirect;
 #endif
 
-    assert(nelem <= PY_SSIZE_T_MAX / elsize);
-    nbytes = nelem * elsize;
-
     if (nelem == 0 || elsize == 0)
         goto redirect;
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May  6 14:28:43 2014
From: python-checkins at python.org (tim.golden)
Date: Tue,  6 May 2014 14:28:43 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue21440_Use_support=2Er?=
 =?utf-8?q?mtree_in_test=5Fzipfile_=26_test=5Ftarfile?=
Message-ID: <3gNKwq4jffz7Ljb@mail.python.org>

http://hg.python.org/cpython/rev/4ebf97299b18
changeset:   90566:4ebf97299b18
user:        Tim Golden <mail at timgolden.me.uk>
date:        Tue May 06 13:24:26 2014 +0100
summary:
  Issue21440 Use support.rmtree in test_zipfile & test_tarfile

files:
  Lib/test/test_tarfile.py |  39 +++++++++++++--------------
  Lib/test/test_zipfile.py |  23 +++++++--------
  2 files changed, 30 insertions(+), 32 deletions(-)


diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py
--- a/Lib/test/test_tarfile.py
+++ b/Lib/test/test_tarfile.py
@@ -1,7 +1,6 @@
 import sys
 import os
 import io
-import shutil
 from hashlib import md5
 
 import unittest
@@ -456,16 +455,16 @@
         # Test hardlink extraction (e.g. bug #857297).
         with tarfile.open(tarname, errorlevel=1, encoding="iso8859-1") as tar:
             tar.extract("ustar/regtype", TEMPDIR)
-            self.addCleanup(os.remove, os.path.join(TEMPDIR, "ustar/regtype"))
+            self.addCleanup(support.unlink, os.path.join(TEMPDIR, "ustar/regtype"))
 
             tar.extract("ustar/lnktype", TEMPDIR)
-            self.addCleanup(os.remove, os.path.join(TEMPDIR, "ustar/lnktype"))
+            self.addCleanup(support.unlink, os.path.join(TEMPDIR, "ustar/lnktype"))
             with open(os.path.join(TEMPDIR, "ustar/lnktype"), "rb") as f:
                 data = f.read()
             self.assertEqual(md5sum(data), md5_regtype)
 
             tar.extract("ustar/symtype", TEMPDIR)
-            self.addCleanup(os.remove, os.path.join(TEMPDIR, "ustar/symtype"))
+            self.addCleanup(support.unlink, os.path.join(TEMPDIR, "ustar/symtype"))
             with open(os.path.join(TEMPDIR, "ustar/symtype"), "rb") as f:
                 data = f.read()
             self.assertEqual(md5sum(data), md5_regtype)
@@ -498,7 +497,7 @@
                 self.assertEqual(tarinfo.mtime, file_mtime, errmsg)
         finally:
             tar.close()
-            shutil.rmtree(DIR)
+            support.rmtree(DIR)
 
     def test_extract_directory(self):
         dirtype = "ustar/dirtype"
@@ -513,7 +512,7 @@
                 if sys.platform != "win32":
                     self.assertEqual(os.stat(extracted).st_mode & 0o777, 0o755)
         finally:
-            shutil.rmtree(DIR)
+            support.rmtree(DIR)
 
     def test_init_close_fobj(self):
         # Issue #7341: Close the internal file object in the TarFile
@@ -877,7 +876,7 @@
                 fobj.seek(4096)
                 fobj.truncate()
             s = os.stat(name)
-            os.remove(name)
+            support.unlink(name)
             return s.st_blocks == 0
         else:
             return False
@@ -1010,7 +1009,7 @@
             finally:
                 tar.close()
         finally:
-            os.rmdir(path)
+            support.rmdir(path)
 
     @unittest.skipUnless(hasattr(os, "link"),
                          "Missing hardlink implementation")
@@ -1030,8 +1029,8 @@
             finally:
                 tar.close()
         finally:
-            os.remove(target)
-            os.remove(link)
+            support.unlink(target)
+            support.unlink(link)
 
     @support.skip_unless_symlink
     def test_symlink_size(self):
@@ -1045,7 +1044,7 @@
             finally:
                 tar.close()
         finally:
-            os.remove(path)
+            support.unlink(path)
 
     def test_add_self(self):
         # Test for #1257255.
@@ -1092,7 +1091,7 @@
             finally:
                 tar.close()
         finally:
-            shutil.rmtree(tempdir)
+            support.rmtree(tempdir)
 
     def test_filter(self):
         tempdir = os.path.join(TEMPDIR, "filter")
@@ -1128,7 +1127,7 @@
             finally:
                 tar.close()
         finally:
-            shutil.rmtree(tempdir)
+            support.rmtree(tempdir)
 
     # Guarantee that stored pathnames are not modified. Don't
     # remove ./ or ../ or double slashes. Still make absolute
@@ -1156,9 +1155,9 @@
             tar.close()
 
         if not dir:
-            os.remove(foo)
+            support.unlink(foo)
         else:
-            os.rmdir(foo)
+            support.rmdir(foo)
 
         self.assertEqual(t.name, cmp_path or path.replace(os.sep, "/"))
 
@@ -1189,8 +1188,8 @@
             finally:
                 tar.close()
         finally:
-            os.unlink(temparchive)
-            shutil.rmtree(tempdir)
+            support.unlink(temparchive)
+            support.rmtree(tempdir)
 
     def test_pathnames(self):
         self._test_pathname("foo")
@@ -1290,7 +1289,7 @@
         # Test for issue #8464: Create files with correct
         # permissions.
         if os.path.exists(tmpname):
-            os.remove(tmpname)
+            support.unlink(tmpname)
 
         original_umask = os.umask(0o022)
         try:
@@ -1644,7 +1643,7 @@
     def setUp(self):
         self.tarname = tmpname
         if os.path.exists(self.tarname):
-            os.remove(self.tarname)
+            support.unlink(self.tarname)
 
     def _create_testtar(self, mode="w:"):
         with tarfile.open(tarname, encoding="iso8859-1") as src:
@@ -2151,7 +2150,7 @@
 
 def tearDownModule():
     if os.path.exists(TEMPDIR):
-        shutil.rmtree(TEMPDIR)
+        support.rmtree(TEMPDIR)
 
 if __name__ == "__main__":
     unittest.main()
diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py
--- a/Lib/test/test_zipfile.py
+++ b/Lib/test/test_zipfile.py
@@ -3,7 +3,6 @@
 import sys
 import importlib.util
 import time
-import shutil
 import struct
 import zipfile
 import unittest
@@ -12,7 +11,7 @@
 from tempfile import TemporaryFile
 from random import randint, random, getrandbits
 
-from test.support import (TESTFN, findfile, unlink,
+from test.support import (TESTFN, findfile, unlink, rmtree,
                           requires_zlib, requires_bz2, requires_lzma,
                           captured_stdout, check_warnings)
 
@@ -691,7 +690,7 @@
                 self.assertNotIn('mod2.txt', names)
 
         finally:
-            shutil.rmtree(TESTFN2)
+            rmtree(TESTFN2)
 
     def test_write_python_directory_filtered(self):
         os.mkdir(TESTFN2)
@@ -711,7 +710,7 @@
                 self.assertNotIn('mod2.py', names)
 
         finally:
-            shutil.rmtree(TESTFN2)
+            rmtree(TESTFN2)
 
     def test_write_non_pyfile(self):
         with TemporaryFile() as t, zipfile.PyZipFile(t, "w") as zipfp:
@@ -741,7 +740,7 @@
                 self.assertNotIn('mod1.pyo', names)
 
         finally:
-            shutil.rmtree(TESTFN2)
+            rmtree(TESTFN2)
 
 
 class ExtractTests(unittest.TestCase):
@@ -767,7 +766,7 @@
                 os.remove(writtenfile)
 
         # remove the test file subdirectories
-        shutil.rmtree(os.path.join(os.getcwd(), 'ziptest2dir'))
+        rmtree(os.path.join(os.getcwd(), 'ziptest2dir'))
 
     def test_extract_all(self):
         with zipfile.ZipFile(TESTFN2, "w", zipfile.ZIP_STORED) as zipfp:
@@ -785,7 +784,7 @@
                 os.remove(outfile)
 
         # remove the test file subdirectories
-        shutil.rmtree(os.path.join(os.getcwd(), 'ziptest2dir'))
+        rmtree(os.path.join(os.getcwd(), 'ziptest2dir'))
 
     def check_file(self, filename, content):
         self.assertTrue(os.path.isfile(filename))
@@ -867,12 +866,12 @@
                                  msg='extract %r: %r != %r' %
                                  (arcname, writtenfile, correctfile))
             self.check_file(correctfile, content)
-            shutil.rmtree('target')
+            rmtree('target')
 
             with zipfile.ZipFile(TESTFN2, 'r') as zipfp:
                 zipfp.extractall(targetpath)
             self.check_file(correctfile, content)
-            shutil.rmtree('target')
+            rmtree('target')
 
             correctfile = os.path.join(os.getcwd(), *fixedname.split('/'))
 
@@ -881,12 +880,12 @@
                 self.assertEqual(writtenfile, correctfile,
                                  msg="extract %r" % arcname)
             self.check_file(correctfile, content)
-            shutil.rmtree(fixedname.split('/')[0])
+            rmtree(fixedname.split('/')[0])
 
             with zipfile.ZipFile(TESTFN2, 'r') as zipfp:
                 zipfp.extractall()
             self.check_file(correctfile, content)
-            shutil.rmtree(fixedname.split('/')[0])
+            rmtree(fixedname.split('/')[0])
 
             os.remove(TESTFN2)
 
@@ -1628,7 +1627,7 @@
         self.assertTrue(zipf.filelist[0].filename.endswith("x/"))
 
     def tearDown(self):
-        shutil.rmtree(TESTFN2)
+        rmtree(TESTFN2)
         if os.path.exists(TESTFN):
             unlink(TESTFN)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May  6 14:30:02 2014
From: python-checkins at python.org (tim.golden)
Date: Tue,  6 May 2014 14:30:02 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue21393_Use_CryptReleas?=
 =?utf-8?q?eContext_to_release_Crypt_handle_on_Windows?=
Message-ID: <3gNKyL3h5Pz7Ljb@mail.python.org>

http://hg.python.org/cpython/rev/2b18f6c37a68
changeset:   90567:2b18f6c37a68
user:        Tim Golden <mail at timgolden.me.uk>
date:        Tue May 06 13:29:45 2014 +0100
summary:
  Issue21393 Use CryptReleaseContext to release Crypt handle on Windows

files:
  Python/random.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Python/random.c b/Python/random.c
--- a/Python/random.c
+++ b/Python/random.c
@@ -298,7 +298,7 @@
 {
 #ifdef MS_WINDOWS
     if (hCryptProv) {
-        CloseHandle(hCryptProv);
+        CryptReleaseContext(hCryptProv, 0);
         hCryptProv = 0;
     }
 #else

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May  6 16:08:22 2014
From: python-checkins at python.org (zach.ware)
Date: Tue,  6 May 2014 16:08:22 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMzY2?=
 =?utf-8?q?=3A_Document_the_fact_that_=60=60return=60=60_in_a_=60=60finall?=
 =?utf-8?q?y=60=60_clause?=
Message-ID: <3gNN7p3Jcpz7LtR@mail.python.org>

http://hg.python.org/cpython/rev/faaa8d569664
changeset:   90568:faaa8d569664
branch:      3.4
parent:      90561:048381e58162
user:        Zachary Ware <zachary.ware at gmail.com>
date:        Tue May 06 09:07:13 2014 -0500
summary:
  Issue #21366: Document the fact that ``return`` in a ``finally`` clause
overrides a ``return`` in the ``try`` suite.

files:
  Doc/reference/compound_stmts.rst |  14 ++++++++++++++
  1 files changed, 14 insertions(+), 0 deletions(-)


diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst
--- a/Doc/reference/compound_stmts.rst
+++ b/Doc/reference/compound_stmts.rst
@@ -337,6 +337,20 @@
 reason is a problem with the current implementation --- this restriction may be
 lifted in the future).
 
+The return value of a function is determined by the last :keyword:`return`
+statement executed.  Since the :keyword:`finally` clause always executes, a
+:keyword:`return` statement executed in the :keyword:`finally` clause will
+always be the last one executed::
+
+   >>> def foo():
+   ...     try:
+   ...         return 'try'
+   ...     finally:
+   ...         return 'finally'
+   ...
+   >>> foo()
+   'finally'
+
 Additional information on exceptions can be found in section :ref:`exceptions`,
 and information on using the :keyword:`raise` statement to generate exceptions
 may be found in section :ref:`raise`.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May  6 16:08:23 2014
From: python-checkins at python.org (zach.ware)
Date: Tue,  6 May 2014 16:08:23 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Issue_=2321366=3A_Document_the_fact_that_=60=60return=60?=
 =?utf-8?q?=60_in_a_=60=60finally=60=60_clause?=
Message-ID: <3gNN7q54GLz7Ltf@mail.python.org>

http://hg.python.org/cpython/rev/685f92aad1dc
changeset:   90569:685f92aad1dc
parent:      90567:2b18f6c37a68
parent:      90568:faaa8d569664
user:        Zachary Ware <zachary.ware at gmail.com>
date:        Tue May 06 09:07:51 2014 -0500
summary:
  Issue #21366: Document the fact that ``return`` in a ``finally`` clause
overrides a ``return`` in the ``try`` suite.

files:
  Doc/reference/compound_stmts.rst |  14 ++++++++++++++
  1 files changed, 14 insertions(+), 0 deletions(-)


diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst
--- a/Doc/reference/compound_stmts.rst
+++ b/Doc/reference/compound_stmts.rst
@@ -337,6 +337,20 @@
 reason is a problem with the current implementation --- this restriction may be
 lifted in the future).
 
+The return value of a function is determined by the last :keyword:`return`
+statement executed.  Since the :keyword:`finally` clause always executes, a
+:keyword:`return` statement executed in the :keyword:`finally` clause will
+always be the last one executed::
+
+   >>> def foo():
+   ...     try:
+   ...         return 'try'
+   ...     finally:
+   ...         return 'finally'
+   ...
+   >>> foo()
+   'finally'
+
 Additional information on exceptions can be found in section :ref:`exceptions`,
 and information on using the :keyword:`raise` statement to generate exceptions
 may be found in section :ref:`raise`.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May  6 16:10:29 2014
From: python-checkins at python.org (zach.ware)
Date: Tue,  6 May 2014 16:10:29 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIxMzY2?=
 =?utf-8?q?=3A_Document_the_fact_that_=60=60return=60=60_in_a_=60=60finall?=
 =?utf-8?q?y=60=60_clause?=
Message-ID: <3gNNBF0WHKzRGY@mail.python.org>

http://hg.python.org/cpython/rev/7fabe3652ee0
changeset:   90570:7fabe3652ee0
branch:      2.7
parent:      90544:b768d41dec0a
user:        Zachary Ware <zachary.ware at gmail.com>
date:        Tue May 06 09:07:13 2014 -0500
summary:
  Issue #21366: Document the fact that ``return`` in a ``finally`` clause
overrides a ``return`` in the ``try`` suite.

files:
  Doc/reference/compound_stmts.rst |  14 ++++++++++++++
  1 files changed, 14 insertions(+), 0 deletions(-)


diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst
--- a/Doc/reference/compound_stmts.rst
+++ b/Doc/reference/compound_stmts.rst
@@ -320,6 +320,20 @@
 reason is a problem with the current implementation --- this restriction may be
 lifted in the future).
 
+The return value of a function is determined by the last :keyword:`return`
+statement executed.  Since the :keyword:`finally` clause always executes, a
+:keyword:`return` statement executed in the :keyword:`finally` clause will
+always be the last one executed::
+
+   >>> def foo():
+   ...     try:
+   ...         return 'try'
+   ...     finally:
+   ...         return 'finally'
+   ...
+   >>> foo()
+   'finally'
+
 Additional information on exceptions can be found in section :ref:`exceptions`,
 and information on using the :keyword:`raise` statement to generate exceptions
 may be found in section :ref:`raise`.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May  6 16:21:57 2014
From: python-checkins at python.org (zach.ware)
Date: Tue,  6 May 2014 16:21:57 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Add_prompts_to?=
	=?utf-8?q?_interactive_example=2E?=
Message-ID: <3gNNRT5Fdjz7LtW@mail.python.org>

http://hg.python.org/cpython/rev/b4b10afd2fe6
changeset:   90571:b4b10afd2fe6
branch:      3.4
parent:      90568:faaa8d569664
user:        Zachary Ware <zachary.ware at gmail.com>
date:        Tue May 06 09:18:17 2014 -0500
summary:
  Add prompts to interactive example.

This makes it match the new example below, and allows Sphinx's
"hide the prompts and output" feature to work.

files:
  Doc/reference/compound_stmts.rst |  16 ++++++++--------
  1 files changed, 8 insertions(+), 8 deletions(-)


diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst
--- a/Doc/reference/compound_stmts.rst
+++ b/Doc/reference/compound_stmts.rst
@@ -313,14 +313,14 @@
 If the :keyword:`finally` clause executes a :keyword:`return` or :keyword:`break`
 statement, the saved exception is discarded::
 
-    def f():
-        try:
-            1/0
-        finally:
-            return 42
-
-    >>> f()
-    42
+   >>> def f():
+   ...     try:
+   ...         1/0
+   ...     finally:
+   ...         return 42
+   ...
+   >>> f()
+   42
 
 The exception information is not available to the program during execution of
 the :keyword:`finally` clause.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May  6 16:21:58 2014
From: python-checkins at python.org (zach.ware)
Date: Tue,  6 May 2014 16:21:58 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?q?=29=3A_Merge_with_3=2E4?=
Message-ID: <3gNNRV6thgz7Lsb@mail.python.org>

http://hg.python.org/cpython/rev/ec609d498566
changeset:   90572:ec609d498566
parent:      90569:685f92aad1dc
parent:      90571:b4b10afd2fe6
user:        Zachary Ware <zachary.ware at gmail.com>
date:        Tue May 06 09:19:16 2014 -0500
summary:
  Merge with 3.4

files:
  Doc/reference/compound_stmts.rst |  16 ++++++++--------
  1 files changed, 8 insertions(+), 8 deletions(-)


diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst
--- a/Doc/reference/compound_stmts.rst
+++ b/Doc/reference/compound_stmts.rst
@@ -313,14 +313,14 @@
 If the :keyword:`finally` clause executes a :keyword:`return` or :keyword:`break`
 statement, the saved exception is discarded::
 
-    def f():
-        try:
-            1/0
-        finally:
-            return 42
-
-    >>> f()
-    42
+   >>> def f():
+   ...     try:
+   ...         1/0
+   ...     finally:
+   ...         return 42
+   ...
+   >>> f()
+   42
 
 The exception information is not available to the program during execution of
 the :keyword:`finally` clause.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May  6 16:22:00 2014
From: python-checkins at python.org (zach.ware)
Date: Tue,  6 May 2014 16:22:00 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Add_prompts_to?=
	=?utf-8?q?_interactive_example=2E?=
Message-ID: <3gNNRX1Sm2z7LtC@mail.python.org>

http://hg.python.org/cpython/rev/87dcfe8427d8
changeset:   90573:87dcfe8427d8
branch:      2.7
parent:      90570:7fabe3652ee0
user:        Zachary Ware <zachary.ware at gmail.com>
date:        Tue May 06 09:18:17 2014 -0500
summary:
  Add prompts to interactive example.

This makes it match the new example below, and allows Sphinx's
"hide the prompts and output" feature to work.

files:
  Doc/reference/compound_stmts.rst |  16 ++++++++--------
  1 files changed, 8 insertions(+), 8 deletions(-)


diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst
--- a/Doc/reference/compound_stmts.rst
+++ b/Doc/reference/compound_stmts.rst
@@ -296,14 +296,14 @@
 exception or executes a :keyword:`return` or :keyword:`break` statement, the
 saved exception is discarded::
 
-    def f():
-        try:
-            1/0
-        finally:
-            return 42
-
-    >>> f()
-    42
+   >>> def f():
+   ...     try:
+   ...         1/0
+   ...     finally:
+   ...         return 42
+   ...
+   >>> f()
+   42
 
 The exception information is not available to the program during execution of
 the :keyword:`finally` clause.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May  6 18:43:12 2014
From: python-checkins at python.org (zach.ware)
Date: Tue,  6 May 2014 18:43:12 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321442=3A_Fix_MSVC?=
 =?utf-8?q?_compiler_warning_introduced_by_issue21377=2E?=
Message-ID: <3gNRZS4cN2z7Ljd@mail.python.org>

http://hg.python.org/cpython/rev/6234f4caba57
changeset:   90574:6234f4caba57
parent:      90572:ec609d498566
user:        Zachary Ware <zachary.ware at gmail.com>
date:        Tue May 06 11:42:37 2014 -0500
summary:
  Issue #21442: Fix MSVC compiler warning introduced by issue21377.

files:
  Objects/bytesobject.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -2809,7 +2809,7 @@
 
     if (Py_REFCNT(*pv) == 1 && PyBytes_CheckExact(*pv)) {
         /* Only one reference, so we can resize in place */
-        size_t oldsize;
+        Py_ssize_t oldsize;
         Py_buffer wb;
         
         wb.len = -1;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May  6 23:47:24 2014
From: python-checkins at python.org (guido.van.rossum)
Date: Tue,  6 May 2014 23:47:24 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogRml4?=
 =?utf-8?q?_the_second_half_of_issue_=2321447=3A_race_in_=5Fwrite=5Fto=5Fs?=
 =?utf-8?b?ZWxmKCku?=
Message-ID: <3gNZKS394Yz7LjM@mail.python.org>

http://hg.python.org/cpython/rev/afe8c4bd3230
changeset:   90575:afe8c4bd3230
branch:      3.4
parent:      90571:b4b10afd2fe6
user:        Guido van Rossum <guido at python.org>
date:        Tue May 06 14:42:40 2014 -0700
summary:
  asyncio: Fix the second half of issue #21447: race in _write_to_self().

files:
  Lib/asyncio/selector_events.py                |  15 +++++++--
  Lib/test/test_asyncio/test_selector_events.py |   5 ++-
  2 files changed, 14 insertions(+), 6 deletions(-)


diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py
--- a/Lib/asyncio/selector_events.py
+++ b/Lib/asyncio/selector_events.py
@@ -87,10 +87,17 @@
             pass
 
     def _write_to_self(self):
-        try:
-            self._csock.send(b'x')
-        except (BlockingIOError, InterruptedError):
-            pass
+        # This may be called from a different thread, possibly after
+        # _close_self_pipe() has been called or even while it is
+        # running.  Guard for self._csock being None or closed.  When
+        # a socket is closed, send() raises OSError (with errno set to
+        # EBADF, but let's not rely on the exact error code).
+        csock = self._csock
+        if csock is not None:
+            try:
+                csock.send(b'x')
+            except OSError:
+                pass
 
     def _start_serving(self, protocol_factory, sock,
                        sslcontext=None, server=None):
diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py
--- a/Lib/test/test_asyncio/test_selector_events.py
+++ b/Lib/test/test_asyncio/test_selector_events.py
@@ -121,8 +121,9 @@
         self.assertIsNone(self.loop._write_to_self())
 
     def test_write_to_self_exception(self):
-        self.loop._csock.send.side_effect = OSError()
-        self.assertRaises(OSError, self.loop._write_to_self)
+        # _write_to_self() swallows OSError
+        self.loop._csock.send.side_effect = RuntimeError()
+        self.assertRaises(RuntimeError, self.loop._write_to_self)
 
     def test_sock_recv(self):
         sock = mock.Mock()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May  6 23:47:25 2014
From: python-checkins at python.org (guido.van.rossum)
Date: Tue,  6 May 2014 23:47:25 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge_3=2E4-=3Edefault=3A_asyncio=3A_Fix_the_second_half?=
 =?utf-8?q?_of_issue_=2321447=3A_race_in?=
Message-ID: <3gNZKT4rXSz7Ljb@mail.python.org>

http://hg.python.org/cpython/rev/c0538334f4df
changeset:   90576:c0538334f4df
parent:      90574:6234f4caba57
parent:      90575:afe8c4bd3230
user:        Guido van Rossum <guido at python.org>
date:        Tue May 06 14:45:39 2014 -0700
summary:
  Merge 3.4->default: asyncio: Fix the second half of issue #21447: race in _write_to_self().

files:
  Lib/asyncio/selector_events.py                |  15 +++++++--
  Lib/test/test_asyncio/test_selector_events.py |   5 ++-
  2 files changed, 14 insertions(+), 6 deletions(-)


diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py
--- a/Lib/asyncio/selector_events.py
+++ b/Lib/asyncio/selector_events.py
@@ -87,10 +87,17 @@
             pass
 
     def _write_to_self(self):
-        try:
-            self._csock.send(b'x')
-        except (BlockingIOError, InterruptedError):
-            pass
+        # This may be called from a different thread, possibly after
+        # _close_self_pipe() has been called or even while it is
+        # running.  Guard for self._csock being None or closed.  When
+        # a socket is closed, send() raises OSError (with errno set to
+        # EBADF, but let's not rely on the exact error code).
+        csock = self._csock
+        if csock is not None:
+            try:
+                csock.send(b'x')
+            except OSError:
+                pass
 
     def _start_serving(self, protocol_factory, sock,
                        sslcontext=None, server=None):
diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py
--- a/Lib/test/test_asyncio/test_selector_events.py
+++ b/Lib/test/test_asyncio/test_selector_events.py
@@ -121,8 +121,9 @@
         self.assertIsNone(self.loop._write_to_self())
 
     def test_write_to_self_exception(self):
-        self.loop._csock.send.side_effect = OSError()
-        self.assertRaises(OSError, self.loop._write_to_self)
+        # _write_to_self() swallows OSError
+        self.loop._csock.send.side_effect = RuntimeError()
+        self.assertRaises(RuntimeError, self.loop._write_to_self)
 
     def test_sock_recv(self):
         sock = mock.Mock()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May  7 03:34:18 2014
From: python-checkins at python.org (r.david.murray)
Date: Wed,  7 May 2014 03:34:18 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogIzIxMzAwOiBDbGVh?=
 =?utf-8?q?n_up_the_docs_for_the_email_=22policy=22_arguments=2E?=
Message-ID: <3gNgMG3v4Gz7Ljc@mail.python.org>

http://hg.python.org/cpython/rev/d994d75cce95
changeset:   90577:d994d75cce95
branch:      3.4
parent:      90575:afe8c4bd3230
user:        R David Murray <rdmurray at bitdance.com>
date:        Tue May 06 21:33:18 2014 -0400
summary:
  #21300: Clean up the docs for the email "policy" arguments.

files:
  Doc/library/email.generator.rst |  10 +++--
  Doc/library/email.parser.rst    |  41 ++++++++++++--------
  Lib/email/generator.py          |   9 +++-
  3 files changed, 37 insertions(+), 23 deletions(-)


diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst
--- a/Doc/library/email.generator.rst
+++ b/Doc/library/email.generator.rst
@@ -112,7 +112,7 @@
 :mod:`email.message`.
 
 .. class:: BytesGenerator(outfp, mangle_from_=True, maxheaderlen=78, *, \
-                          policy=policy.default)
+                          policy=None)
 
    The constructor for the :class:`BytesGenerator` class takes a binary
    :term:`file-like object` called *outfp* for an argument.  *outfp* must
@@ -134,9 +134,11 @@
    wrapping.  The default is 78, as recommended (but not required) by
    :rfc:`2822`.
 
+
    The *policy* keyword specifies a :mod:`~email.policy` object that controls a
-   number of aspects of the generator's operation.  The default policy
-   maintains backward compatibility.
+   number of aspects of the generator's operation.  If no *policy* is specified,
+   then the *policy* attached to the message object passed to :attr:`flatten`
+   is used.
 
    .. versionchanged:: 3.3 Added the *policy* keyword.
 
@@ -174,7 +176,7 @@
 
       Optional *linesep* specifies the line separator character used to
       terminate lines in the output.  If specified it overrides the value
-      specified by the ``Generator``\ 's ``policy``.
+      specified by the ``Generator``\ or *msg*\ 's ``policy``.
 
    .. method:: clone(fp)
 
diff --git a/Doc/library/email.parser.rst b/Doc/library/email.parser.rst
--- a/Doc/library/email.parser.rst
+++ b/Doc/library/email.parser.rst
@@ -60,15 +60,18 @@
 Here is the API for the :class:`FeedParser`:
 
 
-.. class:: FeedParser(_factory=email.message.Message, *, policy=policy.default)
+.. class:: FeedParser(_factory=email.message.Message, *, policy=policy.compat32)
 
    Create a :class:`FeedParser` instance.  Optional *_factory* is a no-argument
    callable that will be called whenever a new message object is needed.  It
    defaults to the :class:`email.message.Message` class.
 
-   The *policy* keyword specifies a :mod:`~email.policy` object that controls a
-   number of aspects of the parser's operation.  The default policy maintains
-   backward compatibility.
+   If *policy* is specified (it must be an instance of a :mod:`~email.policy`
+   class) use the rules it specifies to udpate the representation of the
+   message.  If *policy* is not set, use the :class:`compat32
+   <email.policy.Compat32>` policy, which maintains backward compatibility with
+   the Python 3.2 version of the email package.  For more information see the
+   :mod:`~email.policy` documentation.
 
    .. versionchanged:: 3.3 Added the *policy* keyword.
 
@@ -113,7 +116,7 @@
    The BytesHeaderParser class.
 
 
-.. class:: Parser(_class=email.message.Message, *, policy=policy.default)
+.. class:: Parser(_class=email.message.Message, *, policy=policy.compat32)
 
    The constructor for the :class:`Parser` class takes an optional argument
    *_class*.  This must be a callable factory (such as a function or a class), and
@@ -121,9 +124,12 @@
    :class:`~email.message.Message` (see :mod:`email.message`).  The factory will
    be called without arguments.
 
-   The *policy* keyword specifies a :mod:`~email.policy` object that controls a
-   number of aspects of the parser's operation.  The default policy maintains
-   backward compatibility.
+   If *policy* is specified (it must be an instance of a :mod:`~email.policy`
+   class) use the rules it specifies to udpate the representation of the
+   message.  If *policy* is not set, use the :class:`compat32
+   <email.policy.Compat32>` policy, which maintains backward compatibility with
+   the Python 3.2 version of the email package.  For more information see the
+   :mod:`~email.policy` documentation.
 
    .. versionchanged:: 3.3
       Removed the *strict* argument that was deprecated in 2.4.  Added the
@@ -159,15 +165,18 @@
       Optional *headersonly* is as with the :meth:`parse` method.
 
 
-.. class:: BytesParser(_class=email.message.Message, *, policy=policy.default)
+.. class:: BytesParser(_class=email.message.Message, *, policy=policy.compat32)
 
    This class is exactly parallel to :class:`Parser`, but handles bytes input.
    The *_class* and *strict* arguments are interpreted in the same way as for
    the :class:`Parser` constructor.
 
-   The *policy* keyword specifies a :mod:`~email.policy` object that
-   controls a number of aspects of the parser's operation.  The default
-   policy maintains backward compatibility.
+   If *policy* is specified (it must be an instance of a :mod:`~email.policy`
+   class) use the rules it specifies to udpate the representation of the
+   message.  If *policy* is not set, use the :class:`compat32
+   <email.policy.Compat32>` policy, which maintains backward compatibility with
+   the Python 3.2 version of the email package.  For more information see the
+   :mod:`~email.policy` documentation.
 
    .. versionchanged:: 3.3
       Removed the *strict* argument.  Added the *policy* keyword.
@@ -209,7 +218,7 @@
 .. currentmodule:: email
 
 .. function:: message_from_string(s, _class=email.message.Message, *, \
-                                  policy=policy.default)
+                                  policy=policy.compat32)
 
    Return a message object structure from a string.  This is exactly equivalent to
    ``Parser().parsestr(s)``.  *_class* and *policy* are interpreted as
@@ -219,7 +228,7 @@
       Removed the *strict* argument.  Added the *policy* keyword.
 
 .. function:: message_from_bytes(s, _class=email.message.Message, *, \
-                                 policy=policy.default)
+                                 policy=policy.compat32)
 
    Return a message object structure from a byte string.  This is exactly
    equivalent to ``BytesParser().parsebytes(s)``.  Optional *_class* and
@@ -231,7 +240,7 @@
       Removed the *strict* argument.  Added the *policy* keyword.
 
 .. function:: message_from_file(fp, _class=email.message.Message, *, \
-                                policy=policy.default)
+                                policy=policy.compat32)
 
    Return a message object structure tree from an open :term:`file object`.
    This is exactly equivalent to ``Parser().parse(fp)``.  *_class*
@@ -242,7 +251,7 @@
       Removed the *strict* argument.  Added the *policy* keyword.
 
 .. function:: message_from_binary_file(fp, _class=email.message.Message, *, \
-                                       policy=policy.default)
+                                       policy=policy.compat32)
 
    Return a message object structure tree from an open binary :term:`file
    object`.  This is exactly equivalent to ``BytesParser().parse(fp)``.
diff --git a/Lib/email/generator.py b/Lib/email/generator.py
--- a/Lib/email/generator.py
+++ b/Lib/email/generator.py
@@ -51,8 +51,9 @@
         by RFC 2822.
 
         The policy keyword specifies a policy object that controls a number of
-        aspects of the generator's operation.  The default policy maintains
-        backward compatibility.
+        aspects of the generator's operation.  If no policy is specified,
+        the policy associated with the Message object passed to the
+        flatten method is used.
 
         """
         self._fp = outfp
@@ -76,7 +77,9 @@
         Note that for subobjects, no From_ line is printed.
 
         linesep specifies the characters used to indicate a new line in
-        the output.  The default value is determined by the policy.
+        the output.  The default value is determined by the policy specified
+        when the Generator instance was created or, if none was specified,
+        from the policy associated with the msg.
 
         """
         # We use the _XXX constants for operating on data that comes directly

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May  7 03:34:19 2014
From: python-checkins at python.org (r.david.murray)
Date: Wed,  7 May 2014 03:34:19 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge=3A_=2321300=3A_Clean_up_the_docs_for_the_email_=22?=
 =?utf-8?q?policy=22_arguments=2E?=
Message-ID: <3gNgMH6f5Pz7Lk4@mail.python.org>

http://hg.python.org/cpython/rev/63fa945119cb
changeset:   90578:63fa945119cb
parent:      90576:c0538334f4df
parent:      90577:d994d75cce95
user:        R David Murray <rdmurray at bitdance.com>
date:        Tue May 06 21:33:50 2014 -0400
summary:
  Merge: #21300: Clean up the docs for the email "policy" arguments.

files:
  Doc/library/email.generator.rst |  10 +++--
  Doc/library/email.parser.rst    |  41 ++++++++++++--------
  Lib/email/generator.py          |   9 +++-
  3 files changed, 37 insertions(+), 23 deletions(-)


diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst
--- a/Doc/library/email.generator.rst
+++ b/Doc/library/email.generator.rst
@@ -112,7 +112,7 @@
 :mod:`email.message`.
 
 .. class:: BytesGenerator(outfp, mangle_from_=True, maxheaderlen=78, *, \
-                          policy=policy.default)
+                          policy=None)
 
    The constructor for the :class:`BytesGenerator` class takes a binary
    :term:`file-like object` called *outfp* for an argument.  *outfp* must
@@ -134,9 +134,11 @@
    wrapping.  The default is 78, as recommended (but not required) by
    :rfc:`2822`.
 
+
    The *policy* keyword specifies a :mod:`~email.policy` object that controls a
-   number of aspects of the generator's operation.  The default policy
-   maintains backward compatibility.
+   number of aspects of the generator's operation.  If no *policy* is specified,
+   then the *policy* attached to the message object passed to :attr:`flatten`
+   is used.
 
    .. versionchanged:: 3.3 Added the *policy* keyword.
 
@@ -174,7 +176,7 @@
 
       Optional *linesep* specifies the line separator character used to
       terminate lines in the output.  If specified it overrides the value
-      specified by the ``Generator``\ 's ``policy``.
+      specified by the ``Generator``\ or *msg*\ 's ``policy``.
 
    .. method:: clone(fp)
 
diff --git a/Doc/library/email.parser.rst b/Doc/library/email.parser.rst
--- a/Doc/library/email.parser.rst
+++ b/Doc/library/email.parser.rst
@@ -60,15 +60,18 @@
 Here is the API for the :class:`FeedParser`:
 
 
-.. class:: FeedParser(_factory=email.message.Message, *, policy=policy.default)
+.. class:: FeedParser(_factory=email.message.Message, *, policy=policy.compat32)
 
    Create a :class:`FeedParser` instance.  Optional *_factory* is a no-argument
    callable that will be called whenever a new message object is needed.  It
    defaults to the :class:`email.message.Message` class.
 
-   The *policy* keyword specifies a :mod:`~email.policy` object that controls a
-   number of aspects of the parser's operation.  The default policy maintains
-   backward compatibility.
+   If *policy* is specified (it must be an instance of a :mod:`~email.policy`
+   class) use the rules it specifies to udpate the representation of the
+   message.  If *policy* is not set, use the :class:`compat32
+   <email.policy.Compat32>` policy, which maintains backward compatibility with
+   the Python 3.2 version of the email package.  For more information see the
+   :mod:`~email.policy` documentation.
 
    .. versionchanged:: 3.3 Added the *policy* keyword.
 
@@ -113,7 +116,7 @@
    The BytesHeaderParser class.
 
 
-.. class:: Parser(_class=email.message.Message, *, policy=policy.default)
+.. class:: Parser(_class=email.message.Message, *, policy=policy.compat32)
 
    The constructor for the :class:`Parser` class takes an optional argument
    *_class*.  This must be a callable factory (such as a function or a class), and
@@ -121,9 +124,12 @@
    :class:`~email.message.Message` (see :mod:`email.message`).  The factory will
    be called without arguments.
 
-   The *policy* keyword specifies a :mod:`~email.policy` object that controls a
-   number of aspects of the parser's operation.  The default policy maintains
-   backward compatibility.
+   If *policy* is specified (it must be an instance of a :mod:`~email.policy`
+   class) use the rules it specifies to udpate the representation of the
+   message.  If *policy* is not set, use the :class:`compat32
+   <email.policy.Compat32>` policy, which maintains backward compatibility with
+   the Python 3.2 version of the email package.  For more information see the
+   :mod:`~email.policy` documentation.
 
    .. versionchanged:: 3.3
       Removed the *strict* argument that was deprecated in 2.4.  Added the
@@ -159,15 +165,18 @@
       Optional *headersonly* is as with the :meth:`parse` method.
 
 
-.. class:: BytesParser(_class=email.message.Message, *, policy=policy.default)
+.. class:: BytesParser(_class=email.message.Message, *, policy=policy.compat32)
 
    This class is exactly parallel to :class:`Parser`, but handles bytes input.
    The *_class* and *strict* arguments are interpreted in the same way as for
    the :class:`Parser` constructor.
 
-   The *policy* keyword specifies a :mod:`~email.policy` object that
-   controls a number of aspects of the parser's operation.  The default
-   policy maintains backward compatibility.
+   If *policy* is specified (it must be an instance of a :mod:`~email.policy`
+   class) use the rules it specifies to udpate the representation of the
+   message.  If *policy* is not set, use the :class:`compat32
+   <email.policy.Compat32>` policy, which maintains backward compatibility with
+   the Python 3.2 version of the email package.  For more information see the
+   :mod:`~email.policy` documentation.
 
    .. versionchanged:: 3.3
       Removed the *strict* argument.  Added the *policy* keyword.
@@ -209,7 +218,7 @@
 .. currentmodule:: email
 
 .. function:: message_from_string(s, _class=email.message.Message, *, \
-                                  policy=policy.default)
+                                  policy=policy.compat32)
 
    Return a message object structure from a string.  This is exactly equivalent to
    ``Parser().parsestr(s)``.  *_class* and *policy* are interpreted as
@@ -219,7 +228,7 @@
       Removed the *strict* argument.  Added the *policy* keyword.
 
 .. function:: message_from_bytes(s, _class=email.message.Message, *, \
-                                 policy=policy.default)
+                                 policy=policy.compat32)
 
    Return a message object structure from a byte string.  This is exactly
    equivalent to ``BytesParser().parsebytes(s)``.  Optional *_class* and
@@ -231,7 +240,7 @@
       Removed the *strict* argument.  Added the *policy* keyword.
 
 .. function:: message_from_file(fp, _class=email.message.Message, *, \
-                                policy=policy.default)
+                                policy=policy.compat32)
 
    Return a message object structure tree from an open :term:`file object`.
    This is exactly equivalent to ``Parser().parse(fp)``.  *_class*
@@ -242,7 +251,7 @@
       Removed the *strict* argument.  Added the *policy* keyword.
 
 .. function:: message_from_binary_file(fp, _class=email.message.Message, *, \
-                                       policy=policy.default)
+                                       policy=policy.compat32)
 
    Return a message object structure tree from an open binary :term:`file
    object`.  This is exactly equivalent to ``BytesParser().parse(fp)``.
diff --git a/Lib/email/generator.py b/Lib/email/generator.py
--- a/Lib/email/generator.py
+++ b/Lib/email/generator.py
@@ -51,8 +51,9 @@
         by RFC 2822.
 
         The policy keyword specifies a policy object that controls a number of
-        aspects of the generator's operation.  The default policy maintains
-        backward compatibility.
+        aspects of the generator's operation.  If no policy is specified,
+        the policy associated with the Message object passed to the
+        flatten method is used.
 
         """
         self._fp = outfp
@@ -76,7 +77,9 @@
         Note that for subobjects, no From_ line is printed.
 
         linesep specifies the characters used to indicate a new line in
-        the output.  The default value is determined by the policy.
+        the output.  The default value is determined by the policy specified
+        when the Generator instance was created or, if none was specified,
+        from the policy associated with the msg.
 
         """
         # We use the _XXX constants for operating on data that comes directly

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May  7 04:42:04 2014
From: python-checkins at python.org (matthias.klose)
Date: Wed,  7 May 2014 04:42:04 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogIC0gSXNzdWUgIzE3?=
 =?utf-8?q?752=3A_Fix_distutils_tests_when_run_from_the_installed_location?=
 =?utf-8?q?=2E?=
Message-ID: <3gNhsS0SkFz7LjY@mail.python.org>

http://hg.python.org/cpython/rev/faef1da30c6d
changeset:   90579:faef1da30c6d
branch:      2.7
parent:      90573:87dcfe8427d8
user:        doko at ubuntu.com
date:        Wed May 07 04:41:26 2014 +0200
summary:
   - Issue #17752: Fix distutils tests when run from the installed location.

files:
  Lib/distutils/tests/support.py |   2 +-
  Misc/NEWS                      |   2 ++
  configure                      |  14 +++++++-------
  configure.ac                   |  14 +++++++-------
  4 files changed, 17 insertions(+), 15 deletions(-)


diff --git a/Lib/distutils/tests/support.py b/Lib/distutils/tests/support.py
--- a/Lib/distutils/tests/support.py
+++ b/Lib/distutils/tests/support.py
@@ -218,4 +218,4 @@
                 cmd.library_dirs = []
             else:
                 name, equals, value = runshared.partition('=')
-                cmd.library_dirs = value.split(os.pathsep)
+                cmd.library_dirs = [d for d in value.split(os.pathsep) if d]
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -329,6 +329,8 @@
 Tests
 -----
 
+- Issue #17752: Fix distutils tests when run from the installed location.
+
 - Issue #18604: Consolidated checks for GUI availability.  All platforms now
   at least check whether Tk can be instantiated when the GUI resource is
   requested.
diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -5326,7 +5326,7 @@
 if test "$enable_framework"
 then
   LDLIBRARY='$(PYTHONFRAMEWORKDIR)/Versions/$(VERSION)/$(PYTHONFRAMEWORK)'
-  RUNSHARED=DYLD_FRAMEWORK_PATH="`pwd`:$DYLD_FRAMEWORK_PATH"
+  RUNSHARED=DYLD_FRAMEWORK_PATH=`pwd`${DYLD_FRAMEWORK_PATH:+:${DYLD_FRAMEWORK_PATH}}
   BLDLIBRARY=''
 else
   BLDLIBRARY='$(LDLIBRARY)'
@@ -5348,13 +5348,13 @@
     SunOS*)
 	  LDLIBRARY='libpython$(VERSION).so'
 	  BLDLIBRARY='-Wl,-R,$(LIBDIR) -L. -lpython$(VERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  INSTSONAME="$LDLIBRARY".$SOVERSION
           ;;
     Linux*|GNU*|NetBSD*|FreeBSD*|DragonFly*|OpenBSD*)
 	  LDLIBRARY='libpython$(VERSION).so'
 	  BLDLIBRARY='-L. -lpython$(VERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  case $ac_sys_system in
 	      FreeBSD*)
 		SOVERSION=`echo $SOVERSION|cut -d "." -f 1`
@@ -5372,12 +5372,12 @@
 			;;
 	  esac
 	  BLDLIBRARY='-Wl,+b,$(LIBDIR) -L. -lpython$(VERSION)'
-	  RUNSHARED=SHLIB_PATH=`pwd`:${SHLIB_PATH}
+	  RUNSHARED=SHLIB_PATH=`pwd`${SHLIB_PATH:+:${SHLIB_PATH}}
 	  ;;
     OSF*)
 	  LDLIBRARY='libpython$(VERSION).so'
 	  BLDLIBRARY='-rpath $(LIBDIR) -L. -lpython$(VERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  ;;
     atheos*)
 	  LDLIBRARY='libpython$(VERSION).so'
@@ -5387,11 +5387,11 @@
     Darwin*)
     	LDLIBRARY='libpython$(VERSION).dylib'
 	BLDLIBRARY='-L. -lpython$(VERSION)'
-	RUNSHARED='DYLD_LIBRARY_PATH=`pwd`:${DYLD_LIBRARY_PATH}'
+	RUNSHARED=DYLD_LIBRARY_PATH=`pwd`${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}
 	;;
     AIX*)
 	LDLIBRARY='libpython$(VERSION).so'
-	RUNSHARED=LIBPATH=`pwd`:${LIBPATH}
+	RUNSHARED=LIBPATH=`pwd`${LIBPATH:+:${LIBPATH}}
 	;;
 
   esac
diff --git a/configure.ac b/configure.ac
--- a/configure.ac
+++ b/configure.ac
@@ -860,7 +860,7 @@
 if test "$enable_framework"
 then
   LDLIBRARY='$(PYTHONFRAMEWORKDIR)/Versions/$(VERSION)/$(PYTHONFRAMEWORK)'
-  RUNSHARED=DYLD_FRAMEWORK_PATH="`pwd`:$DYLD_FRAMEWORK_PATH"
+  RUNSHARED=DYLD_FRAMEWORK_PATH=`pwd`${DYLD_FRAMEWORK_PATH:+:${DYLD_FRAMEWORK_PATH}}
   BLDLIBRARY=''
 else
   BLDLIBRARY='$(LDLIBRARY)'
@@ -880,13 +880,13 @@
     SunOS*)
 	  LDLIBRARY='libpython$(VERSION).so'
 	  BLDLIBRARY='-Wl,-R,$(LIBDIR) -L. -lpython$(VERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  INSTSONAME="$LDLIBRARY".$SOVERSION
           ;;
     Linux*|GNU*|NetBSD*|FreeBSD*|DragonFly*|OpenBSD*)
 	  LDLIBRARY='libpython$(VERSION).so'
 	  BLDLIBRARY='-L. -lpython$(VERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  case $ac_sys_system in
 	      FreeBSD*)
 		SOVERSION=`echo $SOVERSION|cut -d "." -f 1`
@@ -904,12 +904,12 @@
 			;;
 	  esac
 	  BLDLIBRARY='-Wl,+b,$(LIBDIR) -L. -lpython$(VERSION)'
-	  RUNSHARED=SHLIB_PATH=`pwd`:${SHLIB_PATH}
+	  RUNSHARED=SHLIB_PATH=`pwd`${SHLIB_PATH:+:${SHLIB_PATH}}
 	  ;;
     OSF*)
 	  LDLIBRARY='libpython$(VERSION).so'
 	  BLDLIBRARY='-rpath $(LIBDIR) -L. -lpython$(VERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  ;;
     atheos*)
 	  LDLIBRARY='libpython$(VERSION).so'
@@ -919,11 +919,11 @@
     Darwin*)
     	LDLIBRARY='libpython$(VERSION).dylib'
 	BLDLIBRARY='-L. -lpython$(VERSION)'
-	RUNSHARED='DYLD_LIBRARY_PATH=`pwd`:${DYLD_LIBRARY_PATH}'
+	RUNSHARED=DYLD_LIBRARY_PATH=`pwd`${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}
 	;;
     AIX*)
 	LDLIBRARY='libpython$(VERSION).so'
-	RUNSHARED=LIBPATH=`pwd`:${LIBPATH}
+	RUNSHARED=LIBPATH=`pwd`${LIBPATH:+:${LIBPATH}}
 	;;
 
   esac

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Wed May  7 09:56:12 2014
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Wed, 07 May 2014 09:56:12 +0200
Subject: [Python-checkins] Daily reference leaks (c0538334f4df): sum=-1
Message-ID: <E1Whwi4-00055e-H5@vds2544.sivit.org>

results for c0538334f4df on branch "default"
--------------------------------------------

test_functools leaked [0, 0, 3] memory blocks, sum=3
test_site leaked [0, 0, -2] references, sum=-2
test_site leaked [0, 0, -2] memory blocks, sum=-2


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogybM3bD', '-x']

From python-checkins at python.org  Wed May  7 13:09:15 2014
From: python-checkins at python.org (matthias.klose)
Date: Wed,  7 May 2014 13:09:15 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogLSBJc3N1ZSAjMTc3?=
 =?utf-8?q?52=3A_Fix_distutils_tests_when_run_from_the_installed_location?=
 =?utf-8?q?=2E?=
Message-ID: <3gNw6g1nKvz7LjR@mail.python.org>

http://hg.python.org/cpython/rev/7d1929cc08dd
changeset:   90580:7d1929cc08dd
branch:      3.3
parent:      90523:0a4b211b927e
user:        doko at ubuntu.com
date:        Wed May 07 04:44:42 2014 +0200
summary:
  - Issue #17752: Fix distutils tests when run from the installed location.

files:
  Lib/distutils/tests/support.py |   2 +-
  Misc/NEWS                      |   2 ++
  configure                      |  12 ++++++------
  configure.ac                   |  12 ++++++------
  4 files changed, 15 insertions(+), 13 deletions(-)


diff --git a/Lib/distutils/tests/support.py b/Lib/distutils/tests/support.py
--- a/Lib/distutils/tests/support.py
+++ b/Lib/distutils/tests/support.py
@@ -207,4 +207,4 @@
                 cmd.library_dirs = []
             else:
                 name, equals, value = runshared.partition('=')
-                cmd.library_dirs = value.split(os.pathsep)
+                cmd.library_dirs = [d for d in value.split(os.pathsep) if d]
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -38,6 +38,8 @@
 Tests
 -----
 
+- Issue #17752: Fix distutils tests when run from the installed location.
+
 - Issue #20946: Correct alignment assumptions of some ctypes tests.
 
 - Issue #20939: Fix test_geturl failure in test_urllibnet due to
diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -5576,7 +5576,7 @@
 if test "$enable_framework"
 then
   LDLIBRARY='$(PYTHONFRAMEWORKDIR)/Versions/$(VERSION)/$(PYTHONFRAMEWORK)'
-  RUNSHARED=DYLD_FRAMEWORK_PATH="`pwd`:$DYLD_FRAMEWORK_PATH"
+  RUNSHARED=DYLD_FRAMEWORK_PATH=`pwd`${DYLD_FRAMEWORK_PATH:+:${DYLD_FRAMEWORK_PATH}}
   BLDLIBRARY=''
 else
   BLDLIBRARY='$(LDLIBRARY)'
@@ -5595,7 +5595,7 @@
     SunOS*)
 	  LDLIBRARY='libpython$(LDVERSION).so'
 	  BLDLIBRARY='-Wl,-R,$(LIBDIR) -L. -lpython$(LDVERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  INSTSONAME="$LDLIBRARY".$SOVERSION
 	  if test "$with_pydebug" != yes
 	  then
@@ -5605,7 +5605,7 @@
     Linux*|GNU*|NetBSD*|FreeBSD*|DragonFly*|OpenBSD*)
 	  LDLIBRARY='libpython$(LDVERSION).so'
 	  BLDLIBRARY='-L. -lpython$(LDVERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  case $ac_sys_system in
 	      FreeBSD*)
 		SOVERSION=`echo $SOVERSION|cut -d "." -f 1`
@@ -5627,16 +5627,16 @@
 			;;
 	  esac
 	  BLDLIBRARY='-Wl,+b,$(LIBDIR) -L. -lpython$(LDVERSION)'
-	  RUNSHARED=SHLIB_PATH=`pwd`:${SHLIB_PATH}
+	  RUNSHARED=SHLIB_PATH=`pwd`${SHLIB_PATH:+:${SHLIB_PATH}}
 	  ;;
     Darwin*)
     	LDLIBRARY='libpython$(LDVERSION).dylib'
 	BLDLIBRARY='-L. -lpython$(LDVERSION)'
-	RUNSHARED='DYLD_LIBRARY_PATH=`pwd`:${DYLD_LIBRARY_PATH}'
+	RUNSHARED='DYLD_LIBRARY_PATH=`pwd`${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}'
 	;;
     AIX*)
 	LDLIBRARY='libpython$(LDVERSION).so'
-	RUNSHARED=LIBPATH=`pwd`:${LIBPATH}
+	RUNSHARED=LIBPATH=`pwd`${LIBPATH:+:${LIBPATH}}
 	;;
 
   esac
diff --git a/configure.ac b/configure.ac
--- a/configure.ac
+++ b/configure.ac
@@ -909,7 +909,7 @@
 if test "$enable_framework"
 then
   LDLIBRARY='$(PYTHONFRAMEWORKDIR)/Versions/$(VERSION)/$(PYTHONFRAMEWORK)'
-  RUNSHARED=DYLD_FRAMEWORK_PATH="`pwd`:$DYLD_FRAMEWORK_PATH"
+  RUNSHARED=DYLD_FRAMEWORK_PATH=`pwd`${DYLD_FRAMEWORK_PATH:+:${DYLD_FRAMEWORK_PATH}}
   BLDLIBRARY=''
 else
   BLDLIBRARY='$(LDLIBRARY)'
@@ -926,7 +926,7 @@
     SunOS*)
 	  LDLIBRARY='libpython$(LDVERSION).so'
 	  BLDLIBRARY='-Wl,-R,$(LIBDIR) -L. -lpython$(LDVERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  INSTSONAME="$LDLIBRARY".$SOVERSION
 	  if test "$with_pydebug" != yes
 	  then
@@ -936,7 +936,7 @@
     Linux*|GNU*|NetBSD*|FreeBSD*|DragonFly*|OpenBSD*)
 	  LDLIBRARY='libpython$(LDVERSION).so'
 	  BLDLIBRARY='-L. -lpython$(LDVERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  case $ac_sys_system in
 	      FreeBSD*)
 		SOVERSION=`echo $SOVERSION|cut -d "." -f 1`
@@ -958,16 +958,16 @@
 			;;
 	  esac
 	  BLDLIBRARY='-Wl,+b,$(LIBDIR) -L. -lpython$(LDVERSION)'
-	  RUNSHARED=SHLIB_PATH=`pwd`:${SHLIB_PATH}
+	  RUNSHARED=SHLIB_PATH=`pwd`${SHLIB_PATH:+:${SHLIB_PATH}}
 	  ;;
     Darwin*)
     	LDLIBRARY='libpython$(LDVERSION).dylib'
 	BLDLIBRARY='-L. -lpython$(LDVERSION)'
-	RUNSHARED='DYLD_LIBRARY_PATH=`pwd`:${DYLD_LIBRARY_PATH}'
+	RUNSHARED='DYLD_LIBRARY_PATH=`pwd`${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}'
 	;;
     AIX*)
 	LDLIBRARY='libpython$(LDVERSION).so'
-	RUNSHARED=LIBPATH=`pwd`:${LIBPATH}
+	RUNSHARED=LIBPATH=`pwd`${LIBPATH:+:${LIBPATH}}
 	;;
 
   esac

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May  7 13:09:16 2014
From: python-checkins at python.org (matthias.klose)
Date: Wed,  7 May 2014 13:09:16 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogLSBJc3N1ZSAjMTc3?=
 =?utf-8?q?52=3A_Fix_distutils_tests_when_run_from_the_installed_location?=
 =?utf-8?q?=2E?=
Message-ID: <3gNw6h5JTNz7Ljy@mail.python.org>

http://hg.python.org/cpython/rev/01e933cb1de9
changeset:   90581:01e933cb1de9
branch:      3.4
parent:      90577:d994d75cce95
user:        doko at ubuntu.com
date:        Wed May 07 12:57:44 2014 +0200
summary:
  - Issue #17752: Fix distutils tests when run from the installed location.

files:
  Lib/distutils/tests/support.py |   2 +-
  Misc/NEWS                      |  13 ++++++++++++-
  configure                      |  12 ++++++------
  configure.ac                   |  12 ++++++------
  4 files changed, 25 insertions(+), 14 deletions(-)


diff --git a/Lib/distutils/tests/support.py b/Lib/distutils/tests/support.py
--- a/Lib/distutils/tests/support.py
+++ b/Lib/distutils/tests/support.py
@@ -207,4 +207,4 @@
                 cmd.library_dirs = []
             else:
                 name, equals, value = runshared.partition('=')
-                cmd.library_dirs = value.split(os.pathsep)
+                cmd.library_dirs = [d for d in value.split(os.pathsep) if d]
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -2,10 +2,21 @@
 Python News
 +++++++++++
 
+What's New in Python 3.4.1?
+==========================
+
+Release date: TBA
+
+Tests
+-----
+
+- Issue #17752: Fix distutils tests when run from the installed location.
+
+
 What's New in Python 3.4.1rc1?
 ==============================
 
-Release date: TBA
+Release date: 2014-05-05
 
 Core and Builtins
 -----------------
diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -5605,7 +5605,7 @@
 if test "$enable_framework"
 then
   LDLIBRARY='$(PYTHONFRAMEWORKDIR)/Versions/$(VERSION)/$(PYTHONFRAMEWORK)'
-  RUNSHARED=DYLD_FRAMEWORK_PATH="`pwd`:$DYLD_FRAMEWORK_PATH"
+  RUNSHARED=DYLD_FRAMEWORK_PATH=`pwd`${DYLD_FRAMEWORK_PATH:+:${DYLD_FRAMEWORK_PATH}}
   BLDLIBRARY=''
 else
   BLDLIBRARY='$(LDLIBRARY)'
@@ -5625,7 +5625,7 @@
     SunOS*)
 	  LDLIBRARY='libpython$(LDVERSION).so'
 	  BLDLIBRARY='-Wl,-R,$(LIBDIR) -L. -lpython$(LDVERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  INSTSONAME="$LDLIBRARY".$SOVERSION
 	  if test "$with_pydebug" != yes
 	  then
@@ -5635,7 +5635,7 @@
     Linux*|GNU*|NetBSD*|FreeBSD*|DragonFly*|OpenBSD*)
 	  LDLIBRARY='libpython$(LDVERSION).so'
 	  BLDLIBRARY='-L. -lpython$(LDVERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  case $ac_sys_system in
 	      FreeBSD*)
 		SOVERSION=`echo $SOVERSION|cut -d "." -f 1`
@@ -5657,16 +5657,16 @@
 			;;
 	  esac
 	  BLDLIBRARY='-Wl,+b,$(LIBDIR) -L. -lpython$(LDVERSION)'
-	  RUNSHARED=SHLIB_PATH=`pwd`:${SHLIB_PATH}
+	  RUNSHARED=SHLIB_PATH=`pwd`${SHLIB_PATH:+:${SHLIB_PATH}}
 	  ;;
     Darwin*)
     	LDLIBRARY='libpython$(LDVERSION).dylib'
 	BLDLIBRARY='-L. -lpython$(LDVERSION)'
-	RUNSHARED='DYLD_LIBRARY_PATH=`pwd`:${DYLD_LIBRARY_PATH}'
+	RUNSHARED=DYLD_LIBRARY_PATH=`pwd`${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}
 	;;
     AIX*)
 	LDLIBRARY='libpython$(LDVERSION).so'
-	RUNSHARED=LIBPATH=`pwd`:${LIBPATH}
+	RUNSHARED=LIBPATH=`pwd`${LIBPATH:+:${LIBPATH}}
 	;;
 
   esac
diff --git a/configure.ac b/configure.ac
--- a/configure.ac
+++ b/configure.ac
@@ -928,7 +928,7 @@
 if test "$enable_framework"
 then
   LDLIBRARY='$(PYTHONFRAMEWORKDIR)/Versions/$(VERSION)/$(PYTHONFRAMEWORK)'
-  RUNSHARED=DYLD_FRAMEWORK_PATH="`pwd`:$DYLD_FRAMEWORK_PATH"
+  RUNSHARED=DYLD_FRAMEWORK_PATH=`pwd`${DYLD_FRAMEWORK_PATH:+:${DYLD_FRAMEWORK_PATH}}
   BLDLIBRARY=''
 else
   BLDLIBRARY='$(LDLIBRARY)'
@@ -946,7 +946,7 @@
     SunOS*)
 	  LDLIBRARY='libpython$(LDVERSION).so'
 	  BLDLIBRARY='-Wl,-R,$(LIBDIR) -L. -lpython$(LDVERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  INSTSONAME="$LDLIBRARY".$SOVERSION
 	  if test "$with_pydebug" != yes
 	  then
@@ -956,7 +956,7 @@
     Linux*|GNU*|NetBSD*|FreeBSD*|DragonFly*|OpenBSD*)
 	  LDLIBRARY='libpython$(LDVERSION).so'
 	  BLDLIBRARY='-L. -lpython$(LDVERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  case $ac_sys_system in
 	      FreeBSD*)
 		SOVERSION=`echo $SOVERSION|cut -d "." -f 1`
@@ -978,16 +978,16 @@
 			;;
 	  esac
 	  BLDLIBRARY='-Wl,+b,$(LIBDIR) -L. -lpython$(LDVERSION)'
-	  RUNSHARED=SHLIB_PATH=`pwd`:${SHLIB_PATH}
+	  RUNSHARED=SHLIB_PATH=`pwd`${SHLIB_PATH:+:${SHLIB_PATH}}
 	  ;;
     Darwin*)
     	LDLIBRARY='libpython$(LDVERSION).dylib'
 	BLDLIBRARY='-L. -lpython$(LDVERSION)'
-	RUNSHARED='DYLD_LIBRARY_PATH=`pwd`:${DYLD_LIBRARY_PATH}'
+	RUNSHARED=DYLD_LIBRARY_PATH=`pwd`${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}
 	;;
     AIX*)
 	LDLIBRARY='libpython$(LDVERSION).so'
-	RUNSHARED=LIBPATH=`pwd`:${LIBPATH}
+	RUNSHARED=LIBPATH=`pwd`${LIBPATH:+:${LIBPATH}}
 	;;
 
   esac

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May  7 13:09:18 2014
From: python-checkins at python.org (matthias.klose)
Date: Wed,  7 May 2014 13:09:18 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_-_Issue_=2317752=3A_Fix_distutils_tests_when_run_from_th?=
 =?utf-8?q?e_installed_location=2E?=
Message-ID: <3gNw6k1Wz5z7LkF@mail.python.org>

http://hg.python.org/cpython/rev/c0bcf1383d77
changeset:   90582:c0bcf1383d77
parent:      90578:63fa945119cb
parent:      90581:01e933cb1de9
user:        doko at ubuntu.com
date:        Wed May 07 13:08:51 2014 +0200
summary:
  - Issue #17752: Fix distutils tests when run from the installed location.

files:
  Lib/distutils/tests/support.py |   2 +-
  Misc/NEWS                      |   2 ++
  configure                      |  12 ++++++------
  configure.ac                   |  12 ++++++------
  4 files changed, 15 insertions(+), 13 deletions(-)


diff --git a/Lib/distutils/tests/support.py b/Lib/distutils/tests/support.py
--- a/Lib/distutils/tests/support.py
+++ b/Lib/distutils/tests/support.py
@@ -207,4 +207,4 @@
                 cmd.library_dirs = []
             else:
                 name, equals, value = runshared.partition('=')
-                cmd.library_dirs = value.split(os.pathsep)
+                cmd.library_dirs = [d for d in value.split(os.pathsep) if d]
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -403,6 +403,8 @@
 Tests
 -----
 
+- Issue #17752: Fix distutils tests when run from the installed location.
+
 - Issue #18604: Consolidated checks for GUI availability.  All platforms now
   at least check whether Tk can be instantiated when the GUI resource is
   requested.
diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -5606,7 +5606,7 @@
 if test "$enable_framework"
 then
   LDLIBRARY='$(PYTHONFRAMEWORKDIR)/Versions/$(VERSION)/$(PYTHONFRAMEWORK)'
-  RUNSHARED=DYLD_FRAMEWORK_PATH="`pwd`:$DYLD_FRAMEWORK_PATH"
+  RUNSHARED=DYLD_FRAMEWORK_PATH=`pwd`${DYLD_FRAMEWORK_PATH:+:${DYLD_FRAMEWORK_PATH}}
   BLDLIBRARY=''
 else
   BLDLIBRARY='$(LDLIBRARY)'
@@ -5626,7 +5626,7 @@
     SunOS*)
 	  LDLIBRARY='libpython$(LDVERSION).so'
 	  BLDLIBRARY='-Wl,-R,$(LIBDIR) -L. -lpython$(LDVERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  INSTSONAME="$LDLIBRARY".$SOVERSION
 	  if test "$with_pydebug" != yes
 	  then
@@ -5636,7 +5636,7 @@
     Linux*|GNU*|NetBSD*|FreeBSD*|DragonFly*|OpenBSD*)
 	  LDLIBRARY='libpython$(LDVERSION).so'
 	  BLDLIBRARY='-L. -lpython$(LDVERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  case $ac_sys_system in
 	      FreeBSD*)
 		SOVERSION=`echo $SOVERSION|cut -d "." -f 1`
@@ -5658,16 +5658,16 @@
 			;;
 	  esac
 	  BLDLIBRARY='-Wl,+b,$(LIBDIR) -L. -lpython$(LDVERSION)'
-	  RUNSHARED=SHLIB_PATH=`pwd`:${SHLIB_PATH}
+	  RUNSHARED=SHLIB_PATH=`pwd`${SHLIB_PATH:+:${SHLIB_PATH}}
 	  ;;
     Darwin*)
     	LDLIBRARY='libpython$(LDVERSION).dylib'
 	BLDLIBRARY='-L. -lpython$(LDVERSION)'
-	RUNSHARED='DYLD_LIBRARY_PATH=`pwd`:${DYLD_LIBRARY_PATH}'
+	RUNSHARED=DYLD_LIBRARY_PATH=`pwd`${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}
 	;;
     AIX*)
 	LDLIBRARY='libpython$(LDVERSION).so'
-	RUNSHARED=LIBPATH=`pwd`:${LIBPATH}
+	RUNSHARED=LIBPATH=`pwd`${LIBPATH:+:${LIBPATH}}
 	;;
 
   esac
diff --git a/configure.ac b/configure.ac
--- a/configure.ac
+++ b/configure.ac
@@ -928,7 +928,7 @@
 if test "$enable_framework"
 then
   LDLIBRARY='$(PYTHONFRAMEWORKDIR)/Versions/$(VERSION)/$(PYTHONFRAMEWORK)'
-  RUNSHARED=DYLD_FRAMEWORK_PATH="`pwd`:$DYLD_FRAMEWORK_PATH"
+  RUNSHARED=DYLD_FRAMEWORK_PATH=`pwd`${DYLD_FRAMEWORK_PATH:+:${DYLD_FRAMEWORK_PATH}}
   BLDLIBRARY=''
 else
   BLDLIBRARY='$(LDLIBRARY)'
@@ -946,7 +946,7 @@
     SunOS*)
 	  LDLIBRARY='libpython$(LDVERSION).so'
 	  BLDLIBRARY='-Wl,-R,$(LIBDIR) -L. -lpython$(LDVERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  INSTSONAME="$LDLIBRARY".$SOVERSION
 	  if test "$with_pydebug" != yes
 	  then
@@ -956,7 +956,7 @@
     Linux*|GNU*|NetBSD*|FreeBSD*|DragonFly*|OpenBSD*)
 	  LDLIBRARY='libpython$(LDVERSION).so'
 	  BLDLIBRARY='-L. -lpython$(LDVERSION)'
-	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
+	  RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
 	  case $ac_sys_system in
 	      FreeBSD*)
 		SOVERSION=`echo $SOVERSION|cut -d "." -f 1`
@@ -978,16 +978,16 @@
 			;;
 	  esac
 	  BLDLIBRARY='-Wl,+b,$(LIBDIR) -L. -lpython$(LDVERSION)'
-	  RUNSHARED=SHLIB_PATH=`pwd`:${SHLIB_PATH}
+	  RUNSHARED=SHLIB_PATH=`pwd`${SHLIB_PATH:+:${SHLIB_PATH}}
 	  ;;
     Darwin*)
     	LDLIBRARY='libpython$(LDVERSION).dylib'
 	BLDLIBRARY='-L. -lpython$(LDVERSION)'
-	RUNSHARED='DYLD_LIBRARY_PATH=`pwd`:${DYLD_LIBRARY_PATH}'
+	RUNSHARED=DYLD_LIBRARY_PATH=`pwd`${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}
 	;;
     AIX*)
 	LDLIBRARY='libpython$(LDVERSION).so'
-	RUNSHARED=LIBPATH=`pwd`:${LIBPATH}
+	RUNSHARED=LIBPATH=`pwd`${LIBPATH:+:${LIBPATH}}
 	;;
 
   esac

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May  7 19:08:14 2014
From: python-checkins at python.org (tim.golden)
Date: Wed,  7 May 2014 19:08:14 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue19643_Add_an_example_?=
 =?utf-8?q?of_shutil=2Ermtree_which_shows_how_to_cope_with?=
Message-ID: <3gP44t65ykz7Ljy@mail.python.org>

http://hg.python.org/cpython/rev/31d63ea5dffa
changeset:   90583:31d63ea5dffa
user:        Tim Golden <mail at timgolden.me.uk>
date:        Wed May 07 18:05:45 2014 +0100
summary:
  Issue19643 Add an example of shutil.rmtree which shows how to cope with readonly files on Windows

files:
  Doc/library/shutil.rst |  20 ++++++++++++++++++++
  1 files changed, 20 insertions(+), 0 deletions(-)


diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst
--- a/Doc/library/shutil.rst
+++ b/Doc/library/shutil.rst
@@ -421,6 +421,26 @@
    copytree(source, destination, ignore=_logpath)
 
 
+.. _shutil-rmtree-example:
+
+rmtree example
+~~~~~~~~~~~~~~
+
+This example shows how to remove a directory tree on Windows where some
+of the files have their read-only bit set. It uses the onerror callback
+to clear the readonly bit and reattempt the remove. Any subsequent failure
+will propagate. ::
+
+    import os, stat
+    import shutil
+    
+    def remove_readonly(func, path, _):
+        "Clear the readonly bit and reattempt the removal"
+        os.chmod(path, stat.S_IWRITE)
+        func(path)   
+   
+    shutil.rmtree(directory, onerror=remove_readonly)
+
 .. _archiving-operations:
 
 Archiving operations

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May  7 19:08:16 2014
From: python-checkins at python.org (tim.golden)
Date: Wed,  7 May 2014 19:08:16 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue19643_Fix_whitespace?=
Message-ID: <3gP44w0JY9z7Ljy@mail.python.org>

http://hg.python.org/cpython/rev/a7560c8f38ee
changeset:   90584:a7560c8f38ee
user:        Tim Golden <mail at timgolden.me.uk>
date:        Wed May 07 18:08:08 2014 +0100
summary:
  Issue19643 Fix whitespace

files:
  Doc/library/shutil.rst |  6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst
--- a/Doc/library/shutil.rst
+++ b/Doc/library/shutil.rst
@@ -433,12 +433,12 @@
 
     import os, stat
     import shutil
-    
+
     def remove_readonly(func, path, _):
         "Clear the readonly bit and reattempt the removal"
         os.chmod(path, stat.S_IWRITE)
-        func(path)   
-   
+        func(path)
+
     shutil.rmtree(directory, onerror=remove_readonly)
 
 .. _archiving-operations:

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Thu May  8 10:01:31 2014
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Thu, 08 May 2014 10:01:31 +0200
Subject: [Python-checkins] Daily reference leaks (a7560c8f38ee): sum=58
Message-ID: <E1WiJGl-0007E1-OV@vds2544.sivit.org>

results for a7560c8f38ee on branch "default"
--------------------------------------------

test_functools leaked [0, 0, 3] memory blocks, sum=3
test_multiprocessing_forkserver leaked [38, 0, 0] references, sum=38
test_multiprocessing_forkserver leaked [17, 0, 0] memory blocks, sum=17
test_site leaked [0, -2, 2] references, sum=0
test_site leaked [0, -2, 2] memory blocks, sum=0


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogopXqFC', '-x']

From python-checkins at python.org  Thu May  8 12:37:49 2014
From: python-checkins at python.org (kristjan.jonsson)
Date: Thu,  8 May 2014 12:37:49 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_The_PyCOND=5FTIMEDWAIT_mus?=
 =?utf-8?q?t_use_microseconds_for_the_timeout_argument?=
Message-ID: <3gPWMx4Nhzz7LjS@mail.python.org>

http://hg.python.org/cpython/rev/80b76c6fad44
changeset:   90585:80b76c6fad44
user:        Kristj?n Valur J?nsson <sweskman at gmail.com>
date:        Thu May 08 10:36:27 2014 +0000
summary:
  The PyCOND_TIMEDWAIT must use microseconds for the timeout argument
in order to have the same resolution as pthreads condition variables.
At the same time, it must be large enough to accept 31 bits of
milliseconds, which is the maximum timeout value in the windows API.
A PY_LONG_LONG of microseconds fullfills both requirements.
This closes issue #20737

files:
  Python/condvar.h   |  12 ++++++------
  Python/thread_nt.h |   2 +-
  2 files changed, 7 insertions(+), 7 deletions(-)


diff --git a/Python/condvar.h b/Python/condvar.h
--- a/Python/condvar.h
+++ b/Python/condvar.h
@@ -60,7 +60,7 @@
 #include <pthread.h>
 
 #define PyCOND_ADD_MICROSECONDS(tv, interval) \
-do { \
+do { /* TODO: add overflow and truncation checks */ \
     tv.tv_usec += (long) interval; \
     tv.tv_sec += tv.tv_usec / 1000000; \
     tv.tv_usec %= 1000000; \
@@ -89,7 +89,7 @@
 
 /* return 0 for success, 1 on timeout, -1 on error */
 Py_LOCAL_INLINE(int)
-PyCOND_TIMEDWAIT(PyCOND_T *cond, PyMUTEX_T *mut, long us)
+PyCOND_TIMEDWAIT(PyCOND_T *cond, PyMUTEX_T *mut, PY_LONG_LONG us)
 {
     int r;
     struct timespec ts;
@@ -270,9 +270,9 @@
 }
 
 Py_LOCAL_INLINE(int)
-PyCOND_TIMEDWAIT(PyCOND_T *cv, PyMUTEX_T *cs, long us)
+PyCOND_TIMEDWAIT(PyCOND_T *cv, PyMUTEX_T *cs, PY_LONG_LONG us)
 {
-    return _PyCOND_WAIT_MS(cv, cs, us/1000);
+    return _PyCOND_WAIT_MS(cv, cs, (DWORD)(us/1000));
 }
 
 Py_LOCAL_INLINE(int)
@@ -363,9 +363,9 @@
  * 2 to indicate that we don't know.
  */
 Py_LOCAL_INLINE(int)
-PyCOND_TIMEDWAIT(PyCOND_T *cv, PyMUTEX_T *cs, long us)
+PyCOND_TIMEDWAIT(PyCOND_T *cv, PyMUTEX_T *cs, PY_LONG_LONG us)
 {
-    return SleepConditionVariableSRW(cv, cs, us/1000, 0) ? 2 : -1;
+    return SleepConditionVariableSRW(cv, cs, (DWORD)(us/1000), 0) ? 2 : -1;
 }
 
 Py_LOCAL_INLINE(int)
diff --git a/Python/thread_nt.h b/Python/thread_nt.h
--- a/Python/thread_nt.h
+++ b/Python/thread_nt.h
@@ -77,7 +77,7 @@
         /* wait at least until the target */
         DWORD now, target = GetTickCount() + milliseconds;
         while (mutex->locked) {
-            if (PyCOND_TIMEDWAIT(&mutex->cv, &mutex->cs, milliseconds*1000) < 0) {
+            if (PyCOND_TIMEDWAIT(&mutex->cv, &mutex->cs, (PY_LONG_LONG)milliseconds*1000) < 0) {
                 result = WAIT_FAILED;
                 break;
             }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu May  8 13:19:31 2014
From: python-checkins at python.org (kristjan.jonsson)
Date: Thu,  8 May 2014 13:19:31 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogVGhlIFB5Q09ORF9U?=
 =?utf-8?q?IMEDWAIT_must_use_microseconds_for_the_timeout_argument?=
Message-ID: <3gPXJ34WHRz7LjR@mail.python.org>

http://hg.python.org/cpython/rev/ab5e2b0fba15
changeset:   90586:ab5e2b0fba15
branch:      3.3
parent:      90580:7d1929cc08dd
user:        Kristj?n Valur J?nsson <sweskman at gmail.com>
date:        Thu May 08 10:36:27 2014 +0000
summary:
  The PyCOND_TIMEDWAIT must use microseconds for the timeout argument
in order to have the same resolution as pthreads condition variables.
At the same time, it must be large enough to accept 31 bits of
milliseconds, which is the maximum timeout value in the windows API.
A PY_LONG_LONG of microseconds fullfills both requirements.
This closes issue #20737

files:
  Python/condvar.h   |  12 ++++++------
  Python/thread_nt.h |   2 +-
  2 files changed, 7 insertions(+), 7 deletions(-)


diff --git a/Python/condvar.h b/Python/condvar.h
--- a/Python/condvar.h
+++ b/Python/condvar.h
@@ -60,7 +60,7 @@
 #include <pthread.h>
 
 #define PyCOND_ADD_MICROSECONDS(tv, interval) \
-do { \
+do { /* TODO: add overflow and truncation checks */ \
     tv.tv_usec += (long) interval; \
     tv.tv_sec += tv.tv_usec / 1000000; \
     tv.tv_usec %= 1000000; \
@@ -89,7 +89,7 @@
 
 /* return 0 for success, 1 on timeout, -1 on error */
 Py_LOCAL_INLINE(int)
-PyCOND_TIMEDWAIT(PyCOND_T *cond, PyMUTEX_T *mut, long us)
+PyCOND_TIMEDWAIT(PyCOND_T *cond, PyMUTEX_T *mut, PY_LONG_LONG us)
 {
     int r;
     struct timespec ts;
@@ -270,9 +270,9 @@
 }
 
 Py_LOCAL_INLINE(int)
-PyCOND_TIMEDWAIT(PyCOND_T *cv, PyMUTEX_T *cs, long us)
+PyCOND_TIMEDWAIT(PyCOND_T *cv, PyMUTEX_T *cs, PY_LONG_LONG us)
 {
-    return _PyCOND_WAIT_MS(cv, cs, us/1000);
+    return _PyCOND_WAIT_MS(cv, cs, (DWORD)(us/1000));
 }
 
 Py_LOCAL_INLINE(int)
@@ -363,9 +363,9 @@
  * 2 to indicate that we don't know.
  */
 Py_LOCAL_INLINE(int)
-PyCOND_TIMEDWAIT(PyCOND_T *cv, PyMUTEX_T *cs, long us)
+PyCOND_TIMEDWAIT(PyCOND_T *cv, PyMUTEX_T *cs, PY_LONG_LONG us)
 {
-    return SleepConditionVariableSRW(cv, cs, us/1000, 0) ? 2 : -1;
+    return SleepConditionVariableSRW(cv, cs, (DWORD)(us/1000), 0) ? 2 : -1;
 }
 
 Py_LOCAL_INLINE(int)
diff --git a/Python/thread_nt.h b/Python/thread_nt.h
--- a/Python/thread_nt.h
+++ b/Python/thread_nt.h
@@ -77,7 +77,7 @@
         /* wait at least until the target */
         DWORD now, target = GetTickCount() + milliseconds;
         while (mutex->locked) {
-            if (PyCOND_TIMEDWAIT(&mutex->cv, &mutex->cs, milliseconds*1000) < 0) {
+            if (PyCOND_TIMEDWAIT(&mutex->cv, &mutex->cs, (PY_LONG_LONG)milliseconds*1000) < 0) {
                 result = WAIT_FAILED;
                 break;
             }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu May  8 13:19:32 2014
From: python-checkins at python.org (kristjan.jonsson)
Date: Thu,  8 May 2014 13:19:32 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuNCk6?=
 =?utf-8?q?_Merging_from_3=2E3=3A_The_PyCOND=5FTIMEDWAIT_must_use_microsec?=
 =?utf-8?q?onds_for_the_timeout?=
Message-ID: <3gPXJ46bBHz7Ljl@mail.python.org>

http://hg.python.org/cpython/rev/7764bb7f2983
changeset:   90587:7764bb7f2983
branch:      3.4
parent:      90581:01e933cb1de9
parent:      90586:ab5e2b0fba15
user:        Kristj?n Valur J?nsson <sweskman at gmail.com>
date:        Thu May 08 10:59:52 2014 +0000
summary:
  Merging from 3.3: The PyCOND_TIMEDWAIT must use microseconds for the timeout argument
in order to have the same resolution as pthreads condition variables.
At the same time, it must be large enough to accept 31 bits of
milliseconds, which is the maximum timeout value in the windows API.
A PY_LONG_LONG of microseconds fullfills both requirements.
This closes issue #20737

files:
  Python/condvar.h   |  12 ++++++------
  Python/thread_nt.h |   2 +-
  2 files changed, 7 insertions(+), 7 deletions(-)


diff --git a/Python/condvar.h b/Python/condvar.h
--- a/Python/condvar.h
+++ b/Python/condvar.h
@@ -60,7 +60,7 @@
 #include <pthread.h>
 
 #define PyCOND_ADD_MICROSECONDS(tv, interval) \
-do { \
+do { /* TODO: add overflow and truncation checks */ \
     tv.tv_usec += (long) interval; \
     tv.tv_sec += tv.tv_usec / 1000000; \
     tv.tv_usec %= 1000000; \
@@ -89,7 +89,7 @@
 
 /* return 0 for success, 1 on timeout, -1 on error */
 Py_LOCAL_INLINE(int)
-PyCOND_TIMEDWAIT(PyCOND_T *cond, PyMUTEX_T *mut, long us)
+PyCOND_TIMEDWAIT(PyCOND_T *cond, PyMUTEX_T *mut, PY_LONG_LONG us)
 {
     int r;
     struct timespec ts;
@@ -270,9 +270,9 @@
 }
 
 Py_LOCAL_INLINE(int)
-PyCOND_TIMEDWAIT(PyCOND_T *cv, PyMUTEX_T *cs, long us)
+PyCOND_TIMEDWAIT(PyCOND_T *cv, PyMUTEX_T *cs, PY_LONG_LONG us)
 {
-    return _PyCOND_WAIT_MS(cv, cs, us/1000);
+    return _PyCOND_WAIT_MS(cv, cs, (DWORD)(us/1000));
 }
 
 Py_LOCAL_INLINE(int)
@@ -363,9 +363,9 @@
  * 2 to indicate that we don't know.
  */
 Py_LOCAL_INLINE(int)
-PyCOND_TIMEDWAIT(PyCOND_T *cv, PyMUTEX_T *cs, long us)
+PyCOND_TIMEDWAIT(PyCOND_T *cv, PyMUTEX_T *cs, PY_LONG_LONG us)
 {
-    return SleepConditionVariableSRW(cv, cs, us/1000, 0) ? 2 : -1;
+    return SleepConditionVariableSRW(cv, cs, (DWORD)(us/1000), 0) ? 2 : -1;
 }
 
 Py_LOCAL_INLINE(int)
diff --git a/Python/thread_nt.h b/Python/thread_nt.h
--- a/Python/thread_nt.h
+++ b/Python/thread_nt.h
@@ -77,7 +77,7 @@
         /* wait at least until the target */
         DWORD now, target = GetTickCount() + milliseconds;
         while (mutex->locked) {
-            if (PyCOND_TIMEDWAIT(&mutex->cv, &mutex->cs, milliseconds*1000) < 0) {
+            if (PyCOND_TIMEDWAIT(&mutex->cv, &mutex->cs, (PY_LONG_LONG)milliseconds*1000) < 0) {
                 result = WAIT_FAILED;
                 break;
             }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu May  8 13:19:34 2014
From: python-checkins at python.org (kristjan.jonsson)
Date: Thu,  8 May 2014 13:19:34 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?q?=29=3A_Merge_with_3=2E4?=
Message-ID: <3gPXJ6239Jz7LjR@mail.python.org>

http://hg.python.org/cpython/rev/219502cf57eb
changeset:   90588:219502cf57eb
parent:      90585:80b76c6fad44
parent:      90587:7764bb7f2983
user:        Kristj?n Valur J?nsson <sweskman at gmail.com>
date:        Thu May 08 11:18:27 2014 +0000
summary:
  Merge with 3.4

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu May  8 16:06:34 2014
From: python-checkins at python.org (r.david.murray)
Date: Thu,  8 May 2014 16:06:34 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogIzIxMzAwOiBmaXgg?=
	=?utf-8?q?typo?=
Message-ID: <3gPc0p4W0dz7Lkb@mail.python.org>

http://hg.python.org/cpython/rev/9e55089aa505
changeset:   90589:9e55089aa505
branch:      3.4
parent:      90587:7764bb7f2983
user:        R David Murray <rdmurray at bitdance.com>
date:        Thu May 08 10:05:47 2014 -0400
summary:
  #21300: fix typo

files:
  Doc/library/email.message.rst |  2 +-
  Doc/library/email.parser.rst  |  6 +++---
  2 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst
--- a/Doc/library/email.message.rst
+++ b/Doc/library/email.message.rst
@@ -34,7 +34,7 @@
 .. class:: Message(policy=compat32)
 
    If *policy* is specified (it must be an instance of a :mod:`~email.policy`
-   class) use the rules it specifies to udpate and serialize the representation
+   class) use the rules it specifies to update and serialize the representation
    of the message.  If *policy* is not set, use the :class:`compat32
    <email.policy.Compat32>` policy, which maintains backward compatibility with
    the Python 3.2 version of the email package.  For more information see the
diff --git a/Doc/library/email.parser.rst b/Doc/library/email.parser.rst
--- a/Doc/library/email.parser.rst
+++ b/Doc/library/email.parser.rst
@@ -67,7 +67,7 @@
    defaults to the :class:`email.message.Message` class.
 
    If *policy* is specified (it must be an instance of a :mod:`~email.policy`
-   class) use the rules it specifies to udpate the representation of the
+   class) use the rules it specifies to update the representation of the
    message.  If *policy* is not set, use the :class:`compat32
    <email.policy.Compat32>` policy, which maintains backward compatibility with
    the Python 3.2 version of the email package.  For more information see the
@@ -125,7 +125,7 @@
    be called without arguments.
 
    If *policy* is specified (it must be an instance of a :mod:`~email.policy`
-   class) use the rules it specifies to udpate the representation of the
+   class) use the rules it specifies to update the representation of the
    message.  If *policy* is not set, use the :class:`compat32
    <email.policy.Compat32>` policy, which maintains backward compatibility with
    the Python 3.2 version of the email package.  For more information see the
@@ -172,7 +172,7 @@
    the :class:`Parser` constructor.
 
    If *policy* is specified (it must be an instance of a :mod:`~email.policy`
-   class) use the rules it specifies to udpate the representation of the
+   class) use the rules it specifies to update the representation of the
    message.  If *policy* is not set, use the :class:`compat32
    <email.policy.Compat32>` policy, which maintains backward compatibility with
    the Python 3.2 version of the email package.  For more information see the

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu May  8 16:06:35 2014
From: python-checkins at python.org (r.david.murray)
Date: Thu,  8 May 2014 16:06:35 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge_=2321300=3A_fix_typo?=
Message-ID: <3gPc0q68htz7Lkb@mail.python.org>

http://hg.python.org/cpython/rev/232938736a31
changeset:   90590:232938736a31
parent:      90588:219502cf57eb
parent:      90589:9e55089aa505
user:        R David Murray <rdmurray at bitdance.com>
date:        Thu May 08 10:06:17 2014 -0400
summary:
  Merge #21300: fix typo

files:
  Doc/library/email.message.rst |  2 +-
  Doc/library/email.parser.rst  |  6 +++---
  2 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst
--- a/Doc/library/email.message.rst
+++ b/Doc/library/email.message.rst
@@ -34,7 +34,7 @@
 .. class:: Message(policy=compat32)
 
    If *policy* is specified (it must be an instance of a :mod:`~email.policy`
-   class) use the rules it specifies to udpate and serialize the representation
+   class) use the rules it specifies to update and serialize the representation
    of the message.  If *policy* is not set, use the :class:`compat32
    <email.policy.Compat32>` policy, which maintains backward compatibility with
    the Python 3.2 version of the email package.  For more information see the
diff --git a/Doc/library/email.parser.rst b/Doc/library/email.parser.rst
--- a/Doc/library/email.parser.rst
+++ b/Doc/library/email.parser.rst
@@ -67,7 +67,7 @@
    defaults to the :class:`email.message.Message` class.
 
    If *policy* is specified (it must be an instance of a :mod:`~email.policy`
-   class) use the rules it specifies to udpate the representation of the
+   class) use the rules it specifies to update the representation of the
    message.  If *policy* is not set, use the :class:`compat32
    <email.policy.Compat32>` policy, which maintains backward compatibility with
    the Python 3.2 version of the email package.  For more information see the
@@ -125,7 +125,7 @@
    be called without arguments.
 
    If *policy* is specified (it must be an instance of a :mod:`~email.policy`
-   class) use the rules it specifies to udpate the representation of the
+   class) use the rules it specifies to update the representation of the
    message.  If *policy* is not set, use the :class:`compat32
    <email.policy.Compat32>` policy, which maintains backward compatibility with
    the Python 3.2 version of the email package.  For more information see the
@@ -172,7 +172,7 @@
    the :class:`Parser` constructor.
 
    If *policy* is specified (it must be an instance of a :mod:`~email.policy`
-   class) use the rules it specifies to udpate the representation of the
+   class) use the rules it specifies to update the representation of the
    message.  If *policy* is not set, use the :class:`compat32
    <email.policy.Compat32>` policy, which maintains backward compatibility with
    the Python 3.2 version of the email package.  For more information see the

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu May  8 19:26:13 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Thu,  8 May 2014 19:26:13 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIxMzUw?=
 =?utf-8?q?=3A_Fix_file=2Ewritelines=28=29_to_accept_arbitrary_buffer_obje?=
 =?utf-8?q?cts=2C_as?=
Message-ID: <3gPhR93tgxz7LjP@mail.python.org>

http://hg.python.org/cpython/rev/db842f730432
changeset:   90591:db842f730432
branch:      2.7
parent:      90579:faef1da30c6d
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu May 08 19:26:04 2014 +0200
summary:
  Issue #21350: Fix file.writelines() to accept arbitrary buffer objects, as advertised.
Patch by Brian Kearns.

files:
  Lib/test/test_file2k.py |   7 +++++++
  Misc/NEWS               |   3 +++
  Objects/fileobject.c    |  14 +++++++-------
  3 files changed, 17 insertions(+), 7 deletions(-)


diff --git a/Lib/test/test_file2k.py b/Lib/test/test_file2k.py
--- a/Lib/test/test_file2k.py
+++ b/Lib/test/test_file2k.py
@@ -89,6 +89,13 @@
         self.assertRaises(TypeError, self.f.writelines,
                           [NonString(), NonString()])
 
+    def testWritelinesBuffer(self):
+        self.f.writelines([array('c', 'abc')])
+        self.f.close()
+        self.f = open(TESTFN, 'rb')
+        buf = self.f.read()
+        self.assertEqual(buf, 'abc')
+
     def testRepr(self):
         # verify repr works
         self.assertTrue(repr(self.f).startswith("<open file '" + TESTFN))
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #21350: Fix file.writelines() to accept arbitrary buffer objects,
+  as advertised.  Patch by Brian Kearns.
+
 - Issue #20437: Fixed 43 potential bugs when deleting objects references.
 
 - Issue #21134: Fix segfault when str is called on an uninitialized
diff --git a/Objects/fileobject.c b/Objects/fileobject.c
--- a/Objects/fileobject.c
+++ b/Objects/fileobject.c
@@ -1941,13 +1941,13 @@
             PyObject *v = PyList_GET_ITEM(list, i);
             if (!PyString_Check(v)) {
                 const char *buffer;
-                if (((f->f_binary &&
-                      PyObject_AsReadBuffer(v,
-                          (const void**)&buffer,
-                                        &len)) ||
-                     PyObject_AsCharBuffer(v,
-                                           &buffer,
-                                           &len))) {
+                int res;
+                if (f->f_binary) {
+                    res = PyObject_AsReadBuffer(v, (const void**)&buffer, &len);
+                } else {
+                    res = PyObject_AsCharBuffer(v, &buffer, &len);
+                }
+                if (res) {
                     PyErr_SetString(PyExc_TypeError,
             "writelines() argument must be a sequence of strings");
                             goto error;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  9 00:10:21 2014
From: python-checkins at python.org (charles-francois.natali)
Date: Fri,  9 May 2014 00:10:21 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321037=3A_Add_a_bu?=
 =?utf-8?q?ild_option_to_enable_AddressSanitizer_support=2E?=
Message-ID: <3gPpl15zbXz7LjQ@mail.python.org>

http://hg.python.org/cpython/rev/17689e43839a
changeset:   90592:17689e43839a
parent:      90590:232938736a31
user:        Charles-Fran?ois Natali <cf.natali at gmail.com>
date:        Thu May 08 23:08:51 2014 +0100
summary:
  Issue #21037: Add a build option to enable AddressSanitizer support.

files:
  Misc/NEWS    |   2 ++
  configure    |  20 ++++++++++++++++++++
  configure.ac |  11 +++++++++++
  3 files changed, 33 insertions(+), 0 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -347,6 +347,8 @@
 Build
 -----
 
+- Issue #21037: Add a build option to enable AddressSanitizer support.
+
 - The Windows build now includes OpenSSL 1.0.1g
 
 - Issue #19962: The Windows build process now creates "python.bat" in the
diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -796,6 +796,7 @@
 enable_profiling
 with_pydebug
 with_hash_algorithm
+with_address_sanitizer
 with_libs
 with_system_expat
 with_system_ffi
@@ -1472,6 +1473,8 @@
   --with-pydebug          build with Py_DEBUG defined
   --with-hash-algorithm=[fnv|siphash24]
                           select hash algorithm
+  --with-address-sanitizer
+                          enable AddressSanitizer
   --with-libs='lib1 ...'  link against additional libs
   --with-system-expat     build pyexpat module using an installed expat
                           library
@@ -9154,6 +9157,23 @@
 fi
 
 
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for --with-address-sanitizer" >&5
+$as_echo_n "checking for --with-address-sanitizer... " >&6; }
+
+# Check whether --with-address_sanitizer was given.
+if test "${with_address_sanitizer+set}" = set; then :
+  withval=$with_address_sanitizer;
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $withval" >&5
+$as_echo "$withval" >&6; }
+BASECFLAGS="-fsanitize=address -fno-omit-frame-pointer $BASECFLAGS"
+LDFLAGS="-fsanitize=address $LDFLAGS"
+
+else
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+fi
+
+
 # Most SVR4 platforms (e.g. Solaris) need -lsocket and -lnsl.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for t_open in -lnsl" >&5
 $as_echo_n "checking for t_open in -lnsl... " >&6; }
diff --git a/configure.ac b/configure.ac
--- a/configure.ac
+++ b/configure.ac
@@ -2314,6 +2314,17 @@
 ],
 [AC_MSG_RESULT(default)])
 
+AC_MSG_CHECKING(for --with-address-sanitizer)
+AC_ARG_WITH(address_sanitizer,
+            AS_HELP_STRING([--with-address-sanitizer],
+                           [enable AddressSanitizer]),
+[
+AC_MSG_RESULT($withval)
+BASECFLAGS="-fsanitize=address -fno-omit-frame-pointer $BASECFLAGS"
+LDFLAGS="-fsanitize=address $LDFLAGS"
+],
+[AC_MSG_RESULT(no)])
+
 # Most SVR4 platforms (e.g. Solaris) need -lsocket and -lnsl.
 AC_CHECK_LIB(nsl, t_open, [LIBS="-lnsl $LIBS"]) # SVR4
 AC_CHECK_LIB(socket, socket, [LIBS="-lsocket $LIBS"], [], $LIBS) # SVR4 sockets

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  9 00:33:29 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Fri,  9 May 2014 00:33:29 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMzk2?=
 =?utf-8?b?OiBGaXggVGV4dElPV3JhcHBlciguLi4sIHdyaXRlX3Rocm91Z2g9VHJ1ZSkg?=
 =?utf-8?q?to_not_force_a_flush=28=29?=
Message-ID: <3gPqFj6gCPz7Ljd@mail.python.org>

http://hg.python.org/cpython/rev/39f2a78f4357
changeset:   90593:39f2a78f4357
branch:      3.4
parent:      90589:9e55089aa505
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Fri May 09 00:24:50 2014 +0200
summary:
  Issue #21396: Fix TextIOWrapper(..., write_through=True) to not force a flush() on the underlying binary stream.

Patch by akira.

files:
  Lib/test/test_io.py         |  32 +++++++++++++++++++++++++
  Lib/test/test_subprocess.py |   1 +
  Misc/ACKS                   |   1 +
  Misc/NEWS                   |   6 ++++
  Modules/_io/textio.c        |   9 +++---
  5 files changed, 45 insertions(+), 4 deletions(-)


diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py
--- a/Lib/test/test_io.py
+++ b/Lib/test/test_io.py
@@ -2615,6 +2615,38 @@
         txt.write('5')
         self.assertEqual(b''.join(raw._write_stack), b'123\n45')
 
+    def test_bufio_write_through(self):
+        # Issue #21396: write_through=True doesn't force a flush()
+        # on the underlying binary buffered object.
+        flush_called, write_called = [], []
+        class BufferedWriter(self.BufferedWriter):
+            def flush(self, *args, **kwargs):
+                flush_called.append(True)
+                return super().flush(*args, **kwargs)
+            def write(self, *args, **kwargs):
+                write_called.append(True)
+                return super().write(*args, **kwargs)
+
+        rawio = self.BytesIO()
+        data = b"a"
+        bufio = BufferedWriter(rawio, len(data)*2)
+        textio = self.TextIOWrapper(bufio, encoding='ascii',
+                                    write_through=True)
+        # write to the buffered io but don't overflow the buffer
+        text = data.decode('ascii')
+        textio.write(text)
+
+        # buffer.flush is not called with write_through=True
+        self.assertFalse(flush_called)
+        # buffer.write *is* called with write_through=True
+        self.assertTrue(write_called)
+        self.assertEqual(rawio.getvalue(), b"") # no flush
+
+        write_called = [] # reset
+        textio.write(text * 10) # total content is larger than bufio buffer
+        self.assertTrue(write_called)
+        self.assertEqual(rawio.getvalue(), data * 11) # all flushed
+
     def test_read_nonbytes(self):
         # Issue #17106
         # Crash when underlying read() returns non-bytes
diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py
--- a/Lib/test/test_subprocess.py
+++ b/Lib/test/test_subprocess.py
@@ -786,6 +786,7 @@
                              stdout=subprocess.PIPE,
                              universal_newlines=1)
         p.stdin.write("line1\n")
+        p.stdin.flush()
         self.assertEqual(p.stdout.readline(), "line1\n")
         p.stdin.write("line3\n")
         p.stdin.close()
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -24,6 +24,7 @@
 Farhan Ahmad
 Matthew Ahrens
 Nir Aides
+Akira
 Yaniv Aknin
 Jyrki Alakuijala
 Steve Alexander
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -7,6 +7,12 @@
 
 Release date: TBA
 
+Library
+-------
+
+- Issue #21396: Fix TextIOWrapper(..., write_through=True) to not force a
+  flush() on the underlying binary stream.  Patch by akira.
+
 Tests
 -----
 
diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c
--- a/Modules/_io/textio.c
+++ b/Modules/_io/textio.c
@@ -1297,7 +1297,7 @@
     PyObject *b;
     Py_ssize_t textlen;
     int haslf = 0;
-    int needflush = 0;
+    int needflush = 0, text_needflush = 0;
 
     CHECK_INITIALIZED(self);
 
@@ -1331,8 +1331,8 @@
     }
 
     if (self->write_through)
-        needflush = 1;
-    else if (self->line_buffering &&
+        text_needflush = 1;
+    if (self->line_buffering &&
         (haslf ||
          PyUnicode_FindChar(text, '\r', 0, PyUnicode_GET_LENGTH(text), 1) != -1))
         needflush = 1;
@@ -1363,7 +1363,8 @@
     }
     self->pending_bytes_count += PyBytes_GET_SIZE(b);
     Py_DECREF(b);
-    if (self->pending_bytes_count > self->chunk_size || needflush) {
+    if (self->pending_bytes_count > self->chunk_size || needflush ||
+        text_needflush) {
         if (_textiowrapper_writeflush(self) < 0)
             return NULL;
     }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  9 00:33:31 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Fri,  9 May 2014 00:33:31 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?b?KTogSXNzdWUgIzIxMzk2OiBGaXggVGV4dElPV3JhcHBlciguLi4sIHdyaXRl?=
 =?utf-8?q?=5Fthrough=3DTrue=29_to_not_force_a_flush=28=29?=
Message-ID: <3gPqFl2h5gz7Ljr@mail.python.org>

http://hg.python.org/cpython/rev/37d0c41ed8ad
changeset:   90594:37d0c41ed8ad
parent:      90592:17689e43839a
parent:      90593:39f2a78f4357
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Fri May 09 00:31:32 2014 +0200
summary:
  Issue #21396: Fix TextIOWrapper(..., write_through=True) to not force a flush() on the underlying binary stream.

Patch by akira.

files:
  Lib/test/test_io.py         |  32 +++++++++++++++++++++++++
  Lib/test/test_subprocess.py |   1 +
  Misc/NEWS                   |   3 ++
  Modules/_io/textio.c        |   9 +++---
  4 files changed, 41 insertions(+), 4 deletions(-)


diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py
--- a/Lib/test/test_io.py
+++ b/Lib/test/test_io.py
@@ -2615,6 +2615,38 @@
         txt.write('5')
         self.assertEqual(b''.join(raw._write_stack), b'123\n45')
 
+    def test_bufio_write_through(self):
+        # Issue #21396: write_through=True doesn't force a flush()
+        # on the underlying binary buffered object.
+        flush_called, write_called = [], []
+        class BufferedWriter(self.BufferedWriter):
+            def flush(self, *args, **kwargs):
+                flush_called.append(True)
+                return super().flush(*args, **kwargs)
+            def write(self, *args, **kwargs):
+                write_called.append(True)
+                return super().write(*args, **kwargs)
+
+        rawio = self.BytesIO()
+        data = b"a"
+        bufio = BufferedWriter(rawio, len(data)*2)
+        textio = self.TextIOWrapper(bufio, encoding='ascii',
+                                    write_through=True)
+        # write to the buffered io but don't overflow the buffer
+        text = data.decode('ascii')
+        textio.write(text)
+
+        # buffer.flush is not called with write_through=True
+        self.assertFalse(flush_called)
+        # buffer.write *is* called with write_through=True
+        self.assertTrue(write_called)
+        self.assertEqual(rawio.getvalue(), b"") # no flush
+
+        write_called = [] # reset
+        textio.write(text * 10) # total content is larger than bufio buffer
+        self.assertTrue(write_called)
+        self.assertEqual(rawio.getvalue(), data * 11) # all flushed
+
     def test_read_nonbytes(self):
         # Issue #17106
         # Crash when underlying read() returns non-bytes
diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py
--- a/Lib/test/test_subprocess.py
+++ b/Lib/test/test_subprocess.py
@@ -786,6 +786,7 @@
                              stdout=subprocess.PIPE,
                              universal_newlines=1)
         p.stdin.write("line1\n")
+        p.stdin.flush()
         self.assertEqual(p.stdout.readline(), "line1\n")
         p.stdin.write("line3\n")
         p.stdin.close()
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -69,6 +69,9 @@
 Library
 -------
 
+- Issue #21396: Fix TextIOWrapper(..., write_through=True) to not force a
+  flush() on the underlying binary stream.  Patch by akira.
+
 - Issue #18314: Unlink now removes junctions on Windows. Patch by Kim Gr?sman
 
 - Issue #21088: Bugfix for curses.window.addch() regression in 3.4.0.
diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c
--- a/Modules/_io/textio.c
+++ b/Modules/_io/textio.c
@@ -1297,7 +1297,7 @@
     PyObject *b;
     Py_ssize_t textlen;
     int haslf = 0;
-    int needflush = 0;
+    int needflush = 0, text_needflush = 0;
 
     CHECK_INITIALIZED(self);
 
@@ -1331,8 +1331,8 @@
     }
 
     if (self->write_through)
-        needflush = 1;
-    else if (self->line_buffering &&
+        text_needflush = 1;
+    if (self->line_buffering &&
         (haslf ||
          PyUnicode_FindChar(text, '\r', 0, PyUnicode_GET_LENGTH(text), 1) != -1))
         needflush = 1;
@@ -1363,7 +1363,8 @@
     }
     self->pending_bytes_count += PyBytes_GET_SIZE(b);
     Py_DECREF(b);
-    if (self->pending_bytes_count > self->chunk_size || needflush) {
+    if (self->pending_bytes_count > self->chunk_size || needflush ||
+        text_needflush) {
         if (_textiowrapper_writeflush(self) < 0)
             return NULL;
     }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  9 00:43:41 2014
From: python-checkins at python.org (tim.peters)
Date: Fri,  9 May 2014 00:43:41 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxNDM1?=
 =?utf-8?q?=3A_Segfault_in_gc_with_cyclic_trash?=
Message-ID: <3gPqTT1pRgz7Lkw@mail.python.org>

http://hg.python.org/cpython/rev/64ba3f2de99c
changeset:   90595:64ba3f2de99c
branch:      3.4
parent:      90593:39f2a78f4357
user:        Tim Peters <tim at python.org>
date:        Thu May 08 17:42:19 2014 -0500
summary:
  Issue #21435: Segfault in gc with cyclic trash
Changed the iteration logic in finalize_garbage() to tolerate objects vanishing
from the list as a side effect of executing a finalizer.

files:
  Lib/test/test_gc.py |  32 +++++++++++++++++++++++++++++++
  Misc/NEWS           |   7 ++++++
  Modules/gcmodule.c  |  34 ++++++++++++++++++++++----------
  3 files changed, 62 insertions(+), 11 deletions(-)


diff --git a/Lib/test/test_gc.py b/Lib/test/test_gc.py
--- a/Lib/test/test_gc.py
+++ b/Lib/test/test_gc.py
@@ -580,6 +580,38 @@
             # would be damaged, with an empty __dict__.
             self.assertEqual(x, None)
 
+    def test_bug21435(self):
+        # This is a poor test - its only virtue is that it happened to
+        # segfault on Tim's Windows box before the patch for 21435 was
+        # applied.  That's a nasty bug relying on specific pieces of cyclic
+        # trash appearing in exactly the right order in finalize_garbage()'s
+        # input list.
+        # But there's no reliable way to force that order from Python code,
+        # so over time chances are good this test won't really be testing much
+        # of anything anymore.  Still, if it blows up, there's _some_
+        # problem ;-)
+        gc.collect()
+
+        class A:
+            pass
+
+        class B:
+            def __init__(self, x):
+                self.x = x
+
+            def __del__(self):
+                self.attr = None
+
+        def do_work():
+            a = A()
+            b = B(A())
+
+            a.attr = b
+            b.attr = a
+
+        do_work()
+        gc.collect() # this blows up (bad C pointer) when it fails
+
     @cpython_only
     def test_garbage_at_shutdown(self):
         import subprocess
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -7,6 +7,13 @@
 
 Release date: TBA
 
+Core and Builtins
+-----------------
+
+- Issue #21435: In rare cases, when running finalizers on objects in cyclic
+  trash a bad pointer dereference could occur due to a subtle flaw in
+  internal iteration logic.
+
 Library
 -------
 
diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -776,28 +776,40 @@
     return 0;
 }
 
+/* Run first-time finalizers (if any) on all the objects in collectable.
+ * Note that this may remove some (or even all) of the objects from the
+ * list, due to refcounts falling to 0.
+ */
 static void
-finalize_garbage(PyGC_Head *collectable, PyGC_Head *old)
+finalize_garbage(PyGC_Head *collectable)
 {
     destructor finalize;
-    PyGC_Head *gc = collectable->gc.gc_next;
+    PyGC_Head seen;
 
-    for (; gc != collectable; gc = gc->gc.gc_next) {
+    /* While we're going through the loop, `finalize(op)` may cause op, or
+     * other objects, to be reclaimed via refcounts falling to zero.  So
+     * there's little we can rely on about the structure of the input
+     * `collectable` list across iterations.  For safety, we always take the
+     * first object in that list and move it to a temporary `seen` list.
+     * If objects vanish from the `collectable` and `seen` lists we don't
+     * care.
+     */
+    gc_list_init(&seen);
+
+    while (!gc_list_is_empty(collectable)) {
+        PyGC_Head *gc = collectable->gc.gc_next;
         PyObject *op = FROM_GC(gc);
-
+        gc_list_move(gc, &seen);
         if (!_PyGCHead_FINALIZED(gc) &&
-            PyType_HasFeature(Py_TYPE(op), Py_TPFLAGS_HAVE_FINALIZE) &&
-            (finalize = Py_TYPE(op)->tp_finalize) != NULL) {
+                PyType_HasFeature(Py_TYPE(op), Py_TPFLAGS_HAVE_FINALIZE) &&
+                (finalize = Py_TYPE(op)->tp_finalize) != NULL) {
             _PyGCHead_SET_FINALIZED(gc, 1);
             Py_INCREF(op);
             finalize(op);
-            if (Py_REFCNT(op) == 1) {
-                /* op will be destroyed */
-                gc = gc->gc.gc_prev;
-            }
             Py_DECREF(op);
         }
     }
+    gc_list_merge(&seen, collectable);
 }
 
 /* Walk the collectable list and check that they are really unreachable
@@ -1006,7 +1018,7 @@
     m += handle_weakrefs(&unreachable, old);
 
     /* Call tp_finalize on objects which have one. */
-    finalize_garbage(&unreachable, old);
+    finalize_garbage(&unreachable);
 
     if (check_garbage(&unreachable)) {
         revive_garbage(&unreachable);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  9 00:43:42 2014
From: python-checkins at python.org (tim.peters)
Date: Fri,  9 May 2014 00:43:42 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?b?KTogTWVyZ2UgZnJvbSAzLjQu?=
Message-ID: <3gPqTV4ksDz7Lk2@mail.python.org>

http://hg.python.org/cpython/rev/cb9a3985df00
changeset:   90596:cb9a3985df00
parent:      90594:37d0c41ed8ad
parent:      90595:64ba3f2de99c
user:        Tim Peters <tim at python.org>
date:        Thu May 08 17:43:25 2014 -0500
summary:
  Merge from 3.4.

Issue #21435: Segfault in gc with cyclic trash
Changed the iteration logic in finalize_garbage() to tolerate objects vanishing
from the list as a side effect of executing a finalizer.

files:
  Lib/test/test_gc.py |  32 +++++++++++++++++++++++++++++++
  Misc/NEWS           |   4 +++
  Modules/gcmodule.c  |  34 ++++++++++++++++++++++----------
  3 files changed, 59 insertions(+), 11 deletions(-)


diff --git a/Lib/test/test_gc.py b/Lib/test/test_gc.py
--- a/Lib/test/test_gc.py
+++ b/Lib/test/test_gc.py
@@ -580,6 +580,38 @@
             # would be damaged, with an empty __dict__.
             self.assertEqual(x, None)
 
+    def test_bug21435(self):
+        # This is a poor test - its only virtue is that it happened to
+        # segfault on Tim's Windows box before the patch for 21435 was
+        # applied.  That's a nasty bug relying on specific pieces of cyclic
+        # trash appearing in exactly the right order in finalize_garbage()'s
+        # input list.
+        # But there's no reliable way to force that order from Python code,
+        # so over time chances are good this test won't really be testing much
+        # of anything anymore.  Still, if it blows up, there's _some_
+        # problem ;-)
+        gc.collect()
+
+        class A:
+            pass
+
+        class B:
+            def __init__(self, x):
+                self.x = x
+
+            def __del__(self):
+                self.attr = None
+
+        def do_work():
+            a = A()
+            b = B(A())
+
+            a.attr = b
+            b.attr = a
+
+        do_work()
+        gc.collect() # this blows up (bad C pointer) when it fails
+
     @cpython_only
     def test_garbage_at_shutdown(self):
         import subprocess
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,10 @@
 Core and Builtins
 -----------------
 
+- Issue #21435: In rare cases, when running finalizers on objects in cyclic
+  trash a bad pointer dereference could occur due to a subtle flaw in
+  internal iteration logic.
+
 - Issue #21233: Add new C functions: PyMem_RawCalloc(), PyMem_Calloc(),
   PyObject_Calloc(), _PyObject_GC_Calloc(). bytes(int) and bytearray(int)
   are now using ``calloc()`` instead of ``malloc()`` for large objects which
diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -776,28 +776,40 @@
     return 0;
 }
 
+/* Run first-time finalizers (if any) on all the objects in collectable.
+ * Note that this may remove some (or even all) of the objects from the
+ * list, due to refcounts falling to 0.
+ */
 static void
-finalize_garbage(PyGC_Head *collectable, PyGC_Head *old)
+finalize_garbage(PyGC_Head *collectable)
 {
     destructor finalize;
-    PyGC_Head *gc = collectable->gc.gc_next;
+    PyGC_Head seen;
 
-    for (; gc != collectable; gc = gc->gc.gc_next) {
+    /* While we're going through the loop, `finalize(op)` may cause op, or
+     * other objects, to be reclaimed via refcounts falling to zero.  So
+     * there's little we can rely on about the structure of the input
+     * `collectable` list across iterations.  For safety, we always take the
+     * first object in that list and move it to a temporary `seen` list.
+     * If objects vanish from the `collectable` and `seen` lists we don't
+     * care.
+     */
+    gc_list_init(&seen);
+
+    while (!gc_list_is_empty(collectable)) {
+        PyGC_Head *gc = collectable->gc.gc_next;
         PyObject *op = FROM_GC(gc);
-
+        gc_list_move(gc, &seen);
         if (!_PyGCHead_FINALIZED(gc) &&
-            PyType_HasFeature(Py_TYPE(op), Py_TPFLAGS_HAVE_FINALIZE) &&
-            (finalize = Py_TYPE(op)->tp_finalize) != NULL) {
+                PyType_HasFeature(Py_TYPE(op), Py_TPFLAGS_HAVE_FINALIZE) &&
+                (finalize = Py_TYPE(op)->tp_finalize) != NULL) {
             _PyGCHead_SET_FINALIZED(gc, 1);
             Py_INCREF(op);
             finalize(op);
-            if (Py_REFCNT(op) == 1) {
-                /* op will be destroyed */
-                gc = gc->gc.gc_prev;
-            }
             Py_DECREF(op);
         }
     }
+    gc_list_merge(&seen, collectable);
 }
 
 /* Walk the collectable list and check that they are really unreachable
@@ -1006,7 +1018,7 @@
     m += handle_weakrefs(&unreachable, old);
 
     /* Call tp_finalize on objects which have one. */
-    finalize_garbage(&unreachable, old);
+    finalize_garbage(&unreachable);
 
     if (check_garbage(&unreachable)) {
         revive_garbage(&unreachable);

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Fri May  9 10:01:40 2014
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Fri, 09 May 2014 10:01:40 +0200
Subject: [Python-checkins] Daily reference leaks (cb9a3985df00): sum=7
Message-ID: <E1WifkS-0002iY-Ac@vds2544.sivit.org>

results for cb9a3985df00 on branch "default"
--------------------------------------------

test_functools leaked [0, 0, 3] memory blocks, sum=3
test_site leaked [2, -2, 2] references, sum=2
test_site leaked [2, -2, 2] memory blocks, sum=2


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogKkAHSU', '-x']

From python-checkins at python.org  Fri May  9 16:10:16 2014
From: python-checkins at python.org (zach.ware)
Date: Fri,  9 May 2014 16:10:16 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321141=3A_The_Wind?=
 =?utf-8?q?ows_build_process_no_longer_attempts_to_find_Perl=2C?=
Message-ID: <3gQD2c4qsgz7Ljn@mail.python.org>

http://hg.python.org/cpython/rev/9ef99fafaadd
changeset:   90597:9ef99fafaadd
user:        Zachary Ware <zachary.ware at gmail.com>
date:        Fri May 09 09:07:50 2014 -0500
summary:
  Issue #21141: The Windows build process no longer attempts to find Perl,
instead relying on OpenSSL source being configured and ready to build.  The
``PCbuild\build_ssl.py`` script has been re-written and re-named to
``PCbuild\prepare_ssl.py``, and takes care of configuring OpenSSL source
for both 32 and 64 bit platforms.  OpenSSL sources obtained from
svn.python.org will always be pre-configured and ready to build.

files:
  Misc/NEWS             |    7 +
  PCbuild/build_ssl.bat |    4 +-
  PCbuild/build_ssl.py  |  192 ++++++++++++-----------------
  PCbuild/readme.txt    |   35 +---
  PCbuild/ssl.vcxproj   |   82 ++++++++----
  5 files changed, 151 insertions(+), 169 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -354,6 +354,13 @@
 Build
 -----
 
+- Issue #21141: The Windows build process no longer attempts to find Perl,
+  instead relying on OpenSSL source being configured and ready to build.  The
+  ``PCbuild\build_ssl.py`` script has been re-written and re-named to
+  ``PCbuild\prepare_ssl.py``, and takes care of configuring OpenSSL source
+  for both 32 and 64 bit platforms.  OpenSSL sources obtained from
+  svn.python.org will always be pre-configured and ready to build.
+
 - Issue #21037: Add a build option to enable AddressSanitizer support.
 
 - The Windows build now includes OpenSSL 1.0.1g
diff --git a/PCbuild/build_ssl.bat b/PCbuild/prepare_ssl.bat
rename from PCbuild/build_ssl.bat
rename to PCbuild/prepare_ssl.bat
--- a/PCbuild/build_ssl.bat
+++ b/PCbuild/prepare_ssl.bat
@@ -1,6 +1,7 @@
 @echo off
 if not defined HOST_PYTHON (
   if %1 EQU Debug (
+    shift
     set HOST_PYTHON=python_d.exe
     if not exist python35_d.dll exit 1
   ) ELSE (
@@ -8,5 +9,4 @@
     if not exist python35.dll exit 1
   )
 )
-%HOST_PYTHON% build_ssl.py %1 %2 %3
-
+%HOST_PYTHON% prepare_ssl.py %1
diff --git a/PCbuild/build_ssl.py b/PCbuild/prepare_ssl.py
rename from PCbuild/build_ssl.py
rename to PCbuild/prepare_ssl.py
--- a/PCbuild/build_ssl.py
+++ b/PCbuild/prepare_ssl.py
@@ -1,29 +1,26 @@
-# Script for building the _ssl and _hashlib modules for Windows.
-# Uses Perl to setup the OpenSSL environment correctly
-# and build OpenSSL, then invokes a simple nmake session
-# for the actual _ssl.pyd and _hashlib.pyd DLLs.
+# Script for preparing OpenSSL for building on Windows.
+# Uses Perl to create nmake makefiles and otherwise prepare the way
+# for building on 32 or 64 bit platforms.
+
+# Script originally authored by Mark Hammond.
+# Major revisions by:
+#   Martin v. L?wis
+#   Christian Heimes
+#   Zachary Ware
 
 # THEORETICALLY, you can:
-# * Unpack the latest SSL release one level above your main Python source
-#   directory.  It is likely you will already find the zlib library and
-#   any other external packages there.
+# * Unpack the latest OpenSSL release where $(opensslDir) in
+#   PCbuild\pyproject.props expects it to be.
 # * Install ActivePerl and ensure it is somewhere on your path.
-# * Run this script from the PCBuild directory.
+# * Run this script with the OpenSSL source dir as the only argument.
 #
-# it should configure and build SSL, then build the _ssl and _hashlib
-# Python extensions without intervention.
+# it should configure OpenSSL such that it is ready to be built by
+# ssl.vcxproj on 32 or 64 bit platforms.
 
-# Modified by Christian Heimes
-# Now this script supports pre-generated makefiles and assembly files.
-# Developers don't need an installation of Perl anymore to build Python. A svn
-# checkout from our svn repository is enough.
-#
-# In Order to create the files in the case of an update you still need Perl.
-# Run build_ssl in this order:
-# python.exe build_ssl.py Release x64
-# python.exe build_ssl.py Release Win32
-
-import os, sys, re, shutil
+import os
+import re
+import sys
+import shutil
 
 # Find all "foo.exe" files on the PATH.
 def find_all_on_path(filename, extras = None):
@@ -63,14 +60,6 @@
     print(" Please install ActivePerl and ensure it appears on your path")
     return None
 
-# Fetch SSL directory from VC properties
-def get_ssl_dir():
-    propfile = (os.path.join(os.path.dirname(__file__), 'pyproject.props'))
-    with open(propfile) as f:
-        m = re.search('openssl-([^<]+)<', f.read())
-        return "..\..\openssl-"+m.group(1)
-
-
 def create_makefile64(makefile, m32):
     """Create and fix makefile for 64bit
 
@@ -138,24 +127,14 @@
         return
     shutil.copy(src, dst)
 
-def main():
-    build_all = "-a" in sys.argv
-    if sys.argv[1] == "Release":
-        debug = False
-    elif sys.argv[1] == "Debug":
-        debug = True
-    else:
-        raise ValueError(str(sys.argv))
-
-    if sys.argv[2] == "Win32":
-        arch = "x86"
+def prep(arch):
+    if arch == "x86":
         configure = "VC-WIN32"
         do_script = "ms\\do_nasm"
         makefile="ms\\nt.mak"
         m32 = makefile
         dirsuffix = "32"
-    elif sys.argv[2] == "x64":
-        arch="amd64"
+    elif arch == "amd64":
         configure = "VC-WIN64A"
         do_script = "ms\\do_win64a"
         makefile = "ms\\nt64.mak"
@@ -163,11 +142,54 @@
         dirsuffix = "64"
         #os.environ["VSEXTCOMP_USECL"] = "MS_OPTERON"
     else:
-        raise ValueError(str(sys.argv))
+        raise ValueError('Unrecognized platform: %s' % arch)
 
-    make_flags = ""
-    if build_all:
-        make_flags = "-a"
+    # rebuild makefile when we do the role over from 32 to 64 build
+    if arch == "amd64" and os.path.isfile(m32) and not os.path.isfile(makefile):
+        os.unlink(m32)
+
+    # If the ssl makefiles do not exist, we invoke Perl to generate them.
+    # Due to a bug in this script, the makefile sometimes ended up empty
+    # Force a regeneration if it is.
+    if not os.path.isfile(makefile) or os.path.getsize(makefile)==0:
+        print("Creating the makefiles...")
+        sys.stdout.flush()
+        run_configure(configure, do_script)
+
+        if arch == "amd64":
+            create_makefile64(makefile, m32)
+        fix_makefile(makefile)
+        copy(r"crypto\buildinf.h", r"crypto\buildinf_%s.h" % arch)
+        copy(r"crypto\opensslconf.h", r"crypto\opensslconf_%s.h" % arch)
+    else:
+        print(makefile, 'already exists!')
+
+    # If the assembler files don't exist in tmpXX, copy them there
+    if os.path.exists("asm"+dirsuffix):
+        if not os.path.exists("tmp"+dirsuffix):
+            os.mkdir("tmp"+dirsuffix)
+        for f in os.listdir("asm"+dirsuffix):
+            if not f.endswith(".asm"): continue
+            if os.path.isfile(r"tmp%s\%s" % (dirsuffix, f)): continue
+            shutil.copy(r"asm%s\%s" % (dirsuffix, f), "tmp"+dirsuffix)
+
+def main():
+    if len(sys.argv) == 1:
+        print("Not enough arguments: directory containing OpenSSL",
+              "sources must be supplied")
+        sys.exit(1)
+
+    if len(sys.argv) > 2:
+        print("Too many arguments supplied, all we need is the directory",
+              "containing OpenSSL sources")
+        sys.exit(1)
+
+    ssl_dir = sys.argv[1]
+
+    if not os.path.exists(ssl_dir) and os.path.isdir(ssl_dir):
+        print(ssl_dir, "is not an existing directory!")
+        sys.exit(1)
+
     # perl should be on the path, but we also look in "\perl" and "c:\\perl"
     # as "well known" locations
     perls = find_all_on_path("perl.exe", ["\\perl\\bin", "C:\\perl\\bin"])
@@ -175,79 +197,21 @@
     if perl:
         print("Found a working perl at '%s'" % (perl,))
     else:
-        print("No Perl installation was found. Existing Makefiles are used.")
+        sys.exit(1)
     sys.stdout.flush()
-    # Look for SSL 2 levels up from pcbuild - ie, same place zlib etc all live.
-    ssl_dir = get_ssl_dir()
-    if ssl_dir is None:
-        sys.exit(1)
 
-    old_cd = os.getcwd()
+    # Put our working Perl at the front of our path
+    os.environ["PATH"] = os.path.dirname(perl) + \
+                                os.pathsep + \
+                                os.environ["PATH"]
+
+    old_cwd = os.getcwd()
     try:
         os.chdir(ssl_dir)
-        # rebuild makefile when we do the role over from 32 to 64 build
-        if arch == "amd64" and os.path.isfile(m32) and not os.path.isfile(makefile):
-            os.unlink(m32)
-
-        # If the ssl makefiles do not exist, we invoke Perl to generate them.
-        # Due to a bug in this script, the makefile sometimes ended up empty
-        # Force a regeneration if it is.
-        if not os.path.isfile(makefile) or os.path.getsize(makefile)==0:
-            if perl is None:
-                print("Perl is required to build the makefiles!")
-                sys.exit(1)
-
-            print("Creating the makefiles...")
-            sys.stdout.flush()
-            # Put our working Perl at the front of our path
-            os.environ["PATH"] = os.path.dirname(perl) + \
-                                          os.pathsep + \
-                                          os.environ["PATH"]
-            run_configure(configure, do_script)
-            if debug:
-                print("OpenSSL debug builds aren't supported.")
-            #if arch=="x86" and debug:
-            #    # the do_masm script in openssl doesn't generate a debug
-            #    # build makefile so we generate it here:
-            #    os.system("perl util\mk1mf.pl debug "+configure+" >"+makefile)
-
-            if arch == "amd64":
-                create_makefile64(makefile, m32)
-            fix_makefile(makefile)
-            copy(r"crypto\buildinf.h", r"crypto\buildinf_%s.h" % arch)
-            copy(r"crypto\opensslconf.h", r"crypto\opensslconf_%s.h" % arch)
-
-        # If the assembler files don't exist in tmpXX, copy them there
-        if perl is None and os.path.exists("asm"+dirsuffix):
-            if not os.path.exists("tmp"+dirsuffix):
-                os.mkdir("tmp"+dirsuffix)
-            for f in os.listdir("asm"+dirsuffix):
-                if not f.endswith(".asm"): continue
-                if os.path.isfile(r"tmp%s\%s" % (dirsuffix, f)): continue
-                shutil.copy(r"asm%s\%s" % (dirsuffix, f), "tmp"+dirsuffix)
-
-        # Now run make.
-        if arch == "amd64":
-            rc = os.system("nasm -f win64 -DNEAR -Ox -g ms\\uptable.asm")
-            if rc:
-                print("nasm assembler has failed.")
-                sys.exit(rc)
-
-        copy(r"crypto\buildinf_%s.h" % arch, r"crypto\buildinf.h")
-        copy(r"crypto\opensslconf_%s.h" % arch, r"crypto\opensslconf.h")
-
-        #makeCommand = "nmake /nologo PERL=\"%s\" -f \"%s\"" %(perl, makefile)
-        makeCommand = "nmake /nologo -f \"%s\"" % makefile
-        print("Executing ssl makefiles:", makeCommand)
-        sys.stdout.flush()
-        rc = os.system(makeCommand)
-        if rc:
-            print("Executing "+makefile+" failed")
-            print(rc)
-            sys.exit(rc)
+        for arch in ['amd64', 'x86']:
+            prep(arch)
     finally:
-        os.chdir(old_cd)
-    sys.exit(rc)
+        os.chdir(old_cwd)
 
 if __name__=='__main__':
     main()
diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt
--- a/PCbuild/readme.txt
+++ b/PCbuild/readme.txt
@@ -181,30 +181,19 @@
     you should first try to update NASM and do a full rebuild of
     OpenSSL.
 
-    If you like to use the official sources instead of the files from
-    python.org's subversion repository, Perl is required to build the
-    necessary makefiles and assembly files.  ActivePerl is available
-    from
+    The ssl sub-project expects your OpenSSL sources to have already
+    been configured and be ready to build.  If you get your sources
+    from svn.python.org as suggested in the "Getting External Sources"
+    section below, the OpenSSL source will already be ready to go.  If
+    you want to build a different version, you will need to run
+
+       PCbuild\prepare_ssl.py path\to\openssl-source-dir
+
+    That script will prepare your OpenSSL sources in the same way that
+    those available on svn.python.org have been prepared.  Note that
+    Perl must be installed and available on your PATH to configure
+    OpenSSL.  ActivePerl is recommended and is available from
         http://www.activestate.com/activeperl/
-    The svn.python.org version contains pre-built makefiles and assembly
-    files.
-
-    The build process makes sure that no patented algorithms are
-    included.  For now RC5, MDC2 and IDEA are excluded from the build.
-    You may have to manually remove $(OBJ_D)\i_*.obj from ms\nt.mak if
-    using official sources; the svn.python.org-hosted version is already
-    fixed.
-
-    The ssl.vcxproj sub-project simply invokes PCbuild/build_ssl.py,
-    which locates and builds OpenSSL.
-
-    build_ssl.py attempts to catch the most common errors (such as not
-    being able to find OpenSSL sources, or not being able to find a Perl
-    that works with OpenSSL) and give a reasonable error message.  If
-    you have a problem that doesn't seem to be handled correctly (e.g.,
-    you know you have ActivePerl but we can't find it), please take a
-    peek at build_ssl.py and suggest patches.  Note that build_ssl.py
-    should be able to be run directly from the command-line.
 
     The ssl sub-project does not have the ability to clean the OpenSSL
     build; if you need to rebuild, you'll have to clean it by hand.
diff --git a/PCbuild/ssl.vcxproj b/PCbuild/ssl.vcxproj
--- a/PCbuild/ssl.vcxproj
+++ b/PCbuild/ssl.vcxproj
@@ -118,9 +118,12 @@
   <PropertyGroup Label="UserMacros" />
   <PropertyGroup>
     <_ProjectFileVersion>10.0.30319.1</_ProjectFileVersion>
-    <NMakeBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">cd "$(SolutionDir)"
-"$(PythonExe)" build_ssl.py Release $(Platform) -a
-</NMakeBuildCommandLine>
+    <NMakeBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
+cd "$(SolutionDir)$(opensslDir)"
+copy /Y crypto\buildinf_x86.h crypto\buildinf.h
+copy /Y crypto\opensslconf_x86.h crypto\opensslconf.h
+nmake /nologo -f "ms\nt.mak"
+	</NMakeBuildCommandLine>
     <NMakeReBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" />
     <NMakeCleanCommandLine Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">echo OpenSSL must be cleaned manually if you want to rebuild it.</NMakeCleanCommandLine>
     <NMakeOutput Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" />
@@ -129,9 +132,13 @@
     <NMakeForcedIncludes Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">$(NMakeForcedIncludes)</NMakeForcedIncludes>
     <NMakeAssemblySearchPath Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">$(NMakeAssemblySearchPath)</NMakeAssemblySearchPath>
     <NMakeForcedUsingAssemblies Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">$(NMakeForcedUsingAssemblies)</NMakeForcedUsingAssemblies>
-    <NMakeBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">cd "$(SolutionDir)"
-"$(PythonExe)" build_ssl.py Release $(Platform) -a
-</NMakeBuildCommandLine>
+    <NMakeBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
+cd "$(SolutionDir)$(opensslDir)"
+nasm -f win64 -DNEAR -Ox -g ms\\uptable.asm || echo nasm failed! &amp;&amp; exit
+copy /Y crypto\buildinf_amd64.h crypto\buildinf.h
+copy /Y crypto\opensslconf_amd64.h crypto\opensslconf.h
+nmake /nologo -f "ms\nt64.mak"
+	</NMakeBuildCommandLine>
     <NMakeReBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" />
     <NMakeCleanCommandLine Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">echo OpenSSL must be cleaned manually if you want to rebuild it.</NMakeCleanCommandLine>
     <NMakeOutput Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" />
@@ -140,9 +147,12 @@
     <NMakeForcedIncludes Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">$(NMakeForcedIncludes)</NMakeForcedIncludes>
     <NMakeAssemblySearchPath Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">$(NMakeAssemblySearchPath)</NMakeAssemblySearchPath>
     <NMakeForcedUsingAssemblies Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">$(NMakeForcedUsingAssemblies)</NMakeForcedUsingAssemblies>
-    <NMakeBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">cd "$(SolutionDir)"
-"$(PythonExe)" build_ssl.py Release $(Platform) -a
-</NMakeBuildCommandLine>
+    <NMakeBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
+cd "$(SolutionDir)$(opensslDir)"
+copy /Y crypto\buildinf_x86.h crypto\buildinf.h
+copy /Y crypto\opensslconf_x86.h crypto\opensslconf.h
+nmake /nologo -f "ms\nt.mak"
+	</NMakeBuildCommandLine>
     <NMakeReBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" />
     <NMakeCleanCommandLine Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">echo OpenSSL must be cleaned manually if you want to rebuild it.</NMakeCleanCommandLine>
     <NMakeOutput Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" />
@@ -151,9 +161,13 @@
     <NMakeForcedIncludes Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">$(NMakeForcedIncludes)</NMakeForcedIncludes>
     <NMakeAssemblySearchPath Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">$(NMakeAssemblySearchPath)</NMakeAssemblySearchPath>
     <NMakeForcedUsingAssemblies Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">$(NMakeForcedUsingAssemblies)</NMakeForcedUsingAssemblies>
-    <NMakeBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='Release|x64'">cd "$(SolutionDir)"
-"$(PythonExe)" build_ssl.py Release $(Platform) -a
-</NMakeBuildCommandLine>
+    <NMakeBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
+cd "$(SolutionDir)$(opensslDir)"
+nasm -f win64 -DNEAR -Ox -g ms\\uptable.asm || echo nasm failed! &amp;&amp; exit
+copy /Y crypto\buildinf_amd64.h crypto\buildinf.h
+copy /Y crypto\opensslconf_amd64.h crypto\opensslconf.h
+nmake /nologo -f "ms\nt64.mak"
+	</NMakeBuildCommandLine>
     <NMakeReBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='Release|x64'" />
     <NMakeCleanCommandLine Condition="'$(Configuration)|$(Platform)'=='Release|x64'">echo OpenSSL must be cleaned manually if you want to rebuild it.</NMakeCleanCommandLine>
     <NMakeOutput Condition="'$(Configuration)|$(Platform)'=='Release|x64'" />
@@ -162,9 +176,12 @@
     <NMakeForcedIncludes Condition="'$(Configuration)|$(Platform)'=='Release|x64'">$(NMakeForcedIncludes)</NMakeForcedIncludes>
     <NMakeAssemblySearchPath Condition="'$(Configuration)|$(Platform)'=='Release|x64'">$(NMakeAssemblySearchPath)</NMakeAssemblySearchPath>
     <NMakeForcedUsingAssemblies Condition="'$(Configuration)|$(Platform)'=='Release|x64'">$(NMakeForcedUsingAssemblies)</NMakeForcedUsingAssemblies>
-    <NMakeBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='PGInstrument|Win32'">cd "$(SolutionDir)"
-"$(PythonExe)" build_ssl.py Release $(Platform) -a
-</NMakeBuildCommandLine>
+    <NMakeBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='PGInstrument|Win32'">
+cd "$(SolutionDir)$(opensslDir)"
+copy /Y crypto\buildinf_x86.h crypto\buildinf.h
+copy /Y crypto\opensslconf_x86.h crypto\opensslconf.h
+nmake /nologo -f "ms\nt.mak"
+	</NMakeBuildCommandLine>
     <NMakeReBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='PGInstrument|Win32'" />
     <NMakeCleanCommandLine Condition="'$(Configuration)|$(Platform)'=='PGInstrument|Win32'">echo OpenSSL must be cleaned manually if you want to rebuild it.</NMakeCleanCommandLine>
     <NMakeOutput Condition="'$(Configuration)|$(Platform)'=='PGInstrument|Win32'" />
@@ -173,9 +190,13 @@
     <NMakeForcedIncludes Condition="'$(Configuration)|$(Platform)'=='PGInstrument|Win32'">$(NMakeForcedIncludes)</NMakeForcedIncludes>
     <NMakeAssemblySearchPath Condition="'$(Configuration)|$(Platform)'=='PGInstrument|Win32'">$(NMakeAssemblySearchPath)</NMakeAssemblySearchPath>
     <NMakeForcedUsingAssemblies Condition="'$(Configuration)|$(Platform)'=='PGInstrument|Win32'">$(NMakeForcedUsingAssemblies)</NMakeForcedUsingAssemblies>
-    <NMakeBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='PGInstrument|x64'">cd "$(SolutionDir)"
-"$(PythonExe)" build_ssl.py Release $(Platform) -a
-</NMakeBuildCommandLine>
+    <NMakeBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='PGInstrument|x64'">
+cd "$(SolutionDir)$(opensslDir)"
+nasm -f win64 -DNEAR -Ox -g ms\\uptable.asm || echo nasm failed! &amp;&amp; exit
+copy /Y crypto\buildinf_amd64.h crypto\buildinf.h
+copy /Y crypto\opensslconf_amd64.h crypto\opensslconf.h
+nmake /nologo -f "ms\nt64.mak"
+	</NMakeBuildCommandLine>
     <NMakeReBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='PGInstrument|x64'" />
     <NMakeCleanCommandLine Condition="'$(Configuration)|$(Platform)'=='PGInstrument|x64'">echo OpenSSL must be cleaned manually if you want to rebuild it.</NMakeCleanCommandLine>
     <NMakeOutput Condition="'$(Configuration)|$(Platform)'=='PGInstrument|x64'" />
@@ -184,9 +205,12 @@
     <NMakeForcedIncludes Condition="'$(Configuration)|$(Platform)'=='PGInstrument|x64'">$(NMakeForcedIncludes)</NMakeForcedIncludes>
     <NMakeAssemblySearchPath Condition="'$(Configuration)|$(Platform)'=='PGInstrument|x64'">$(NMakeAssemblySearchPath)</NMakeAssemblySearchPath>
     <NMakeForcedUsingAssemblies Condition="'$(Configuration)|$(Platform)'=='PGInstrument|x64'">$(NMakeForcedUsingAssemblies)</NMakeForcedUsingAssemblies>
-    <NMakeBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='PGUpdate|Win32'">cd "$(SolutionDir)"
-"$(PythonExe)" build_ssl.py Release $(Platform) -a
-</NMakeBuildCommandLine>
+    <NMakeBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='PGUpdate|Win32'">
+cd "$(SolutionDir)$(opensslDir)"
+copy /Y crypto\buildinf_x86.h crypto\buildinf.h
+copy /Y crypto\opensslconf_x86.h crypto\opensslconf.h
+nmake /nologo -f "ms\nt.mak"
+	</NMakeBuildCommandLine>
     <NMakeReBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='PGUpdate|Win32'" />
     <NMakeCleanCommandLine Condition="'$(Configuration)|$(Platform)'=='PGUpdate|Win32'">echo OpenSSL must be cleaned manually if you want to rebuild it.</NMakeCleanCommandLine>
     <NMakeOutput Condition="'$(Configuration)|$(Platform)'=='PGUpdate|Win32'" />
@@ -195,9 +219,13 @@
     <NMakeForcedIncludes Condition="'$(Configuration)|$(Platform)'=='PGUpdate|Win32'">$(NMakeForcedIncludes)</NMakeForcedIncludes>
     <NMakeAssemblySearchPath Condition="'$(Configuration)|$(Platform)'=='PGUpdate|Win32'">$(NMakeAssemblySearchPath)</NMakeAssemblySearchPath>
     <NMakeForcedUsingAssemblies Condition="'$(Configuration)|$(Platform)'=='PGUpdate|Win32'">$(NMakeForcedUsingAssemblies)</NMakeForcedUsingAssemblies>
-    <NMakeBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='PGUpdate|x64'">cd "$(SolutionDir)"
-"$(PythonExe)" build_ssl.py Release $(Platform) -a
-</NMakeBuildCommandLine>
+    <NMakeBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='PGUpdate|x64'">
+cd "$(SolutionDir)$(opensslDir)"
+nasm -f win64 -DNEAR -Ox -g ms\\uptable.asm || echo nasm failed! &amp;&amp; exit
+copy /Y crypto\buildinf_amd64.h crypto\buildinf.h
+copy /Y crypto\opensslconf_amd64.h crypto\opensslconf.h
+nmake /nologo -f "ms\nt64.mak"
+	</NMakeBuildCommandLine>
     <NMakeReBuildCommandLine Condition="'$(Configuration)|$(Platform)'=='PGUpdate|x64'" />
     <NMakeCleanCommandLine Condition="'$(Configuration)|$(Platform)'=='PGUpdate|x64'">echo OpenSSL must be cleaned manually if you want to rebuild it.</NMakeCleanCommandLine>
     <NMakeOutput Condition="'$(Configuration)|$(Platform)'=='PGUpdate|x64'" />
@@ -209,12 +237,6 @@
   </PropertyGroup>
   <ItemDefinitionGroup>
   </ItemDefinitionGroup>
-  <ItemGroup>
-    <ProjectReference Include="python.vcxproj">
-      <Project>{b11d750f-cd1f-4a96-85ce-e69a5c5259f9}</Project>
-      <ReferenceOutputAssembly>false</ReferenceOutputAssembly>
-    </ProjectReference>
-  </ItemGroup>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
   <ImportGroup Label="ExtensionTargets">
   </ImportGroup>

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  9 16:38:08 2014
From: python-checkins at python.org (brett.cannon)
Date: Fri,  9 May 2014 16:38:08 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMTU3?=
 =?utf-8?q?=3A_Touch_up_imp_docs_to_be_more_explicit_about_importlib?=
Message-ID: <3gQDfm0tXsz7LjR@mail.python.org>

http://hg.python.org/cpython/rev/9809a791436d
changeset:   90598:9809a791436d
branch:      3.4
parent:      90595:64ba3f2de99c
user:        Brett Cannon <brett at python.org>
date:        Fri May 09 10:37:31 2014 -0400
summary:
  Issue #21157: Touch up imp docs to be more explicit about importlib
alternatives.

files:
  Doc/library/imp.rst |  12 ++++++++----
  1 files changed, 8 insertions(+), 4 deletions(-)


diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst
--- a/Doc/library/imp.rst
+++ b/Doc/library/imp.rst
@@ -79,7 +79,9 @@
    When *P* itself has a dotted name, apply this recipe recursively.
 
    .. deprecated:: 3.3
-      Use :func:`importlib.find_loader` instead.
+      Use :func:`importlib.util.find_spec` instead unless Python 3.3
+      compatibility is required, in which case use
+      :func:`importlib.find_loader`.
 
 
 .. function:: load_module(name, file, pathname, description)
@@ -104,9 +106,11 @@
 
    .. deprecated:: 3.3
       If previously used in conjunction with :func:`imp.find_module` then
-      call ``load_module()`` on the returned loader. If you wish to load a
-      module from a specific file, then use one of the file-based loaders found
-      in :mod:`importlib.machinery`.
+      consider using :func:`importlib.import_module`, otherwise use the loader
+      returned by the replacement you chose for :func:`imp.find_module`. If you
+      called :func:`imp.load_module` and related functions directly then use the
+      classes in :mod:`importlib.machinery`, e.g.
+      ``importlib.machinery.SourceFileLoader(name, path).load_module()``.
 
 
 .. function:: new_module(name)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  9 16:38:09 2014
From: python-checkins at python.org (brett.cannon)
Date: Fri,  9 May 2014 16:38:09 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge_for_issue_=2321157?=
Message-ID: <3gQDfn2d5wz7Ljl@mail.python.org>

http://hg.python.org/cpython/rev/418780d59502
changeset:   90599:418780d59502
parent:      90597:9ef99fafaadd
parent:      90598:9809a791436d
user:        Brett Cannon <brett at python.org>
date:        Fri May 09 10:37:55 2014 -0400
summary:
  Merge for issue #21157

files:
  Doc/library/imp.rst |  12 ++++++++----
  1 files changed, 8 insertions(+), 4 deletions(-)


diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst
--- a/Doc/library/imp.rst
+++ b/Doc/library/imp.rst
@@ -79,7 +79,9 @@
    When *P* itself has a dotted name, apply this recipe recursively.
 
    .. deprecated:: 3.3
-      Use :func:`importlib.find_loader` instead.
+      Use :func:`importlib.util.find_spec` instead unless Python 3.3
+      compatibility is required, in which case use
+      :func:`importlib.find_loader`.
 
 
 .. function:: load_module(name, file, pathname, description)
@@ -104,9 +106,11 @@
 
    .. deprecated:: 3.3
       If previously used in conjunction with :func:`imp.find_module` then
-      call ``load_module()`` on the returned loader. If you wish to load a
-      module from a specific file, then use one of the file-based loaders found
-      in :mod:`importlib.machinery`.
+      consider using :func:`importlib.import_module`, otherwise use the loader
+      returned by the replacement you chose for :func:`imp.find_module`. If you
+      called :func:`imp.load_module` and related functions directly then use the
+      classes in :mod:`importlib.machinery`, e.g.
+      ``importlib.machinery.SourceFileLoader(name, path).load_module()``.
 
 
 .. function:: new_module(name)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  9 17:56:15 2014
From: python-checkins at python.org (brett.cannon)
Date: Fri,  9 May 2014 17:56:15 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxNDM4?=
 =?utf-8?q?=3A_Document_what_loaders_don=27t_require_a_module_name_for?=
Message-ID: <3gQGNv4Bg9z7Ll0@mail.python.org>

http://hg.python.org/cpython/rev/86042348b38a
changeset:   90600:86042348b38a
branch:      3.4
parent:      90598:9809a791436d
user:        Brett Cannon <brett at python.org>
date:        Fri May 09 11:55:49 2014 -0400
summary:
  Issue #21438: Document what loaders don't require a module name for
load_module().

files:
  Doc/library/importlib.rst |  12 +++++++++++-
  1 files changed, 11 insertions(+), 1 deletions(-)


diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst
--- a/Doc/library/importlib.rst
+++ b/Doc/library/importlib.rst
@@ -887,6 +887,11 @@
 
       Concrete implementation of :meth:`importlib.abc.SourceLoader.set_data`.
 
+   .. method:: load_module(name=None)
+
+      Concrete implementation of :meth:`importlib.abc.Loader.load_module` where
+      specifying the name of the module to load is optional.
+
 
 .. class:: SourcelessFileLoader(fullname, path)
 
@@ -921,6 +926,11 @@
       Returns ``None`` as bytecode files have no source when this loader is
       used.
 
+   .. method:: load_module(name=None)
+
+   Concrete implementation of :meth:`importlib.abc.Loader.load_module` where
+   specifying the name of the module to load is optional.
+
 
 .. class:: ExtensionFileLoader(fullname, path)
 
@@ -940,7 +950,7 @@
 
       Path to the extension module.
 
-   .. method:: load_module(fullname)
+   .. method:: load_module(name=None)
 
       Loads the extension module if and only if *fullname* is the same as
       :attr:`name` or is ``None``.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  9 17:56:16 2014
From: python-checkins at python.org (brett.cannon)
Date: Fri,  9 May 2014 17:56:16 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge_for_issue_=2321438?=
Message-ID: <3gQGNw5YTYz7Lkb@mail.python.org>

http://hg.python.org/cpython/rev/e9453f6fa787
changeset:   90601:e9453f6fa787
parent:      90599:418780d59502
parent:      90600:86042348b38a
user:        Brett Cannon <brett at python.org>
date:        Fri May 09 11:56:07 2014 -0400
summary:
  Merge for issue #21438

files:
  Doc/library/importlib.rst |  12 +++++++++++-
  1 files changed, 11 insertions(+), 1 deletions(-)


diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst
--- a/Doc/library/importlib.rst
+++ b/Doc/library/importlib.rst
@@ -887,6 +887,11 @@
 
       Concrete implementation of :meth:`importlib.abc.SourceLoader.set_data`.
 
+   .. method:: load_module(name=None)
+
+      Concrete implementation of :meth:`importlib.abc.Loader.load_module` where
+      specifying the name of the module to load is optional.
+
 
 .. class:: SourcelessFileLoader(fullname, path)
 
@@ -921,6 +926,11 @@
       Returns ``None`` as bytecode files have no source when this loader is
       used.
 
+   .. method:: load_module(name=None)
+
+   Concrete implementation of :meth:`importlib.abc.Loader.load_module` where
+   specifying the name of the module to load is optional.
+
 
 .. class:: ExtensionFileLoader(fullname, path)
 
@@ -940,7 +950,7 @@
 
       Path to the extension module.
 
-   .. method:: load_module(fullname)
+   .. method:: load_module(name=None)
 
       Loads the extension module if and only if *fullname* is the same as
       :attr:`name` or is ``None``.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  9 18:28:30 2014
From: python-checkins at python.org (brett.cannon)
Date: Fri,  9 May 2014 18:28:30 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321156=3A_importli?=
 =?utf-8?q?b=2Eabc=2EInspectLoader=2Esource=5Fto=5Fcode=28=29_is_now_a?=
Message-ID: <3gQH662mP7z7LjT@mail.python.org>

http://hg.python.org/cpython/rev/9bd844792b32
changeset:   90602:9bd844792b32
user:        Brett Cannon <brett at python.org>
date:        Fri May 09 12:28:22 2014 -0400
summary:
  Issue #21156: importlib.abc.InspectLoader.source_to_code() is now a
staticmethod.

files:
  Doc/library/importlib.rst |  8 +++++++-
  Doc/whatsnew/3.5.rst      |  5 +++++
  Lib/importlib/abc.py      |  3 ++-
  Misc/NEWS                 |  3 +++
  4 files changed, 17 insertions(+), 2 deletions(-)


diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst
--- a/Doc/library/importlib.rst
+++ b/Doc/library/importlib.rst
@@ -499,7 +499,7 @@
         .. versionchanged:: 3.4
            Raises :exc:`ImportError` instead of :exc:`NotImplementedError`.
 
-    .. method:: source_to_code(data, path='<string>')
+    .. staticmethod:: source_to_code(data, path='<string>')
 
         Create a code object from Python source.
 
@@ -508,8 +508,14 @@
         the "path" to where the source code originated from, which can be an
         abstract concept (e.g. location in a zip file).
 
+        With the subsequent code object one can execute it in a module by
+        running ``exec(code, module.__dict__)``.
+
         .. versionadded:: 3.4
 
+        .. versionchanged:: 3.5
+           Made the method static.
+
     .. method:: exec_module(module)
 
        Implementation of :meth:`Loader.exec_module`.
diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst
--- a/Doc/whatsnew/3.5.rst
+++ b/Doc/whatsnew/3.5.rst
@@ -158,6 +158,11 @@
   *module* contains no docstrings instead of raising :exc:`ValueError`
   (contributed by Glenn Jones in :issue:`15916`).
 
+* :func:`importlib.abc.InspectLoader.source_to_code` is now a
+  static method to make it easier to work with source code in a string.
+  With a module object that you want to initialize you can then use
+  ``exec(code, module.__dict__)`` to execute the code in the module.
+
 
 Optimizations
 =============
diff --git a/Lib/importlib/abc.py b/Lib/importlib/abc.py
--- a/Lib/importlib/abc.py
+++ b/Lib/importlib/abc.py
@@ -217,7 +217,8 @@
         """
         raise ImportError
 
-    def source_to_code(self, data, path='<string>'):
+    @staticmethod
+    def source_to_code(data, path='<string>'):
         """Compile 'data' into a code object.
 
         The 'data' argument can be anything that compile() can handle. The'path'
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -73,6 +73,9 @@
 Library
 -------
 
+- Issue #21156: importlib.abc.InspectLoader.source_to_code() is now a
+  staticmethod.
+
 - Issue #21396: Fix TextIOWrapper(..., write_through=True) to not force a
   flush() on the underlying binary stream.  Patch by akira.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  9 19:03:21 2014
From: python-checkins at python.org (tim.golden)
Date: Fri,  9 May 2014 19:03:21 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Issue10752_Be_?=
 =?utf-8?q?more_robust_when_finding_a_PERL_interpreter_to_build_OpenSSL=2E?=
Message-ID: <3gQHtK0wb4z7LjM@mail.python.org>

http://hg.python.org/cpython/rev/160f32753b0c
changeset:   90603:160f32753b0c
branch:      3.4
parent:      90600:86042348b38a
user:        Tim Golden <mail at timgolden.me.uk>
date:        Fri May 09 18:01:19 2014 +0100
summary:
  Issue10752 Be more robust when finding a PERL interpreter to build OpenSSL. Initial patch by Gabi Davar

files:
  PCbuild/build_ssl.py |  18 +++++++++---------
  1 files changed, 9 insertions(+), 9 deletions(-)


diff --git a/PCbuild/build_ssl.py b/PCbuild/build_ssl.py
--- a/PCbuild/build_ssl.py
+++ b/PCbuild/build_ssl.py
@@ -24,6 +24,7 @@
 # python.exe build_ssl.py Release Win32
 
 import os, sys, re, shutil
+import subprocess
 
 # Find all "foo.exe" files on the PATH.
 def find_all_on_path(filename, extras = None):
@@ -46,22 +47,21 @@
 # is available.
 def find_working_perl(perls):
     for perl in perls:
-        fh = os.popen('"%s" -e "use Win32;"' % perl)
-        fh.read()
-        rc = fh.close()
-        if rc:
+        try:
+            subprocess.check_output([perl, "-e", "use Win32;"])
+        except subprocess.CalledProcessError:
             continue
-        return perl
-    print("Can not find a suitable PERL:")
+        else:
+            return perl
+
     if perls:
-        print(" the following perl interpreters were found:")
+        print("The following perl interpreters were found:")
         for p in perls:
             print(" ", p)
         print(" None of these versions appear suitable for building OpenSSL")
     else:
-        print(" NO perl interpreters were found on this machine at all!")
+        print("NO perl interpreters were found on this machine at all!")
     print(" Please install ActivePerl and ensure it appears on your path")
-    return None
 
 # Fetch SSL directory from VC properties
 def get_ssl_dir():

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  9 19:03:22 2014
From: python-checkins at python.org (tim.golden)
Date: Fri,  9 May 2014 19:03:22 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Issue10752_Be_more_robust_when_finding_a_PERL_interprete?=
 =?utf-8?q?r_to_build_OpenSSL=2E?=
Message-ID: <3gQHtL2HcYz7Ljm@mail.python.org>

http://hg.python.org/cpython/rev/e492d0ac9abb
changeset:   90604:e492d0ac9abb
parent:      90602:9bd844792b32
parent:      90603:160f32753b0c
user:        Tim Golden <mail at timgolden.me.uk>
date:        Fri May 09 18:01:44 2014 +0100
summary:
  Issue10752 Be more robust when finding a PERL interpreter to build OpenSSL. Initial patch by Gabi Davar

files:
  PCbuild/prepare_ssl.py |  18 +++++++++---------
  1 files changed, 9 insertions(+), 9 deletions(-)


diff --git a/PCbuild/prepare_ssl.py b/PCbuild/prepare_ssl.py
--- a/PCbuild/prepare_ssl.py
+++ b/PCbuild/prepare_ssl.py
@@ -21,6 +21,7 @@
 import re
 import sys
 import shutil
+import subprocess
 
 # Find all "foo.exe" files on the PATH.
 def find_all_on_path(filename, extras = None):
@@ -43,22 +44,21 @@
 # is available.
 def find_working_perl(perls):
     for perl in perls:
-        fh = os.popen('"%s" -e "use Win32;"' % perl)
-        fh.read()
-        rc = fh.close()
-        if rc:
+        try:
+            subprocess.check_output([perl, "-e", "use Win32;"])
+        except subprocess.CalledProcessError:
             continue
-        return perl
-    print("Can not find a suitable PERL:")
+        else:
+            return perl
+
     if perls:
-        print(" the following perl interpreters were found:")
+        print("The following perl interpreters were found:")
         for p in perls:
             print(" ", p)
         print(" None of these versions appear suitable for building OpenSSL")
     else:
-        print(" NO perl interpreters were found on this machine at all!")
+        print("NO perl interpreters were found on this machine at all!")
     print(" Please install ActivePerl and ensure it appears on your path")
-    return None
 
 def create_makefile64(makefile, m32):
     """Create and fix makefile for 64bit

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  9 19:19:55 2014
From: python-checkins at python.org (tim.golden)
Date: Fri,  9 May 2014 19:19:55 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Issue21452_Add?=
 =?utf-8?q?_missing_backslash_to_build_path_for_make=5Fbuildinfo?=
Message-ID: <3gQJFR3bFgz7Ljv@mail.python.org>

http://hg.python.org/cpython/rev/469837abe5ca
changeset:   90605:469837abe5ca
branch:      3.4
parent:      90600:86042348b38a
user:        Tim Golden <mail at timgolden.me.uk>
date:        Fri May 09 18:18:11 2014 +0100
summary:
  Issue21452 Add missing backslash to build path for make_buildinfo

files:
  PCbuild/pythoncore.vcxproj |  12 ++++++------
  1 files changed, 6 insertions(+), 6 deletions(-)


diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj
--- a/PCbuild/pythoncore.vcxproj
+++ b/PCbuild/pythoncore.vcxproj
@@ -173,7 +173,7 @@
     </ResourceCompile>
     <PreLinkEvent>
       <Message>Generate build information...</Message>
-      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)"</Command>
+      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)\"</Command>
     </PreLinkEvent>
     <Link>
       <AdditionalDependencies>$(IntDir)getbuildinfo.o;%(AdditionalDependencies)</AdditionalDependencies>
@@ -209,7 +209,7 @@
     </ResourceCompile>
     <PreLinkEvent>
       <Message>Generate build information...</Message>
-      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)"</Command>
+      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)\"</Command>
     </PreLinkEvent>
     <Link>
       <AdditionalDependencies>$(IntDir)getbuildinfo.o;%(AdditionalDependencies)</AdditionalDependencies>
@@ -314,7 +314,7 @@
     </ResourceCompile>
     <PreLinkEvent>
       <Message>Generate build information...</Message>
-      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)"</Command>
+      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)\"</Command>
     </PreLinkEvent>
     <Link>
       <AdditionalDependencies>$(IntDir)getbuildinfo.o;%(AdditionalDependencies)</AdditionalDependencies>
@@ -350,7 +350,7 @@
     </ResourceCompile>
     <PreLinkEvent>
       <Message>Generate build information...</Message>
-      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)"</Command>
+      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)\"</Command>
     </PreLinkEvent>
     <Link>
       <AdditionalDependencies>$(IntDir)getbuildinfo.o;%(AdditionalDependencies)</AdditionalDependencies>
@@ -383,7 +383,7 @@
     </ResourceCompile>
     <PreLinkEvent>
       <Message>Generate build information...</Message>
-      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)"</Command>
+      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)\"</Command>
     </PreLinkEvent>
     <Link>
       <AdditionalDependencies>$(IntDir)getbuildinfo.o;%(AdditionalDependencies)</AdditionalDependencies>
@@ -419,7 +419,7 @@
     </ResourceCompile>
     <PreLinkEvent>
       <Message>Generate build information...</Message>
-      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)"</Command>
+      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)\"</Command>
     </PreLinkEvent>
     <Link>
       <AdditionalDependencies>$(IntDir)getbuildinfo.o;%(AdditionalDependencies)</AdditionalDependencies>

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  9 19:19:56 2014
From: python-checkins at python.org (tim.golden)
Date: Fri,  9 May 2014 19:19:56 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy40IC0+IDMuNCk6?=
	=?utf-8?q?_Merge?=
Message-ID: <3gQJFS5HnVz7Lk2@mail.python.org>

http://hg.python.org/cpython/rev/3dea57884dcd
changeset:   90606:3dea57884dcd
branch:      3.4
parent:      90605:469837abe5ca
parent:      90603:160f32753b0c
user:        Tim Golden <mail at timgolden.me.uk>
date:        Fri May 09 18:19:13 2014 +0100
summary:
  Merge

files:
  PCbuild/build_ssl.py |  18 +++++++++---------
  1 files changed, 9 insertions(+), 9 deletions(-)


diff --git a/PCbuild/build_ssl.py b/PCbuild/build_ssl.py
--- a/PCbuild/build_ssl.py
+++ b/PCbuild/build_ssl.py
@@ -24,6 +24,7 @@
 # python.exe build_ssl.py Release Win32
 
 import os, sys, re, shutil
+import subprocess
 
 # Find all "foo.exe" files on the PATH.
 def find_all_on_path(filename, extras = None):
@@ -46,22 +47,21 @@
 # is available.
 def find_working_perl(perls):
     for perl in perls:
-        fh = os.popen('"%s" -e "use Win32;"' % perl)
-        fh.read()
-        rc = fh.close()
-        if rc:
+        try:
+            subprocess.check_output([perl, "-e", "use Win32;"])
+        except subprocess.CalledProcessError:
             continue
-        return perl
-    print("Can not find a suitable PERL:")
+        else:
+            return perl
+
     if perls:
-        print(" the following perl interpreters were found:")
+        print("The following perl interpreters were found:")
         for p in perls:
             print(" ", p)
         print(" None of these versions appear suitable for building OpenSSL")
     else:
-        print(" NO perl interpreters were found on this machine at all!")
+        print("NO perl interpreters were found on this machine at all!")
     print(" Please install ActivePerl and ensure it appears on your path")
-    return None
 
 # Fetch SSL directory from VC properties
 def get_ssl_dir():

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  9 19:19:57 2014
From: python-checkins at python.org (tim.golden)
Date: Fri,  9 May 2014 19:19:57 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Issue21452_Add_missing_backslash_to_build_path_for_make?=
 =?utf-8?q?=5Fbuildinfo?=
Message-ID: <3gQJFT6qs5z7LkP@mail.python.org>

http://hg.python.org/cpython/rev/a14420d8b556
changeset:   90607:a14420d8b556
parent:      90604:e492d0ac9abb
parent:      90606:3dea57884dcd
user:        Tim Golden <mail at timgolden.me.uk>
date:        Fri May 09 18:19:31 2014 +0100
summary:
  Issue21452 Add missing backslash to build path for make_buildinfo

files:
  PCbuild/pythoncore.vcxproj |  12 ++++++------
  1 files changed, 6 insertions(+), 6 deletions(-)


diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj
--- a/PCbuild/pythoncore.vcxproj
+++ b/PCbuild/pythoncore.vcxproj
@@ -173,7 +173,7 @@
     </ResourceCompile>
     <PreLinkEvent>
       <Message>Generate build information...</Message>
-      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)"</Command>
+      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)\"</Command>
     </PreLinkEvent>
     <Link>
       <AdditionalDependencies>$(IntDir)getbuildinfo.o;%(AdditionalDependencies)</AdditionalDependencies>
@@ -209,7 +209,7 @@
     </ResourceCompile>
     <PreLinkEvent>
       <Message>Generate build information...</Message>
-      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)"</Command>
+      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)\"</Command>
     </PreLinkEvent>
     <Link>
       <AdditionalDependencies>$(IntDir)getbuildinfo.o;%(AdditionalDependencies)</AdditionalDependencies>
@@ -314,7 +314,7 @@
     </ResourceCompile>
     <PreLinkEvent>
       <Message>Generate build information...</Message>
-      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)"</Command>
+      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)\"</Command>
     </PreLinkEvent>
     <Link>
       <AdditionalDependencies>$(IntDir)getbuildinfo.o;%(AdditionalDependencies)</AdditionalDependencies>
@@ -350,7 +350,7 @@
     </ResourceCompile>
     <PreLinkEvent>
       <Message>Generate build information...</Message>
-      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)"</Command>
+      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)\"</Command>
     </PreLinkEvent>
     <Link>
       <AdditionalDependencies>$(IntDir)getbuildinfo.o;%(AdditionalDependencies)</AdditionalDependencies>
@@ -383,7 +383,7 @@
     </ResourceCompile>
     <PreLinkEvent>
       <Message>Generate build information...</Message>
-      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)"</Command>
+      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)\"</Command>
     </PreLinkEvent>
     <Link>
       <AdditionalDependencies>$(IntDir)getbuildinfo.o;%(AdditionalDependencies)</AdditionalDependencies>
@@ -419,7 +419,7 @@
     </ResourceCompile>
     <PreLinkEvent>
       <Message>Generate build information...</Message>
-      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)"</Command>
+      <Command>"$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)\"</Command>
     </PreLinkEvent>
     <Link>
       <AdditionalDependencies>$(IntDir)getbuildinfo.o;%(AdditionalDependencies)</AdditionalDependencies>

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  9 19:38:20 2014
From: python-checkins at python.org (brett.cannon)
Date: Fri,  9 May 2014 19:38:20 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320776=3A_Flesh_ou?=
 =?utf-8?q?t_tests_for_importlib=2Emachinery=2EPathFinder=2E?=
Message-ID: <3gQJfh4cjdz7LjR@mail.python.org>

http://hg.python.org/cpython/rev/fa439bb9d705
changeset:   90608:fa439bb9d705
user:        Brett Cannon <brett at python.org>
date:        Fri May 09 13:38:11 2014 -0400
summary:
  Issue #20776: Flesh out tests for importlib.machinery.PathFinder.

files:
  Lib/test/test_importlib/import_/test_path.py |  47 ++++++++++
  1 files changed, 47 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_importlib/import_/test_path.py b/Lib/test/test_importlib/import_/test_path.py
--- a/Lib/test/test_importlib/import_/test_path.py
+++ b/Lib/test/test_importlib/import_/test_path.py
@@ -112,6 +112,53 @@
             if email is not missing:
                 sys.modules['email'] = email
 
+    def test_finder_with_find_module(self):
+        class TestFinder:
+            def find_module(self, fullname):
+                return self.to_return
+        failing_finder = TestFinder()
+        failing_finder.to_return = None
+        path = 'testing path'
+        with util.import_state(path_importer_cache={path: failing_finder}):
+            self.assertIsNone(
+                    self.machinery.PathFinder.find_spec('whatever', [path]))
+        success_finder = TestFinder()
+        success_finder.to_return = __loader__
+        with util.import_state(path_importer_cache={path: success_finder}):
+            spec = self.machinery.PathFinder.find_spec('whatever', [path])
+        self.assertEqual(spec.loader, __loader__)
+
+    def test_finder_with_find_loader(self):
+        class TestFinder:
+            loader = None
+            portions = []
+            def find_loader(self, fullname):
+                return self.loader, self.portions
+        path = 'testing path'
+        with util.import_state(path_importer_cache={path: TestFinder()}):
+            self.assertIsNone(
+                    self.machinery.PathFinder.find_spec('whatever', [path]))
+        success_finder = TestFinder()
+        success_finder.loader = __loader__
+        with util.import_state(path_importer_cache={path: success_finder}):
+            spec = self.machinery.PathFinder.find_spec('whatever', [path])
+        self.assertEqual(spec.loader, __loader__)
+
+    def test_finder_with_find_spec(self):
+        class TestFinder:
+            spec = None
+            def find_spec(self, fullname, target=None):
+                return self.spec
+        path = 'testing path'
+        with util.import_state(path_importer_cache={path: TestFinder()}):
+            self.assertIsNone(
+                    self.machinery.PathFinder.find_spec('whatever', [path]))
+        success_finder = TestFinder()
+        success_finder.spec = self.machinery.ModuleSpec('whatever', __loader__)
+        with util.import_state(path_importer_cache={path: success_finder}):
+            got = self.machinery.PathFinder.find_spec('whatever', [path])
+        self.assertEqual(got, success_finder.spec)
+
 Frozen_FinderTests, Source_FinderTests = util.test_both(
         FinderTests, importlib=importlib, machinery=machinery)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May  9 20:33:05 2014
From: python-checkins at python.org (brett.cannon)
Date: Fri,  9 May 2014 20:33:05 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319721=3A_Consolid?=
 =?utf-8?q?ate_test=5Fimportlib_utility_code_into_a_single?=
Message-ID: <3gQKss65Tyz7Ljl@mail.python.org>

http://hg.python.org/cpython/rev/4e243b399307
changeset:   90609:4e243b399307
user:        Brett Cannon <brett at python.org>
date:        Fri May 09 14:32:57 2014 -0400
summary:
  Issue #19721: Consolidate test_importlib utility code into a single
module.

files:
  Lib/test/test_importlib/builtin/test_finder.py             |   19 +-
  Lib/test/test_importlib/builtin/test_loader.py             |   25 +-
  Lib/test/test_importlib/builtin/util.py                    |    7 -
  Lib/test/test_importlib/extension/test_case_sensitivity.py |    7 +-
  Lib/test/test_importlib/extension/test_finder.py           |   11 +-
  Lib/test/test_importlib/extension/test_loader.py           |   31 +-
  Lib/test/test_importlib/extension/test_path_hook.py        |    9 +-
  Lib/test/test_importlib/extension/util.py                  |   19 -
  Lib/test/test_importlib/import_/test___loader__.py         |    5 +-
  Lib/test/test_importlib/import_/test___package__.py        |    7 +-
  Lib/test/test_importlib/import_/test_api.py                |    5 +-
  Lib/test/test_importlib/import_/test_caching.py            |    5 +-
  Lib/test/test_importlib/import_/test_fromlist.py           |    5 +-
  Lib/test/test_importlib/import_/test_meta_path.py          |    7 +-
  Lib/test/test_importlib/import_/test_packages.py           |    3 +-
  Lib/test/test_importlib/import_/test_path.py               |    5 +-
  Lib/test/test_importlib/import_/test_relative_imports.py   |    3 +-
  Lib/test/test_importlib/import_/util.py                    |   20 -
  Lib/test/test_importlib/source/test_case_sensitivity.py    |    3 +-
  Lib/test/test_importlib/source/test_file_loader.py         |   61 ++--
  Lib/test/test_importlib/source/test_finder.py              |   11 +-
  Lib/test/test_importlib/source/test_path_hook.py           |    3 +-
  Lib/test/test_importlib/source/test_source_encoding.py     |    5 +-
  Lib/test/test_importlib/source/util.py                     |   96 ------
  Lib/test/test_importlib/util.py                            |  147 +++++++++-
  25 files changed, 249 insertions(+), 270 deletions(-)


diff --git a/Lib/test/test_importlib/builtin/test_finder.py b/Lib/test/test_importlib/builtin/test_finder.py
--- a/Lib/test/test_importlib/builtin/test_finder.py
+++ b/Lib/test/test_importlib/builtin/test_finder.py
@@ -1,6 +1,5 @@
 from .. import abc
 from .. import util
-from . import util as builtin_util
 
 frozen_machinery, source_machinery = util.import_importlib('importlib.machinery')
 
@@ -8,14 +7,15 @@
 import unittest
 
 
+ at unittest.skipIf(util.BUILTINS.good_name is None, 'no reasonable builtin module')
 class FindSpecTests(abc.FinderTests):
 
     """Test find_spec() for built-in modules."""
 
     def test_module(self):
         # Common case.
-        with util.uncache(builtin_util.NAME):
-            found = self.machinery.BuiltinImporter.find_spec(builtin_util.NAME)
+        with util.uncache(util.BUILTINS.good_name):
+            found = self.machinery.BuiltinImporter.find_spec(util.BUILTINS.good_name)
             self.assertTrue(found)
             self.assertEqual(found.origin, 'built-in')
 
@@ -39,8 +39,8 @@
 
     def test_ignore_path(self):
         # The value for 'path' should always trigger a failed import.
-        with util.uncache(builtin_util.NAME):
-            spec = self.machinery.BuiltinImporter.find_spec(builtin_util.NAME,
+        with util.uncache(util.BUILTINS.good_name):
+            spec = self.machinery.BuiltinImporter.find_spec(util.BUILTINS.good_name,
                                                             ['pkg'])
             self.assertIsNone(spec)
 
@@ -48,14 +48,15 @@
         machinery=[frozen_machinery, source_machinery])
 
 
+ at unittest.skipIf(util.BUILTINS.good_name is None, 'no reasonable builtin module')
 class FinderTests(abc.FinderTests):
 
     """Test find_module() for built-in modules."""
 
     def test_module(self):
         # Common case.
-        with util.uncache(builtin_util.NAME):
-            found = self.machinery.BuiltinImporter.find_module(builtin_util.NAME)
+        with util.uncache(util.BUILTINS.good_name):
+            found = self.machinery.BuiltinImporter.find_module(util.BUILTINS.good_name)
             self.assertTrue(found)
             self.assertTrue(hasattr(found, 'load_module'))
 
@@ -72,8 +73,8 @@
 
     def test_ignore_path(self):
         # The value for 'path' should always trigger a failed import.
-        with util.uncache(builtin_util.NAME):
-            loader = self.machinery.BuiltinImporter.find_module(builtin_util.NAME,
+        with util.uncache(util.BUILTINS.good_name):
+            loader = self.machinery.BuiltinImporter.find_module(util.BUILTINS.good_name,
                                                             ['pkg'])
             self.assertIsNone(loader)
 
diff --git a/Lib/test/test_importlib/builtin/test_loader.py b/Lib/test/test_importlib/builtin/test_loader.py
--- a/Lib/test/test_importlib/builtin/test_loader.py
+++ b/Lib/test/test_importlib/builtin/test_loader.py
@@ -1,6 +1,5 @@
 from .. import abc
 from .. import util
-from . import util as builtin_util
 
 frozen_machinery, source_machinery = util.import_importlib('importlib.machinery')
 
@@ -8,7 +7,7 @@
 import types
 import unittest
 
-
+ at unittest.skipIf(util.BUILTINS.good_name is None, 'no reasonable builtin module')
 class LoaderTests(abc.LoaderTests):
 
     """Test load_module() for built-in modules."""
@@ -29,8 +28,8 @@
 
     def test_module(self):
         # Common case.
-        with util.uncache(builtin_util.NAME):
-            module = self.load_module(builtin_util.NAME)
+        with util.uncache(util.BUILTINS.good_name):
+            module = self.load_module(util.BUILTINS.good_name)
             self.verify(module)
 
     # Built-in modules cannot be a package.
@@ -41,9 +40,9 @@
 
     def test_module_reuse(self):
         # Test that the same module is used in a reload.
-        with util.uncache(builtin_util.NAME):
-            module1 = self.load_module(builtin_util.NAME)
-            module2 = self.load_module(builtin_util.NAME)
+        with util.uncache(util.BUILTINS.good_name):
+            module1 = self.load_module(util.BUILTINS.good_name)
+            module2 = self.load_module(util.BUILTINS.good_name)
             self.assertIs(module1, module2)
 
     def test_unloadable(self):
@@ -70,32 +69,34 @@
         machinery=[frozen_machinery, source_machinery])
 
 
+ at unittest.skipIf(util.BUILTINS.good_name is None, 'no reasonable builtin module')
 class InspectLoaderTests:
 
     """Tests for InspectLoader methods for BuiltinImporter."""
 
     def test_get_code(self):
         # There is no code object.
-        result = self.machinery.BuiltinImporter.get_code(builtin_util.NAME)
+        result = self.machinery.BuiltinImporter.get_code(util.BUILTINS.good_name)
         self.assertIsNone(result)
 
     def test_get_source(self):
         # There is no source.
-        result = self.machinery.BuiltinImporter.get_source(builtin_util.NAME)
+        result = self.machinery.BuiltinImporter.get_source(util.BUILTINS.good_name)
         self.assertIsNone(result)
 
     def test_is_package(self):
         # Cannot be a package.
-        result = self.machinery.BuiltinImporter.is_package(builtin_util.NAME)
+        result = self.machinery.BuiltinImporter.is_package(util.BUILTINS.good_name)
         self.assertTrue(not result)
 
+    @unittest.skipIf(util.BUILTINS.bad_name is None, 'all modules are built in')
     def test_not_builtin(self):
         # Modules not built-in should raise ImportError.
         for meth_name in ('get_code', 'get_source', 'is_package'):
             method = getattr(self.machinery.BuiltinImporter, meth_name)
         with self.assertRaises(ImportError) as cm:
-            method(builtin_util.BAD_NAME)
-        self.assertRaises(builtin_util.BAD_NAME)
+            method(util.BUILTINS.bad_name)
+        self.assertRaises(util.BUILTINS.bad_name)
 
 Frozen_InspectLoaderTests, Source_InspectLoaderTests = util.test_both(
         InspectLoaderTests,
diff --git a/Lib/test/test_importlib/builtin/util.py b/Lib/test/test_importlib/builtin/util.py
deleted file mode 100644
--- a/Lib/test/test_importlib/builtin/util.py
+++ /dev/null
@@ -1,7 +0,0 @@
-import sys
-
-assert 'errno' in sys.builtin_module_names
-NAME = 'errno'
-
-assert 'importlib' not in sys.builtin_module_names
-BAD_NAME = 'importlib'
diff --git a/Lib/test/test_importlib/extension/test_case_sensitivity.py b/Lib/test/test_importlib/extension/test_case_sensitivity.py
--- a/Lib/test/test_importlib/extension/test_case_sensitivity.py
+++ b/Lib/test/test_importlib/extension/test_case_sensitivity.py
@@ -4,22 +4,21 @@
 import unittest
 
 from .. import util
-from . import util as ext_util
 
 frozen_machinery, source_machinery = util.import_importlib('importlib.machinery')
 
 
 # XXX find_spec tests
 
- at unittest.skipIf(ext_util.FILENAME is None, '_testcapi not available')
+ at unittest.skipIf(util.EXTENSIONS.filename is None, '_testcapi not available')
 @util.case_insensitive_tests
 class ExtensionModuleCaseSensitivityTest:
 
     def find_module(self):
-        good_name = ext_util.NAME
+        good_name = util.EXTENSIONS.name
         bad_name = good_name.upper()
         assert good_name != bad_name
-        finder = self.machinery.FileFinder(ext_util.PATH,
+        finder = self.machinery.FileFinder(util.EXTENSIONS.path,
                                           (self.machinery.ExtensionFileLoader,
                                            self.machinery.EXTENSION_SUFFIXES))
         return finder.find_module(bad_name)
diff --git a/Lib/test/test_importlib/extension/test_finder.py b/Lib/test/test_importlib/extension/test_finder.py
--- a/Lib/test/test_importlib/extension/test_finder.py
+++ b/Lib/test/test_importlib/extension/test_finder.py
@@ -1,8 +1,7 @@
 from .. import abc
-from .. import util as test_util
-from . import util
+from .. import util
 
-machinery = test_util.import_importlib('importlib.machinery')
+machinery = util.import_importlib('importlib.machinery')
 
 import unittest
 import warnings
@@ -14,7 +13,7 @@
     """Test the finder for extension modules."""
 
     def find_module(self, fullname):
-        importer = self.machinery.FileFinder(util.PATH,
+        importer = self.machinery.FileFinder(util.EXTENSIONS.path,
                                             (self.machinery.ExtensionFileLoader,
                                              self.machinery.EXTENSION_SUFFIXES))
         with warnings.catch_warnings():
@@ -22,7 +21,7 @@
             return importer.find_module(fullname)
 
     def test_module(self):
-        self.assertTrue(self.find_module(util.NAME))
+        self.assertTrue(self.find_module(util.EXTENSIONS.name))
 
     # No extension module as an __init__ available for testing.
     test_package = test_package_in_package = None
@@ -36,7 +35,7 @@
     def test_failure(self):
         self.assertIsNone(self.find_module('asdfjkl;'))
 
-Frozen_FinderTests, Source_FinderTests = test_util.test_both(
+Frozen_FinderTests, Source_FinderTests = util.test_both(
         FinderTests, machinery=machinery)
 
 
diff --git a/Lib/test/test_importlib/extension/test_loader.py b/Lib/test/test_importlib/extension/test_loader.py
--- a/Lib/test/test_importlib/extension/test_loader.py
+++ b/Lib/test/test_importlib/extension/test_loader.py
@@ -1,4 +1,3 @@
-from . import util as ext_util
 from .. import abc
 from .. import util
 
@@ -15,8 +14,8 @@
     """Test load_module() for extension modules."""
 
     def setUp(self):
-        self.loader = self.machinery.ExtensionFileLoader(ext_util.NAME,
-                                                         ext_util.FILEPATH)
+        self.loader = self.machinery.ExtensionFileLoader(util.EXTENSIONS.name,
+                                                         util.EXTENSIONS.file_path)
 
     def load_module(self, fullname):
         return self.loader.load_module(fullname)
@@ -29,23 +28,23 @@
             self.load_module('XXX')
 
     def test_equality(self):
-        other = self.machinery.ExtensionFileLoader(ext_util.NAME,
-                                                   ext_util.FILEPATH)
+        other = self.machinery.ExtensionFileLoader(util.EXTENSIONS.name,
+                                                   util.EXTENSIONS.file_path)
         self.assertEqual(self.loader, other)
 
     def test_inequality(self):
-        other = self.machinery.ExtensionFileLoader('_' + ext_util.NAME,
-                                                   ext_util.FILEPATH)
+        other = self.machinery.ExtensionFileLoader('_' + util.EXTENSIONS.name,
+                                                   util.EXTENSIONS.file_path)
         self.assertNotEqual(self.loader, other)
 
     def test_module(self):
-        with util.uncache(ext_util.NAME):
-            module = self.load_module(ext_util.NAME)
-            for attr, value in [('__name__', ext_util.NAME),
-                                ('__file__', ext_util.FILEPATH),
+        with util.uncache(util.EXTENSIONS.name):
+            module = self.load_module(util.EXTENSIONS.name)
+            for attr, value in [('__name__', util.EXTENSIONS.name),
+                                ('__file__', util.EXTENSIONS.file_path),
                                 ('__package__', '')]:
                 self.assertEqual(getattr(module, attr), value)
-            self.assertIn(ext_util.NAME, sys.modules)
+            self.assertIn(util.EXTENSIONS.name, sys.modules)
             self.assertIsInstance(module.__loader__,
                                   self.machinery.ExtensionFileLoader)
 
@@ -56,9 +55,9 @@
     test_lacking_parent = None
 
     def test_module_reuse(self):
-        with util.uncache(ext_util.NAME):
-            module1 = self.load_module(ext_util.NAME)
-            module2 = self.load_module(ext_util.NAME)
+        with util.uncache(util.EXTENSIONS.name):
+            module1 = self.load_module(util.EXTENSIONS.name)
+            module2 = self.load_module(util.EXTENSIONS.name)
             self.assertIs(module1, module2)
 
     # No easy way to trigger a failure after a successful import.
@@ -71,7 +70,7 @@
         self.assertEqual(cm.exception.name, name)
 
     def test_is_package(self):
-        self.assertFalse(self.loader.is_package(ext_util.NAME))
+        self.assertFalse(self.loader.is_package(util.EXTENSIONS.name))
         for suffix in self.machinery.EXTENSION_SUFFIXES:
             path = os.path.join('some', 'path', 'pkg', '__init__' + suffix)
             loader = self.machinery.ExtensionFileLoader('pkg', path)
diff --git a/Lib/test/test_importlib/extension/test_path_hook.py b/Lib/test/test_importlib/extension/test_path_hook.py
--- a/Lib/test/test_importlib/extension/test_path_hook.py
+++ b/Lib/test/test_importlib/extension/test_path_hook.py
@@ -1,7 +1,6 @@
-from .. import util as test_util
-from . import util
+from .. import util
 
-machinery = test_util.import_importlib('importlib.machinery')
+machinery = util.import_importlib('importlib.machinery')
 
 import collections
 import sys
@@ -22,9 +21,9 @@
     def test_success(self):
         # Path hook should handle a directory where a known extension module
         # exists.
-        self.assertTrue(hasattr(self.hook(util.PATH), 'find_module'))
+        self.assertTrue(hasattr(self.hook(util.EXTENSIONS.path), 'find_module'))
 
-Frozen_PathHooksTests, Source_PathHooksTests = test_util.test_both(
+Frozen_PathHooksTests, Source_PathHooksTests = util.test_both(
         PathHookTests, machinery=machinery)
 
 
diff --git a/Lib/test/test_importlib/extension/util.py b/Lib/test/test_importlib/extension/util.py
deleted file mode 100644
--- a/Lib/test/test_importlib/extension/util.py
+++ /dev/null
@@ -1,19 +0,0 @@
-from importlib import machinery
-import os
-import sys
-
-PATH = None
-EXT = None
-FILENAME = None
-NAME = '_testcapi'
-try:
-    for PATH in sys.path:
-        for EXT in machinery.EXTENSION_SUFFIXES:
-            FILENAME = NAME + EXT
-            FILEPATH = os.path.join(PATH, FILENAME)
-            if os.path.exists(os.path.join(PATH, FILENAME)):
-                raise StopIteration
-    else:
-        PATH = EXT = FILENAME = FILEPATH = None
-except StopIteration:
-    pass
diff --git a/Lib/test/test_importlib/import_/test___loader__.py b/Lib/test/test_importlib/import_/test___loader__.py
--- a/Lib/test/test_importlib/import_/test___loader__.py
+++ b/Lib/test/test_importlib/import_/test___loader__.py
@@ -4,7 +4,6 @@
 import unittest
 
 from .. import util
-from . import util as import_util
 
 
 class SpecLoaderMock:
@@ -25,7 +24,7 @@
         self.assertEqual(loader, module.__loader__)
 
 Frozen_SpecTests, Source_SpecTests = util.test_both(
-        SpecLoaderAttributeTests, __import__=import_util.__import__)
+        SpecLoaderAttributeTests, __import__=util.__import__)
 
 
 class LoaderMock:
@@ -63,7 +62,7 @@
 
 
 Frozen_Tests, Source_Tests = util.test_both(LoaderAttributeTests,
-                                            __import__=import_util.__import__)
+                                            __import__=util.__import__)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/import_/test___package__.py b/Lib/test/test_importlib/import_/test___package__.py
--- a/Lib/test/test_importlib/import_/test___package__.py
+++ b/Lib/test/test_importlib/import_/test___package__.py
@@ -6,7 +6,6 @@
 """
 import unittest
 from .. import util
-from . import util as import_util
 
 
 class Using__package__:
@@ -74,13 +73,13 @@
     mock_modules = util.mock_modules
 
 Frozen_UsingPackagePEP302, Source_UsingPackagePEP302 = util.test_both(
-        Using__package__PEP302, __import__=import_util.__import__)
+        Using__package__PEP302, __import__=util.__import__)
 
 class Using__package__PEP302(Using__package__):
     mock_modules = util.mock_spec
 
 Frozen_UsingPackagePEP451, Source_UsingPackagePEP451 = util.test_both(
-        Using__package__PEP302, __import__=import_util.__import__)
+        Using__package__PEP302, __import__=util.__import__)
 
 
 class Setting__package__:
@@ -95,7 +94,7 @@
 
     """
 
-    __import__ = import_util.__import__[1]
+    __import__ = util.__import__[1]
 
     # [top-level]
     def test_top_level(self):
diff --git a/Lib/test/test_importlib/import_/test_api.py b/Lib/test/test_importlib/import_/test_api.py
--- a/Lib/test/test_importlib/import_/test_api.py
+++ b/Lib/test/test_importlib/import_/test_api.py
@@ -1,5 +1,4 @@
 from .. import util
-from . import util as import_util
 
 from importlib import machinery
 import sys
@@ -80,14 +79,14 @@
     bad_finder_loader = BadLoaderFinder
 
 Frozen_OldAPITests, Source_OldAPITests = util.test_both(
-        OldAPITests, __import__=import_util.__import__)
+        OldAPITests, __import__=util.__import__)
 
 
 class SpecAPITests(APITest):
     bad_finder_loader = BadSpecFinderLoader
 
 Frozen_SpecAPITests, Source_SpecAPITests = util.test_both(
-        SpecAPITests, __import__=import_util.__import__)
+        SpecAPITests, __import__=util.__import__)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/import_/test_caching.py b/Lib/test/test_importlib/import_/test_caching.py
--- a/Lib/test/test_importlib/import_/test_caching.py
+++ b/Lib/test/test_importlib/import_/test_caching.py
@@ -1,6 +1,5 @@
 """Test that sys.modules is used properly by import."""
 from .. import util
-from . import util as import_util
 import sys
 from types import MethodType
 import unittest
@@ -40,14 +39,14 @@
             self.assertEqual(cm.exception.name, name)
 
 Frozen_UseCache, Source_UseCache = util.test_both(
-        UseCache, __import__=import_util.__import__)
+        UseCache, __import__=util.__import__)
 
 
 class ImportlibUseCache(UseCache, unittest.TestCase):
 
     # Pertinent only to PEP 302; exec_module() doesn't return a module.
 
-    __import__ = import_util.__import__[1]
+    __import__ = util.__import__[1]
 
     def create_mock(self, *names, return_=None):
         mock = util.mock_modules(*names)
diff --git a/Lib/test/test_importlib/import_/test_fromlist.py b/Lib/test/test_importlib/import_/test_fromlist.py
--- a/Lib/test/test_importlib/import_/test_fromlist.py
+++ b/Lib/test/test_importlib/import_/test_fromlist.py
@@ -1,6 +1,5 @@
 """Test that the semantics relating to the 'fromlist' argument are correct."""
 from .. import util
-from . import util as import_util
 import unittest
 
 
@@ -30,7 +29,7 @@
                 self.assertEqual(module.__name__, 'pkg.module')
 
 Frozen_ReturnValue, Source_ReturnValue = util.test_both(
-        ReturnValue, __import__=import_util.__import__)
+        ReturnValue, __import__=util.__import__)
 
 
 class HandlingFromlist:
@@ -122,7 +121,7 @@
                 self.assertEqual(module.module2.__name__, 'pkg.module2')
 
 Frozen_FromList, Source_FromList = util.test_both(
-        HandlingFromlist, __import__=import_util.__import__)
+        HandlingFromlist, __import__=util.__import__)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/import_/test_meta_path.py b/Lib/test/test_importlib/import_/test_meta_path.py
--- a/Lib/test/test_importlib/import_/test_meta_path.py
+++ b/Lib/test/test_importlib/import_/test_meta_path.py
@@ -1,5 +1,4 @@
 from .. import util
-from . import util as import_util
 import importlib._bootstrap
 import sys
 from types import MethodType
@@ -47,7 +46,7 @@
                 self.assertTrue(issubclass(w[-1].category, ImportWarning))
 
 Frozen_CallingOrder, Source_CallingOrder = util.test_both(
-        CallingOrder, __import__=import_util.__import__)
+        CallingOrder, __import__=util.__import__)
 
 
 class CallSignature:
@@ -105,14 +104,14 @@
     finder_name = 'find_module'
 
 Frozen_CallSignaturePEP302, Source_CallSignaturePEP302 = util.test_both(
-        CallSignaturePEP302, __import__=import_util.__import__)
+        CallSignaturePEP302, __import__=util.__import__)
 
 class CallSignaturePEP451(CallSignature):
     mock_modules = util.mock_spec
     finder_name = 'find_spec'
 
 Frozen_CallSignaturePEP451, Source_CallSignaturePEP451 = util.test_both(
-        CallSignaturePEP451, __import__=import_util.__import__)
+        CallSignaturePEP451, __import__=util.__import__)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/import_/test_packages.py b/Lib/test/test_importlib/import_/test_packages.py
--- a/Lib/test/test_importlib/import_/test_packages.py
+++ b/Lib/test/test_importlib/import_/test_packages.py
@@ -1,5 +1,4 @@
 from .. import util
-from . import util as import_util
 import sys
 import unittest
 import importlib
@@ -103,7 +102,7 @@
                     support.unload(subname)
 
 Frozen_ParentTests, Source_ParentTests = util.test_both(
-        ParentModuleTests, __import__=import_util.__import__)
+        ParentModuleTests, __import__=util.__import__)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/import_/test_path.py b/Lib/test/test_importlib/import_/test_path.py
--- a/Lib/test/test_importlib/import_/test_path.py
+++ b/Lib/test/test_importlib/import_/test_path.py
@@ -1,5 +1,4 @@
 from .. import util
-from . import util as import_util
 
 importlib = util.import_importlib('importlib')
 machinery = util.import_importlib('importlib.machinery')
@@ -58,7 +57,7 @@
         module = '<test module>'
         path = '<test path>'
         importer = util.mock_spec(module)
-        hook = import_util.mock_path_hook(path, importer=importer)
+        hook = util.mock_path_hook(path, importer=importer)
         with util.import_state(path_hooks=[hook]):
             loader = self.machinery.PathFinder.find_module(module, [path])
             self.assertIs(loader, importer)
@@ -83,7 +82,7 @@
         path = ''
         module = '<test module>'
         importer = util.mock_spec(module)
-        hook = import_util.mock_path_hook(os.getcwd(), importer=importer)
+        hook = util.mock_path_hook(os.getcwd(), importer=importer)
         with util.import_state(path=[path], path_hooks=[hook]):
             loader = self.machinery.PathFinder.find_module(module)
             self.assertIs(loader, importer)
diff --git a/Lib/test/test_importlib/import_/test_relative_imports.py b/Lib/test/test_importlib/import_/test_relative_imports.py
--- a/Lib/test/test_importlib/import_/test_relative_imports.py
+++ b/Lib/test/test_importlib/import_/test_relative_imports.py
@@ -1,6 +1,5 @@
 """Test relative imports (PEP 328)."""
 from .. import util
-from . import util as import_util
 import sys
 import unittest
 
@@ -209,7 +208,7 @@
             self.__import__('sys', level=1)
 
 Frozen_RelativeImports, Source_RelativeImports = util.test_both(
-        RelativeImports, __import__=import_util.__import__)
+        RelativeImports, __import__=util.__import__)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/import_/util.py b/Lib/test/test_importlib/import_/util.py
deleted file mode 100644
--- a/Lib/test/test_importlib/import_/util.py
+++ /dev/null
@@ -1,20 +0,0 @@
-from .. import util
-
-frozen_importlib, source_importlib = util.import_importlib('importlib')
-
-import builtins
-import functools
-import importlib
-import unittest
-
-
-__import__ = staticmethod(builtins.__import__), staticmethod(source_importlib.__import__)
-
-
-def mock_path_hook(*entries, importer):
-    """A mock sys.path_hooks entry."""
-    def hook(entry):
-        if entry not in entries:
-            raise ImportError
-        return importer
-    return hook
diff --git a/Lib/test/test_importlib/source/test_case_sensitivity.py b/Lib/test/test_importlib/source/test_case_sensitivity.py
--- a/Lib/test/test_importlib/source/test_case_sensitivity.py
+++ b/Lib/test/test_importlib/source/test_case_sensitivity.py
@@ -1,6 +1,5 @@
 """Test case-sensitivity (PEP 235)."""
 from .. import util
-from . import util as source_util
 
 importlib = util.import_importlib('importlib')
 machinery = util.import_importlib('importlib.machinery')
@@ -32,7 +31,7 @@
         """Look for a module with matching and non-matching sensitivity."""
         sensitive_pkg = 'sensitive.{0}'.format(self.name)
         insensitive_pkg = 'insensitive.{0}'.format(self.name.lower())
-        context = source_util.create_modules(insensitive_pkg, sensitive_pkg)
+        context = util.create_modules(insensitive_pkg, sensitive_pkg)
         with context as mapping:
             sensitive_path = os.path.join(mapping['.root'], 'sensitive')
             insensitive_path = os.path.join(mapping['.root'], 'insensitive')
diff --git a/Lib/test/test_importlib/source/test_file_loader.py b/Lib/test/test_importlib/source/test_file_loader.py
--- a/Lib/test/test_importlib/source/test_file_loader.py
+++ b/Lib/test/test_importlib/source/test_file_loader.py
@@ -1,6 +1,5 @@
 from .. import abc
 from .. import util
-from . import util as source_util
 
 importlib = util.import_importlib('importlib')
 importlib_abc = util.import_importlib('importlib.abc')
@@ -71,7 +70,7 @@
 
     # [basic]
     def test_module(self):
-        with source_util.create_modules('_temp') as mapping:
+        with util.create_modules('_temp') as mapping:
             loader = self.machinery.SourceFileLoader('_temp', mapping['_temp'])
             with warnings.catch_warnings():
                 warnings.simplefilter('ignore', DeprecationWarning)
@@ -83,7 +82,7 @@
                 self.assertEqual(getattr(module, attr), value)
 
     def test_package(self):
-        with source_util.create_modules('_pkg.__init__') as mapping:
+        with util.create_modules('_pkg.__init__') as mapping:
             loader = self.machinery.SourceFileLoader('_pkg',
                                                  mapping['_pkg.__init__'])
             with warnings.catch_warnings():
@@ -98,7 +97,7 @@
 
 
     def test_lacking_parent(self):
-        with source_util.create_modules('_pkg.__init__', '_pkg.mod')as mapping:
+        with util.create_modules('_pkg.__init__', '_pkg.mod')as mapping:
             loader = self.machinery.SourceFileLoader('_pkg.mod',
                                                     mapping['_pkg.mod'])
             with warnings.catch_warnings():
@@ -115,7 +114,7 @@
         return lambda name: fxn(name) + 1
 
     def test_module_reuse(self):
-        with source_util.create_modules('_temp') as mapping:
+        with util.create_modules('_temp') as mapping:
             loader = self.machinery.SourceFileLoader('_temp', mapping['_temp'])
             with warnings.catch_warnings():
                 warnings.simplefilter('ignore', DeprecationWarning)
@@ -139,7 +138,7 @@
         attributes = ('__file__', '__path__', '__package__')
         value = '<test>'
         name = '_temp'
-        with source_util.create_modules(name) as mapping:
+        with util.create_modules(name) as mapping:
             orig_module = types.ModuleType(name)
             for attr in attributes:
                 setattr(orig_module, attr, value)
@@ -159,7 +158,7 @@
 
     # [syntax error]
     def test_bad_syntax(self):
-        with source_util.create_modules('_temp') as mapping:
+        with util.create_modules('_temp') as mapping:
             with open(mapping['_temp'], 'w') as file:
                 file.write('=')
             loader = self.machinery.SourceFileLoader('_temp', mapping['_temp'])
@@ -190,11 +189,11 @@
             if os.path.exists(pycache):
                 shutil.rmtree(pycache)
 
-    @source_util.writes_bytecode_files
+    @util.writes_bytecode_files
     def test_timestamp_overflow(self):
         # When a modification timestamp is larger than 2**32, it should be
         # truncated rather than raise an OverflowError.
-        with source_util.create_modules('_temp') as mapping:
+        with util.create_modules('_temp') as mapping:
             source = mapping['_temp']
             compiled = self.util.cache_from_source(source)
             with open(source, 'w') as f:
@@ -275,45 +274,45 @@
         return bytecode_path
 
     def _test_empty_file(self, test, *, del_source=False):
-        with source_util.create_modules('_temp') as mapping:
+        with util.create_modules('_temp') as mapping:
             bc_path = self.manipulate_bytecode('_temp', mapping,
                                                 lambda bc: b'',
                                                 del_source=del_source)
             test('_temp', mapping, bc_path)
 
-    @source_util.writes_bytecode_files
+    @util.writes_bytecode_files
     def _test_partial_magic(self, test, *, del_source=False):
         # When their are less than 4 bytes to a .pyc, regenerate it if
         # possible, else raise ImportError.
-        with source_util.create_modules('_temp') as mapping:
+        with util.create_modules('_temp') as mapping:
             bc_path = self.manipulate_bytecode('_temp', mapping,
                                                 lambda bc: bc[:3],
                                                 del_source=del_source)
             test('_temp', mapping, bc_path)
 
     def _test_magic_only(self, test, *, del_source=False):
-        with source_util.create_modules('_temp') as mapping:
+        with util.create_modules('_temp') as mapping:
             bc_path = self.manipulate_bytecode('_temp', mapping,
                                                 lambda bc: bc[:4],
                                                 del_source=del_source)
             test('_temp', mapping, bc_path)
 
     def _test_partial_timestamp(self, test, *, del_source=False):
-        with source_util.create_modules('_temp') as mapping:
+        with util.create_modules('_temp') as mapping:
             bc_path = self.manipulate_bytecode('_temp', mapping,
                                                 lambda bc: bc[:7],
                                                 del_source=del_source)
             test('_temp', mapping, bc_path)
 
     def _test_partial_size(self, test, *, del_source=False):
-        with source_util.create_modules('_temp') as mapping:
+        with util.create_modules('_temp') as mapping:
             bc_path = self.manipulate_bytecode('_temp', mapping,
                                                 lambda bc: bc[:11],
                                                 del_source=del_source)
             test('_temp', mapping, bc_path)
 
     def _test_no_marshal(self, *, del_source=False):
-        with source_util.create_modules('_temp') as mapping:
+        with util.create_modules('_temp') as mapping:
             bc_path = self.manipulate_bytecode('_temp', mapping,
                                                 lambda bc: bc[:12],
                                                 del_source=del_source)
@@ -322,7 +321,7 @@
                 self.import_(file_path, '_temp')
 
     def _test_non_code_marshal(self, *, del_source=False):
-        with source_util.create_modules('_temp') as mapping:
+        with util.create_modules('_temp') as mapping:
             bytecode_path = self.manipulate_bytecode('_temp', mapping,
                                     lambda bc: bc[:12] + marshal.dumps(b'abcd'),
                                     del_source=del_source)
@@ -333,7 +332,7 @@
             self.assertEqual(cm.exception.path, bytecode_path)
 
     def _test_bad_marshal(self, *, del_source=False):
-        with source_util.create_modules('_temp') as mapping:
+        with util.create_modules('_temp') as mapping:
             bytecode_path = self.manipulate_bytecode('_temp', mapping,
                                                 lambda bc: bc[:12] + b'<test>',
                                                 del_source=del_source)
@@ -342,7 +341,7 @@
                 self.import_(file_path, '_temp')
 
     def _test_bad_magic(self, test, *, del_source=False):
-        with source_util.create_modules('_temp') as mapping:
+        with util.create_modules('_temp') as mapping:
             bc_path = self.manipulate_bytecode('_temp', mapping,
                                     lambda bc: b'\x00\x00\x00\x00' + bc[4:])
             test('_temp', mapping, bc_path)
@@ -371,7 +370,7 @@
     def setUpClass(cls):
         cls.loader = cls.machinery.SourceFileLoader
 
-    @source_util.writes_bytecode_files
+    @util.writes_bytecode_files
     def test_empty_file(self):
         # When a .pyc is empty, regenerate it if possible, else raise
         # ImportError.
@@ -390,7 +389,7 @@
 
         self._test_partial_magic(test)
 
-    @source_util.writes_bytecode_files
+    @util.writes_bytecode_files
     def test_magic_only(self):
         # When there is only the magic number, regenerate the .pyc if possible,
         # else raise EOFError.
@@ -401,7 +400,7 @@
 
         self._test_magic_only(test)
 
-    @source_util.writes_bytecode_files
+    @util.writes_bytecode_files
     def test_bad_magic(self):
         # When the magic number is different, the bytecode should be
         # regenerated.
@@ -413,7 +412,7 @@
 
         self._test_bad_magic(test)
 
-    @source_util.writes_bytecode_files
+    @util.writes_bytecode_files
     def test_partial_timestamp(self):
         # When the timestamp is partial, regenerate the .pyc, else
         # raise EOFError.
@@ -424,7 +423,7 @@
 
         self._test_partial_timestamp(test)
 
-    @source_util.writes_bytecode_files
+    @util.writes_bytecode_files
     def test_partial_size(self):
         # When the size is partial, regenerate the .pyc, else
         # raise EOFError.
@@ -435,29 +434,29 @@
 
         self._test_partial_size(test)
 
-    @source_util.writes_bytecode_files
+    @util.writes_bytecode_files
     def test_no_marshal(self):
         # When there is only the magic number and timestamp, raise EOFError.
         self._test_no_marshal()
 
-    @source_util.writes_bytecode_files
+    @util.writes_bytecode_files
     def test_non_code_marshal(self):
         self._test_non_code_marshal()
         # XXX ImportError when sourceless
 
     # [bad marshal]
-    @source_util.writes_bytecode_files
+    @util.writes_bytecode_files
     def test_bad_marshal(self):
         # Bad marshal data should raise a ValueError.
         self._test_bad_marshal()
 
     # [bad timestamp]
-    @source_util.writes_bytecode_files
+    @util.writes_bytecode_files
     def test_old_timestamp(self):
         # When the timestamp is older than the source, bytecode should be
         # regenerated.
         zeros = b'\x00\x00\x00\x00'
-        with source_util.create_modules('_temp') as mapping:
+        with util.create_modules('_temp') as mapping:
             py_compile.compile(mapping['_temp'])
             bytecode_path = self.util.cache_from_source(mapping['_temp'])
             with open(bytecode_path, 'r+b') as bytecode_file:
@@ -471,10 +470,10 @@
                 self.assertEqual(bytecode_file.read(4), source_timestamp)
 
     # [bytecode read-only]
-    @source_util.writes_bytecode_files
+    @util.writes_bytecode_files
     def test_read_only_bytecode(self):
         # When bytecode is read-only but should be rewritten, fail silently.
-        with source_util.create_modules('_temp') as mapping:
+        with util.create_modules('_temp') as mapping:
             # Create bytecode that will need to be re-created.
             py_compile.compile(mapping['_temp'])
             bytecode_path = self.util.cache_from_source(mapping['_temp'])
diff --git a/Lib/test/test_importlib/source/test_finder.py b/Lib/test/test_importlib/source/test_finder.py
--- a/Lib/test/test_importlib/source/test_finder.py
+++ b/Lib/test/test_importlib/source/test_finder.py
@@ -1,6 +1,5 @@
 from .. import abc
 from .. import util
-from . import util as source_util
 
 machinery = util.import_importlib('importlib.machinery')
 
@@ -60,7 +59,7 @@
         """
         if create is None:
             create = {test}
-        with source_util.create_modules(*create) as mapping:
+        with util.create_modules(*create) as mapping:
             if compile_:
                 for name in compile_:
                     py_compile.compile(mapping[name])
@@ -100,14 +99,14 @@
 
     # [sub module]
     def test_module_in_package(self):
-        with source_util.create_modules('pkg.__init__', 'pkg.sub') as mapping:
+        with util.create_modules('pkg.__init__', 'pkg.sub') as mapping:
             pkg_dir = os.path.dirname(mapping['pkg.__init__'])
             loader = self.import_(pkg_dir, 'pkg.sub')
             self.assertTrue(hasattr(loader, 'load_module'))
 
     # [sub package]
     def test_package_in_package(self):
-        context = source_util.create_modules('pkg.__init__', 'pkg.sub.__init__')
+        context = util.create_modules('pkg.__init__', 'pkg.sub.__init__')
         with context as mapping:
             pkg_dir = os.path.dirname(mapping['pkg.__init__'])
             loader = self.import_(pkg_dir, 'pkg.sub')
@@ -120,7 +119,7 @@
         self.assertIn('__init__', loader.get_filename(name))
 
     def test_failure(self):
-        with source_util.create_modules('blah') as mapping:
+        with util.create_modules('blah') as mapping:
             nothing = self.import_(mapping['.root'], 'sdfsadsadf')
             self.assertIsNone(nothing)
 
@@ -147,7 +146,7 @@
     # Regression test for http://bugs.python.org/issue14846
     def test_dir_removal_handling(self):
         mod = 'mod'
-        with source_util.create_modules(mod) as mapping:
+        with util.create_modules(mod) as mapping:
             finder = self.get_finder(mapping['.root'])
             found = self._find(finder, 'mod', loader_only=True)
             self.assertIsNotNone(found)
diff --git a/Lib/test/test_importlib/source/test_path_hook.py b/Lib/test/test_importlib/source/test_path_hook.py
--- a/Lib/test/test_importlib/source/test_path_hook.py
+++ b/Lib/test/test_importlib/source/test_path_hook.py
@@ -1,5 +1,4 @@
 from .. import util
-from . import util as source_util
 
 machinery = util.import_importlib('importlib.machinery')
 
@@ -15,7 +14,7 @@
             self.machinery.SOURCE_SUFFIXES))
 
     def test_success(self):
-        with source_util.create_modules('dummy') as mapping:
+        with util.create_modules('dummy') as mapping:
             self.assertTrue(hasattr(self.path_hook()(mapping['.root']),
                                  'find_module'))
 
diff --git a/Lib/test/test_importlib/source/test_source_encoding.py b/Lib/test/test_importlib/source/test_source_encoding.py
--- a/Lib/test/test_importlib/source/test_source_encoding.py
+++ b/Lib/test/test_importlib/source/test_source_encoding.py
@@ -1,5 +1,4 @@
 from .. import util
-from . import util as source_util
 
 machinery = util.import_importlib('importlib.machinery')
 
@@ -37,7 +36,7 @@
     module_name = '_temp'
 
     def run_test(self, source):
-        with source_util.create_modules(self.module_name) as mapping:
+        with util.create_modules(self.module_name) as mapping:
             with open(mapping[self.module_name], 'wb') as file:
                 file.write(source)
             loader = self.machinery.SourceFileLoader(self.module_name,
@@ -120,7 +119,7 @@
         module_name = '_temp'
         source_lines = [b"a = 42", b"b = -13", b'']
         source = line_ending.join(source_lines)
-        with source_util.create_modules(module_name) as mapping:
+        with util.create_modules(module_name) as mapping:
             with open(mapping[module_name], 'wb') as file:
                 file.write(source)
             loader = self.machinery.SourceFileLoader(module_name,
diff --git a/Lib/test/test_importlib/source/util.py b/Lib/test/test_importlib/source/util.py
deleted file mode 100644
--- a/Lib/test/test_importlib/source/util.py
+++ /dev/null
@@ -1,96 +0,0 @@
-from .. import util
-import contextlib
-import errno
-import functools
-import os
-import os.path
-import sys
-import tempfile
-from test import support
-
-
-def writes_bytecode_files(fxn):
-    """Decorator to protect sys.dont_write_bytecode from mutation and to skip
-    tests that require it to be set to False."""
-    if sys.dont_write_bytecode:
-        return lambda *args, **kwargs: None
-    @functools.wraps(fxn)
-    def wrapper(*args, **kwargs):
-        original = sys.dont_write_bytecode
-        sys.dont_write_bytecode = False
-        try:
-            to_return = fxn(*args, **kwargs)
-        finally:
-            sys.dont_write_bytecode = original
-        return to_return
-    return wrapper
-
-
-def ensure_bytecode_path(bytecode_path):
-    """Ensure that the __pycache__ directory for PEP 3147 pyc file exists.
-
-    :param bytecode_path: File system path to PEP 3147 pyc file.
-    """
-    try:
-        os.mkdir(os.path.dirname(bytecode_path))
-    except OSError as error:
-        if error.errno != errno.EEXIST:
-            raise
-
-
- at contextlib.contextmanager
-def create_modules(*names):
-    """Temporarily create each named module with an attribute (named 'attr')
-    that contains the name passed into the context manager that caused the
-    creation of the module.
-
-    All files are created in a temporary directory returned by
-    tempfile.mkdtemp(). This directory is inserted at the beginning of
-    sys.path. When the context manager exits all created files (source and
-    bytecode) are explicitly deleted.
-
-    No magic is performed when creating packages! This means that if you create
-    a module within a package you must also create the package's __init__ as
-    well.
-
-    """
-    source = 'attr = {0!r}'
-    created_paths = []
-    mapping = {}
-    state_manager = None
-    uncache_manager = None
-    try:
-        temp_dir = tempfile.mkdtemp()
-        mapping['.root'] = temp_dir
-        import_names = set()
-        for name in names:
-            if not name.endswith('__init__'):
-                import_name = name
-            else:
-                import_name = name[:-len('.__init__')]
-            import_names.add(import_name)
-            if import_name in sys.modules:
-                del sys.modules[import_name]
-            name_parts = name.split('.')
-            file_path = temp_dir
-            for directory in name_parts[:-1]:
-                file_path = os.path.join(file_path, directory)
-                if not os.path.exists(file_path):
-                    os.mkdir(file_path)
-                    created_paths.append(file_path)
-            file_path = os.path.join(file_path, name_parts[-1] + '.py')
-            with open(file_path, 'w') as file:
-                file.write(source.format(name))
-            created_paths.append(file_path)
-            mapping[name] = file_path
-        uncache_manager = util.uncache(*import_names)
-        uncache_manager.__enter__()
-        state_manager = util.import_state(path=[temp_dir])
-        state_manager.__enter__()
-        yield mapping
-    finally:
-        if state_manager is not None:
-            state_manager.__exit__(None, None, None)
-        if uncache_manager is not None:
-            uncache_manager.__exit__(None, None, None)
-        support.rmtree(temp_dir)
diff --git a/Lib/test/test_importlib/util.py b/Lib/test/test_importlib/util.py
--- a/Lib/test/test_importlib/util.py
+++ b/Lib/test/test_importlib/util.py
@@ -1,12 +1,49 @@
-from contextlib import contextmanager
-from importlib import util, invalidate_caches
+import builtins
+import contextlib
+import errno
+import functools
+import importlib
+from importlib import machinery, util, invalidate_caches
+import os
 import os.path
 from test import support
 import unittest
 import sys
+import tempfile
 import types
 
 
+BUILTINS = types.SimpleNamespace()
+BUILTINS.good_name = None
+BUILTINS.bad_name = None
+if 'errno' in sys.builtin_module_names:
+    BUILTINS.good_name = 'errno'
+if 'importlib' not in sys.builtin_module_names:
+    BUILTINS.bad_name = 'importlib'
+
+EXTENSIONS = types.SimpleNamespace()
+EXTENSIONS.path = None
+EXTENSIONS.ext = None
+EXTENSIONS.filename = None
+EXTENSIONS.file_path = None
+EXTENSIONS.name = '_testcapi'
+
+def _extension_details():
+    global EXTENSIONS
+    for path in sys.path:
+        for ext in machinery.EXTENSION_SUFFIXES:
+            filename = EXTENSIONS.name + ext
+            file_path = os.path.join(path, filename)
+            if os.path.exists(file_path):
+                EXTENSIONS.path = path
+                EXTENSIONS.ext = ext
+                EXTENSIONS.filename = filename
+                EXTENSIONS.file_path = file_path
+                return
+
+_extension_details()
+
+
 def import_importlib(module_name):
     """Import a module from importlib both w/ and w/o _frozen_importlib."""
     fresh = ('importlib',) if '.' in module_name else ()
@@ -38,6 +75,9 @@
     if not os.path.exists(changed_name):
         CASE_INSENSITIVE_FS = False
 
+_, source_importlib = import_importlib('importlib')
+__import__ = staticmethod(builtins.__import__), staticmethod(source_importlib.__import__)
+
 
 def case_insensitive_tests(test):
     """Class decorator that nullifies tests requiring a case-insensitive
@@ -53,7 +93,7 @@
     return '{}.{}'.format(parent, name), path
 
 
- at contextmanager
+ at contextlib.contextmanager
 def uncache(*names):
     """Uncache a module from sys.modules.
 
@@ -79,7 +119,7 @@
                 pass
 
 
- at contextmanager
+ at contextlib.contextmanager
 def temp_module(name, content='', *, pkg=False):
     conflicts = [n for n in sys.modules if n.partition('.')[0] == name]
     with support.temp_cwd(None) as cwd:
@@ -103,7 +143,7 @@
                 yield location
 
 
- at contextmanager
+ at contextlib.contextmanager
 def import_state(**kwargs):
     """Context manager to manage the various importers and stored state in the
     sys module.
@@ -198,6 +238,7 @@
                     raise
             return self.modules[fullname]
 
+
 class mock_spec(_ImporterMock):
 
     """Importer mock using PEP 451 APIs."""
@@ -223,3 +264,99 @@
             self.module_code[module.__spec__.name]()
         except KeyError:
             pass
+
+
+def writes_bytecode_files(fxn):
+    """Decorator to protect sys.dont_write_bytecode from mutation and to skip
+    tests that require it to be set to False."""
+    if sys.dont_write_bytecode:
+        return lambda *args, **kwargs: None
+    @functools.wraps(fxn)
+    def wrapper(*args, **kwargs):
+        original = sys.dont_write_bytecode
+        sys.dont_write_bytecode = False
+        try:
+            to_return = fxn(*args, **kwargs)
+        finally:
+            sys.dont_write_bytecode = original
+        return to_return
+    return wrapper
+
+
+def ensure_bytecode_path(bytecode_path):
+    """Ensure that the __pycache__ directory for PEP 3147 pyc file exists.
+
+    :param bytecode_path: File system path to PEP 3147 pyc file.
+    """
+    try:
+        os.mkdir(os.path.dirname(bytecode_path))
+    except OSError as error:
+        if error.errno != errno.EEXIST:
+            raise
+
+
+ at contextlib.contextmanager
+def create_modules(*names):
+    """Temporarily create each named module with an attribute (named 'attr')
+    that contains the name passed into the context manager that caused the
+    creation of the module.
+
+    All files are created in a temporary directory returned by
+    tempfile.mkdtemp(). This directory is inserted at the beginning of
+    sys.path. When the context manager exits all created files (source and
+    bytecode) are explicitly deleted.
+
+    No magic is performed when creating packages! This means that if you create
+    a module within a package you must also create the package's __init__ as
+    well.
+
+    """
+    source = 'attr = {0!r}'
+    created_paths = []
+    mapping = {}
+    state_manager = None
+    uncache_manager = None
+    try:
+        temp_dir = tempfile.mkdtemp()
+        mapping['.root'] = temp_dir
+        import_names = set()
+        for name in names:
+            if not name.endswith('__init__'):
+                import_name = name
+            else:
+                import_name = name[:-len('.__init__')]
+            import_names.add(import_name)
+            if import_name in sys.modules:
+                del sys.modules[import_name]
+            name_parts = name.split('.')
+            file_path = temp_dir
+            for directory in name_parts[:-1]:
+                file_path = os.path.join(file_path, directory)
+                if not os.path.exists(file_path):
+                    os.mkdir(file_path)
+                    created_paths.append(file_path)
+            file_path = os.path.join(file_path, name_parts[-1] + '.py')
+            with open(file_path, 'w') as file:
+                file.write(source.format(name))
+            created_paths.append(file_path)
+            mapping[name] = file_path
+        uncache_manager = uncache(*import_names)
+        uncache_manager.__enter__()
+        state_manager = import_state(path=[temp_dir])
+        state_manager.__enter__()
+        yield mapping
+    finally:
+        if state_manager is not None:
+            state_manager.__exit__(None, None, None)
+        if uncache_manager is not None:
+            uncache_manager.__exit__(None, None, None)
+        support.rmtree(temp_dir)
+
+
+def mock_path_hook(*entries, importer):
+    """A mock sys.path_hooks entry."""
+    def hook(entry):
+        if entry not in entries:
+            raise ImportError
+        return importer
+    return hook

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat May 10 02:58:22 2014
From: python-checkins at python.org (eli.bendersky)
Date: Sat, 10 May 2014 02:58:22 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319655=3A_Replace_?=
 =?utf-8?q?the_ASDL_parser_carried_with_CPython?=
Message-ID: <3gQVQQ3qSjz7Ljs@mail.python.org>

http://hg.python.org/cpython/rev/b769352e2922
changeset:   90610:b769352e2922
user:        Eli Bendersky <eliben at gmail.com>
date:        Fri May 09 17:58:22 2014 -0700
summary:
  Issue #19655: Replace the ASDL parser carried with CPython

The new parser does not rely on Spark (which is now removed from our repo),
uses modern 3.x idioms and is significantly smaller and simpler.

It generates exactly the same AST files (.h and .c), so in practice no builds
should be affected.

files:
  Misc/NEWS        |    5 +
  Parser/asdl.py   |  572 ++++++++++-------------
  Parser/asdl_c.py |   54 +-
  Parser/spark.py  |  849 -----------------------------------
  4 files changed, 286 insertions(+), 1194 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -57,6 +57,11 @@
 
 - Issue #19995: %c, %o, %x, and %X now raise TypeError on non-integer input.
 
+- Issue #19655: The ASDL parser - used by the build process to generate code for
+  managing the Python AST in C - was rewritten. The new parser is self contained
+  and does not require to carry long the spark.py parser-generator library;
+  spark.py was removed from the source base.
+
 - Issue #12546: Allow \x00 to be used as a fill character when using str, int,
   float, and complex __format__ methods.
 
diff --git a/Parser/asdl.py b/Parser/asdl.py
--- a/Parser/asdl.py
+++ b/Parser/asdl.py
@@ -1,255 +1,53 @@
-"""An implementation of the Zephyr Abstract Syntax Definition Language.
+#-------------------------------------------------------------------------------
+# Parser for ASDL [1] definition files. Reads in an ASDL description and parses
+# it into an AST that describes it.
+#
+# The EBNF we're parsing here: Figure 1 of the paper [1]. Extended to support
+# modules and attributes after a product. Words starting with Capital letters
+# are terminals. Literal tokens are in "double quotes". Others are
+# non-terminals. Id is either TokenId or ConstructorId.
+#
+# module        ::= "module" Id "{" [definitions] "}"
+# definitions   ::= { TypeId "=" type }
+# type          ::= product | sum
+# product       ::= fields ["attributes" fields]
+# fields        ::= "(" { field, "," } field ")"
+# field         ::= TypeId ["?" | "*"] [Id]
+# sum           ::= constructor { "|" constructor } ["attributes" fields]
+# constructor   ::= ConstructorId [fields]
+#
+# [1] "The Zephyr Abstract Syntax Description Language" by Wang, et. al. See
+#     http://asdl.sourceforge.net/
+#-------------------------------------------------------------------------------
+from collections import namedtuple
+import re
 
-See http://asdl.sourceforge.net/ and
-http://www.cs.princeton.edu/research/techreps/TR-554-97
+__all__ = [
+    'builtin_types', 'parse', 'AST', 'Module', 'Type', 'Constructor',
+    'Field', 'Sum', 'Product', 'VisitorBase', 'Check', 'check']
 
-Only supports top level module decl, not view.  I'm guessing that view
-is intended to support the browser and I'm not interested in the
-browser.
+# The following classes define nodes into which the ASDL description is parsed.
+# Note: this is a "meta-AST". ASDL files (such as Python.asdl) describe the AST
+# structure used by a programming language. But ASDL files themselves need to be
+# parsed. This module parses ASDL files and uses a simple AST to represent them.
+# See the EBNF at the top of the file to understand the logical connection
+# between the various node types.
 
-Changes for Python: Add support for module versions
-"""
+builtin_types = set(
+    ['identifier', 'string', 'bytes', 'int', 'object', 'singleton'])
 
-import os
-import sys
-import traceback
-
-import spark
-
-def output(*strings):
-    for s in strings:
-        sys.stdout.write(str(s) + "\n")
-
-
-class Token(object):
-    # spark seems to dispatch in the parser based on a token's
-    # type attribute
-    def __init__(self, type, lineno):
-        self.type = type
-        self.lineno = lineno
-
-    def __str__(self):
-        return self.type
-
+class AST:
     def __repr__(self):
-        return str(self)
-
-class Id(Token):
-    def __init__(self, value, lineno):
-        self.type = 'Id'
-        self.value = value
-        self.lineno = lineno
-
-    def __str__(self):
-        return self.value
-
-class String(Token):
-    def __init__(self, value, lineno):
-        self.type = 'String'
-        self.value = value
-        self.lineno = lineno
-
-class ASDLSyntaxError(Exception):
-
-    def __init__(self, lineno, token=None, msg=None):
-        self.lineno = lineno
-        self.token = token
-        self.msg = msg
-
-    def __str__(self):
-        if self.msg is None:
-            return "Error at '%s', line %d" % (self.token, self.lineno)
-        else:
-            return "%s, line %d" % (self.msg, self.lineno)
-
-class ASDLScanner(spark.GenericScanner, object):
-
-    def tokenize(self, input):
-        self.rv = []
-        self.lineno = 1
-        super(ASDLScanner, self).tokenize(input)
-        return self.rv
-
-    def t_id(self, s):
-        r"[\w\.]+"
-        # XXX doesn't distinguish upper vs. lower, which is
-        # significant for ASDL.
-        self.rv.append(Id(s, self.lineno))
-
-    def t_string(self, s):
-        r'"[^"]*"'
-        self.rv.append(String(s, self.lineno))
-
-    def t_xxx(self, s): # not sure what this production means
-        r"<="
-        self.rv.append(Token(s, self.lineno))
-
-    def t_punctuation(self, s):
-        r"[\{\}\*\=\|\(\)\,\?\:]"
-        self.rv.append(Token(s, self.lineno))
-
-    def t_comment(self, s):
-        r"\-\-[^\n]*"
-        pass
-
-    def t_newline(self, s):
-        r"\n"
-        self.lineno += 1
-
-    def t_whitespace(self, s):
-        r"[ \t]+"
-        pass
-
-    def t_default(self, s):
-        r" . +"
-        raise ValueError("unmatched input: %r" % s)
-
-class ASDLParser(spark.GenericParser, object):
-    def __init__(self):
-        super(ASDLParser, self).__init__("module")
-
-    def typestring(self, tok):
-        return tok.type
-
-    def error(self, tok):
-        raise ASDLSyntaxError(tok.lineno, tok)
-
-    def p_module_0(self, info):
-        " module ::= Id Id { } "
-        module, name, _0, _1 = info
-        if module.value != "module":
-            raise ASDLSyntaxError(module.lineno,
-                                  msg="expected 'module', found %s" % module)
-        return Module(name, None)
-
-    def p_module(self, info):
-        " module ::= Id Id { definitions } "
-        module, name, _0, definitions, _1 = info
-        if module.value != "module":
-            raise ASDLSyntaxError(module.lineno,
-                                  msg="expected 'module', found %s" % module)
-        return Module(name, definitions)
-
-    def p_definition_0(self, definition):
-        " definitions ::= definition "
-        return definition[0]
-
-    def p_definition_1(self, definitions):
-        " definitions ::= definition definitions "
-        return definitions[0] + definitions[1]
-
-    def p_definition(self, info):
-        " definition ::= Id = type "
-        id, _, type = info
-        return [Type(id, type)]
-
-    def p_type_0(self, product):
-        " type ::= product "
-        return product[0]
-
-    def p_type_1(self, sum):
-        " type ::= sum "
-        return Sum(sum[0])
-
-    def p_type_2(self, info):
-        " type ::= sum Id ( fields ) "
-        sum, id, _0, attributes, _1 = info
-        if id.value != "attributes":
-            raise ASDLSyntaxError(id.lineno,
-                                  msg="expected attributes, found %s" % id)
-        return Sum(sum, attributes)
-
-    def p_product_0(self, info):
-        " product ::= ( fields ) "
-        _0, fields, _1 = info
-        return Product(fields)
-
-    def p_product_1(self, info):
-        " product ::= ( fields ) Id ( fields ) "
-        _0, fields, _1, id, _2, attributes, _3 = info
-        if id.value != "attributes":
-            raise ASDLSyntaxError(id.lineno,
-                                  msg="expected attributes, found %s" % id)
-        return Product(fields, attributes)
-
-    def p_sum_0(self, constructor):
-        " sum ::= constructor "
-        return [constructor[0]]
-
-    def p_sum_1(self, info):
-        " sum ::= constructor | sum "
-        constructor, _, sum = info
-        return [constructor] + sum
-
-    def p_sum_2(self, info):
-        " sum ::= constructor | sum "
-        constructor, _, sum = info
-        return [constructor] + sum
-
-    def p_constructor_0(self, id):
-        " constructor ::= Id "
-        return Constructor(id[0])
-
-    def p_constructor_1(self, info):
-        " constructor ::= Id ( fields ) "
-        id, _0, fields, _1 = info
-        return Constructor(id, fields)
-
-    def p_fields_0(self, field):
-        " fields ::= field "
-        return [field[0]]
-
-    def p_fields_1(self, info):
-        " fields ::= fields , field "
-        fields, _, field = info
-        return fields + [field]
-
-    def p_field_0(self, type_):
-        " field ::= Id "
-        return Field(type_[0])
-
-    def p_field_1(self, info):
-        " field ::= Id Id "
-        type, name = info
-        return Field(type, name)
-
-    def p_field_2(self, info):
-        " field ::= Id * Id "
-        type, _, name = info
-        return Field(type, name, seq=True)
-
-    def p_field_3(self, info):
-        " field ::= Id ? Id "
-        type, _, name = info
-        return Field(type, name, opt=True)
-
-    def p_field_4(self, type_):
-        " field ::= Id * "
-        return Field(type_[0], seq=True)
-
-    def p_field_5(self, type_):
-        " field ::= Id ? "
-        return Field(type[0], opt=True)
-
-builtin_types = ("identifier", "string", "bytes", "int", "object", "singleton")
-
-# below is a collection of classes to capture the AST of an AST :-)
-# not sure if any of the methods are useful yet, but I'm adding them
-# piecemeal as they seem helpful
-
-class AST(object):
-    pass # a marker class
+        raise NotImplementedError
 
 class Module(AST):
     def __init__(self, name, dfns):
         self.name = name
         self.dfns = dfns
-        self.types = {} # maps type name to value (from dfns)
-        for type in dfns:
-            self.types[type.name.value] = type.value
+        self.types = {type.name: type.value for type in dfns}
 
     def __repr__(self):
-        return "Module(%s, %s)" % (self.name, self.dfns)
+        return 'Module({0.name}, {0.dfns})'.format(self)
 
 class Type(AST):
     def __init__(self, name, value):
@@ -257,7 +55,7 @@
         self.value = value
 
     def __repr__(self):
-        return "Type(%s, %s)" % (self.name, self.value)
+        return 'Type({0.name}, {0.value})'.format(self)
 
 class Constructor(AST):
     def __init__(self, name, fields=None):
@@ -265,7 +63,7 @@
         self.fields = fields or []
 
     def __repr__(self):
-        return "Constructor(%s, %s)" % (self.name, self.fields)
+        return 'Constructor({0.name}, {0.fields})'.format(self)
 
 class Field(AST):
     def __init__(self, type, name=None, seq=False, opt=False):
@@ -282,9 +80,9 @@
         else:
             extra = ""
         if self.name is None:
-            return "Field(%s%s)" % (self.type, extra)
+            return 'Field({0.type}{1})'.format(self, extra)
         else:
-            return "Field(%s, %s%s)" % (self.type, self.name, extra)
+            return 'Field({0.type}, {0.name}{1})'.format(self, extra)
 
 class Sum(AST):
     def __init__(self, types, attributes=None):
@@ -292,10 +90,10 @@
         self.attributes = attributes or []
 
     def __repr__(self):
-        if self.attributes is None:
-            return "Sum(%s)" % self.types
+        if self.attributes:
+            return 'Sum({0.types}, {0.attributes})'.format(self)
         else:
-            return "Sum(%s, %s)" % (self.types, self.attributes)
+            return 'Sum({0.types})'.format(self)
 
 class Product(AST):
     def __init__(self, fields, attributes=None):
@@ -303,49 +101,43 @@
         self.attributes = attributes or []
 
     def __repr__(self):
-        if self.attributes is None:
-            return "Product(%s)" % self.fields
+        if self.attributes:
+            return 'Product({0.fields}, {0.attributes})'.format(self)
         else:
-            return "Product(%s, %s)" % (self.fields, self.attributes)
+            return 'Product({0.fields})'.format(self)
 
-class VisitorBase(object):
+# A generic visitor for the meta-AST that describes ASDL. This can be used by
+# emitters. Note that this visitor does not provide a generic visit method, so a
+# subclass needs to define visit methods from visitModule to as deep as the
+# interesting node.
+# We also define a Check visitor that makes sure the parsed ASDL is well-formed.
 
-    def __init__(self, skip=False):
+class VisitorBase:
+    """Generic tree visitor for ASTs."""
+    def __init__(self):
         self.cache = {}
-        self.skip = skip
 
-    def visit(self, object, *args):
-        meth = self._dispatch(object)
-        if meth is None:
-            return
-        try:
-            meth(object, *args)
-        except Exception:
-            output("Error visiting" + repr(object))
-            output(str(sys.exc_info()[1]))
-            traceback.print_exc()
-            # XXX hack
-            if hasattr(self, 'file'):
-                self.file.flush()
-            os._exit(1)
-
-    def _dispatch(self, object):
-        assert isinstance(object, AST), repr(object)
-        klass = object.__class__
+    def visit(self, obj, *args):
+        klass = obj.__class__
         meth = self.cache.get(klass)
         if meth is None:
             methname = "visit" + klass.__name__
-            if self.skip:
-                meth = getattr(self, methname, None)
-            else:
-                meth = getattr(self, methname)
+            meth = getattr(self, methname, None)
             self.cache[klass] = meth
-        return meth
+        if meth:
+            try:
+                meth(obj, *args)
+            except Exception as e:
+                print("Error visiting %r: %s" % (obj, e))
+                raise
 
 class Check(VisitorBase):
+    """A visitor that checks a parsed ASDL tree for correctness.
 
+    Errors are printed and accumulated.
+    """
     def __init__(self):
-        super(Check, self).__init__(skip=True)
+        super().__init__()
         self.cons = {}
         self.errors = 0
         self.types = {}
@@ -367,8 +159,8 @@
         if conflict is None:
             self.cons[key] = name
         else:
-            output("Redefinition of constructor %s" % key)
-            output("Defined in %s and %s" % (conflict, name))
+            print('Redefinition of constructor {}'.format(key))
+            print('Defined in {} and {}'.format(conflict, name))
             self.errors += 1
         for f in cons.fields:
             self.visit(f, key)
@@ -383,6 +175,11 @@
             self.visit(f, name)
 
 def check(mod):
+    """Check the parsed ASDL tree for correctness.
+
+    Return True if success. For failure, the errors are printed out and False
+    is returned.
+    """
     v = Check()
     v.visit(mod)
 
@@ -390,47 +187,190 @@
         if t not in mod.types and not t in builtin_types:
             v.errors += 1
             uses = ", ".join(v.types[t])
-            output("Undefined type %s, used in %s" % (t, uses))
-
+            print('Undefined type {}, used in {}'.format(t, uses))
     return not v.errors
 
-def parse(file):
-    scanner = ASDLScanner()
-    parser = ASDLParser()
+# The ASDL parser itself comes next. The only interesting external interface
+# here is the top-level parse function.
 
-    f = open(file)
-    try:
-        buf = f.read()
-    finally:
-        f.close()
-    tokens = scanner.tokenize(buf)
-    try:
-        return parser.parse(tokens)
-    except ASDLSyntaxError:
-        err = sys.exc_info()[1]
-        output(str(err))
-        lines = buf.split("\n")
-        output(lines[err.lineno - 1]) # lines starts at 0, files at 1
+def parse(filename):
+    """Parse ASDL from the given file and return a Module node describing it."""
+    with open(filename) as f:
+        parser = ASDLParser()
+        return parser.parse(f.read())
 
-if __name__ == "__main__":
-    import glob
-    import sys
+# Types for describing tokens in an ASDL specification.
+class TokenKind:
+    """TokenKind is provides a scope for enumerated token kinds."""
+    (ConstructorId, TypeId, Equals, Comma, Question, Pipe, Asterisk,
+     LParen, RParen, LBrace, RBrace) = range(11)
 
-    if len(sys.argv) > 1:
-        files = sys.argv[1:]
-    else:
-        testdir = "tests"
-        files = glob.glob(testdir + "/*.asdl")
+    operator_table = {
+        '=': Equals, ',': Comma,    '?': Question, '|': Pipe,    '(': LParen,
+        ')': RParen, '*': Asterisk, '{': LBrace,   '}': RBrace}
 
-    for file in files:
-        output(file)
-        mod = parse(file)
-        if not mod:
-            break
-        output("module", mod.name)
-        output(len(mod.dfns), "definitions")
-        if not check(mod):
-            output("Check failed")
+Token = namedtuple('Token', 'kind value lineno')
+
+class ASDLSyntaxError(Exception):
+    def __init__(self, msg, lineno=None):
+        self.msg = msg
+        self.lineno = lineno or '<unknown>'
+
+    def __str__(self):
+        return 'Syntax error on line {0.lineno}: {0.msg}'.format(self)
+
+def tokenize_asdl(buf):
+    """Tokenize the given buffer. Yield Token objects."""
+    for lineno, line in enumerate(buf.splitlines(), 1):
+        for m in re.finditer(r'\s*(\w+|--.*|.)', line.strip()):
+            c = m.group(1)
+            if c[0].isalpha():
+                # Some kind of identifier
+                if c[0].isupper():
+                    yield Token(TokenKind.ConstructorId, c, lineno)
+                else:
+                    yield Token(TokenKind.TypeId, c, lineno)
+            elif c[:2] == '--':
+                # Comment
+                break
+            else:
+                # Operators
+                try:
+                    op_kind = TokenKind.operator_table[c]
+                except KeyError:
+                    raise ASDLSyntaxError('Invalid operator %s' % c, lineno)
+                yield Token(op_kind, c, lineno)
+
+class ASDLParser:
+    """Parser for ASDL files.
+
+    Create, then call the parse method on a buffer containing ASDL.
+    This is a simple recursive descent parser that uses tokenize_asdl for the
+    lexing.
+    """
+    def __init__(self):
+        self._tokenizer = None
+        self.cur_token = None
+
+    def parse(self, buf):
+        """Parse the ASDL in the buffer and return an AST with a Module root.
+        """
+        self._tokenizer = tokenize_asdl(buf)
+        self._advance()
+        return self._parse_module()
+
+    def _parse_module(self):
+        if self._at_keyword('module'):
+            self._advance()
         else:
-            for dfn in mod.dfns:
-                output(dfn.name, dfn.value)
+            raise ASDLSyntaxError(
+                'Expected "module" (found {})'.format(self.cur_token.value),
+                self.cur_token.lineno)
+        name = self._match(self._id_kinds)
+        self._match(TokenKind.LBrace)
+        defs = self._parse_definitions()
+        self._match(TokenKind.RBrace)
+        return Module(name, defs)
+
+    def _parse_definitions(self):
+        defs = []
+        while self.cur_token.kind == TokenKind.TypeId:
+            typename = self._advance()
+            self._match(TokenKind.Equals)
+            type = self._parse_type()
+            defs.append(Type(typename, type))
+        return defs
+
+    def _parse_type(self):
+        if self.cur_token.kind == TokenKind.LParen:
+            # If we see a (, it's a product
+            return self._parse_product()
+        else:
+            # Otherwise it's a sum. Look for ConstructorId
+            sumlist = [Constructor(self._match(TokenKind.ConstructorId),
+                                   self._parse_optional_fields())]
+            while self.cur_token.kind  == TokenKind.Pipe:
+                # More constructors
+                self._advance()
+                sumlist.append(Constructor(
+                                self._match(TokenKind.ConstructorId),
+                                self._parse_optional_fields()))
+            return Sum(sumlist, self._parse_optional_attributes())
+
+    def _parse_product(self):
+        return Product(self._parse_fields(), self._parse_optional_attributes())
+
+    def _parse_fields(self):
+        fields = []
+        self._match(TokenKind.LParen)
+        while self.cur_token.kind == TokenKind.TypeId:
+            typename = self._advance()
+            is_seq, is_opt = self._parse_optional_field_quantifier()
+            id = (self._advance() if self.cur_token.kind in self._id_kinds
+                                  else None)
+            fields.append(Field(typename, id, seq=is_seq, opt=is_opt))
+            if self.cur_token.kind == TokenKind.RParen:
+                break
+            elif self.cur_token.kind == TokenKind.Comma:
+                self._advance()
+        self._match(TokenKind.RParen)
+        return fields
+
+    def _parse_optional_fields(self):
+        if self.cur_token.kind == TokenKind.LParen:
+            return self._parse_fields()
+        else:
+            return None
+
+    def _parse_optional_attributes(self):
+        if self._at_keyword('attributes'):
+            self._advance()
+            return self._parse_fields()
+        else:
+            return None
+
+    def _parse_optional_field_quantifier(self):
+        is_seq, is_opt = False, False
+        if self.cur_token.kind == TokenKind.Asterisk:
+            is_seq = True
+            self._advance()
+        elif self.cur_token.kind == TokenKind.Question:
+            is_opt = True
+            self._advance()
+        return is_seq, is_opt
+
+    def _advance(self):
+        """ Return the value of the current token and read the next one into
+            self.cur_token.
+        """
+        cur_val = None if self.cur_token is None else self.cur_token.value
+        try:
+            self.cur_token = next(self._tokenizer)
+        except StopIteration:
+            self.cur_token = None
+        return cur_val
+
+    _id_kinds = (TokenKind.ConstructorId, TokenKind.TypeId)
+
+    def _match(self, kind):
+        """The 'match' primitive of RD parsers.
+
+        * Verifies that the current token is of the given kind (kind can
+          be a tuple, in which the kind must match one of its members).
+        * Returns the value of the current token
+        * Reads in the next token
+        """
+        if (isinstance(kind, tuple) and self.cur_token.kind in kind or
+            self.cur_token.kind == kind
+            ):
+            value = self.cur_token.value
+            self._advance()
+            return value
+        else:
+            raise ASDLSyntaxError(
+                'Unmatched {} (found {})'.format(kind, self.cur_token.kind),
+                self.cur_token.lineno)
+
+    def _at_keyword(self, keyword):
+        return (self.cur_token.kind == TokenKind.TypeId and
+                self.cur_token.value == keyword)
diff --git a/Parser/asdl_c.py b/Parser/asdl_c.py
--- a/Parser/asdl_c.py
+++ b/Parser/asdl_c.py
@@ -1,9 +1,6 @@
 #! /usr/bin/env python
 """Generate C code from an ASDL description."""
 
-# TO DO
-# handle fields that have a type but no name
-
 import os, sys
 
 import asdl
@@ -14,12 +11,8 @@
 def get_c_type(name):
     """Return a string for the C name of the type.
 
-    This function special cases the default types provided by asdl:
-    identifier, string, int.
+    This function special cases the default types provided by asdl.
     """
-    # XXX ack!  need to figure out where Id is useful and where string
-    if isinstance(name, asdl.Id):
-        name = name.value
     if name in asdl.builtin_types:
         return name
     else:
@@ -144,7 +137,7 @@
 
 
 class StructVisitor(EmitVisitor):
-    """Visitor to generate typdefs for AST."""
+    """Visitor to generate typedefs for AST."""
 
     def visitModule(self, mod):
         for dfn in mod.dfns:
@@ -188,9 +181,6 @@
                 self.visit(f, depth + 1)
             self.emit("} %s;" % cons.name, depth)
             self.emit("", depth)
-        else:
-            # XXX not sure what I want here, nothing is probably fine
-            pass
 
     def visitField(self, field, depth):
         # XXX need to lookup field.type, because it might be something
@@ -198,7 +188,7 @@
         ctype = get_c_type(field.type)
         name = field.name
         if field.seq:
-            if field.type.value in ('cmpop',):
+            if field.type == 'cmpop':
                 self.emit("asdl_int_seq *%(name)s;" % locals(), depth)
             else:
                 self.emit("asdl_seq *%(name)s;" % locals(), depth)
@@ -253,7 +243,7 @@
                 name = f.name
             # XXX should extend get_c_type() to handle this
             if f.seq:
-                if f.type.value in ('cmpop',):
+                if f.type == 'cmpop':
                     ctype = "asdl_int_seq *"
                 else:
                     ctype = "asdl_seq *"
@@ -437,7 +427,7 @@
             self.emit("", 0)
             for f in t.fields:
                 self.visitField(f, t.name, sum=sum, depth=2)
-            args = [f.name.value for f in t.fields] + [a.name.value for a in sum.attributes]
+            args = [f.name for f in t.fields] + [a.name for a in sum.attributes]
             self.emit("*out = %s(%s);" % (t.name, self.buildArgs(args)), 2)
             self.emit("if (*out == NULL) goto failed;", 2)
             self.emit("return 0;", 2)
@@ -465,7 +455,7 @@
         self.emit("", 0)
         for f in prod.fields:
             self.visitField(f, name, prod=prod, depth=1)
-        args = [f.name.value for f in prod.fields]
+        args = [f.name for f in prod.fields]
         self.emit("*out = %s(%s);" % (name, self.buildArgs(args)), 1)
         self.emit("return 0;", 1)
         self.emit("failed:", 0)
@@ -487,8 +477,8 @@
 
     def isSimpleSum(self, field):
         # XXX can the members of this list be determined automatically?
-        return field.type.value in ('expr_context', 'boolop', 'operator',
-                                    'unaryop', 'cmpop')
+        return field.type in ('expr_context', 'boolop', 'operator',
+                              'unaryop', 'cmpop')
 
     def isNumeric(self, field):
         return get_c_type(field.type) in ("int", "bool")
@@ -960,7 +950,7 @@
 
     def visitProduct(self, prod, name):
         if prod.fields:
-            fields = name.value+"_fields"
+            fields = name+"_fields"
         else:
             fields = "NULL"
         self.emit('%s_type = make_type("%s", &AST_type, %s, %d);' %
@@ -987,7 +977,7 @@
 
     def visitConstructor(self, cons, name, simple):
         if cons.fields:
-            fields = cons.name.value+"_fields"
+            fields = cons.name+"_fields"
         else:
             fields = "NULL"
         self.emit('%s_type = make_type("%s", %s_type, %s, %d);' %
@@ -1170,7 +1160,7 @@
     def set(self, field, value, depth):
         if field.seq:
             # XXX should really check for is_simple, but that requires a symbol table
-            if field.type.value == "cmpop":
+            if field.type == "cmpop":
                 # While the sequence elements are stored as void*,
                 # ast2obj_cmpop expects an enum
                 self.emit("{", depth)
@@ -1249,12 +1239,15 @@
 
 common_msg = "/* File automatically generated by %s. */\n\n"
 
-def main(srcfile):
+def main(srcfile, dump_module=False):
     argv0 = sys.argv[0]
     components = argv0.split(os.sep)
     argv0 = os.sep.join(components[-2:])
     auto_gen_msg = common_msg % argv0
     mod = asdl.parse(srcfile)
+    if dump_module:
+        print('Parsed Module:')
+        print(mod)
     if not asdl.check(mod):
         sys.exit(1)
     if INC_DIR:
@@ -1301,16 +1294,19 @@
 
     INC_DIR = ''
     SRC_DIR = ''
-    opts, args = getopt.getopt(sys.argv[1:], "h:c:")
-    if len(opts) != 1:
-        sys.stdout.write("Must specify exactly one output file\n")
-        sys.exit(1)
+    dump_module = False
+    opts, args = getopt.getopt(sys.argv[1:], "dh:c:")
     for o, v in opts:
         if o == '-h':
             INC_DIR = v
         if o == '-c':
             SRC_DIR = v
-    if len(args) != 1:
-        sys.stdout.write("Must specify single input file\n")
+        if o == '-d':
+            dump_module = True
+    if INC_DIR and SRC_DIR:
+        print('Must specify exactly one output file')
         sys.exit(1)
-    main(args[0])
+    elif len(args) != 1:
+        print('Must specify single input file')
+        sys.exit(1)
+    main(args[0], dump_module)
diff --git a/Parser/spark.py b/Parser/spark.py
deleted file mode 100644
--- a/Parser/spark.py
+++ /dev/null
@@ -1,849 +0,0 @@
-#  Copyright (c) 1998-2002 John Aycock
-#
-#  Permission is hereby granted, free of charge, to any person obtaining
-#  a copy of this software and associated documentation files (the
-#  "Software"), to deal in the Software without restriction, including
-#  without limitation the rights to use, copy, modify, merge, publish,
-#  distribute, sublicense, and/or sell copies of the Software, and to
-#  permit persons to whom the Software is furnished to do so, subject to
-#  the following conditions:
-#
-#  The above copyright notice and this permission notice shall be
-#  included in all copies or substantial portions of the Software.
-#
-#  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-#  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-#  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
-#  IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
-#  CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
-#  TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
-#  SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
-__version__ = 'SPARK-0.7 (pre-alpha-5)'
-
-import re
-
-# Compatibility with older pythons.
-def output(string='', end='\n'):
-    sys.stdout.write(string + end)
-
-try:
-    sorted
-except NameError:
-    def sorted(seq):
-        seq2 = seq[:]
-        seq2.sort()
-        return seq2
-
-def _namelist(instance):
-    namelist, namedict, classlist = [], {}, [instance.__class__]
-    for c in classlist:
-        for b in c.__bases__:
-            classlist.append(b)
-        for name in c.__dict__.keys():
-            if name not in namedict:
-                namelist.append(name)
-                namedict[name] = 1
-    return namelist
-
-class GenericScanner:
-    def __init__(self, flags=0):
-        pattern = self.reflect()
-        self.re = re.compile(pattern, re.VERBOSE|flags)
-
-        self.index2func = {}
-        for name, number in self.re.groupindex.items():
-            self.index2func[number-1] = getattr(self, 't_' + name)
-
-    def makeRE(self, name):
-        doc = getattr(self, name).__doc__
-        rv = '(?P<%s>%s)' % (name[2:], doc)
-        return rv
-
-    def reflect(self):
-        rv = []
-        for name in _namelist(self):
-            if name[:2] == 't_' and name != 't_default':
-                rv.append(self.makeRE(name))
-
-        rv.append(self.makeRE('t_default'))
-        return '|'.join(rv)
-
-    def error(self, s, pos):
-        output("Lexical error at position %s" % pos)
-        raise SystemExit
-
-    def tokenize(self, s):
-        pos = 0
-        n = len(s)
-        while pos < n:
-            m = self.re.match(s, pos)
-            if m is None:
-                self.error(s, pos)
-
-            groups = m.groups()
-            for i in range(len(groups)):
-                if groups[i] and i in self.index2func:
-                    self.index2func[i](groups[i])
-            pos = m.end()
-
-    def t_default(self, s):
-        r'( . | \n )+'
-        output("Specification error: unmatched input")
-        raise SystemExit
-
-#
-#  Extracted from GenericParser and made global so that [un]picking works.
-#
-class _State:
-    def __init__(self, stateno, items):
-        self.T, self.complete, self.items = [], [], items
-        self.stateno = stateno
-
-class GenericParser:
-    #
-    #  An Earley parser, as per J. Earley, "An Efficient Context-Free
-    #  Parsing Algorithm", CACM 13(2), pp. 94-102.  Also J. C. Earley,
-    #  "An Efficient Context-Free Parsing Algorithm", Ph.D. thesis,
-    #  Carnegie-Mellon University, August 1968.  New formulation of
-    #  the parser according to J. Aycock, "Practical Earley Parsing
-    #  and the SPARK Toolkit", Ph.D. thesis, University of Victoria,
-    #  2001, and J. Aycock and R. N. Horspool, "Practical Earley
-    #  Parsing", unpublished paper, 2001.
-    #
-
-    def __init__(self, start):
-        self.rules = {}
-        self.rule2func = {}
-        self.rule2name = {}
-        self.collectRules()
-        self.augment(start)
-        self.ruleschanged = 1
-
-    _NULLABLE = '\e_'
-    _START = 'START'
-    _BOF = '|-'
-
-    #
-    #  When pickling, take the time to generate the full state machine;
-    #  some information is then extraneous, too.  Unfortunately we
-    #  can't save the rule2func map.
-    #
-    def __getstate__(self):
-        if self.ruleschanged:
-            #
-            #  XXX - duplicated from parse()
-            #
-            self.computeNull()
-            self.newrules = {}
-            self.new2old = {}
-            self.makeNewRules()
-            self.ruleschanged = 0
-            self.edges, self.cores = {}, {}
-            self.states = { 0: self.makeState0() }
-            self.makeState(0, self._BOF)
-        #
-        #  XXX - should find a better way to do this..
-        #
-        changes = 1
-        while changes:
-            changes = 0
-            for k, v in self.edges.items():
-                if v is None:
-                    state, sym = k
-                    if state in self.states:
-                        self.goto(state, sym)
-                        changes = 1
-        rv = self.__dict__.copy()
-        for s in self.states.values():
-            del s.items
-        del rv['rule2func']
-        del rv['nullable']
-        del rv['cores']
-        return rv
-
-    def __setstate__(self, D):
-        self.rules = {}
-        self.rule2func = {}
-        self.rule2name = {}
-        self.collectRules()
-        start = D['rules'][self._START][0][1][1]        # Blech.
-        self.augment(start)
-        D['rule2func'] = self.rule2func
-        D['makeSet'] = self.makeSet_fast
-        self.__dict__ = D
-
-    #
-    #  A hook for GenericASTBuilder and GenericASTMatcher.  Mess
-    #  thee not with this; nor shall thee toucheth the _preprocess
-    #  argument to addRule.
-    #
-    def preprocess(self, rule, func):       return rule, func
-
-    def addRule(self, doc, func, _preprocess=1):
-        fn = func
-        rules = doc.split()
-
-        index = []
-        for i in range(len(rules)):
-            if rules[i] == '::=':
-                index.append(i-1)
-        index.append(len(rules))
-
-        for i in range(len(index)-1):
-            lhs = rules[index[i]]
-            rhs = rules[index[i]+2:index[i+1]]
-            rule = (lhs, tuple(rhs))
-
-            if _preprocess:
-                rule, fn = self.preprocess(rule, func)
-
-            if lhs in self.rules:
-                self.rules[lhs].append(rule)
-            else:
-                self.rules[lhs] = [ rule ]
-            self.rule2func[rule] = fn
-            self.rule2name[rule] = func.__name__[2:]
-        self.ruleschanged = 1
-
-    def collectRules(self):
-        for name in _namelist(self):
-            if name[:2] == 'p_':
-                func = getattr(self, name)
-                doc = func.__doc__
-                self.addRule(doc, func)
-
-    def augment(self, start):
-        rule = '%s ::= %s %s' % (self._START, self._BOF, start)
-        self.addRule(rule, lambda args: args[1], 0)
-
-    def computeNull(self):
-        self.nullable = {}
-        tbd = []
-
-        for rulelist in self.rules.values():
-            lhs = rulelist[0][0]
-            self.nullable[lhs] = 0
-            for rule in rulelist:
-                rhs = rule[1]
-                if len(rhs) == 0:
-                    self.nullable[lhs] = 1
-                    continue
-                #
-                #  We only need to consider rules which
-                #  consist entirely of nonterminal symbols.
-                #  This should be a savings on typical
-                #  grammars.
-                #
-                for sym in rhs:
-                    if sym not in self.rules:
-                        break
-                else:
-                    tbd.append(rule)
-        changes = 1
-        while changes:
-            changes = 0
-            for lhs, rhs in tbd:
-                if self.nullable[lhs]:
-                    continue
-                for sym in rhs:
-                    if not self.nullable[sym]:
-                        break
-                else:
-                    self.nullable[lhs] = 1
-                    changes = 1
-
-    def makeState0(self):
-        s0 = _State(0, [])
-        for rule in self.newrules[self._START]:
-            s0.items.append((rule, 0))
-        return s0
-
-    def finalState(self, tokens):
-        #
-        #  Yuck.
-        #
-        if len(self.newrules[self._START]) == 2 and len(tokens) == 0:
-            return 1
-        start = self.rules[self._START][0][1][1]
-        return self.goto(1, start)
-
-    def makeNewRules(self):
-        worklist = []
-        for rulelist in self.rules.values():
-            for rule in rulelist:
-                worklist.append((rule, 0, 1, rule))
-
-        for rule, i, candidate, oldrule in worklist:
-            lhs, rhs = rule
-            n = len(rhs)
-            while i < n:
-                sym = rhs[i]
-                if sym not in self.rules or \
-                   not self.nullable[sym]:
-                    candidate = 0
-                    i = i + 1
-                    continue
-
-                newrhs = list(rhs)
-                newrhs[i] = self._NULLABLE+sym
-                newrule = (lhs, tuple(newrhs))
-                worklist.append((newrule, i+1,
-                                 candidate, oldrule))
-                candidate = 0
-                i = i + 1
-            else:
-                if candidate:
-                    lhs = self._NULLABLE+lhs
-                    rule = (lhs, rhs)
-                if lhs in self.newrules:
-                    self.newrules[lhs].append(rule)
-                else:
-                    self.newrules[lhs] = [ rule ]
-                self.new2old[rule] = oldrule
-
-    def typestring(self, token):
-        return None
-
-    def error(self, token):
-        output("Syntax error at or near `%s' token" % token)
-        raise SystemExit
-
-    def parse(self, tokens):
-        sets = [ [(1,0), (2,0)] ]
-        self.links = {}
-
-        if self.ruleschanged:
-            self.computeNull()
-            self.newrules = {}
-            self.new2old = {}
-            self.makeNewRules()
-            self.ruleschanged = 0
-            self.edges, self.cores = {}, {}
-            self.states = { 0: self.makeState0() }
-            self.makeState(0, self._BOF)
-
-        for i in range(len(tokens)):
-            sets.append([])
-
-            if sets[i] == []:
-                break
-            self.makeSet(tokens[i], sets, i)
-        else:
-            sets.append([])
-            self.makeSet(None, sets, len(tokens))
-
-        #_dump(tokens, sets, self.states)
-
-        finalitem = (self.finalState(tokens), 0)
-        if finalitem not in sets[-2]:
-            if len(tokens) > 0:
-                self.error(tokens[i-1])
-            else:
-                self.error(None)
-
-        return self.buildTree(self._START, finalitem,
-                              tokens, len(sets)-2)
-
-    def isnullable(self, sym):
-        #
-        #  For symbols in G_e only.  If we weren't supporting 1.5,
-        #  could just use sym.startswith().
-        #
-        return self._NULLABLE == sym[0:len(self._NULLABLE)]
-
-    def skip(self, hs, pos=0):
-        n = len(hs[1])
-        while pos < n:
-            if not self.isnullable(hs[1][pos]):
-                break
-            pos = pos + 1
-        return pos
-
-    def makeState(self, state, sym):
-        assert sym is not None
-        #
-        #  Compute \epsilon-kernel state's core and see if
-        #  it exists already.
-        #
-        kitems = []
-        for rule, pos in self.states[state].items:
-            lhs, rhs = rule
-            if rhs[pos:pos+1] == (sym,):
-                kitems.append((rule, self.skip(rule, pos+1)))
-        core = kitems
-
-        core.sort()
-        tcore = tuple(core)
-        if tcore in self.cores:
-            return self.cores[tcore]
-        #
-        #  Nope, doesn't exist.  Compute it and the associated
-        #  \epsilon-nonkernel state together; we'll need it right away.
-        #
-        k = self.cores[tcore] = len(self.states)
-        K, NK = _State(k, kitems), _State(k+1, [])
-        self.states[k] = K
-        predicted = {}
-
-        edges = self.edges
-        rules = self.newrules
-        for X in K, NK:
-            worklist = X.items
-            for item in worklist:
-                rule, pos = item
-                lhs, rhs = rule
-                if pos == len(rhs):
-                    X.complete.append(rule)
-                    continue
-
-                nextSym = rhs[pos]
-                key = (X.stateno, nextSym)
-                if nextSym not in rules:
-                    if key not in edges:
-                        edges[key] = None
-                        X.T.append(nextSym)
-                else:
-                    edges[key] = None
-                    if nextSym not in predicted:
-                        predicted[nextSym] = 1
-                        for prule in rules[nextSym]:
-                            ppos = self.skip(prule)
-                            new = (prule, ppos)
-                            NK.items.append(new)
-            #
-            #  Problem: we know K needs generating, but we
-            #  don't yet know about NK.  Can't commit anything
-            #  regarding NK to self.edges until we're sure.  Should
-            #  we delay committing on both K and NK to avoid this
-            #  hacky code?  This creates other problems..
-            #
-            if X is K:
-                edges = {}
-
-        if NK.items == []:
-            return k
-
-        #
-        #  Check for \epsilon-nonkernel's core.  Unfortunately we
-        #  need to know the entire set of predicted nonterminals
-        #  to do this without accidentally duplicating states.
-        #
-        core = sorted(predicted.keys())
-        tcore = tuple(core)
-        if tcore in self.cores:
-            self.edges[(k, None)] = self.cores[tcore]
-            return k
-
-        nk = self.cores[tcore] = self.edges[(k, None)] = NK.stateno
-        self.edges.update(edges)
-        self.states[nk] = NK
-        return k
-
-    def goto(self, state, sym):
-        key = (state, sym)
-        if key not in self.edges:
-            #
-            #  No transitions from state on sym.
-            #
-            return None
-
-        rv = self.edges[key]
-        if rv is None:
-            #
-            #  Target state isn't generated yet.  Remedy this.
-            #
-            rv = self.makeState(state, sym)
-            self.edges[key] = rv
-        return rv
-
-    def gotoT(self, state, t):
-        return [self.goto(state, t)]
-
-    def gotoST(self, state, st):
-        rv = []
-        for t in self.states[state].T:
-            if st == t:
-                rv.append(self.goto(state, t))
-        return rv
-
-    def add(self, set, item, i=None, predecessor=None, causal=None):
-        if predecessor is None:
-            if item not in set:
-                set.append(item)
-        else:
-            key = (item, i)
-            if item not in set:
-                self.links[key] = []
-                set.append(item)
-            self.links[key].append((predecessor, causal))
-
-    def makeSet(self, token, sets, i):
-        cur, next = sets[i], sets[i+1]
-
-        ttype = token is not None and self.typestring(token) or None
-        if ttype is not None:
-            fn, arg = self.gotoT, ttype
-        else:
-            fn, arg = self.gotoST, token
-
-        for item in cur:
-            ptr = (item, i)
-            state, parent = item
-            add = fn(state, arg)
-            for k in add:
-                if k is not None:
-                    self.add(next, (k, parent), i+1, ptr)
-                    nk = self.goto(k, None)
-                    if nk is not None:
-                        self.add(next, (nk, i+1))
-
-            if parent == i:
-                continue
-
-            for rule in self.states[state].complete:
-                lhs, rhs = rule
-                for pitem in sets[parent]:
-                    pstate, pparent = pitem
-                    k = self.goto(pstate, lhs)
-                    if k is not None:
-                        why = (item, i, rule)
-                        pptr = (pitem, parent)
-                        self.add(cur, (k, pparent),
-                                 i, pptr, why)
-                        nk = self.goto(k, None)
-                        if nk is not None:
-                            self.add(cur, (nk, i))
-
-    def makeSet_fast(self, token, sets, i):
-        #
-        #  Call *only* when the entire state machine has been built!
-        #  It relies on self.edges being filled in completely, and
-        #  then duplicates and inlines code to boost speed at the
-        #  cost of extreme ugliness.
-        #
-        cur, next = sets[i], sets[i+1]
-        ttype = token is not None and self.typestring(token) or None
-
-        for item in cur:
-            ptr = (item, i)
-            state, parent = item
-            if ttype is not None:
-                k = self.edges.get((state, ttype), None)
-                if k is not None:
-                    #self.add(next, (k, parent), i+1, ptr)
-                    #INLINED --v
-                    new = (k, parent)
-                    key = (new, i+1)
-                    if new not in next:
-                        self.links[key] = []
-                        next.append(new)
-                    self.links[key].append((ptr, None))
-                    #INLINED --^
-                    #nk = self.goto(k, None)
-                    nk = self.edges.get((k, None), None)
-                    if nk is not None:
-                        #self.add(next, (nk, i+1))
-                        #INLINED --v
-                        new = (nk, i+1)
-                        if new not in next:
-                            next.append(new)
-                        #INLINED --^
-            else:
-                add = self.gotoST(state, token)
-                for k in add:
-                    if k is not None:
-                        self.add(next, (k, parent), i+1, ptr)
-                        #nk = self.goto(k, None)
-                        nk = self.edges.get((k, None), None)
-                        if nk is not None:
-                            self.add(next, (nk, i+1))
-
-            if parent == i:
-                continue
-
-            for rule in self.states[state].complete:
-                lhs, rhs = rule
-                for pitem in sets[parent]:
-                    pstate, pparent = pitem
-                    #k = self.goto(pstate, lhs)
-                    k = self.edges.get((pstate, lhs), None)
-                    if k is not None:
-                        why = (item, i, rule)
-                        pptr = (pitem, parent)
-                        #self.add(cur, (k, pparent),
-                        #        i, pptr, why)
-                        #INLINED --v
-                        new = (k, pparent)
-                        key = (new, i)
-                        if new not in cur:
-                            self.links[key] = []
-                            cur.append(new)
-                        self.links[key].append((pptr, why))
-                        #INLINED --^
-                        #nk = self.goto(k, None)
-                        nk = self.edges.get((k, None), None)
-                        if nk is not None:
-                            #self.add(cur, (nk, i))
-                            #INLINED --v
-                            new = (nk, i)
-                            if new not in cur:
-                                cur.append(new)
-                            #INLINED --^
-
-    def predecessor(self, key, causal):
-        for p, c in self.links[key]:
-            if c == causal:
-                return p
-        assert 0
-
-    def causal(self, key):
-        links = self.links[key]
-        if len(links) == 1:
-            return links[0][1]
-        choices = []
-        rule2cause = {}
-        for p, c in links:
-            rule = c[2]
-            choices.append(rule)
-            rule2cause[rule] = c
-        return rule2cause[self.ambiguity(choices)]
-
-    def deriveEpsilon(self, nt):
-        if len(self.newrules[nt]) > 1:
-            rule = self.ambiguity(self.newrules[nt])
-        else:
-            rule = self.newrules[nt][0]
-        #output(rule)
-
-        rhs = rule[1]
-        attr = [None] * len(rhs)
-
-        for i in range(len(rhs)-1, -1, -1):
-            attr[i] = self.deriveEpsilon(rhs[i])
-        return self.rule2func[self.new2old[rule]](attr)
-
-    def buildTree(self, nt, item, tokens, k):
-        state, parent = item
-
-        choices = []
-        for rule in self.states[state].complete:
-            if rule[0] == nt:
-                choices.append(rule)
-        rule = choices[0]
-        if len(choices) > 1:
-            rule = self.ambiguity(choices)
-        #output(rule)
-
-        rhs = rule[1]
-        attr = [None] * len(rhs)
-
-        for i in range(len(rhs)-1, -1, -1):
-            sym = rhs[i]
-            if sym not in self.newrules:
-                if sym != self._BOF:
-                    attr[i] = tokens[k-1]
-                    key = (item, k)
-                    item, k = self.predecessor(key, None)
-            #elif self.isnullable(sym):
-            elif self._NULLABLE == sym[0:len(self._NULLABLE)]:
-                attr[i] = self.deriveEpsilon(sym)
-            else:
-                key = (item, k)
-                why = self.causal(key)
-                attr[i] = self.buildTree(sym, why[0],
-                                         tokens, why[1])
-                item, k = self.predecessor(key, why)
-        return self.rule2func[self.new2old[rule]](attr)
-
-    def ambiguity(self, rules):
-        #
-        #  XXX - problem here and in collectRules() if the same rule
-        #        appears in >1 method.  Also undefined results if rules
-        #        causing the ambiguity appear in the same method.
-        #
-        sortlist = []
-        name2index = {}
-        for i in range(len(rules)):
-            lhs, rhs = rule = rules[i]
-            name = self.rule2name[self.new2old[rule]]
-            sortlist.append((len(rhs), name))
-            name2index[name] = i
-        sortlist.sort()
-        list = [b for a, b in sortlist]
-        return rules[name2index[self.resolve(list)]]
-
-    def resolve(self, list):
-        #
-        #  Resolve ambiguity in favor of the shortest RHS.
-        #  Since we walk the tree from the top down, this
-        #  should effectively resolve in favor of a "shift".
-        #
-        return list[0]
-
-#
-#  GenericASTBuilder automagically constructs a concrete/abstract syntax tree
-#  for a given input.  The extra argument is a class (not an instance!)
-#  which supports the "__setslice__" and "__len__" methods.
-#
-#  XXX - silently overrides any user code in methods.
-#
-
-class GenericASTBuilder(GenericParser):
-    def __init__(self, AST, start):
-        GenericParser.__init__(self, start)
-        self.AST = AST
-
-    def preprocess(self, rule, func):
-        rebind = lambda lhs, self=self: \
-                        lambda args, lhs=lhs, self=self: \
-                                self.buildASTNode(args, lhs)
-        lhs, rhs = rule
-        return rule, rebind(lhs)
-
-    def buildASTNode(self, args, lhs):
-        children = []
-        for arg in args:
-            if isinstance(arg, self.AST):
-                children.append(arg)
-            else:
-                children.append(self.terminal(arg))
-        return self.nonterminal(lhs, children)
-
-    def terminal(self, token):      return token
-
-    def nonterminal(self, type, args):
-        rv = self.AST(type)
-        rv[:len(args)] = args
-        return rv
-
-#
-#  GenericASTTraversal is a Visitor pattern according to Design Patterns.  For
-#  each node it attempts to invoke the method n_<node type>, falling
-#  back onto the default() method if the n_* can't be found.  The preorder
-#  traversal also looks for an exit hook named n_<node type>_exit (no default
-#  routine is called if it's not found).  To prematurely halt traversal
-#  of a subtree, call the prune() method -- this only makes sense for a
-#  preorder traversal.  Node type is determined via the typestring() method.
-#
-
-class GenericASTTraversalPruningException:
-    pass
-
-class GenericASTTraversal:
-    def __init__(self, ast):
-        self.ast = ast
-
-    def typestring(self, node):
-        return node.type
-
-    def prune(self):
-        raise GenericASTTraversalPruningException
-
-    def preorder(self, node=None):
-        if node is None:
-            node = self.ast
-
-        try:
-            name = 'n_' + self.typestring(node)
-            if hasattr(self, name):
-                func = getattr(self, name)
-                func(node)
-            else:
-                self.default(node)
-        except GenericASTTraversalPruningException:
-            return
-
-        for kid in node:
-            self.preorder(kid)
-
-        name = name + '_exit'
-        if hasattr(self, name):
-            func = getattr(self, name)
-            func(node)
-
-    def postorder(self, node=None):
-        if node is None:
-            node = self.ast
-
-        for kid in node:
-            self.postorder(kid)
-
-        name = 'n_' + self.typestring(node)
-        if hasattr(self, name):
-            func = getattr(self, name)
-            func(node)
-        else:
-            self.default(node)
-
-
-    def default(self, node):
-        pass
-
-#
-#  GenericASTMatcher.  AST nodes must have "__getitem__" and "__cmp__"
-#  implemented.
-#
-#  XXX - makes assumptions about how GenericParser walks the parse tree.
-#
-
-class GenericASTMatcher(GenericParser):
-    def __init__(self, start, ast):
-        GenericParser.__init__(self, start)
-        self.ast = ast
-
-    def preprocess(self, rule, func):
-        rebind = lambda func, self=self: \
-                        lambda args, func=func, self=self: \
-                                self.foundMatch(args, func)
-        lhs, rhs = rule
-        rhslist = list(rhs)
-        rhslist.reverse()
-
-        return (lhs, tuple(rhslist)), rebind(func)
-
-    def foundMatch(self, args, func):
-        func(args[-1])
-        return args[-1]
-
-    def match_r(self, node):
-        self.input.insert(0, node)
-        children = 0
-
-        for child in node:
-            if children == 0:
-                self.input.insert(0, '(')
-            children = children + 1
-            self.match_r(child)
-
-        if children > 0:
-            self.input.insert(0, ')')
-
-    def match(self, ast=None):
-        if ast is None:
-            ast = self.ast
-        self.input = []
-
-        self.match_r(ast)
-        self.parse(self.input)
-
-    def resolve(self, list):
-        #
-        #  Resolve ambiguity in favor of the longest RHS.
-        #
-        return list[-1]
-
-def _dump(tokens, sets, states):
-    for i in range(len(sets)):
-        output('set %d' % i)
-        for item in sets[i]:
-            output('\t', item)
-            for (lhs, rhs), pos in states[item[0]].items:
-                output('\t\t', lhs, '::=', end='')
-                output(' '.join(rhs[:pos]), end='')
-                output('.', end='')
-                output(' '.join(rhs[pos:]))
-        if i < len(tokens):
-            output()
-            output('token %s' % str(tokens[i]))
-            output()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat May 10 03:00:09 2014
From: python-checkins at python.org (benjamin.peterson)
Date: Sat, 10 May 2014 03:00:09 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?peps=3A_push_2=2E7=2E7_back_a_week?=
Message-ID: <3gQVST5drQz7Lk6@mail.python.org>

http://hg.python.org/peps/rev/6a043f98b47e
changeset:   5471:6a043f98b47e
user:        Benjamin Peterson <benjamin at python.org>
date:        Fri May 09 17:59:11 2014 -0700
summary:
  push 2.7.7 back a week

files:
  pep-0373.txt |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/pep-0373.txt b/pep-0373.txt
--- a/pep-0373.txt
+++ b/pep-0373.txt
@@ -71,8 +71,8 @@
 
 Planned future release dates:
 
-- 2.7.7 rc1 2014-05-12
-- 2.7.7 final 2014-05-26
+- 2.7.7 rc1 2014-05-17
+- 2.7.7 final 2014-05-31
 - 2.7.8 November 2014
 - 2.7.9 May 2015
 - beyond this date, releases as needed

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Sat May 10 03:00:10 2014
From: python-checkins at python.org (benjamin.peterson)
Date: Sat, 10 May 2014 03:00:10 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?peps=3A_Steve_Dower_assumes_the_win_i?=
	=?utf-8?q?nstaller_mantel?=
Message-ID: <3gQVSV73qjz7LkR@mail.python.org>

http://hg.python.org/peps/rev/581815f1c5af
changeset:   5472:581815f1c5af
user:        Benjamin Peterson <benjamin at python.org>
date:        Fri May 09 17:59:52 2014 -0700
summary:
  Steve Dower assumes the win installer mantel

files:
  pep-0373.txt |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/pep-0373.txt b/pep-0373.txt
--- a/pep-0373.txt
+++ b/pep-0373.txt
@@ -41,7 +41,7 @@
 Position                     Name
 ============================ ==================
 2.7 Release Manager          Benjamin Peterson
-Windows installers           Martin v. Loewis
+Windows installers           Steve Dower
 Mac installers               Ned Deily
 ============================ ==================
 

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Sat May 10 04:03:25 2014
From: python-checkins at python.org (eli.bendersky)
Date: Sat, 10 May 2014 04:03:25 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319655=3A_Add_test?=
 =?utf-8?q?s_for_the_new_asdl_parser=2E?=
Message-ID: <3gQWsT0BCYz7LjS@mail.python.org>

http://hg.python.org/cpython/rev/604e1b1a7777
changeset:   90611:604e1b1a7777
user:        Eli Bendersky <eliben at gmail.com>
date:        Fri May 09 19:03:25 2014 -0700
summary:
  Issue #19655: Add tests for the new asdl parser.

This unit test runs only for source builds of Python, similarly to test_tools.

files:
  Lib/test/test_asdl_parser.py |  122 +++++++++++++++++++++++
  1 files changed, 122 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_asdl_parser.py b/Lib/test/test_asdl_parser.py
new file mode 100644
--- /dev/null
+++ b/Lib/test/test_asdl_parser.py
@@ -0,0 +1,122 @@
+"""Tests for the asdl parser in Parser/asdl.py"""
+
+import importlib.machinery
+import os
+from os.path import dirname
+import sys
+import sysconfig
+import unittest
+
+
+# This test is only relevant for from-source builds of Python.
+if not sysconfig.is_python_build():
+    raise unittest.SkipTest('test irrelevant for an installed Python')
+
+src_base = dirname(dirname(dirname(__file__)))
+parser_dir = os.path.join(src_base, 'Parser')
+
+
+class TestAsdlParser(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        # Loads the asdl module dynamically, since it's not in a real importable
+        # package.
+        # Parses Python.asdl into a ast.Module and run the check on it.
+        # There's no need to do this for each test method, hence setUpClass.
+        sys.path.insert(0, parser_dir)
+        loader = importlib.machinery.SourceFileLoader(
+                'asdl', os.path.join(parser_dir, 'asdl.py'))
+        cls.asdl = loader.load_module()
+        cls.mod = cls.asdl.parse(os.path.join(parser_dir, 'Python.asdl'))
+        cls.assertTrue(cls.asdl.check(cls.mod), 'Module validation failed')
+
+    @classmethod
+    def tearDownClass(cls):
+        del sys.path[0]
+
+    def setUp(self):
+        # alias stuff from the class, for convenience
+        self.asdl = TestAsdlParser.asdl
+        self.mod = TestAsdlParser.mod
+        self.types = self.mod.types
+
+    def test_module(self):
+        self.assertEqual(self.mod.name, 'Python')
+        self.assertIn('stmt', self.types)
+        self.assertIn('expr', self.types)
+        self.assertIn('mod', self.types)
+
+    def test_definitions(self):
+        defs = self.mod.dfns
+        self.assertIsInstance(defs[0], self.asdl.Type)
+        self.assertIsInstance(defs[0].value, self.asdl.Sum)
+
+        self.assertIsInstance(self.types['withitem'], self.asdl.Product)
+        self.assertIsInstance(self.types['alias'], self.asdl.Product)
+
+    def test_product(self):
+        alias = self.types['alias']
+        self.assertEqual(
+            str(alias),
+            'Product([Field(identifier, name), Field(identifier, asname, opt=True)])')
+
+    def test_attributes(self):
+        stmt = self.types['stmt']
+        self.assertEqual(len(stmt.attributes), 2)
+        self.assertEqual(str(stmt.attributes[0]), 'Field(int, lineno)')
+        self.assertEqual(str(stmt.attributes[1]), 'Field(int, col_offset)')
+
+    def test_constructor_fields(self):
+        ehandler = self.types['excepthandler']
+        self.assertEqual(len(ehandler.types), 1)
+        self.assertEqual(len(ehandler.attributes), 2)
+
+        cons = ehandler.types[0]
+        self.assertIsInstance(cons, self.asdl.Constructor)
+        self.assertEqual(len(cons.fields), 3)
+
+        f0 = cons.fields[0]
+        self.assertEqual(f0.type, 'expr')
+        self.assertEqual(f0.name, 'type')
+        self.assertTrue(f0.opt)
+
+        f1 = cons.fields[1]
+        self.assertEqual(f1.type, 'identifier')
+        self.assertEqual(f1.name, 'name')
+        self.assertTrue(f1.opt)
+
+        f2 = cons.fields[2]
+        self.assertEqual(f2.type, 'stmt')
+        self.assertEqual(f2.name, 'body')
+        self.assertFalse(f2.opt)
+        self.assertTrue(f2.seq)
+
+    def test_visitor(self):
+        class CustomVisitor(self.asdl.VisitorBase):
+            def __init__(self):
+                super().__init__()
+                self.names_with_seq = []
+
+            def visitModule(self, mod):
+                for dfn in mod.dfns:
+                    self.visit(dfn)
+
+            def visitType(self, type):
+                self.visit(type.value)
+
+            def visitSum(self, sum):
+                for t in sum.types:
+                    self.visit(t)
+
+            def visitConstructor(self, cons):
+                for f in cons.fields:
+                    if f.seq:
+                        self.names_with_seq.append(cons.name)
+
+        v = CustomVisitor()
+        v.visit(self.types['mod'])
+        self.assertEqual(v.names_with_seq, ['Module', 'Interactive', 'Suite'])
+
+
+if __name__ == '__main__':
+    unittest.main()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat May 10 19:25:48 2014
From: python-checkins at python.org (jason.coombs)
Date: Sat, 10 May 2014 19:25:48 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Clean_up_style_in_distutil?=
	=?utf-8?q?s_upload_command?=
Message-ID: <3gQwKm2fPrz7LjP@mail.python.org>

http://hg.python.org/cpython/rev/7bb9e1bd613d
changeset:   90612:7bb9e1bd613d
user:        Jason R. Coombs <jaraco at jaraco.com>
date:        Sat May 10 13:20:28 2014 -0400
summary:
  Clean up style in distutils upload command

files:
  Lib/distutils/command/upload.py |  35 ++++++++++++---------
  1 files changed, 20 insertions(+), 15 deletions(-)


diff --git a/Lib/distutils/command/upload.py b/Lib/distutils/command/upload.py
--- a/Lib/distutils/command/upload.py
+++ b/Lib/distutils/command/upload.py
@@ -1,18 +1,21 @@
-"""distutils.command.upload
+"""
+distutils.command.upload
 
-Implements the Distutils 'upload' subcommand (upload package to PyPI)."""
+Implements the Distutils 'upload' subcommand (upload package to a package
+index).
+"""
 
+import sys
+import os
+import io
+import platform
+from base64 import standard_b64encode
+from urllib.request import urlopen, Request, HTTPError
+from urllib.parse import urlparse
 from distutils.errors import *
 from distutils.core import PyPIRCCommand
 from distutils.spawn import spawn
 from distutils import log
-import sys
-import os, io
-import socket
-import platform
-from base64 import standard_b64encode
-from urllib.request import urlopen, Request, HTTPError
-from urllib.parse import urlparse
 
 # this keeps compatibility for 2.3 and 2.4
 if sys.version < "2.5":
@@ -106,7 +109,7 @@
             'md5_digest': md5(content).hexdigest(),
 
             # additional meta-data
-            'metadata_version' : '1.0',
+            'metadata_version': '1.0',
             'summary': meta.get_description(),
             'home_page': meta.get_url(),
             'author': meta.get_contact(),
@@ -167,13 +170,15 @@
         body.write(b"\n")
         body = body.getvalue()
 
-        self.announce("Submitting %s to %s" % (filename, self.repository), log.INFO)
+        msg = "Submitting %s to %s" % (filename, self.repository)
+        self.announce(msg, log.INFO)
 
         # build the Request
-        headers = {'Content-type':
-                        'multipart/form-data; boundary=%s' % boundary,
-                   'Content-length': str(len(body)),
-                   'Authorization': auth}
+        headers = {
+            'Content-type': 'multipart/form-data; boundary=%s' % boundary,
+            'Content-length': str(len(body)),
+            'Authorization': auth,
+        }
 
         request = Request(self.repository, data=body,
                           headers=headers)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat May 10 19:25:49 2014
From: python-checkins at python.org (jason.coombs)
Date: Sat, 10 May 2014 19:25:49 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Replace_import_*_with_expl?=
	=?utf-8?q?icit_import?=
Message-ID: <3gQwKn4LmZz7LjV@mail.python.org>

http://hg.python.org/cpython/rev/dc2559f5f571
changeset:   90613:dc2559f5f571
user:        Jason R. Coombs <jaraco at jaraco.com>
date:        Sat May 10 13:21:02 2014 -0400
summary:
  Replace import * with explicit import

files:
  Lib/distutils/command/upload.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/distutils/command/upload.py b/Lib/distutils/command/upload.py
--- a/Lib/distutils/command/upload.py
+++ b/Lib/distutils/command/upload.py
@@ -12,7 +12,7 @@
 from base64 import standard_b64encode
 from urllib.request import urlopen, Request, HTTPError
 from urllib.parse import urlparse
-from distutils.errors import *
+from distutils.errors import DistutilsOptionError
 from distutils.core import PyPIRCCommand
 from distutils.spawn import spawn
 from distutils import log

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat May 10 19:25:50 2014
From: python-checkins at python.org (jason.coombs)
Date: Sat, 10 May 2014 19:25:50 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Drop_support_for_Python_2?=
	=?utf-8?q?=2E4_in_upload_command=2E?=
Message-ID: <3gQwKp631bz7LjV@mail.python.org>

http://hg.python.org/cpython/rev/796a80ac1c3f
changeset:   90614:796a80ac1c3f
user:        Jason R. Coombs <jaraco at jaraco.com>
date:        Sat May 10 13:22:43 2014 -0400
summary:
  Drop support for Python 2.4 in upload command.

files:
  Lib/distutils/command/upload.py |  10 ++--------
  1 files changed, 2 insertions(+), 8 deletions(-)


diff --git a/Lib/distutils/command/upload.py b/Lib/distutils/command/upload.py
--- a/Lib/distutils/command/upload.py
+++ b/Lib/distutils/command/upload.py
@@ -5,10 +5,10 @@
 index).
 """
 
-import sys
 import os
 import io
 import platform
+import hashlib
 from base64 import standard_b64encode
 from urllib.request import urlopen, Request, HTTPError
 from urllib.parse import urlparse
@@ -17,12 +17,6 @@
 from distutils.spawn import spawn
 from distutils import log
 
-# this keeps compatibility for 2.3 and 2.4
-if sys.version < "2.5":
-    from md5 import md5
-else:
-    from hashlib import md5
-
 class upload(PyPIRCCommand):
 
     description = "upload binary package to PyPI"
@@ -106,7 +100,7 @@
             'content': (os.path.basename(filename),content),
             'filetype': command,
             'pyversion': pyversion,
-            'md5_digest': md5(content).hexdigest(),
+            'md5_digest': hashlib.md5(content).hexdigest(),
 
             # additional meta-data
             'metadata_version': '1.0',

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat May 10 19:25:52 2014
From: python-checkins at python.org (jason.coombs)
Date: Sat, 10 May 2014 19:25:52 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Replace_overly-aggressive_?=
 =?utf-8?q?comparison_for_type_equality_with_an_isinstance_check=2E?=
Message-ID: <3gQwKr0bpVz7LjX@mail.python.org>

http://hg.python.org/cpython/rev/05e364315de2
changeset:   90615:05e364315de2
user:        Jason R. Coombs <jaraco at jaraco.com>
date:        Sat May 10 13:24:18 2014 -0400
summary:
  Replace overly-aggressive comparison for type equality with an isinstance check.

files:
  Lib/distutils/command/upload.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/distutils/command/upload.py b/Lib/distutils/command/upload.py
--- a/Lib/distutils/command/upload.py
+++ b/Lib/distutils/command/upload.py
@@ -146,7 +146,7 @@
         for key, value in data.items():
             title = '\nContent-Disposition: form-data; name="%s"' % key
             # handle multiple entries for the same name
-            if type(value) != type([]):
+            if not isinstance(value, list):
                 value = [value]
             for value in value:
                 if type(value) is tuple:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat May 10 19:25:53 2014
From: python-checkins at python.org (jason.coombs)
Date: Sat, 10 May 2014 19:25:53 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Reindent_long_line?=
Message-ID: <3gQwKs2Jkpz7Ljf@mail.python.org>

http://hg.python.org/cpython/rev/ce070040e1a6
changeset:   90616:ce070040e1a6
user:        Jason R. Coombs <jaraco at jaraco.com>
date:        Sat May 10 13:24:58 2014 -0400
summary:
  Reindent long line

files:
  Lib/distutils/command/upload.py |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Lib/distutils/command/upload.py b/Lib/distutils/command/upload.py
--- a/Lib/distutils/command/upload.py
+++ b/Lib/distutils/command/upload.py
@@ -57,7 +57,8 @@
 
     def run(self):
         if not self.distribution.dist_files:
-            raise DistutilsOptionError("No dist file created in earlier command")
+            msg = "No dist file created in earlier command"
+            raise DistutilsOptionError(msg)
         for command, pyversion, filename in self.distribution.dist_files:
             self.upload_file(command, pyversion, filename)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat May 10 19:53:10 2014
From: python-checkins at python.org (brian.curtin)
Date: Sat, 10 May 2014 19:53:10 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Backport_4e9f1?=
	=?utf-8?q?017355f_from_=233561=2E?=
Message-ID: <3gQwxL4Nk9z7LjW@mail.python.org>

http://hg.python.org/cpython/rev/a9d34685ec47
changeset:   90617:a9d34685ec47
branch:      2.7
parent:      90591:db842f730432
user:        Brian Curtin <brian at python.org>
date:        Sat May 10 12:52:59 2014 -0500
summary:
  Backport 4e9f1017355f from #3561.

This brings the option to install Python on the Windows Path.
Committed per Benjamin Peterson's approval on python-dev.

files:
  Misc/NEWS        |   4 ++++
  Tools/msi/msi.py |  22 +++++++++++++++++++---
  2 files changed, 23 insertions(+), 3 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -289,6 +289,10 @@
 Tools/Demos
 -----------
 
+- Issue #3561: The Windows installer now has an option, off by default, for
+  placing the Python installation into the system "Path" environment variable.
+  This was backported from Python 3.3.
+
 - Add support for ``yield from`` to 2to3.
 
 - Add support for the PEP 465 matrix multiplication operator to 2to3.
diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py
--- a/Tools/msi/msi.py
+++ b/Tools/msi/msi.py
@@ -445,6 +445,10 @@
               ("SetDLLDirToTarget", 'DLLDIR=""', 751),
              ])
 
+    # Prepend TARGETDIR to the system path, and remove it on uninstall.
+    add_data(db, "Environment",
+             [("PathAddition", "=-*Path", "[TARGETDIR];[~]", "REGISTRY.path")])
+
     # Execute Sequences
     add_data(db, "InstallExecuteSequence",
             [("InitialTargetDir", 'TARGETDIR=""', 750),
@@ -668,11 +672,11 @@
     c=features.xbutton("Advanced", "Advanced", None, 0.30)
     c.event("SpawnDialog", "AdvancedDlg")
 
-    c=features.text("ItemDescription", 140, 180, 210, 30, 3,
+    c=features.text("ItemDescription", 140, 180, 210, 40, 3,
                   "Multiline description of the currently selected item.")
     c.mapping("SelectionDescription","Text")
 
-    c=features.text("ItemSize", 140, 210, 210, 45, 3,
+    c=features.text("ItemSize", 140, 225, 210, 33, 3,
                     "The size of the currently selected item.")
     c.mapping("SelectionSize", "Text")
 
@@ -826,7 +830,7 @@
     # (i.e. additional Python libraries) need to follow the parent feature.
     # Features that have no advertisement trigger (e.g. the test suite)
     # must not support advertisement
-    global default_feature, tcltk, htmlfiles, tools, testsuite, ext_feature, private_crt
+    global default_feature, tcltk, htmlfiles, tools, testsuite, ext_feature, private_crt, prepend_path
     default_feature = Feature(db, "DefaultFeature", "Python",
                               "Python Interpreter and Libraries",
                               1, directory = "TARGETDIR")
@@ -851,6 +855,15 @@
     testsuite = Feature(db, "Testsuite", "Test suite",
                         "Python test suite (Lib/test/)", 11,
                         parent = default_feature, attributes=2|8)
+    # prepend_path is an additional feature which is to be off by default.
+    # Since the default level for the above features is 1, this needs to be
+    # at least level higher.
+    prepend_path = Feature(db, "PrependPath", "Add python.exe to Path",
+                           "Prepend [TARGETDIR] to the system Path variable. "
+                           "This allows you to type 'python' into a command "
+                           "prompt without needing the full path.", 13,
+                           parent = default_feature, attributes=2|8,
+                           level=2)
 
 def extract_msvcr90():
     # Find the redistributable files
@@ -1168,6 +1181,8 @@
                "InstallPath"),
               ("REGISTRY.doc", msilib.gen_uuid(), "TARGETDIR", registry_component, None,
                "Documentation"),
+              ("REGISTRY.path", msilib.gen_uuid(), "TARGETDIR", registry_component, None,
+               None),
               ("REGISTRY.def", msilib.gen_uuid(), "TARGETDIR", registry_component,
                None, None)] + tcldata)
     # See "FeatureComponents Table".
@@ -1184,6 +1199,7 @@
     add_data(db, "FeatureComponents",
              [(default_feature.id, "REGISTRY"),
               (htmlfiles.id, "REGISTRY.doc"),
+              (prepend_path.id, "REGISTRY.path"),
               (ext_feature.id, "REGISTRY.def")] +
               tcldata
               )

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat May 10 23:05:39 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Sat, 10 May 2014 23:05:39 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Remove_the_redundant_and_p?=
 =?utf-8?q?oorly_worded_warning_message=2E?=
Message-ID: <3gR1CR0CDcz7LjY@mail.python.org>

http://hg.python.org/cpython/rev/b466dc34b86e
changeset:   90618:b466dc34b86e
parent:      90616:ce070040e1a6
user:        Raymond Hettinger <python at rcn.com>
date:        Sat May 10 14:05:28 2014 -0700
summary:
  Remove the redundant and poorly worded warning message.

The paragraph above already says, clearly and correctly, that
"However, being completely deterministic, it is not suitable for
all purposes, and is completely unsuitable for cryptographic purposes."

Also we should make any promises about SystemRandom or os.urandom()
being cryptographically secure (they may be, but be can't validate
that promise).  Further, those are actual random number generators
not psuedo-random number generators.

files:
  Doc/library/random.rst |  6 ------
  1 files changed, 0 insertions(+), 6 deletions(-)


diff --git a/Doc/library/random.rst b/Doc/library/random.rst
--- a/Doc/library/random.rst
+++ b/Doc/library/random.rst
@@ -43,12 +43,6 @@
 uses the system function :func:`os.urandom` to generate random numbers
 from sources provided by the operating system.
 
-.. warning::
-
-   The pseudo-random generators of this module should not be used for
-   security purposes.  Use :func:`os.urandom` or :class:`SystemRandom` if
-   you require a cryptographically secure pseudo-random number generator.
-
 
 Bookkeeping functions:
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 00:48:11 2014
From: python-checkins at python.org (guido.van.rossum)
Date: Sun, 11 May 2014 00:48:11 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogVXBz?=
 =?utf-8?q?tream_issue_=23167=3A_remove_dead_code=2C_by_Marc_Schlaich=2E?=
Message-ID: <3gR3Tl1WKvz7Ljd@mail.python.org>

http://hg.python.org/cpython/rev/6a4f908338a8
changeset:   90619:6a4f908338a8
branch:      3.4
parent:      90606:3dea57884dcd
user:        Guido van Rossum <guido at python.org>
date:        Sat May 10 15:47:15 2014 -0700
summary:
  asyncio: Upstream issue #167: remove dead code, by Marc Schlaich.

files:
  Lib/asyncio/base_events.py |  6 +-----
  1 files changed, 1 insertions(+), 5 deletions(-)


diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py
--- a/Lib/asyncio/base_events.py
+++ b/Lib/asyncio/base_events.py
@@ -775,11 +775,7 @@
         elif self._scheduled:
             # Compute the desired timeout.
             when = self._scheduled[0]._when
-            deadline = max(0, when - self.time())
-            if timeout is None:
-                timeout = deadline
-            else:
-                timeout = min(timeout, deadline)
+            timeout = max(0, when - self.time())
 
         # TODO: Instrumentation only in debug mode?
         if logger.isEnabledFor(logging.INFO):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 00:48:12 2014
From: python-checkins at python.org (guido.van.rossum)
Date: Sun, 11 May 2014 00:48:12 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge_3=2E4_-=3E_default=3A_asyncio=3A_Upstream_issue_?=
 =?utf-8?q?=23167=3A_remove_dead_code=2C_by_Marc?=
Message-ID: <3gR3Tm3Jj2z7LkP@mail.python.org>

http://hg.python.org/cpython/rev/1320ad8d5222
changeset:   90620:1320ad8d5222
parent:      90618:b466dc34b86e
parent:      90619:6a4f908338a8
user:        Guido van Rossum <guido at python.org>
date:        Sat May 10 15:48:03 2014 -0700
summary:
  Merge 3.4 -> default: asyncio: Upstream issue #167: remove dead code, by Marc Schlaich.

files:
  Lib/asyncio/base_events.py |  6 +-----
  1 files changed, 1 insertions(+), 5 deletions(-)


diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py
--- a/Lib/asyncio/base_events.py
+++ b/Lib/asyncio/base_events.py
@@ -775,11 +775,7 @@
         elif self._scheduled:
             # Compute the desired timeout.
             when = self._scheduled[0]._when
-            deadline = max(0, when - self.time())
-            if timeout is None:
-                timeout = deadline
-            else:
-                timeout = min(timeout, deadline)
+            timeout = max(0, when - self.time())
 
         # TODO: Instrumentation only in debug mode?
         if logger.isEnabledFor(logging.INFO):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 06:04:56 2014
From: python-checkins at python.org (steve.dower)
Date: Sun, 11 May 2014 06:04:56 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?devguide=3A_Add_myself_to_developer_l?=
 =?utf-8?q?og_and_as_a_Windows_expert=2E?=
Message-ID: <3gRBWD3q7bz7Ljd@mail.python.org>

http://hg.python.org/devguide/rev/8d5d1f2c7378
changeset:   698:8d5d1f2c7378
user:        Steve Dower <steve.dower at microsoft.com>
date:        Sat May 10 21:01:39 2014 -0700
summary:
  Add myself to developer log and as a Windows expert.

files:
  developers.rst |  3 +++
  experts.rst    |  2 +-
  2 files changed, 4 insertions(+), 1 deletions(-)


diff --git a/developers.rst b/developers.rst
--- a/developers.rst
+++ b/developers.rst
@@ -25,6 +25,9 @@
 Permissions History
 -------------------
 
+- Steve Dower was given push privileges on May 10, 2014 by Antoine Pitrou, on
+  recommendation by Martin v. Loewis.
+
 - Kushal Das was given push privileges on Apr 14, 2014 by BAC, for
   general patches, on recommendation by Michael Foord.
 
diff --git a/experts.rst b/experts.rst
--- a/experts.rst
+++ b/experts.rst
@@ -289,7 +289,7 @@
 NetBSD1
 OS2/EMX               aimacintyre
 Solaris/OpenIndiana   jcea
-Windows               tim.golden, zach.ware
+Windows               tim.golden, zach.ware, steve.dower
 JVM/Java              frank.wierzbicki
 ===================   ===========
 

-- 
Repository URL: http://hg.python.org/devguide

From python-checkins at python.org  Sun May 11 06:27:00 2014
From: python-checkins at python.org (steve.dower)
Date: Sun, 11 May 2014 06:27:00 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIxNDc2?=
 =?utf-8?q?_Include_idle_icon_files_in_Windows_installer?=
Message-ID: <3gRC0h3kMYz7Ljb@mail.python.org>

http://hg.python.org/cpython/rev/730eeb10cd81
changeset:   90621:730eeb10cd81
branch:      2.7
parent:      90617:a9d34685ec47
user:        Steve Dower <steve.dower at microsoft.com>
date:        Sat May 10 21:25:54 2014 -0700
summary:
  Issue #21476 Include idle icon files in Windows installer

files:
  Tools/msi/msi.py |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py
--- a/Tools/msi/msi.py
+++ b/Tools/msi/msi.py
@@ -1047,6 +1047,7 @@
             lib.add_file("idle.bat")
         if dir=="Icons":
             lib.glob("*.gif")
+            lib.glob("*.ico")
             lib.add_file("idle.icns")
         if dir=="command" and parent.physical=="distutils":
             lib.glob("wininst*.exe")

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 10:56:08 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Sun, 11 May 2014 10:56:08 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321424=3A__Optimiz?=
 =?utf-8?q?e_heaqp=2Enlargest=28=29_to_make_fewer_tuple_comparisons=2E?=
Message-ID: <3gRJzD5LM9z7Ljd@mail.python.org>

http://hg.python.org/cpython/rev/fadc06047314
changeset:   90622:fadc06047314
parent:      90620:1320ad8d5222
user:        Raymond Hettinger <python at rcn.com>
date:        Sun May 11 01:55:46 2014 -0700
summary:
  Issue #21424:  Optimize heaqp.nlargest() to make fewer tuple comparisons.

Consolidates the logic for nlargest() into a single function so that
decoration tuples (elem,order) or (key, order, elem) only need to
be formed when a new element is added to the heap.  Formerly, a tuple
was created for every element regardless of whether it was added to
the heap.

The change reduces the number of tuples created, the number of ordering
integers created, and total number of tuple comparisons.

files:
  Lib/heapq.py           |  125 ++++++++--------------------
  Lib/test/test_heapq.py |    2 +-
  Misc/NEWS              |    3 +
  Modules/_heapqmodule.c |   85 -------------------
  4 files changed, 41 insertions(+), 174 deletions(-)


diff --git a/Lib/heapq.py b/Lib/heapq.py
--- a/Lib/heapq.py
+++ b/Lib/heapq.py
@@ -192,81 +192,6 @@
     for i in reversed(range(n//2)):
         _siftup_max(x, i)
 
-
-# Algorithm notes for nlargest() and nsmallest()
-# ==============================================
-#
-# Makes just one pass over the data while keeping the n most extreme values
-# in a heap.  Memory consumption is limited to keeping n values in a list.
-#
-# Number of comparisons for n random inputs, keeping the k smallest values:
-# -----------------------------------------------------------
-# Step   Comparisons                 Action
-#  1        1.66*k                   heapify the first k-inputs
-#  2        n - k                    compare new input elements to top of heap
-#  3        k*lg2(k)*(ln(n)-ln(k))   add new extreme values to the heap
-#  4        k*lg2(k)                 final sort of the k most extreme values
-#
-#                                      number of comparisons
-# n-random inputs   k-extreme values    average of 5 trials     % more than min()
-# ---------------   ----------------    -------------------     -----------------
-#       10,000            100                   14,046                 40.5%
-#      100,000            100                  105,749                  5.7%
-#    1,000,000            100                1,007,751                  0.8%
-#
-# Computing the number of comparisons for step 3:
-# -----------------------------------------------
-# * For the i-th new value from the iterable, the probability of being in the
-#   k most extreme values is k/i.  For example, the probability of the 101st
-#   value seen being in the 100 most extreme values is 100/101.
-# * If the value is a new extreme value, the cost of inserting it into the
-#   heap is log(k, 2).
-# * The probabilty times the cost gives:
-#            (k/i) * log(k, 2)
-# * Summing across the remaining n-k elements gives:
-#            sum((k/i) * log(k, 2) for xrange(k+1, n+1))
-# * This reduces to:
-#            (H(n) - H(k)) * k * log(k, 2)
-# * Where H(n) is the n-th harmonic number estimated by:
-#            H(n) = log(n, e) + gamma + 1.0 / (2.0 * n)
-#            gamma = 0.5772156649
-#   http://en.wikipedia.org/wiki/Harmonic_series_(mathematics)#Rate_of_divergence
-# * Substituting the H(n) formula and ignoring the (1/2*n) fraction gives:
-#            comparisons = k * log(k, 2) * (log(n,e) - log(k, e))
-#
-# Worst-case for step 3:
-# ----------------------
-# In the worst case, the input data is reversed sorted so that every new element
-# must be inserted in the heap:
-#             comparisons = log(k, 2) * (n - k)
-#
-# Alternative Algorithms
-# ----------------------
-# Other algorithms were not used because they:
-# 1) Took much more auxiliary memory,
-# 2) Made multiple passes over the data.
-# 3) Made more comparisons in common cases (small k, large n, semi-random input).
-# See detailed comparisons at:
-# http://code.activestate.com/recipes/577573-compare-algorithms-for-heapqsmallest
-
-def nlargest(n, iterable):
-    """Find the n largest elements in a dataset.
-
-    Equivalent to:  sorted(iterable, reverse=True)[:n]
-    """
-    if n <= 0:
-        return []
-    it = iter(iterable)
-    result = list(islice(it, n))
-    if not result:
-        return result
-    heapify(result)
-    _heappushpop = heappushpop
-    for elem in it:
-        _heappushpop(result, elem)
-    result.sort(reverse=True)
-    return result
-
 def nsmallest(n, iterable):
     """Find the n smallest elements in a dataset.
 
@@ -480,7 +405,6 @@
     result = _nsmallest(n, it)
     return [r[2] for r in result]                           # undecorate
 
-_nlargest = nlargest
 def nlargest(n, iterable, key=None):
     """Find the n largest elements in a dataset.
 
@@ -490,12 +414,12 @@
     # Short-cut for n==1 is to use max() when len(iterable)>0
     if n == 1:
         it = iter(iterable)
-        head = list(islice(it, 1))
-        if not head:
-            return []
+        sentinel = object()
         if key is None:
-            return [max(chain(head, it))]
-        return [max(chain(head, it), key=key)]
+            result = max(it, default=sentinel)
+        else:
+            result = max(it, default=sentinel, key=key)
+        return [] if result is sentinel else [result]
 
     # When n>=size, it's faster to use sorted()
     try:
@@ -508,15 +432,40 @@
 
     # When key is none, use simpler decoration
     if key is None:
-        it = zip(iterable, count(0,-1))                     # decorate
-        result = _nlargest(n, it)
-        return [r[0] for r in result]                       # undecorate
+        it = iter(iterable)
+        result = list(islice(zip(it, count(0, -1)), n))
+        if not result:
+            return result
+        heapify(result)
+        order = -n
+        top = result[0][0]
+        _heapreplace = heapreplace
+        for elem in it:
+            if top < elem:
+                order -= 1
+                _heapreplace(result, (elem, order))
+                top = result[0][0]
+        result.sort(reverse=True)
+        return [r[0] for r in result]
 
     # General case, slowest method
-    in1, in2 = tee(iterable)
-    it = zip(map(key, in1), count(0,-1), in2)               # decorate
-    result = _nlargest(n, it)
-    return [r[2] for r in result]                           # undecorate
+    it = iter(iterable)
+    result = [(key(elem), i, elem) for i, elem in zip(range(0, -n, -1), it)]
+    if not result:
+        return result
+    heapify(result)
+    order = -n
+    top = result[0][0]
+    _heapreplace = heapreplace
+    for elem in it:
+        k = key(elem)
+        if top < k:
+            order -= 1
+            _heapreplace(result, (k, order, elem))
+            top = result[0][0]
+    result.sort(reverse=True)
+    return [r[2] for r in result]
+
 
 if __name__ == "__main__":
     # Simple sanity test
diff --git a/Lib/test/test_heapq.py b/Lib/test/test_heapq.py
--- a/Lib/test/test_heapq.py
+++ b/Lib/test/test_heapq.py
@@ -13,7 +13,7 @@
 # _heapq.nlargest/nsmallest are saved in heapq._nlargest/_smallest when
 # _heapq is imported, so check them there
 func_names = ['heapify', 'heappop', 'heappush', 'heappushpop',
-              'heapreplace', '_nlargest', '_nsmallest']
+              'heapreplace', '_nsmallest']
 
 class TestModules(TestCase):
     def test_py_functions(self):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -81,6 +81,9 @@
 - Issue #21156: importlib.abc.InspectLoader.source_to_code() is now a
   staticmethod.
 
+- Issue #21424: Simplified and optimized heaqp.nlargest() to make fewer
+  tuple comparisons.
+
 - Issue #21396: Fix TextIOWrapper(..., write_through=True) to not force a
   flush() on the underlying binary stream.  Patch by akira.
 
diff --git a/Modules/_heapqmodule.c b/Modules/_heapqmodule.c
--- a/Modules/_heapqmodule.c
+++ b/Modules/_heapqmodule.c
@@ -267,89 +267,6 @@
 PyDoc_STRVAR(heapify_doc,
 "Transform list into a heap, in-place, in O(len(heap)) time.");
 
-static PyObject *
-nlargest(PyObject *self, PyObject *args)
-{
-    PyObject *heap=NULL, *elem, *iterable, *sol, *it, *oldelem;
-    Py_ssize_t i, n;
-    int cmp;
-
-    if (!PyArg_ParseTuple(args, "nO:nlargest", &n, &iterable))
-        return NULL;
-
-    it = PyObject_GetIter(iterable);
-    if (it == NULL)
-        return NULL;
-
-    heap = PyList_New(0);
-    if (heap == NULL)
-        goto fail;
-
-    for (i=0 ; i<n ; i++ ){
-        elem = PyIter_Next(it);
-        if (elem == NULL) {
-            if (PyErr_Occurred())
-                goto fail;
-            else
-                goto sortit;
-        }
-        if (PyList_Append(heap, elem) == -1) {
-            Py_DECREF(elem);
-            goto fail;
-        }
-        Py_DECREF(elem);
-    }
-    if (PyList_GET_SIZE(heap) == 0)
-        goto sortit;
-
-    for (i=n/2-1 ; i>=0 ; i--)
-        if(_siftup((PyListObject *)heap, i) == -1)
-            goto fail;
-
-    sol = PyList_GET_ITEM(heap, 0);
-    while (1) {
-        elem = PyIter_Next(it);
-        if (elem == NULL) {
-            if (PyErr_Occurred())
-                goto fail;
-            else
-                goto sortit;
-        }
-        cmp = PyObject_RichCompareBool(sol, elem, Py_LT);
-        if (cmp == -1) {
-            Py_DECREF(elem);
-            goto fail;
-        }
-        if (cmp == 0) {
-            Py_DECREF(elem);
-            continue;
-        }
-        oldelem = PyList_GET_ITEM(heap, 0);
-        PyList_SET_ITEM(heap, 0, elem);
-        Py_DECREF(oldelem);
-        if (_siftup((PyListObject *)heap, 0) == -1)
-            goto fail;
-        sol = PyList_GET_ITEM(heap, 0);
-    }
-sortit:
-    if (PyList_Sort(heap) == -1)
-        goto fail;
-    if (PyList_Reverse(heap) == -1)
-        goto fail;
-    Py_DECREF(it);
-    return heap;
-
-fail:
-    Py_DECREF(it);
-    Py_XDECREF(heap);
-    return NULL;
-}
-
-PyDoc_STRVAR(nlargest_doc,
-"Find the n largest elements in a dataset.\n\
-\n\
-Equivalent to:  sorted(iterable, reverse=True)[:n]\n");
-
 static int
 _siftdownmax(PyListObject *heap, Py_ssize_t startpos, Py_ssize_t pos)
 {
@@ -531,8 +448,6 @@
         METH_VARARGS,           heapreplace_doc},
     {"heapify",         (PyCFunction)heapify,
         METH_O,                 heapify_doc},
-    {"nlargest",        (PyCFunction)nlargest,
-        METH_VARARGS,           nlargest_doc},
     {"nsmallest",       (PyCFunction)nsmallest,
         METH_VARARGS,           nsmallest_doc},
     {NULL,              NULL}           /* sentinel */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 11:26:32 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Sun, 11 May 2014 11:26:32 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_By_popular_demand=2C_add_b?=
	=?utf-8?q?ack_a_security_warning=2E?=
Message-ID: <3gRKfJ08mgz7LjY@mail.python.org>

http://hg.python.org/cpython/rev/365674ea3613
changeset:   90623:365674ea3613
user:        Raymond Hettinger <python at rcn.com>
date:        Sun May 11 02:26:23 2014 -0700
summary:
  By popular demand, add back a security warning.

files:
  Doc/library/random.rst |  5 +++++
  1 files changed, 5 insertions(+), 0 deletions(-)


diff --git a/Doc/library/random.rst b/Doc/library/random.rst
--- a/Doc/library/random.rst
+++ b/Doc/library/random.rst
@@ -43,6 +43,11 @@
 uses the system function :func:`os.urandom` to generate random numbers
 from sources provided by the operating system.
 
+.. warning::
+
+   The pseudo-random generators of this module should not be used for
+   security purposes.
+
 
 Bookkeeping functions:
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 13:45:21 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 11 May 2014 13:45:21 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxNDI1?=
 =?utf-8?q?=3A_Fix_flushing_of_standard_streams_in_the_interactive_interpr?=
 =?utf-8?q?eter=2E?=
Message-ID: <3gRNkT0c8rz7Ljd@mail.python.org>

http://hg.python.org/cpython/rev/ab3e012c45d0
changeset:   90624:ab3e012c45d0
branch:      3.4
parent:      90619:6a4f908338a8
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun May 11 13:42:17 2014 +0200
summary:
  Issue #21425: Fix flushing of standard streams in the interactive interpreter.

files:
  Lib/test/script_helper.py        |   4 +-
  Lib/test/test_cmd_line_script.py |  49 ++++++++++++++++++++
  Misc/NEWS                        |   5 +-
  Python/pythonrun.c               |   3 +-
  4 files changed, 57 insertions(+), 4 deletions(-)


diff --git a/Lib/test/script_helper.py b/Lib/test/script_helper.py
--- a/Lib/test/script_helper.py
+++ b/Lib/test/script_helper.py
@@ -78,7 +78,7 @@
     """
     return _assert_python(False, *args, **env_vars)
 
-def spawn_python(*args, **kw):
+def spawn_python(*args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, **kw):
     """Run a Python subprocess with the given arguments.
 
     kw is extra keyword args to pass to subprocess.Popen. Returns a Popen
@@ -87,7 +87,7 @@
     cmd_line = [sys.executable, '-E']
     cmd_line.extend(args)
     return subprocess.Popen(cmd_line, stdin=subprocess.PIPE,
-                            stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                            stdout=stdout, stderr=stderr,
                             **kw)
 
 def kill_python(p):
diff --git a/Lib/test/test_cmd_line_script.py b/Lib/test/test_cmd_line_script.py
--- a/Lib/test/test_cmd_line_script.py
+++ b/Lib/test/test_cmd_line_script.py
@@ -1,5 +1,6 @@
 # tests command line execution of scripts
 
+import contextlib
 import importlib
 import importlib.machinery
 import zipimport
@@ -8,6 +9,7 @@
 import os
 import os.path
 import py_compile
+import subprocess
 
 import textwrap
 from test import support
@@ -173,6 +175,53 @@
         expected = repr(importlib.machinery.BuiltinImporter).encode("utf-8")
         self.assertIn(expected, out)
 
+    @contextlib.contextmanager
+    def interactive_python(self, separate_stderr=False):
+        if separate_stderr:
+            p = spawn_python('-i', bufsize=1, stderr=subprocess.PIPE)
+            stderr = p.stderr
+        else:
+            p = spawn_python('-i', bufsize=1, stderr=subprocess.STDOUT)
+            stderr = p.stdout
+        try:
+            # Drain stderr until prompt
+            while True:
+                data = stderr.read(4)
+                if data == b">>> ":
+                    break
+                stderr.readline()
+            yield p
+        finally:
+            kill_python(p)
+            stderr.close()
+
+    def check_repl_stdout_flush(self, separate_stderr=False):
+        with self.interactive_python(separate_stderr) as p:
+            p.stdin.write(b"print('foo')\n")
+            p.stdin.flush()
+            self.assertEqual(b'foo', p.stdout.readline().strip())
+
+    def check_repl_stderr_flush(self, separate_stderr=False):
+        with self.interactive_python(separate_stderr) as p:
+            p.stdin.write(b"1/0\n")
+            p.stdin.flush()
+            stderr = p.stderr if separate_stderr else p.stdout
+            self.assertIn(b'Traceback ', stderr.readline())
+            self.assertIn(b'File "<stdin>"', stderr.readline())
+            self.assertIn(b'ZeroDivisionError', stderr.readline())
+
+    def test_repl_stdout_flush(self):
+        self.check_repl_stdout_flush()
+
+    def test_repl_stdout_flush_separate_stderr(self):
+        self.check_repl_stdout_flush(True)
+
+    def test_repl_stderr_flush(self):
+        self.check_repl_stderr_flush()
+
+    def test_repl_stderr_flush_separate_stderr(self):
+        self.check_repl_stderr_flush(True)
+
     def test_basic_script(self):
         with temp_dir() as script_dir:
             script_name = _make_test_script(script_dir, 'script')
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -3,13 +3,16 @@
 +++++++++++
 
 What's New in Python 3.4.1?
-==========================
+===========================
 
 Release date: TBA
 
 Core and Builtins
 -----------------
 
+- Issue #21425: Fix flushing of standard streams in the interactive
+  interpreter.
+
 - Issue #21435: In rare cases, when running finalizers on objects in cyclic
   trash a bad pointer dereference could occur due to a subtle flaw in
   internal iteration logic.
diff --git a/Python/pythonrun.c b/Python/pythonrun.c
--- a/Python/pythonrun.c
+++ b/Python/pythonrun.c
@@ -1444,12 +1444,13 @@
     d = PyModule_GetDict(m);
     v = run_mod(mod, filename, d, d, flags, arena);
     PyArena_Free(arena);
-    flush_io();
     if (v == NULL) {
         PyErr_Print();
+        flush_io();
         return -1;
     }
     Py_DECREF(v);
+    flush_io();
     return 0;
 }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 13:45:22 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 11 May 2014 13:45:22 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Issue_=2321425=3A_Fix_flushing_of_standard_streams_in_th?=
 =?utf-8?q?e_interactive_interpreter=2E?=
Message-ID: <3gRNkV3W4tz7Ljk@mail.python.org>

http://hg.python.org/cpython/rev/d1c0cf44160c
changeset:   90625:d1c0cf44160c
parent:      90623:365674ea3613
parent:      90624:ab3e012c45d0
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun May 11 13:43:31 2014 +0200
summary:
  Issue #21425: Fix flushing of standard streams in the interactive interpreter.

files:
  Lib/test/script_helper.py        |   4 +-
  Lib/test/test_cmd_line_script.py |  49 ++++++++++++++++++++
  Misc/NEWS                        |   3 +
  Python/pythonrun.c               |   3 +-
  4 files changed, 56 insertions(+), 3 deletions(-)


diff --git a/Lib/test/script_helper.py b/Lib/test/script_helper.py
--- a/Lib/test/script_helper.py
+++ b/Lib/test/script_helper.py
@@ -78,7 +78,7 @@
     """
     return _assert_python(False, *args, **env_vars)
 
-def spawn_python(*args, **kw):
+def spawn_python(*args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, **kw):
     """Run a Python subprocess with the given arguments.
 
     kw is extra keyword args to pass to subprocess.Popen. Returns a Popen
@@ -87,7 +87,7 @@
     cmd_line = [sys.executable, '-E']
     cmd_line.extend(args)
     return subprocess.Popen(cmd_line, stdin=subprocess.PIPE,
-                            stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                            stdout=stdout, stderr=stderr,
                             **kw)
 
 def kill_python(p):
diff --git a/Lib/test/test_cmd_line_script.py b/Lib/test/test_cmd_line_script.py
--- a/Lib/test/test_cmd_line_script.py
+++ b/Lib/test/test_cmd_line_script.py
@@ -1,5 +1,6 @@
 # tests command line execution of scripts
 
+import contextlib
 import importlib
 import importlib.machinery
 import zipimport
@@ -8,6 +9,7 @@
 import os
 import os.path
 import py_compile
+import subprocess
 
 import textwrap
 from test import support
@@ -173,6 +175,53 @@
         expected = repr(importlib.machinery.BuiltinImporter).encode("utf-8")
         self.assertIn(expected, out)
 
+    @contextlib.contextmanager
+    def interactive_python(self, separate_stderr=False):
+        if separate_stderr:
+            p = spawn_python('-i', bufsize=1, stderr=subprocess.PIPE)
+            stderr = p.stderr
+        else:
+            p = spawn_python('-i', bufsize=1, stderr=subprocess.STDOUT)
+            stderr = p.stdout
+        try:
+            # Drain stderr until prompt
+            while True:
+                data = stderr.read(4)
+                if data == b">>> ":
+                    break
+                stderr.readline()
+            yield p
+        finally:
+            kill_python(p)
+            stderr.close()
+
+    def check_repl_stdout_flush(self, separate_stderr=False):
+        with self.interactive_python(separate_stderr) as p:
+            p.stdin.write(b"print('foo')\n")
+            p.stdin.flush()
+            self.assertEqual(b'foo', p.stdout.readline().strip())
+
+    def check_repl_stderr_flush(self, separate_stderr=False):
+        with self.interactive_python(separate_stderr) as p:
+            p.stdin.write(b"1/0\n")
+            p.stdin.flush()
+            stderr = p.stderr if separate_stderr else p.stdout
+            self.assertIn(b'Traceback ', stderr.readline())
+            self.assertIn(b'File "<stdin>"', stderr.readline())
+            self.assertIn(b'ZeroDivisionError', stderr.readline())
+
+    def test_repl_stdout_flush(self):
+        self.check_repl_stdout_flush()
+
+    def test_repl_stdout_flush_separate_stderr(self):
+        self.check_repl_stdout_flush(True)
+
+    def test_repl_stderr_flush(self):
+        self.check_repl_stderr_flush()
+
+    def test_repl_stderr_flush_separate_stderr(self):
+        self.check_repl_stderr_flush(True)
+
     def test_basic_script(self):
         with temp_dir() as script_dir:
             script_name = _make_test_script(script_dir, 'script')
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #21425: Fix flushing of standard streams in the interactive
+  interpreter.
+
 - Issue #21435: In rare cases, when running finalizers on objects in cyclic
   trash a bad pointer dereference could occur due to a subtle flaw in
   internal iteration logic.
diff --git a/Python/pythonrun.c b/Python/pythonrun.c
--- a/Python/pythonrun.c
+++ b/Python/pythonrun.c
@@ -1453,12 +1453,13 @@
     d = PyModule_GetDict(m);
     v = run_mod(mod, filename, d, d, flags, arena);
     PyArena_Free(arena);
-    flush_io();
     if (v == NULL) {
         PyErr_Print();
+        flush_io();
         return -1;
     }
     Py_DECREF(v);
+    flush_io();
     return 0;
 }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 16:09:25 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 11 May 2014 16:09:25 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_debugging_output_for_?=
	=?utf-8?q?=2321425?=
Message-ID: <3gRRwj3mDRz7LjW@mail.python.org>

http://hg.python.org/cpython/rev/ffae7aad9dfa
changeset:   90626:ffae7aad9dfa
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun May 11 16:09:15 2014 +0200
summary:
  Add debugging output for #21425

files:
  Lib/test/test_cmd_line_script.py |  3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_cmd_line_script.py b/Lib/test/test_cmd_line_script.py
--- a/Lib/test/test_cmd_line_script.py
+++ b/Lib/test/test_cmd_line_script.py
@@ -187,6 +187,9 @@
             # Drain stderr until prompt
             while True:
                 data = stderr.read(4)
+                if support.verbose:
+                    print("repl stderr[:4]:", data)
+                    sys.stdout.flush()
                 if data == b">>> ":
                     break
                 stderr.readline()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 16:24:55 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 11 May 2014 16:24:55 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_debugging_output_to_wo?=
	=?utf-8?q?rk_with_-bb?=
Message-ID: <3gRSGb5xr3z7LjP@mail.python.org>

http://hg.python.org/cpython/rev/a4b796be7786
changeset:   90627:a4b796be7786
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun May 11 16:24:45 2014 +0200
summary:
  Fix debugging output to work with -bb

files:
  Lib/test/test_cmd_line_script.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_cmd_line_script.py b/Lib/test/test_cmd_line_script.py
--- a/Lib/test/test_cmd_line_script.py
+++ b/Lib/test/test_cmd_line_script.py
@@ -188,7 +188,7 @@
             while True:
                 data = stderr.read(4)
                 if support.verbose:
-                    print("repl stderr[:4]:", data)
+                    print("repl stderr[:4]:", repr(data))
                     sys.stdout.flush()
                 if data == b">>> ":
                     break

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 16:36:27 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 11 May 2014 16:36:27 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_debugging_output_to_wo?=
	=?utf-8?q?rk_with_-W?=
Message-ID: <3gRSWv1Kvfz7LjR@mail.python.org>

http://hg.python.org/cpython/rev/02ffcf30e758
changeset:   90628:02ffcf30e758
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun May 11 16:36:22 2014 +0200
summary:
  Fix debugging output to work with -W

files:
  Lib/test/test_cmd_line_script.py |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_cmd_line_script.py b/Lib/test/test_cmd_line_script.py
--- a/Lib/test/test_cmd_line_script.py
+++ b/Lib/test/test_cmd_line_script.py
@@ -188,8 +188,8 @@
             while True:
                 data = stderr.read(4)
                 if support.verbose:
-                    print("repl stderr[:4]:", repr(data))
-                    sys.stdout.flush()
+                    print("repl stderr[:4]:", repr(data), file=sys.__stdout__)
+                    sys.__stdout__.flush()
                 if data == b">>> ":
                     break
                 stderr.readline()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 16:59:22 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 11 May 2014 16:59:22 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Try_workaround_for_test_is?=
	=?utf-8?q?sues_in_=2321425?=
Message-ID: <3gRT2L3nQyz7LjM@mail.python.org>

http://hg.python.org/cpython/rev/e57718ac8ff8
changeset:   90629:e57718ac8ff8
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun May 11 16:59:16 2014 +0200
summary:
  Try workaround for test issues in #21425

files:
  Lib/test/script_helper.py |  8 ++++++++
  1 files changed, 8 insertions(+), 0 deletions(-)


diff --git a/Lib/test/script_helper.py b/Lib/test/script_helper.py
--- a/Lib/test/script_helper.py
+++ b/Lib/test/script_helper.py
@@ -86,6 +86,14 @@
     """
     cmd_line = [sys.executable, '-E']
     cmd_line.extend(args)
+    # Under Fedora (?), GNU readline can output junk on stderr when initialized,
+    # depending on the TERM setting.  Setting TERM=vt100 is supposed to disable
+    # that.  References:
+    # - http://reinout.vanrees.org/weblog/2009/08/14/readline-invisible-character-hack.html
+    # - http://stackoverflow.com/questions/15760712/python-readline-module-prints-escape-character-during-import
+    # - http://lists.gnu.org/archive/html/bug-readline/2007-08/msg00004.html
+    env = kw.setdefault('env', {})
+    env.setdefault('TERM', 'vt100')
     return subprocess.Popen(cmd_line, stdin=subprocess.PIPE,
                             stdout=stdout, stderr=stderr,
                             **kw)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 17:30:48 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 11 May 2014 17:30:48 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Remove_debugging_output?=
Message-ID: <3gRTkc4Hxsz7Ljc@mail.python.org>

http://hg.python.org/cpython/rev/eb32214af9ae
changeset:   90630:eb32214af9ae
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun May 11 17:29:57 2014 +0200
summary:
  Remove debugging output

files:
  Lib/test/test_cmd_line_script.py |  3 ---
  1 files changed, 0 insertions(+), 3 deletions(-)


diff --git a/Lib/test/test_cmd_line_script.py b/Lib/test/test_cmd_line_script.py
--- a/Lib/test/test_cmd_line_script.py
+++ b/Lib/test/test_cmd_line_script.py
@@ -187,9 +187,6 @@
             # Drain stderr until prompt
             while True:
                 data = stderr.read(4)
-                if support.verbose:
-                    print("repl stderr[:4]:", repr(data), file=sys.__stdout__)
-                    sys.__stdout__.flush()
                 if data == b">>> ":
                     break
                 stderr.readline()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 17:30:49 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 11 May 2014 17:30:49 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Try_workaround?=
 =?utf-8?q?_for_test_issues_in_=2321425?=
Message-ID: <3gRTkd5q5Wz7Ljj@mail.python.org>

http://hg.python.org/cpython/rev/58e2116576cf
changeset:   90631:58e2116576cf
branch:      3.4
parent:      90624:ab3e012c45d0
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun May 11 16:59:16 2014 +0200
summary:
  Try workaround for test issues in #21425

files:
  Lib/test/script_helper.py |  8 ++++++++
  1 files changed, 8 insertions(+), 0 deletions(-)


diff --git a/Lib/test/script_helper.py b/Lib/test/script_helper.py
--- a/Lib/test/script_helper.py
+++ b/Lib/test/script_helper.py
@@ -86,6 +86,14 @@
     """
     cmd_line = [sys.executable, '-E']
     cmd_line.extend(args)
+    # Under Fedora (?), GNU readline can output junk on stderr when initialized,
+    # depending on the TERM setting.  Setting TERM=vt100 is supposed to disable
+    # that.  References:
+    # - http://reinout.vanrees.org/weblog/2009/08/14/readline-invisible-character-hack.html
+    # - http://stackoverflow.com/questions/15760712/python-readline-module-prints-escape-character-during-import
+    # - http://lists.gnu.org/archive/html/bug-readline/2007-08/msg00004.html
+    env = kw.setdefault('env', {})
+    env.setdefault('TERM', 'vt100')
     return subprocess.Popen(cmd_line, stdin=subprocess.PIPE,
                             stdout=stdout, stderr=stderr,
                             **kw)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 17:30:51 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 11 May 2014 17:30:51 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?q?=29=3A_Null_merge?=
Message-ID: <3gRTkg02BJz7Ljr@mail.python.org>

http://hg.python.org/cpython/rev/2b292d6e536a
changeset:   90632:2b292d6e536a
parent:      90630:eb32214af9ae
parent:      90631:58e2116576cf
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun May 11 17:30:41 2014 +0200
summary:
  Null merge

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 19:09:02 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 11 May 2014 19:09:02 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Fix_printing_o?=
 =?utf-8?q?ut_error_message_when_test_fails_and_run_with_-bb?=
Message-ID: <3gRWvy4Vvkz7Lk3@mail.python.org>

http://hg.python.org/cpython/rev/8e9a00257305
changeset:   90633:8e9a00257305
branch:      3.4
parent:      90631:58e2116576cf
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun May 11 19:05:23 2014 +0200
summary:
  Fix printing out error message when test fails and run with -bb

files:
  Lib/test/test_signal.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_signal.py b/Lib/test/test_signal.py
--- a/Lib/test/test_signal.py
+++ b/Lib/test/test_signal.py
@@ -454,7 +454,7 @@
                 stdout = first_line + stdout
                 exitcode = process.wait()
                 if exitcode not in (2, 3):
-                    raise Exception("Child error (exit code %s): %s"
+                    raise Exception("Child error (exit code %s): %r"
                                     % (exitcode, stdout))
                 return (exitcode == 3)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 19:09:03 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 11 May 2014 19:09:03 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Fix_printing_out_error_message_when_test_fails_and_run_w?=
 =?utf-8?q?ith_-bb?=
Message-ID: <3gRWvz68Knz7Lk4@mail.python.org>

http://hg.python.org/cpython/rev/201fa77cef98
changeset:   90634:201fa77cef98
parent:      90632:2b292d6e536a
parent:      90633:8e9a00257305
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun May 11 19:05:50 2014 +0200
summary:
  Fix printing out error message when test fails and run with -bb

files:
  Lib/test/test_signal.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_signal.py b/Lib/test/test_signal.py
--- a/Lib/test/test_signal.py
+++ b/Lib/test/test_signal.py
@@ -473,7 +473,7 @@
                 stdout = first_line + stdout
                 exitcode = process.wait()
                 if exitcode not in (2, 3):
-                    raise Exception("Child error (exit code %s): %s"
+                    raise Exception("Child error (exit code %s): %r"
                                     % (exitcode, stdout))
                 return (exitcode == 3)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 19:13:50 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 11 May 2014 19:13:50 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Try_to_fix_issue_=2321425_?=
 =?utf-8?q?workaround_for_shared_library_builds?=
Message-ID: <3gRX1V1LC0z7LjM@mail.python.org>

http://hg.python.org/cpython/rev/5a1b2566d68e
changeset:   90635:5a1b2566d68e
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun May 11 19:13:43 2014 +0200
summary:
  Try to fix issue #21425 workaround for shared library builds

files:
  Lib/test/script_helper.py |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Lib/test/script_helper.py b/Lib/test/script_helper.py
--- a/Lib/test/script_helper.py
+++ b/Lib/test/script_helper.py
@@ -92,8 +92,8 @@
     # - http://reinout.vanrees.org/weblog/2009/08/14/readline-invisible-character-hack.html
     # - http://stackoverflow.com/questions/15760712/python-readline-module-prints-escape-character-during-import
     # - http://lists.gnu.org/archive/html/bug-readline/2007-08/msg00004.html
-    env = kw.setdefault('env', {})
-    env.setdefault('TERM', 'vt100')
+    env = kw.setdefault('env', dict(os.environ))
+    env['TERM'] = 'vt100'
     return subprocess.Popen(cmd_line, stdin=subprocess.PIPE,
                             stdout=stdout, stderr=stderr,
                             **kw)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 19:39:24 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 11 May 2014 19:39:24 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Try_to_fix_iss?=
 =?utf-8?q?ue_=2321425_workaround_for_shared_library_builds?=
Message-ID: <3gRXb02JSVz7LjX@mail.python.org>

http://hg.python.org/cpython/rev/974c0718c7e0
changeset:   90636:974c0718c7e0
branch:      3.4
parent:      90633:8e9a00257305
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun May 11 19:13:43 2014 +0200
summary:
  Try to fix issue #21425 workaround for shared library builds

files:
  Lib/test/script_helper.py |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Lib/test/script_helper.py b/Lib/test/script_helper.py
--- a/Lib/test/script_helper.py
+++ b/Lib/test/script_helper.py
@@ -92,8 +92,8 @@
     # - http://reinout.vanrees.org/weblog/2009/08/14/readline-invisible-character-hack.html
     # - http://stackoverflow.com/questions/15760712/python-readline-module-prints-escape-character-during-import
     # - http://lists.gnu.org/archive/html/bug-readline/2007-08/msg00004.html
-    env = kw.setdefault('env', {})
-    env.setdefault('TERM', 'vt100')
+    env = kw.setdefault('env', dict(os.environ))
+    env['TERM'] = 'vt100'
     return subprocess.Popen(cmd_line, stdin=subprocess.PIPE,
                             stdout=stdout, stderr=stderr,
                             **kw)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 19:39:25 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 11 May 2014 19:39:25 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?q?=29=3A_Null_merge?=
Message-ID: <3gRXb13ynhz7Ljc@mail.python.org>

http://hg.python.org/cpython/rev/4d2cbf332ec0
changeset:   90637:4d2cbf332ec0
parent:      90635:5a1b2566d68e
parent:      90636:974c0718c7e0
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun May 11 19:39:13 2014 +0200
summary:
  Null merge

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 20:19:23 2014
From: python-checkins at python.org (benjamin.peterson)
Date: Sun, 11 May 2014 20:19:23 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_remove_confusi?=
 =?utf-8?q?ng_delete_indexing_=28closes_=2321466=29?=
Message-ID: <3gRYT76kt4z7Ljd@mail.python.org>

http://hg.python.org/cpython/rev/951775c68b1b
changeset:   90638:951775c68b1b
branch:      2.7
parent:      90621:730eeb10cd81
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun May 11 11:18:51 2014 -0700
summary:
  remove confusing delete indexing (closes #21466)

files:
  Doc/reference/datamodel.rst |  2 --
  1 files changed, 0 insertions(+), 2 deletions(-)


diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst
--- a/Doc/reference/datamodel.rst
+++ b/Doc/reference/datamodel.rst
@@ -356,8 +356,6 @@
          object: mutable sequence
          object: mutable
          pair: assignment; statement
-         single: delete
-         statement: del
          single: subscription
          single: slicing
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 20:19:25 2014
From: python-checkins at python.org (benjamin.peterson)
Date: Sun, 11 May 2014 20:19:25 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_remove_confusi?=
 =?utf-8?q?ng_delete_indexing_=28closes_=2321466=29?=
Message-ID: <3gRYT91Ppgz7Ljh@mail.python.org>

http://hg.python.org/cpython/rev/2e26703d7d2a
changeset:   90639:2e26703d7d2a
branch:      3.4
parent:      90636:974c0718c7e0
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun May 11 11:18:51 2014 -0700
summary:
  remove confusing delete indexing (closes #21466)

files:
  Doc/reference/datamodel.rst |  2 --
  1 files changed, 0 insertions(+), 2 deletions(-)


diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst
--- a/Doc/reference/datamodel.rst
+++ b/Doc/reference/datamodel.rst
@@ -323,8 +323,6 @@
          object: mutable sequence
          object: mutable
          pair: assignment; statement
-         single: delete
-         statement: del
          single: subscription
          single: slicing
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 20:19:26 2014
From: python-checkins at python.org (benjamin.peterson)
Date: Sun, 11 May 2014 20:19:26 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?b?KTogbWVyZ2UgMy40?=
Message-ID: <3gRYTB3Rjhz7Ljr@mail.python.org>

http://hg.python.org/cpython/rev/8ff31935ae07
changeset:   90640:8ff31935ae07
parent:      90637:4d2cbf332ec0
parent:      90639:2e26703d7d2a
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun May 11 11:19:17 2014 -0700
summary:
  merge 3.4

files:
  Doc/reference/datamodel.rst |  2 --
  1 files changed, 0 insertions(+), 2 deletions(-)


diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst
--- a/Doc/reference/datamodel.rst
+++ b/Doc/reference/datamodel.rst
@@ -323,8 +323,6 @@
          object: mutable sequence
          object: mutable
          pair: assignment; statement
-         single: delete
-         statement: del
          single: subscription
          single: slicing
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 22:29:44 2014
From: python-checkins at python.org (gregory.p.smith)
Date: Sun, 11 May 2014 22:29:44 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Remove_the_war?=
 =?utf-8?q?ning-soup_from_the_subprocess_documentation_by_adding?=
Message-ID: <3gRcMX3psRz7Ljc@mail.python.org>

http://hg.python.org/cpython/rev/ccdc70b66bd5
changeset:   90641:ccdc70b66bd5
branch:      3.4
parent:      90639:2e26703d7d2a
user:        Gregory P. Smith <greg at krypto.org>
date:        Sun May 11 13:26:21 2014 -0700
summary:
  Remove the warning-soup from the subprocess documentation by adding
a Security Considerations section as preferred by both the devguide
and documentation users who do not wish to go insane.

files:
  Doc/library/subprocess.rst |  127 +++++++++---------------
  1 files changed, 48 insertions(+), 79 deletions(-)


diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst
--- a/Doc/library/subprocess.rst
+++ b/Doc/library/subprocess.rst
@@ -54,18 +54,12 @@
       >>> subprocess.call("exit 1", shell=True)
       1
 
-   .. warning::
-
-      Invoking the system shell with ``shell=True`` can be a security hazard
-      if combined with untrusted input. See the warning under
-      :ref:`frequently-used-arguments` for details.
-
    .. note::
 
-      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this function. As
-      the pipes are not being read in the current process, the child
-      process may block if it generates enough output to a pipe to fill up
-      the OS pipe buffer.
+      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this
+      function.  The child process will block if it generates enough
+      output to a pipe to fill up the OS pipe buffer as the pipes are
+      not being read from.
 
    .. versionchanged:: 3.3
       *timeout* was added.
@@ -99,18 +93,12 @@
          ...
       subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1
 
-   .. warning::
-
-      Invoking the system shell with ``shell=True`` can be a security hazard
-      if combined with untrusted input. See the warning under
-      :ref:`frequently-used-arguments` for details.
-
    .. note::
 
-      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this function. As
-      the pipes are not being read in the current process, the child
-      process may block if it generates enough output to a pipe to fill up
-      the OS pipe buffer.
+      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this
+      function.  The child process will block if it generates enough
+      output to a pipe to fill up the OS pipe buffer as the pipes are
+      not being read from.
 
    .. versionchanged:: 3.3
       *timeout* was added.
@@ -177,17 +165,12 @@
       ...     shell=True)
       'ls: non_existent_file: No such file or directory\n'
 
-   .. warning::
-
-      Invoking the system shell with ``shell=True`` can be a security hazard
-      if combined with untrusted input. See the warning under
-      :ref:`frequently-used-arguments` for details.
-
    .. note::
 
-      Do not use ``stderr=PIPE`` with this function. As the pipe is not being
-      read in the current process, the child process may block if it
-      generates enough output to the pipe to fill up the OS pipe buffer.
+      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this
+      function.  The child process will block if it generates enough
+      output to a pipe to fill up the OS pipe buffer as the pipes are
+      not being read from.
 
    .. versionadded:: 3.1
 
@@ -210,7 +193,7 @@
 
    Special value that can be used as the *stdin*, *stdout* or *stderr* argument
    to :class:`Popen` and indicates that a pipe to the standard stream should be
-   opened.
+   opened.  Most useful with :meth:`Popen.communicate`.
 
 
 .. data:: STDOUT
@@ -336,28 +319,9 @@
       instead of ``locale.getpreferredencoding()``.  See the
       :class:`io.TextIOWrapper` class for more information on this change.
 
-   .. warning::
+   .. note::
 
-      Executing shell commands that incorporate unsanitized input from an
-      untrusted source makes a program vulnerable to `shell injection
-      <http://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_,
-      a serious security flaw which can result in arbitrary command execution.
-      For this reason, the use of ``shell=True`` is **strongly discouraged**
-      in cases where the command string is constructed from external input::
-
-         >>> from subprocess import call
-         >>> filename = input("What file would you like to display?\n")
-         What file would you like to display?
-         non_existent; rm -rf / #
-         >>> call("cat " + filename, shell=True) # Uh-oh. This will end badly...
-
-      ``shell=False`` disables all shell based features, but does not suffer
-      from this vulnerability; see the Note in the :class:`Popen` constructor
-      documentation for helpful hints in getting ``shell=False`` to work.
-
-      When using ``shell=True``, :func:`shlex.quote` can be used to properly
-      escape whitespace and shell metacharacters in strings that are going to
-      be used to construct shell commands.
+      Read the `Security Considerations`_ section before using ``shell=True``.
 
 These options, along with all of the other options, are described in more
 detail in the :class:`Popen` constructor documentation.
@@ -438,11 +402,9 @@
    into the shell (e.g. :command:`dir` or :command:`copy`).  You do not need
    ``shell=True`` to run a batch file or console-based executable.
 
-   .. warning::
+   .. note::
 
-      Passing ``shell=True`` can be a security hazard if combined with
-      untrusted input.  See the warning under :ref:`frequently-used-arguments`
-      for details.
+      Read the `Security Considerations`_ section before using ``shell=True``.
 
    *bufsize* will be supplied as the corresponding argument to the :func:`open`
    function when creating the stdin/stdout/stderr pipe file objects: :const:`0`
@@ -598,14 +560,21 @@
       The :exc:`SubprocessError` base class was added.
 
 
-Security
-^^^^^^^^
+Security Considerations
+-----------------------
 
-Unlike some other popen functions, this implementation will never call a
-system shell implicitly.  This means that all characters, including shell
-metacharacters, can safely be passed to child processes. Obviously, if the
-shell is invoked explicitly, then it is the application's responsibility to
-ensure that all whitespace and metacharacters are quoted appropriately.
+Unlike some other popen functions, this implementation will never
+implicitly call a system shell.  This means that all characters,
+including shell metacharacters, can safely be passed to child processes.
+If the shell is invoked explicitly, via ``shell=True``, it is the application's
+responsibility to ensure that all whitespace and metacharacters are
+quoted appropriately to avoid
+`shell injection <http://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_
+vulnerabilities.
+
+When using ``shell=True``, the :func:`shlex.quote` function can be
+used to properly escape whitespace and shell metacharacters in strings
+that are going to be used to construct shell commands.
 
 
 Popen Objects
@@ -631,25 +600,25 @@
 
    .. note::
 
+      This will deadlock when using ``stdout=PIPE`` or ``stderr=PIPE``
+      and the child process generates enough output to a pipe such that
+      it blocks waiting for the OS pipe buffer to accept more data.
+      Use :meth:`Popen.communicate` when using pipes to avoid that.
+
+   .. note::
+
       The function is implemented using a busy loop (non-blocking call and
       short sleeps). Use the :mod:`asyncio` module for an asynchronous wait:
       see :class:`asyncio.create_subprocess_exec`.
 
-   .. warning::
-
-      This will deadlock when using ``stdout=PIPE`` and/or
-      ``stderr=PIPE`` and the child process generates enough output to
-      a pipe such that it blocks waiting for the OS pipe buffer to
-      accept more data.  Use :meth:`communicate` to avoid that.
-
    .. versionchanged:: 3.3
       *timeout* was added.
 
    .. deprecated:: 3.4
 
-      Do not use the undocumented *endtime* parameter.  It is was
-      unintentionally exposed in 3.3 but was intended to be private
-      for internal use.  Use *timeout* instead.
+      Do not use the *endtime* parameter.  It is was unintentionally
+      exposed in 3.3 but was left undocumented as it was intended to be
+      private for internal use.  Use *timeout* instead.
 
 .. method:: Popen.communicate(input=None, timeout=None)
 
@@ -716,13 +685,6 @@
 
 The following attributes are also available:
 
-.. warning::
-
-   Use :meth:`~Popen.communicate` rather than :attr:`.stdin.write <Popen.stdin>`,
-   :attr:`.stdout.read <Popen.stdout>` or :attr:`.stderr.read <Popen.stderr>` to avoid
-   deadlocks due to any of the other OS pipe buffers filling up and blocking the
-   child process.
-
 .. attribute:: Popen.args
 
    The *args* argument as it was passed to :class:`Popen` -- a
@@ -756,6 +718,13 @@
    ``True``, the stream is a text stream, otherwise it is a byte stream. If the
    *stderr* argument was not :data:`PIPE`, this attribute is ``None``.
 
+.. warning::
+
+   Use :meth:`~Popen.communicate` rather than :attr:`.stdin.write <Popen.stdin>`,
+   :attr:`.stdout.read <Popen.stdout>` or :attr:`.stderr.read <Popen.stderr>` to avoid
+   deadlocks due to any of the other OS pipe buffers filling up and blocking the
+   child process.
+
 
 .. attribute:: Popen.pid
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 22:29:45 2014
From: python-checkins at python.org (gregory.p.smith)
Date: Sun, 11 May 2014 22:29:45 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Change_all_ref?=
 =?utf-8?q?erences_to_Unix_to_POSIX_in_the_subprocess_docs=2E__It=27s?=
Message-ID: <3gRcMY6gNGz7Ljc@mail.python.org>

http://hg.python.org/cpython/rev/585dfa357fc4
changeset:   90642:585dfa357fc4
branch:      3.4
user:        Gregory P. Smith <greg at krypto.org>
date:        Sun May 11 13:28:35 2014 -0700
summary:
  Change all references to Unix to POSIX in the subprocess docs.  It's
more accurate and sounds less like a strange tale of yore.

files:
  Doc/library/subprocess.rst |  28 +++++++++++++-------------
  1 files changed, 14 insertions(+), 14 deletions(-)


diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst
--- a/Doc/library/subprocess.rst
+++ b/Doc/library/subprocess.rst
@@ -342,7 +342,7 @@
                  startupinfo=None, creationflags=0, restore_signals=True, \
                  start_new_session=False, pass_fds=())
 
-   Execute a child program in a new process.  On Unix, the class uses
+   Execute a child program in a new process.  On POSIX, the class uses
    :meth:`os.execvp`-like behavior to execute the child program.  On Windows,
    the class uses the Windows ``CreateProcess()`` function.  The arguments to
    :class:`Popen` are as follows.
@@ -354,7 +354,7 @@
    arguments for additional differences from the default behavior.  Unless
    otherwise stated, it is recommended to pass *args* as a sequence.
 
-   On Unix, if *args* is a string, the string is interpreted as the name or
+   On POSIX, if *args* is a string, the string is interpreted as the name or
    path of the program to execute.  However, this can only be done if not
    passing arguments to the program.
 
@@ -385,7 +385,7 @@
    the shell as the program to execute.  If *shell* is *True*, it is
    recommended to pass *args* as a string rather than as a sequence.
 
-   On Unix with ``shell=True``, the shell defaults to :file:`/bin/sh`.  If
+   On POSIX with ``shell=True``, the shell defaults to :file:`/bin/sh`.  If
    *args* is a string, the string specifies the command
    to execute through the shell.  This means that the string must be
    formatted exactly as it would be when typed at the shell prompt.  This
@@ -425,9 +425,9 @@
    program to execute specified by *args*.  However, the original *args* is
    still passed to the program.  Most programs treat the program specified
    by *args* as the command name, which can then be different from the program
-   actually executed.  On Unix, the *args* name
+   actually executed.  On POSIX, the *args* name
    becomes the display name for the executable in utilities such as
-   :program:`ps`.  If ``shell=True``, on Unix the *executable* argument
+   :program:`ps`.  If ``shell=True``, on POSIX the *executable* argument
    specifies a replacement shell for the default :file:`/bin/sh`.
 
    *stdin*, *stdout* and *stderr* specify the executed program's standard input,
@@ -443,7 +443,7 @@
 
    If *preexec_fn* is set to a callable object, this object will be called in the
    child process just before the child is executed.
-   (Unix only)
+   (POSIX only)
 
    .. warning::
 
@@ -461,8 +461,8 @@
       common use of *preexec_fn* to call os.setsid() in the child.
 
    If *close_fds* is true, all file descriptors except :const:`0`, :const:`1` and
-   :const:`2` will be closed before the child process is executed. (Unix only).
-   The default varies by platform:  Always true on Unix.  On Windows it is
+   :const:`2` will be closed before the child process is executed. (POSIX only).
+   The default varies by platform:  Always true on POSIX.  On Windows it is
    true when *stdin*/*stdout*/*stderr* are :const:`None`, false otherwise.
    On Windows, if *close_fds* is true then no handles will be inherited by the
    child process.  Note that on Windows, you cannot set *close_fds* to true and
@@ -474,7 +474,7 @@
 
    *pass_fds* is an optional sequence of file descriptors to keep open
    between the parent and child.  Providing any *pass_fds* forces
-   *close_fds* to be :const:`True`.  (Unix only)
+   *close_fds* to be :const:`True`.  (POSIX only)
 
    .. versionadded:: 3.2
       The *pass_fds* parameter was added.
@@ -487,13 +487,13 @@
    If *restore_signals* is true (the default) all signals that Python has set to
    SIG_IGN are restored to SIG_DFL in the child process before the exec.
    Currently this includes the SIGPIPE, SIGXFZ and SIGXFSZ signals.
-   (Unix only)
+   (POSIX only)
 
    .. versionchanged:: 3.2
       *restore_signals* was added.
 
    If *start_new_session* is true the setsid() system call will be made in the
-   child process prior to the execution of the subprocess.  (Unix only)
+   child process prior to the execution of the subprocess.  (POSIX only)
 
    .. versionchanged:: 3.2
       *start_new_session* was added.
@@ -741,7 +741,7 @@
    hasn't terminated yet.
 
    A negative value ``-N`` indicates that the child was terminated by signal
-   ``N`` (Unix only).
+   ``N`` (POSIX only).
 
 
 Windows Popen Helpers
@@ -1066,7 +1066,7 @@
       >>> subprocess.getstatusoutput('/bin/junk')
       (256, 'sh: /bin/junk: not found')
 
-   Availability: Unix & Windows
+   Availability: POSIX & Windows
 
    .. versionchanged:: 3.3.4
       Windows support added
@@ -1082,7 +1082,7 @@
       >>> subprocess.getoutput('ls /bin/ls')
       '/bin/ls'
 
-   Availability: Unix & Windows
+   Availability: POSIX & Windows
 
    .. versionchanged:: 3.3.4
       Windows support added

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 22:29:47 2014
From: python-checkins at python.org (gregory.p.smith)
Date: Sun, 11 May 2014 22:29:47 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_merge_from_3=2E4_-_clean_up_the_subprocess_docs_warning-?=
 =?utf-8?q?soup_and?=
Message-ID: <3gRcMb2ZcYz7Ljs@mail.python.org>

http://hg.python.org/cpython/rev/d7d4a3c68d6b
changeset:   90643:d7d4a3c68d6b
parent:      90640:8ff31935ae07
parent:      90642:585dfa357fc4
user:        Gregory P. Smith <greg at krypto.org>
date:        Sun May 11 13:29:36 2014 -0700
summary:
  merge from 3.4 - clean up the subprocess docs warning-soup and
s/Unix/POSIX/.

files:
  Doc/library/subprocess.rst |  155 ++++++++++---------------
  1 files changed, 62 insertions(+), 93 deletions(-)


diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst
--- a/Doc/library/subprocess.rst
+++ b/Doc/library/subprocess.rst
@@ -54,18 +54,12 @@
       >>> subprocess.call("exit 1", shell=True)
       1
 
-   .. warning::
-
-      Invoking the system shell with ``shell=True`` can be a security hazard
-      if combined with untrusted input. See the warning under
-      :ref:`frequently-used-arguments` for details.
-
    .. note::
 
-      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this function. As
-      the pipes are not being read in the current process, the child
-      process may block if it generates enough output to a pipe to fill up
-      the OS pipe buffer.
+      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this
+      function.  The child process will block if it generates enough
+      output to a pipe to fill up the OS pipe buffer as the pipes are
+      not being read from.
 
    .. versionchanged:: 3.3
       *timeout* was added.
@@ -99,18 +93,12 @@
          ...
       subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1
 
-   .. warning::
-
-      Invoking the system shell with ``shell=True`` can be a security hazard
-      if combined with untrusted input. See the warning under
-      :ref:`frequently-used-arguments` for details.
-
    .. note::
 
-      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this function. As
-      the pipes are not being read in the current process, the child
-      process may block if it generates enough output to a pipe to fill up
-      the OS pipe buffer.
+      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this
+      function.  The child process will block if it generates enough
+      output to a pipe to fill up the OS pipe buffer as the pipes are
+      not being read from.
 
    .. versionchanged:: 3.3
       *timeout* was added.
@@ -177,17 +165,12 @@
       ...     shell=True)
       'ls: non_existent_file: No such file or directory\n'
 
-   .. warning::
-
-      Invoking the system shell with ``shell=True`` can be a security hazard
-      if combined with untrusted input. See the warning under
-      :ref:`frequently-used-arguments` for details.
-
    .. note::
 
-      Do not use ``stderr=PIPE`` with this function. As the pipe is not being
-      read in the current process, the child process may block if it
-      generates enough output to the pipe to fill up the OS pipe buffer.
+      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this
+      function.  The child process will block if it generates enough
+      output to a pipe to fill up the OS pipe buffer as the pipes are
+      not being read from.
 
    .. versionadded:: 3.1
 
@@ -210,7 +193,7 @@
 
    Special value that can be used as the *stdin*, *stdout* or *stderr* argument
    to :class:`Popen` and indicates that a pipe to the standard stream should be
-   opened.
+   opened.  Most useful with :meth:`Popen.communicate`.
 
 
 .. data:: STDOUT
@@ -336,28 +319,9 @@
       instead of ``locale.getpreferredencoding()``.  See the
       :class:`io.TextIOWrapper` class for more information on this change.
 
-   .. warning::
+   .. note::
 
-      Executing shell commands that incorporate unsanitized input from an
-      untrusted source makes a program vulnerable to `shell injection
-      <http://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_,
-      a serious security flaw which can result in arbitrary command execution.
-      For this reason, the use of ``shell=True`` is **strongly discouraged**
-      in cases where the command string is constructed from external input::
-
-         >>> from subprocess import call
-         >>> filename = input("What file would you like to display?\n")
-         What file would you like to display?
-         non_existent; rm -rf / #
-         >>> call("cat " + filename, shell=True) # Uh-oh. This will end badly...
-
-      ``shell=False`` disables all shell based features, but does not suffer
-      from this vulnerability; see the Note in the :class:`Popen` constructor
-      documentation for helpful hints in getting ``shell=False`` to work.
-
-      When using ``shell=True``, :func:`shlex.quote` can be used to properly
-      escape whitespace and shell metacharacters in strings that are going to
-      be used to construct shell commands.
+      Read the `Security Considerations`_ section before using ``shell=True``.
 
 These options, along with all of the other options, are described in more
 detail in the :class:`Popen` constructor documentation.
@@ -378,7 +342,7 @@
                  startupinfo=None, creationflags=0, restore_signals=True, \
                  start_new_session=False, pass_fds=())
 
-   Execute a child program in a new process.  On Unix, the class uses
+   Execute a child program in a new process.  On POSIX, the class uses
    :meth:`os.execvp`-like behavior to execute the child program.  On Windows,
    the class uses the Windows ``CreateProcess()`` function.  The arguments to
    :class:`Popen` are as follows.
@@ -390,7 +354,7 @@
    arguments for additional differences from the default behavior.  Unless
    otherwise stated, it is recommended to pass *args* as a sequence.
 
-   On Unix, if *args* is a string, the string is interpreted as the name or
+   On POSIX, if *args* is a string, the string is interpreted as the name or
    path of the program to execute.  However, this can only be done if not
    passing arguments to the program.
 
@@ -421,7 +385,7 @@
    the shell as the program to execute.  If *shell* is *True*, it is
    recommended to pass *args* as a string rather than as a sequence.
 
-   On Unix with ``shell=True``, the shell defaults to :file:`/bin/sh`.  If
+   On POSIX with ``shell=True``, the shell defaults to :file:`/bin/sh`.  If
    *args* is a string, the string specifies the command
    to execute through the shell.  This means that the string must be
    formatted exactly as it would be when typed at the shell prompt.  This
@@ -438,11 +402,9 @@
    into the shell (e.g. :command:`dir` or :command:`copy`).  You do not need
    ``shell=True`` to run a batch file or console-based executable.
 
-   .. warning::
+   .. note::
 
-      Passing ``shell=True`` can be a security hazard if combined with
-      untrusted input.  See the warning under :ref:`frequently-used-arguments`
-      for details.
+      Read the `Security Considerations`_ section before using ``shell=True``.
 
    *bufsize* will be supplied as the corresponding argument to the :func:`open`
    function when creating the stdin/stdout/stderr pipe file objects: :const:`0`
@@ -463,9 +425,9 @@
    program to execute specified by *args*.  However, the original *args* is
    still passed to the program.  Most programs treat the program specified
    by *args* as the command name, which can then be different from the program
-   actually executed.  On Unix, the *args* name
+   actually executed.  On POSIX, the *args* name
    becomes the display name for the executable in utilities such as
-   :program:`ps`.  If ``shell=True``, on Unix the *executable* argument
+   :program:`ps`.  If ``shell=True``, on POSIX the *executable* argument
    specifies a replacement shell for the default :file:`/bin/sh`.
 
    *stdin*, *stdout* and *stderr* specify the executed program's standard input,
@@ -481,7 +443,7 @@
 
    If *preexec_fn* is set to a callable object, this object will be called in the
    child process just before the child is executed.
-   (Unix only)
+   (POSIX only)
 
    .. warning::
 
@@ -499,8 +461,8 @@
       common use of *preexec_fn* to call os.setsid() in the child.
 
    If *close_fds* is true, all file descriptors except :const:`0`, :const:`1` and
-   :const:`2` will be closed before the child process is executed. (Unix only).
-   The default varies by platform:  Always true on Unix.  On Windows it is
+   :const:`2` will be closed before the child process is executed. (POSIX only).
+   The default varies by platform:  Always true on POSIX.  On Windows it is
    true when *stdin*/*stdout*/*stderr* are :const:`None`, false otherwise.
    On Windows, if *close_fds* is true then no handles will be inherited by the
    child process.  Note that on Windows, you cannot set *close_fds* to true and
@@ -512,7 +474,7 @@
 
    *pass_fds* is an optional sequence of file descriptors to keep open
    between the parent and child.  Providing any *pass_fds* forces
-   *close_fds* to be :const:`True`.  (Unix only)
+   *close_fds* to be :const:`True`.  (POSIX only)
 
    .. versionadded:: 3.2
       The *pass_fds* parameter was added.
@@ -525,13 +487,13 @@
    If *restore_signals* is true (the default) all signals that Python has set to
    SIG_IGN are restored to SIG_DFL in the child process before the exec.
    Currently this includes the SIGPIPE, SIGXFZ and SIGXFSZ signals.
-   (Unix only)
+   (POSIX only)
 
    .. versionchanged:: 3.2
       *restore_signals* was added.
 
    If *start_new_session* is true the setsid() system call will be made in the
-   child process prior to the execution of the subprocess.  (Unix only)
+   child process prior to the execution of the subprocess.  (POSIX only)
 
    .. versionchanged:: 3.2
       *start_new_session* was added.
@@ -598,14 +560,21 @@
       The :exc:`SubprocessError` base class was added.
 
 
-Security
-^^^^^^^^
+Security Considerations
+-----------------------
 
-Unlike some other popen functions, this implementation will never call a
-system shell implicitly.  This means that all characters, including shell
-metacharacters, can safely be passed to child processes. Obviously, if the
-shell is invoked explicitly, then it is the application's responsibility to
-ensure that all whitespace and metacharacters are quoted appropriately.
+Unlike some other popen functions, this implementation will never
+implicitly call a system shell.  This means that all characters,
+including shell metacharacters, can safely be passed to child processes.
+If the shell is invoked explicitly, via ``shell=True``, it is the application's
+responsibility to ensure that all whitespace and metacharacters are
+quoted appropriately to avoid
+`shell injection <http://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_
+vulnerabilities.
+
+When using ``shell=True``, the :func:`shlex.quote` function can be
+used to properly escape whitespace and shell metacharacters in strings
+that are going to be used to construct shell commands.
 
 
 Popen Objects
@@ -631,25 +600,25 @@
 
    .. note::
 
+      This will deadlock when using ``stdout=PIPE`` or ``stderr=PIPE``
+      and the child process generates enough output to a pipe such that
+      it blocks waiting for the OS pipe buffer to accept more data.
+      Use :meth:`Popen.communicate` when using pipes to avoid that.
+
+   .. note::
+
       The function is implemented using a busy loop (non-blocking call and
       short sleeps). Use the :mod:`asyncio` module for an asynchronous wait:
       see :class:`asyncio.create_subprocess_exec`.
 
-   .. warning::
-
-      This will deadlock when using ``stdout=PIPE`` and/or
-      ``stderr=PIPE`` and the child process generates enough output to
-      a pipe such that it blocks waiting for the OS pipe buffer to
-      accept more data.  Use :meth:`communicate` to avoid that.
-
    .. versionchanged:: 3.3
       *timeout* was added.
 
    .. deprecated:: 3.4
 
-      Do not use the undocumented *endtime* parameter.  It is was
-      unintentionally exposed in 3.3 but was intended to be private
-      for internal use.  Use *timeout* instead.
+      Do not use the *endtime* parameter.  It is was unintentionally
+      exposed in 3.3 but was left undocumented as it was intended to be
+      private for internal use.  Use *timeout* instead.
 
 .. method:: Popen.communicate(input=None, timeout=None)
 
@@ -717,13 +686,6 @@
 
 The following attributes are also available:
 
-.. warning::
-
-   Use :meth:`~Popen.communicate` rather than :attr:`.stdin.write <Popen.stdin>`,
-   :attr:`.stdout.read <Popen.stdout>` or :attr:`.stderr.read <Popen.stderr>` to avoid
-   deadlocks due to any of the other OS pipe buffers filling up and blocking the
-   child process.
-
 .. attribute:: Popen.args
 
    The *args* argument as it was passed to :class:`Popen` -- a
@@ -757,6 +719,13 @@
    ``True``, the stream is a text stream, otherwise it is a byte stream. If the
    *stderr* argument was not :data:`PIPE`, this attribute is ``None``.
 
+.. warning::
+
+   Use :meth:`~Popen.communicate` rather than :attr:`.stdin.write <Popen.stdin>`,
+   :attr:`.stdout.read <Popen.stdout>` or :attr:`.stderr.read <Popen.stderr>` to avoid
+   deadlocks due to any of the other OS pipe buffers filling up and blocking the
+   child process.
+
 
 .. attribute:: Popen.pid
 
@@ -773,7 +742,7 @@
    hasn't terminated yet.
 
    A negative value ``-N`` indicates that the child was terminated by signal
-   ``N`` (Unix only).
+   ``N`` (POSIX only).
 
 
 Windows Popen Helpers
@@ -1098,7 +1067,7 @@
       >>> subprocess.getstatusoutput('/bin/junk')
       (256, 'sh: /bin/junk: not found')
 
-   Availability: Unix & Windows
+   Availability: POSIX & Windows
 
    .. versionchanged:: 3.3.4
       Windows support added
@@ -1114,7 +1083,7 @@
       >>> subprocess.getoutput('ls /bin/ls')
       '/bin/ls'
 
-   Availability: Unix & Windows
+   Availability: POSIX & Windows
 
    .. versionchanged:: 3.3.4
       Windows support added

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun May 11 23:21:30 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Sun, 11 May 2014 23:21:30 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_21424=3A__Apply_the_?=
 =?utf-8?q?nlargest=28=29_optimizations_to_nsmallest=28=29_as_well=2E?=
Message-ID: <3gRdWG3Zdkz7LjY@mail.python.org>

http://hg.python.org/cpython/rev/31950174f60f
changeset:   90644:31950174f60f
user:        Raymond Hettinger <python at rcn.com>
date:        Sun May 11 14:21:23 2014 -0700
summary:
  Issue 21424:  Apply the nlargest() optimizations to nsmallest() as well.

files:
  Lib/heapq.py           |  156 +++++++++++++++++++++-------
  Lib/test/test_heapq.py |    2 +-
  Misc/NEWS              |    4 +-
  Modules/_heapqmodule.c |   94 +++-------------
  4 files changed, 138 insertions(+), 118 deletions(-)


diff --git a/Lib/heapq.py b/Lib/heapq.py
--- a/Lib/heapq.py
+++ b/Lib/heapq.py
@@ -127,7 +127,7 @@
 __all__ = ['heappush', 'heappop', 'heapify', 'heapreplace', 'merge',
            'nlargest', 'nsmallest', 'heappushpop']
 
-from itertools import islice, count, tee, chain
+from itertools import islice, count
 
 def heappush(heap, item):
     """Push item onto heap, maintaining the heap invariant."""
@@ -179,12 +179,12 @@
     for i in reversed(range(n//2)):
         _siftup(x, i)
 
-def _heappushpop_max(heap, item):
-    """Maxheap version of a heappush followed by a heappop."""
-    if heap and item < heap[0]:
-        item, heap[0] = heap[0], item
-        _siftup_max(heap, 0)
-    return item
+def _heapreplace_max(heap, item):
+    """Maxheap version of a heappop followed by a heappush."""
+    returnitem = heap[0]    # raises appropriate IndexError if heap is empty
+    heap[0] = item
+    _siftup_max(heap, 0)
+    return returnitem
 
 def _heapify_max(x):
     """Transform list into a maxheap, in-place, in O(len(x)) time."""
@@ -192,24 +192,6 @@
     for i in reversed(range(n//2)):
         _siftup_max(x, i)
 
-def nsmallest(n, iterable):
-    """Find the n smallest elements in a dataset.
-
-    Equivalent to:  sorted(iterable)[:n]
-    """
-    if n <= 0:
-        return []
-    it = iter(iterable)
-    result = list(islice(it, n))
-    if not result:
-        return result
-    _heapify_max(result)
-    _heappushpop = _heappushpop_max
-    for elem in it:
-        _heappushpop(result, elem)
-    result.sort()
-    return result
-
 # 'heap' is a heap at all indices >= startpos, except possibly for pos.  pos
 # is the index of a leaf with a possibly out-of-order value.  Restore the
 # heap invariant.
@@ -327,6 +309,10 @@
     from _heapq import *
 except ImportError:
     pass
+try:
+    from _heapq import _heapreplace_max
+except ImportError:
+    pass
 
 def merge(*iterables):
     '''Merge multiple sorted inputs into a single sorted output.
@@ -367,22 +353,86 @@
         yield v
         yield from next.__self__
 
-# Extend the implementations of nsmallest and nlargest to use a key= argument
-_nsmallest = nsmallest
+
+# Algorithm notes for nlargest() and nsmallest()
+# ==============================================
+#
+# Makes just a single pass over the data while keeping the k most extreme values
+# in a heap.  Memory consumption is limited to keeping k values in a list.
+#
+# Measured performance for random inputs:
+#
+#                                   number of comparisons
+#    n inputs     k-extreme values  (average of 5 trials)   % more than min()
+# -------------   ----------------  - -------------------   -----------------
+#      1,000           100                  3,317               133.2%
+#     10,000           100                 14,046                40.5%
+#    100,000           100                105,749                 5.7%
+#  1,000,000           100              1,007,751                 0.8%
+# 10,000,000           100             10,009,401                 0.1%
+#
+# Theoretical number of comparisons for k smallest of n random inputs:
+#
+# Step   Comparisons                  Action
+# ----   --------------------------   ---------------------------
+#  1     1.66 * k                     heapify the first k-inputs
+#  2     n - k                        compare remaining elements to top of heap
+#  3     k * (1 + lg2(k)) * ln(n/k)   replace the topmost value on the heap
+#  4     k * lg2(k) - (k/2)           final sort of the k most extreme values
+# Combining and simplifying for a rough estimate gives:
+#        comparisons = n + k * (1 + log(n/k)) * (1 + log(k, 2))
+#
+# Computing the number of comparisons for step 3:
+# -----------------------------------------------
+# * For the i-th new value from the iterable, the probability of being in the
+#   k most extreme values is k/i.  For example, the probability of the 101st
+#   value seen being in the 100 most extreme values is 100/101.
+# * If the value is a new extreme value, the cost of inserting it into the
+#   heap is 1 + log(k, 2).
+# * The probabilty times the cost gives:
+#            (k/i) * (1 + log(k, 2))
+# * Summing across the remaining n-k elements gives:
+#            sum((k/i) * (1 + log(k, 2)) for xrange(k+1, n+1))
+# * This reduces to:
+#            (H(n) - H(k)) * k * (1 + log(k, 2))
+# * Where H(n) is the n-th harmonic number estimated by:
+#            gamma = 0.5772156649
+#            H(n) = log(n, e) + gamma + 1.0 / (2.0 * n)
+#   http://en.wikipedia.org/wiki/Harmonic_series_(mathematics)#Rate_of_divergence
+# * Substituting the H(n) formula:
+#            comparisons = k * (1 + log(k, 2)) * (log(n/k, e) + (1/n - 1/k) / 2)
+#
+# Worst-case for step 3:
+# ----------------------
+# In the worst case, the input data is reversed sorted so that every new element
+# must be inserted in the heap:
+#
+#             comparisons = 1.66 * k + log(k, 2) * (n - k)
+#
+# Alternative Algorithms
+# ----------------------
+# Other algorithms were not used because they:
+# 1) Took much more auxiliary memory,
+# 2) Made multiple passes over the data.
+# 3) Made more comparisons in common cases (small k, large n, semi-random input).
+# See the more detailed comparison of approach at:
+# http://code.activestate.com/recipes/577573-compare-algorithms-for-heapqsmallest
+
 def nsmallest(n, iterable, key=None):
     """Find the n smallest elements in a dataset.
 
     Equivalent to:  sorted(iterable, key=key)[:n]
     """
+
     # Short-cut for n==1 is to use min() when len(iterable)>0
     if n == 1:
         it = iter(iterable)
-        head = list(islice(it, 1))
-        if not head:
-            return []
+        sentinel = object()
         if key is None:
-            return [min(chain(head, it))]
-        return [min(chain(head, it), key=key)]
+            result = min(it, default=sentinel)
+        else:
+            result = min(it, default=sentinel, key=key)
+        return [] if result is sentinel else [result]
 
     # When n>=size, it's faster to use sorted()
     try:
@@ -395,15 +445,39 @@
 
     # When key is none, use simpler decoration
     if key is None:
-        it = zip(iterable, count())                         # decorate
-        result = _nsmallest(n, it)
-        return [r[0] for r in result]                       # undecorate
+        it = iter(iterable)
+        result = list(islice(zip(it, count()), n))
+        if not result:
+            return result
+        _heapify_max(result)
+        order = n
+        top = result[0][0]
+        _heapreplace = _heapreplace_max
+        for elem in it:
+            if elem < top:
+                _heapreplace(result, (elem, order))
+                top = result[0][0]
+                order += 1
+        result.sort()
+        return [r[0] for r in result]
 
     # General case, slowest method
-    in1, in2 = tee(iterable)
-    it = zip(map(key, in1), count(), in2)                   # decorate
-    result = _nsmallest(n, it)
-    return [r[2] for r in result]                           # undecorate
+    it = iter(iterable)
+    result = [(key(elem), i, elem) for i, elem in zip(range(n), it)]
+    if not result:
+        return result
+    _heapify_max(result)
+    order = n
+    top = result[0][0]
+    _heapreplace = _heapreplace_max
+    for elem in it:
+        k = key(elem)
+        if k < top:
+            _heapreplace(result, (k, order, elem))
+            top = result[0][0]
+            order += 1
+    result.sort()
+    return [r[2] for r in result]
 
 def nlargest(n, iterable, key=None):
     """Find the n largest elements in a dataset.
@@ -442,9 +516,9 @@
         _heapreplace = heapreplace
         for elem in it:
             if top < elem:
-                order -= 1
                 _heapreplace(result, (elem, order))
                 top = result[0][0]
+                order -= 1
         result.sort(reverse=True)
         return [r[0] for r in result]
 
@@ -460,9 +534,9 @@
     for elem in it:
         k = key(elem)
         if top < k:
-            order -= 1
             _heapreplace(result, (k, order, elem))
             top = result[0][0]
+            order -= 1
     result.sort(reverse=True)
     return [r[2] for r in result]
 
diff --git a/Lib/test/test_heapq.py b/Lib/test/test_heapq.py
--- a/Lib/test/test_heapq.py
+++ b/Lib/test/test_heapq.py
@@ -13,7 +13,7 @@
 # _heapq.nlargest/nsmallest are saved in heapq._nlargest/_smallest when
 # _heapq is imported, so check them there
 func_names = ['heapify', 'heappop', 'heappush', 'heappushpop',
-              'heapreplace', '_nsmallest']
+              'heapreplace', '_heapreplace_max']
 
 class TestModules(TestCase):
     def test_py_functions(self):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -84,8 +84,8 @@
 - Issue #21156: importlib.abc.InspectLoader.source_to_code() is now a
   staticmethod.
 
-- Issue #21424: Simplified and optimized heaqp.nlargest() to make fewer
-  tuple comparisons.
+- Issue #21424: Simplified and optimized heaqp.nlargest() and nmsmallest()
+  to make fewer tuple comparisons.
 
 - Issue #21396: Fix TextIOWrapper(..., write_through=True) to not force a
   flush() on the underlying binary stream.  Patch by akira.
diff --git a/Modules/_heapqmodule.c b/Modules/_heapqmodule.c
--- a/Modules/_heapqmodule.c
+++ b/Modules/_heapqmodule.c
@@ -354,88 +354,34 @@
 }
 
 static PyObject *
-nsmallest(PyObject *self, PyObject *args)
+_heapreplace_max(PyObject *self, PyObject *args)
 {
-    PyObject *heap=NULL, *elem, *iterable, *los, *it, *oldelem;
-    Py_ssize_t i, n;
-    int cmp;
+    PyObject *heap, *item, *returnitem;
 
-    if (!PyArg_ParseTuple(args, "nO:nsmallest", &n, &iterable))
+    if (!PyArg_UnpackTuple(args, "_heapreplace_max", 2, 2, &heap, &item))
         return NULL;
 
-    it = PyObject_GetIter(iterable);
-    if (it == NULL)
+    if (!PyList_Check(heap)) {
+        PyErr_SetString(PyExc_TypeError, "heap argument must be a list");
         return NULL;
-
-    heap = PyList_New(0);
-    if (heap == NULL)
-        goto fail;
-
-    for (i=0 ; i<n ; i++ ){
-        elem = PyIter_Next(it);
-        if (elem == NULL) {
-            if (PyErr_Occurred())
-                goto fail;
-            else
-                goto sortit;
-        }
-        if (PyList_Append(heap, elem) == -1) {
-            Py_DECREF(elem);
-            goto fail;
-        }
-        Py_DECREF(elem);
-    }
-    n = PyList_GET_SIZE(heap);
-    if (n == 0)
-        goto sortit;
-
-    for (i=n/2-1 ; i>=0 ; i--)
-        if(_siftupmax((PyListObject *)heap, i) == -1)
-            goto fail;
-
-    los = PyList_GET_ITEM(heap, 0);
-    while (1) {
-        elem = PyIter_Next(it);
-        if (elem == NULL) {
-            if (PyErr_Occurred())
-                goto fail;
-            else
-                goto sortit;
-        }
-        cmp = PyObject_RichCompareBool(elem, los, Py_LT);
-        if (cmp == -1) {
-            Py_DECREF(elem);
-            goto fail;
-        }
-        if (cmp == 0) {
-            Py_DECREF(elem);
-            continue;
-        }
-
-        oldelem = PyList_GET_ITEM(heap, 0);
-        PyList_SET_ITEM(heap, 0, elem);
-        Py_DECREF(oldelem);
-        if (_siftupmax((PyListObject *)heap, 0) == -1)
-            goto fail;
-        los = PyList_GET_ITEM(heap, 0);
     }
 
-sortit:
-    if (PyList_Sort(heap) == -1)
-        goto fail;
-    Py_DECREF(it);
-    return heap;
+    if (PyList_GET_SIZE(heap) < 1) {
+        PyErr_SetString(PyExc_IndexError, "index out of range");
+        return NULL;
+    }
 
-fail:
-    Py_DECREF(it);
-    Py_XDECREF(heap);
-    return NULL;
+    returnitem = PyList_GET_ITEM(heap, 0);
+    Py_INCREF(item);
+    PyList_SET_ITEM(heap, 0, item);
+    if (_siftupmax((PyListObject *)heap, 0) == -1) {
+        Py_DECREF(returnitem);
+        return NULL;
+    }
+    return returnitem;
 }
 
-PyDoc_STRVAR(nsmallest_doc,
-"Find the n smallest elements in a dataset.\n\
-\n\
-Equivalent to:  sorted(iterable)[:n]\n");
+PyDoc_STRVAR(heapreplace_max_doc, "Maxheap variant of heapreplace");
 
 static PyMethodDef heapq_methods[] = {
     {"heappush",        (PyCFunction)heappush,
@@ -448,8 +394,8 @@
         METH_VARARGS,           heapreplace_doc},
     {"heapify",         (PyCFunction)heapify,
         METH_O,                 heapify_doc},
-    {"nsmallest",       (PyCFunction)nsmallest,
-        METH_VARARGS,           nsmallest_doc},
+    {"_heapreplace_max",(PyCFunction)_heapreplace_max,
+        METH_VARARGS,           heapreplace_max_doc},
     {NULL,              NULL}           /* sentinel */
 };
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 01:11:52 2014
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 12 May 2014 01:11:52 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_backport_hmac?=
 =?utf-8?q?=2Ecompare=5Fdigest_to_partially_implement_PEP_466_=28closes_?=
 =?utf-8?q?=2321306=29?=
Message-ID: <3gRgyc1KnQz7LkC@mail.python.org>

http://hg.python.org/cpython/rev/b40f1a00b134
changeset:   90645:b40f1a00b134
branch:      2.7
parent:      90638:951775c68b1b
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun May 11 16:11:44 2014 -0700
summary:
  backport hmac.compare_digest to partially implement PEP 466 (closes #21306)

Backport from Alex Gaynor.

files:
  Doc/library/hmac.rst  |   33 +++++++
  Lib/hmac.py           |    3 +
  Lib/test/test_hmac.py |  112 ++++++++++++++++++++++++++-
  Misc/NEWS             |    3 +
  Modules/operator.c    |  128 ++++++++++++++++++++++++++++++
  5 files changed, 278 insertions(+), 1 deletions(-)


diff --git a/Doc/library/hmac.rst b/Doc/library/hmac.rst
--- a/Doc/library/hmac.rst
+++ b/Doc/library/hmac.rst
@@ -38,6 +38,13 @@
    This string will be the same length as the *digest_size* of the digest given to
    the constructor.  It may contain non-ASCII characters, including NUL bytes.
 
+   .. warning::
+
+      When comparing the output of :meth:`digest` to an externally-supplied
+      digest during a verification routine, it is recommended to use the
+      :func:`compare_digest` function instead of the ``==`` operator
+      to reduce the vulnerability to timing attacks.
+
 
 .. method:: HMAC.hexdigest()
 
@@ -45,6 +52,13 @@
    containing only hexadecimal digits.  This may be used to exchange the value
    safely in email or other non-binary environments.
 
+   .. warning::
+
+      When comparing the output of :meth:`hexdigest` to an externally-supplied
+      digest during a verification routine, it is recommended to use the
+      :func:`compare_digest` function instead of the ``==`` operator
+      to reduce the vulnerability to timing attacks.
+
 
 .. method:: HMAC.copy()
 
@@ -52,6 +66,25 @@
    compute the digests of strings that share a common initial substring.
 
 
+This module also provides the following helper function:
+
+.. function:: compare_digest(a, b)
+
+   Return ``a == b``.  This function uses an approach designed to prevent
+   timing analysis by avoiding content-based short circuiting behaviour,
+   making it appropriate for cryptography.  *a* and *b* must both be of the
+   same type: either :class:`unicode` or a :term:`bytes-like object`.
+
+   .. note::
+
+      If *a* and *b* are of different lengths, or if an error occurs,
+      a timing attack could theoretically reveal information about the
+      types and lengths of *a* and *b*--but not their values.
+
+
+   .. versionadded:: 2.7.7
+
+
 .. seealso::
 
    Module :mod:`hashlib`
diff --git a/Lib/hmac.py b/Lib/hmac.py
--- a/Lib/hmac.py
+++ b/Lib/hmac.py
@@ -5,6 +5,9 @@
 
 import warnings as _warnings
 
+from operator import _compare_digest as compare_digest
+
+
 trans_5C = "".join ([chr (x ^ 0x5C) for x in xrange(256)])
 trans_36 = "".join ([chr (x ^ 0x36) for x in xrange(256)])
 
diff --git a/Lib/test/test_hmac.py b/Lib/test/test_hmac.py
--- a/Lib/test/test_hmac.py
+++ b/Lib/test/test_hmac.py
@@ -302,12 +302,122 @@
         self.assertTrue(h1.hexdigest() == h2.hexdigest(),
             "Hexdigest of copy doesn't match original hexdigest.")
 
+
+class CompareDigestTestCase(unittest.TestCase):
+
+    def test_compare_digest(self):
+        # Testing input type exception handling
+        a, b = 100, 200
+        self.assertRaises(TypeError, hmac.compare_digest, a, b)
+        a, b = 100, b"foobar"
+        self.assertRaises(TypeError, hmac.compare_digest, a, b)
+        a, b = b"foobar", 200
+        self.assertRaises(TypeError, hmac.compare_digest, a, b)
+        a, b = u"foobar", b"foobar"
+        self.assertRaises(TypeError, hmac.compare_digest, a, b)
+        a, b = b"foobar", u"foobar"
+        self.assertRaises(TypeError, hmac.compare_digest, a, b)
+
+        # Testing bytes of different lengths
+        a, b = b"foobar", b"foo"
+        self.assertFalse(hmac.compare_digest(a, b))
+        a, b = b"\xde\xad\xbe\xef", b"\xde\xad"
+        self.assertFalse(hmac.compare_digest(a, b))
+
+        # Testing bytes of same lengths, different values
+        a, b = b"foobar", b"foobaz"
+        self.assertFalse(hmac.compare_digest(a, b))
+        a, b = b"\xde\xad\xbe\xef", b"\xab\xad\x1d\xea"
+        self.assertFalse(hmac.compare_digest(a, b))
+
+        # Testing bytes of same lengths, same values
+        a, b = b"foobar", b"foobar"
+        self.assertTrue(hmac.compare_digest(a, b))
+        a, b = b"\xde\xad\xbe\xef", b"\xde\xad\xbe\xef"
+        self.assertTrue(hmac.compare_digest(a, b))
+
+        # Testing bytearrays of same lengths, same values
+        a, b = bytearray(b"foobar"), bytearray(b"foobar")
+        self.assertTrue(hmac.compare_digest(a, b))
+
+        # Testing bytearrays of diffeent lengths
+        a, b = bytearray(b"foobar"), bytearray(b"foo")
+        self.assertFalse(hmac.compare_digest(a, b))
+
+        # Testing bytearrays of same lengths, different values
+        a, b = bytearray(b"foobar"), bytearray(b"foobaz")
+        self.assertFalse(hmac.compare_digest(a, b))
+
+        # Testing byte and bytearray of same lengths, same values
+        a, b = bytearray(b"foobar"), b"foobar"
+        self.assertTrue(hmac.compare_digest(a, b))
+        self.assertTrue(hmac.compare_digest(b, a))
+
+        # Testing byte bytearray of diffeent lengths
+        a, b = bytearray(b"foobar"), b"foo"
+        self.assertFalse(hmac.compare_digest(a, b))
+        self.assertFalse(hmac.compare_digest(b, a))
+
+        # Testing byte and bytearray of same lengths, different values
+        a, b = bytearray(b"foobar"), b"foobaz"
+        self.assertFalse(hmac.compare_digest(a, b))
+        self.assertFalse(hmac.compare_digest(b, a))
+
+        # Testing str of same lengths
+        a, b = "foobar", "foobar"
+        self.assertTrue(hmac.compare_digest(a, b))
+
+        # Testing str of diffeent lengths
+        a, b = "foo", "foobar"
+        self.assertFalse(hmac.compare_digest(a, b))
+
+        # Testing bytes of same lengths, different values
+        a, b = "foobar", "foobaz"
+        self.assertFalse(hmac.compare_digest(a, b))
+
+        # Testing error cases
+        a, b = u"foobar", b"foobar"
+        self.assertRaises(TypeError, hmac.compare_digest, a, b)
+        a, b = b"foobar", u"foobar"
+        self.assertRaises(TypeError, hmac.compare_digest, a, b)
+        a, b = b"foobar", 1
+        self.assertRaises(TypeError, hmac.compare_digest, a, b)
+        a, b = 100, 200
+        self.assertRaises(TypeError, hmac.compare_digest, a, b)
+        a, b = "foo?", "foo?"
+        self.assertTrue(hmac.compare_digest(a, b))
+
+        # subclasses are supported by ignore __eq__
+        class mystr(str):
+            def __eq__(self, other):
+                return False
+
+        a, b = mystr("foobar"), mystr("foobar")
+        self.assertTrue(hmac.compare_digest(a, b))
+        a, b = mystr("foobar"), "foobar"
+        self.assertTrue(hmac.compare_digest(a, b))
+        a, b = mystr("foobar"), mystr("foobaz")
+        self.assertFalse(hmac.compare_digest(a, b))
+
+        class mybytes(bytes):
+            def __eq__(self, other):
+                return False
+
+        a, b = mybytes(b"foobar"), mybytes(b"foobar")
+        self.assertTrue(hmac.compare_digest(a, b))
+        a, b = mybytes(b"foobar"), b"foobar"
+        self.assertTrue(hmac.compare_digest(a, b))
+        a, b = mybytes(b"foobar"), mybytes(b"foobaz")
+        self.assertFalse(hmac.compare_digest(a, b))
+
+
 def test_main():
     test_support.run_unittest(
         TestVectorsTestCase,
         ConstructorTestCase,
         SanityTestCase,
-        CopyTestCase
+        CopyTestCase,
+        CompareDigestTestCase,
     )
 
 if __name__ == "__main__":
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -49,6 +49,9 @@
 Library
 -------
 
+- Issue #21306: Backport hmac.compare_digest from Python 3. This is part of PEP
+  466.
+
 - Issue #21321: itertools.islice() now releases the reference to the source
   iterator when the slice is exhausted.  Patch by Anton Afanasyev.
 
diff --git a/Modules/operator.c b/Modules/operator.c
--- a/Modules/operator.c
+++ b/Modules/operator.c
@@ -235,6 +235,132 @@
 #define spam2o(OP,ALTOP,DOC) {#OP, op_##OP, METH_O, PyDoc_STR(DOC)}, \
                            {#ALTOP, op_##OP, METH_O, PyDoc_STR(DOC)},
 
+
+
+/* compare_digest **********************************************************/
+
+/*
+ * timing safe compare
+ *
+ * Returns 1 of the strings are equal.
+ * In case of len(a) != len(b) the function tries to keep the timing
+ * dependent on the length of b. CPU cache locally may still alter timing
+ * a bit.
+ */
+static int
+_tscmp(const unsigned char *a, const unsigned char *b,
+        Py_ssize_t len_a, Py_ssize_t len_b)
+{
+    /* The volatile type declarations make sure that the compiler has no
+     * chance to optimize and fold the code in any way that may change
+     * the timing.
+     */
+    volatile Py_ssize_t length;
+    volatile const unsigned char *left;
+    volatile const unsigned char *right;
+    Py_ssize_t i;
+    unsigned char result;
+
+    /* loop count depends on length of b */
+    length = len_b;
+    left = NULL;
+    right = b;
+
+    /* don't use else here to keep the amount of CPU instructions constant,
+     * volatile forces re-evaluation
+     *  */
+    if (len_a == length) {
+        left = *((volatile const unsigned char**)&a);
+        result = 0;
+    }
+    if (len_a != length) {
+        left = b;
+        result = 1;
+    }
+
+    for (i=0; i < length; i++) {
+        result |= *left++ ^ *right++;
+    }
+
+    return (result == 0);
+}
+
+PyDoc_STRVAR(compare_digest__doc__,
+"compare_digest(a, b) -> bool\n"
+"\n"
+"Return 'a == b'.  This function uses an approach designed to prevent\n"
+"timing analysis, making it appropriate for cryptography.\n"
+"a and b must both be of the same type: either str (ASCII only),\n"
+"or any type that supports the buffer protocol (e.g. bytes).\n"
+"\n"
+"Note: If a and b are of different lengths, or if an error occurs,\n"
+"a timing attack could theoretically reveal information about the\n"
+"types and lengths of a and b--but not their values.\n");
+
+static PyObject*
+compare_digest(PyObject *self, PyObject *args)
+{
+    PyObject *a, *b;
+    int rc;
+
+    if (!PyArg_ParseTuple(args, "OO:compare_digest", &a, &b)) {
+        return NULL;
+    }
+
+    /* Unicode string */
+    if (PyUnicode_Check(a) && PyUnicode_Check(b)) {
+        rc = _tscmp(PyUnicode_AS_DATA(a),
+                    PyUnicode_AS_DATA(b),
+                    PyUnicode_GET_DATA_SIZE(a),
+                    PyUnicode_GET_DATA_SIZE(b));
+    }
+    /* fallback to buffer interface for bytes, bytesarray and other */
+    else {
+        Py_buffer view_a;
+        Py_buffer view_b;
+
+        if ((PyObject_CheckBuffer(a) == 0) & (PyObject_CheckBuffer(b) == 0)) {
+            PyErr_Format(PyExc_TypeError,
+                         "unsupported operand types(s) or combination of types: "
+                         "'%.100s' and '%.100s'",
+                         Py_TYPE(a)->tp_name, Py_TYPE(b)->tp_name);
+            return NULL;
+        }
+
+        if (PyObject_GetBuffer(a, &view_a, PyBUF_SIMPLE) == -1) {
+            return NULL;
+        }
+        if (view_a.ndim > 1) {
+            PyErr_SetString(PyExc_BufferError,
+                            "Buffer must be single dimension");
+            PyBuffer_Release(&view_a);
+            return NULL;
+        }
+
+        if (PyObject_GetBuffer(b, &view_b, PyBUF_SIMPLE) == -1) {
+            PyBuffer_Release(&view_a);
+            return NULL;
+        }
+        if (view_b.ndim > 1) {
+            PyErr_SetString(PyExc_BufferError,
+                            "Buffer must be single dimension");
+            PyBuffer_Release(&view_a);
+            PyBuffer_Release(&view_b);
+            return NULL;
+        }
+
+        rc = _tscmp((const unsigned char*)view_a.buf,
+                    (const unsigned char*)view_b.buf,
+                    view_a.len,
+                    view_b.len);
+
+        PyBuffer_Release(&view_a);
+        PyBuffer_Release(&view_b);
+    }
+
+    return PyBool_FromLong(rc);
+}
+
 static struct PyMethodDef operator_methods[] = {
 
 spam1o(isCallable,
@@ -318,6 +444,8 @@
 spam2(gt,__gt__, "gt(a, b) -- Same as a>b.")
 spam2(ge,__ge__, "ge(a, b) -- Same as a>=b.")
 
+    {"_compare_digest", (PyCFunction)compare_digest, METH_VARARGS,
+     compare_digest__doc__},
     {NULL,              NULL}           /* sentinel */
 
 };

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 01:14:57 2014
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 12 May 2014 01:14:57 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_cast_away_warn?=
	=?utf-8?q?ings?=
Message-ID: <3gRh292mVmz7LjP@mail.python.org>

http://hg.python.org/cpython/rev/75a0a8c121a4
changeset:   90646:75a0a8c121a4
branch:      2.7
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun May 11 16:14:00 2014 -0700
summary:
  cast away warnings

files:
  Modules/operator.c |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Modules/operator.c b/Modules/operator.c
--- a/Modules/operator.c
+++ b/Modules/operator.c
@@ -309,8 +309,8 @@
 
     /* Unicode string */
     if (PyUnicode_Check(a) && PyUnicode_Check(b)) {
-        rc = _tscmp(PyUnicode_AS_DATA(a),
-                    PyUnicode_AS_DATA(b),
+        rc = _tscmp((const unsigned char *)PyUnicode_AS_DATA(a),
+                    (const unsigned char *)PyUnicode_AS_DATA(b),
                     PyUnicode_GET_DATA_SIZE(a),
                     PyUnicode_GET_DATA_SIZE(b));
     }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 01:18:30 2014
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 12 May 2014 01:18:30 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_this_file_now_?=
	=?utf-8?q?has_utf-8_chars?=
Message-ID: <3gRh6G0JFGz7LjP@mail.python.org>

http://hg.python.org/cpython/rev/7c0b69ebbd95
changeset:   90647:7c0b69ebbd95
branch:      2.7
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun May 11 16:16:27 2014 -0700
summary:
  this file now has utf-8 chars

files:
  Lib/test/test_hmac.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_hmac.py b/Lib/test/test_hmac.py
--- a/Lib/test/test_hmac.py
+++ b/Lib/test/test_hmac.py
@@ -1,3 +1,5 @@
+# coding: utf-8
+
 import hmac
 import hashlib
 import unittest

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 01:18:31 2014
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 12 May 2014 01:18:31 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_use_logical_ra?=
	=?utf-8?q?ther_than_bit_and?=
Message-ID: <3gRh6H26jFz7LjP@mail.python.org>

http://hg.python.org/cpython/rev/9f03b1149f78
changeset:   90648:9f03b1149f78
branch:      2.7
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun May 11 16:17:02 2014 -0700
summary:
  use logical rather than bit and

files:
  Modules/operator.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Modules/operator.c b/Modules/operator.c
--- a/Modules/operator.c
+++ b/Modules/operator.c
@@ -319,7 +319,7 @@
         Py_buffer view_a;
         Py_buffer view_b;
 
-        if ((PyObject_CheckBuffer(a) == 0) & (PyObject_CheckBuffer(b) == 0)) {
+        if (PyObject_CheckBuffer(a) == 0 && PyObject_CheckBuffer(b) == 0) {
             PyErr_Format(PyExc_TypeError,
                          "unsupported operand types(s) or combination of types: "
                          "'%.100s' and '%.100s'",

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 01:18:32 2014
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 12 May 2014 01:18:32 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_use_logical_ra?=
	=?utf-8?q?ther_than_bit_and?=
Message-ID: <3gRh6J3ydnz7Ljk@mail.python.org>

http://hg.python.org/cpython/rev/6a26c1bce331
changeset:   90649:6a26c1bce331
branch:      3.4
parent:      90642:585dfa357fc4
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun May 11 16:17:02 2014 -0700
summary:
  use logical rather than bit and

files:
  Modules/_operator.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Modules/_operator.c b/Modules/_operator.c
--- a/Modules/_operator.c
+++ b/Modules/_operator.c
@@ -277,7 +277,7 @@
         Py_buffer view_a;
         Py_buffer view_b;
 
-        if ((PyObject_CheckBuffer(a) == 0) & (PyObject_CheckBuffer(b) == 0)) {
+        if (PyObject_CheckBuffer(a) == 0 && PyObject_CheckBuffer(b) == 0) {
             PyErr_Format(PyExc_TypeError,
                          "unsupported operand types(s) or combination of types: "
                          "'%.100s' and '%.100s'",

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 01:18:34 2014
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 12 May 2014 01:18:34 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?b?KTogbWVyZ2UgMy40?=
Message-ID: <3gRh6L0rVVz7Ljg@mail.python.org>

http://hg.python.org/cpython/rev/a6fc2405fdbc
changeset:   90650:a6fc2405fdbc
parent:      90644:31950174f60f
parent:      90649:6a26c1bce331
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun May 11 16:17:34 2014 -0700
summary:
  merge 3.4

files:
  Modules/_operator.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Modules/_operator.c b/Modules/_operator.c
--- a/Modules/_operator.c
+++ b/Modules/_operator.c
@@ -279,7 +279,7 @@
         Py_buffer view_a;
         Py_buffer view_b;
 
-        if ((PyObject_CheckBuffer(a) == 0) & (PyObject_CheckBuffer(b) == 0)) {
+        if (PyObject_CheckBuffer(a) == 0 && PyObject_CheckBuffer(b) == 0) {
             PyErr_Format(PyExc_TypeError,
                          "unsupported operand types(s) or combination of types: "
                          "'%.100s' and '%.100s'",

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 05:37:51 2014
From: python-checkins at python.org (terry.reedy)
Date: Mon, 12 May 2014 05:37:51 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE4MTA0?=
 =?utf-8?q?=3A_Add_idlelib/idle=5Ftest/htest=2Epy_with_a_few_sample_tests_?=
 =?utf-8?q?to_begin?=
Message-ID: <3gRnsW1lgFz7LjX@mail.python.org>

http://hg.python.org/cpython/rev/460203eaf731
changeset:   90651:460203eaf731
branch:      2.7
parent:      90648:9f03b1149f78
user:        Terry Jan Reedy <tjreedy at udel.edu>
date:        Sun May 11 23:32:20 2014 -0400
summary:
  Issue #18104: Add idlelib/idle_test/htest.py with a few sample tests to begin
consolidating and improving human-validated tests of Idle. Change other files
as needed to work with htest.  Running the module as __main__ runs all tests.

files:
  Lib/idlelib/EditorWindow.py            |  14 +-
  Lib/idlelib/aboutDialog.py             |  11 +-
  Lib/idlelib/configSectionNameDialog.py |  28 +--
  Lib/idlelib/idle_test/htest.py         |  91 ++++++++++++++
  Misc/NEWS                              |   4 +
  5 files changed, 117 insertions(+), 31 deletions(-)


diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py
--- a/Lib/idlelib/EditorWindow.py
+++ b/Lib/idlelib/EditorWindow.py
@@ -108,6 +108,8 @@
         self.parent = None
 
 helpDialog = HelpDialog()  # singleton instance
+def _Help_dialog(parent):  # wrapper for htest
+    helpDialog.show_dialog(parent)
 
 
 class EditorWindow(object):
@@ -1709,19 +1711,21 @@
     tk.call('set', 'tcl_nonwordchars', '[^a-zA-Z0-9_]')
 
 
-def test():
-    root = Tk()
+def _Editor_window(parent):
+    root = parent
     fixwordbreaks(root)
     root.withdraw()
     if sys.argv[1:]:
         filename = sys.argv[1]
     else:
         filename = None
+    macosxSupport.setupApp(root, None)
     edit = EditorWindow(root=root, filename=filename)
     edit.set_close_hook(root.quit)
     edit.text.bind("<<close-all-windows>>", edit.close_event)
-    root.mainloop()
-    root.destroy()
 
 if __name__ == '__main__':
-    test()
+    from idlelib.idle_test.htest import run
+    if len(sys.argv) <= 1:
+        run(_Help_dialog)
+    run(_Editor_window)
diff --git a/Lib/idlelib/aboutDialog.py b/Lib/idlelib/aboutDialog.py
--- a/Lib/idlelib/aboutDialog.py
+++ b/Lib/idlelib/aboutDialog.py
@@ -12,7 +12,7 @@
     """Modal about dialog for idle
 
     """
-    def __init__(self,parent,title):
+    def __init__(self, parent, title):
         Toplevel.__init__(self, parent)
         self.configure(borderwidth=5)
         self.geometry("+%d+%d" % (parent.winfo_rootx()+30,
@@ -136,10 +136,5 @@
         self.destroy()
 
 if __name__ == '__main__':
-    # test the dialog
-    root = Tk()
-    def run():
-        from idlelib import aboutDialog
-        aboutDialog.AboutDialog(root, 'About')
-    Button(root, text='Dialog', command=run).pack()
-    root.mainloop()
+    from idlelib.idle_test.htest import run
+    run(AboutDialog)
diff --git a/Lib/idlelib/configSectionNameDialog.py b/Lib/idlelib/configSectionNameDialog.py
--- a/Lib/idlelib/configSectionNameDialog.py
+++ b/Lib/idlelib/configSectionNameDialog.py
@@ -7,10 +7,11 @@
 from Tkinter import *
 import tkMessageBox
 class GetCfgSectionNameDialog(Toplevel):
-    def __init__(self, parent, title, message, used_names):
+    def __init__(self, parent, title, message, used_names, _htest=False):
         """
         message - string, informational message to display
         used_names - string collection, names already in use for validity check
+        _htest - bool, change box location when running htest
         """
         Toplevel.__init__(self, parent)
         self.configure(borderwidth=5)
@@ -29,11 +30,12 @@
         self.messageInfo.config(width=self.frameMain.winfo_reqwidth())
         self.geometry(
                 "+%d+%d" % (
-                parent.winfo_rootx() +
-                (parent.winfo_width()/2 - self.winfo_reqwidth()/2),
-                parent.winfo_rooty() +
-                (parent.winfo_height()/2 - self.winfo_reqheight()/2)
-                ) )  #centre dialog over parent
+                    parent.winfo_rootx() +
+                    (parent.winfo_width()/2 - self.winfo_reqwidth()/2),
+                    parent.winfo_rooty() +
+                    ((parent.winfo_height()/2 - self.winfo_reqheight()/2)
+                    if not _htest else 100)
+                ) )  #centre dialog over parent (or below htest box)
         self.deiconify()  #geometry set, unhide
         self.wait_window()
     def create_widgets(self):
@@ -86,15 +88,5 @@
     import unittest
     unittest.main('idlelib.idle_test.test_config_name', verbosity=2, exit=False)
 
-    # also human test the dialog
-    root = Tk()
-    def run():
-        dlg=GetCfgSectionNameDialog(root,'Get Name',
-                "After the text entered with [Ok] is stripped, <nothing>, "
-                "'abc', or more that 30 chars are errors. "
-                "Close with a valid entry (printed), [Cancel], or [X]",
-                {'abc'})
-        print dlg.result
-    Message(root, text='').pack()  # will be needed for oher dialog tests
-    Button(root, text='Click to begin dialog test', command=run).pack()
-    root.mainloop()
+    from idlelib.idle_test.htest import run
+    run(GetCfgSectionNameDialog)
diff --git a/Lib/idlelib/idle_test/htest.py b/Lib/idlelib/idle_test/htest.py
new file mode 100644
--- /dev/null
+++ b/Lib/idlelib/idle_test/htest.py
@@ -0,0 +1,91 @@
+'''Run a human test of Idle wndow, dialog, and other widget classes.
+
+run(klass) runs a test for one class.
+runall() runs all the defined tests
+
+The file wih the widget class should end with
+if __name__ == '__main__':
+    <unittest, if there is one>
+    from idlelib.idle_test.htest import run
+    run(X)
+where X is a global object of the module. X must be a callable with a
+.__name__ attribute that accepts a 'parent' attribute. X will usually be
+a widget class, but a callable instance with .__name__ or a wrapper
+function also work. The name of wrapper functions, like _Editor_Window,
+should start with '_'.
+
+This file must then contain an instance of this template.
+_spec = {
+    'file': '',
+    'kwds': {'title': ''},
+    'msg': ""
+    }
+with X.__name__ prepended to _spec.
+File (no .py) is used in runall() to import the file and get the class.
+Kwds is passed to X (**kwds) after 'parent' is added, to initialize X.
+Msg. displayed is a window with a start button. hint as to how the user
+might test the widget. Closing The box skips or ends the test.
+'''
+from importlib import import_module
+import Tkinter as tk
+
+# Template for class_spec dicts, copy and uncomment
+
+_Editor_window_spec = {
+    'file': 'EditorWindow',
+    'kwds': {},
+    'msg': "Test editor functions of interest"
+    }
+
+_Help_dialog_spec = {
+    'file': 'EditorWindow',
+    'kwds': {},
+    'msg': "If the help text displays, this works"
+    }
+
+AboutDialog_spec = {
+    'file': 'aboutDialog',
+    'kwds': {'title': 'About test'},
+    'msg': "Try each button"
+    }
+
+
+GetCfgSectionNameDialog_spec = {
+    'file': 'configSectionNameDialog',
+    'kwds': {'title':'Get Name',
+                 'message':'Enter something',
+                 'used_names': {'abc'},
+                 '_htest': True},
+    'msg': "After the text entered with [Ok] is stripped, <nothing>, "
+              "'abc', or more that 30 chars are errors.\n"
+              "Close 'Get Name' with a valid entry (printed to Shell), [Cancel], or [X]",
+    }
+
+def run(klas):
+    "Test the widget class klas using _spec dict"
+    root = tk.Tk()
+    klas_spec = globals()[klas.__name__+'_spec']
+    klas_kwds = klas_spec['kwds']
+    klas_kwds['parent'] = root
+    # This presumes that Idle consistently uses 'parent'
+    def run_klas():
+        widget = klas(**klas_kwds)
+        try:
+            print(widget.result)
+        except AttributeError:
+            pass
+    tk.Label(root, text=klas_spec['msg'], justify='left').pack()
+    tk.Button(root, text='Test ' + klas.__name__, command=run_klas).pack()
+    root.mainloop()
+
+def runall():
+    'Run all tests. Quick and dirty version.'
+    for k, d in globals().items():
+        if k.endswith('_spec'):
+            mod = import_module('idlelib.' + d['file'])
+            klas = getattr(mod, k[:-5])
+            run(klas)
+
+if __name__ == '__main__':
+    runall()
+
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -308,6 +308,10 @@
 IDLE
 ----
 
+- Issue #18104: Add idlelib/idle_test/htest.py with a few sample tests to begin
+  consolidating and improving human-validated tests of Idle. Change other files
+  as needed to work with htest.  Running the module as __main__ runs all tests.
+
 - Issue #21139: Change default paragraph width to 72, the PEP 8 recommendation.
 
 - Issue #21284: Paragraph reformat test passes after user changes reformat width.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 05:37:53 2014
From: python-checkins at python.org (terry.reedy)
Date: Mon, 12 May 2014 05:37:53 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE4MTA0?=
 =?utf-8?q?=3A_Add_idlelib/idle=5Ftest/htest=2Epy_with_a_few_sample_tests_?=
 =?utf-8?q?to_begin?=
Message-ID: <3gRnsY18Qpz7Ljw@mail.python.org>

http://hg.python.org/cpython/rev/617656f7b538
changeset:   90652:617656f7b538
branch:      3.4
parent:      90649:6a26c1bce331
user:        Terry Jan Reedy <tjreedy at udel.edu>
date:        Sun May 11 23:32:32 2014 -0400
summary:
  Issue #18104: Add idlelib/idle_test/htest.py with a few sample tests to begin
consolidating and improving human-validated tests of Idle. Change other files
as needed to work with htest.  Running the module as __main__ runs all tests.

files:
  Lib/idlelib/EditorWindow.py            |  16 +-
  Lib/idlelib/aboutDialog.py             |  11 +-
  Lib/idlelib/configSectionNameDialog.py |  28 +--
  Lib/idlelib/idle_test/htest.py         |  91 ++++++++++++++
  4 files changed, 114 insertions(+), 32 deletions(-)


diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py
--- a/Lib/idlelib/EditorWindow.py
+++ b/Lib/idlelib/EditorWindow.py
@@ -79,6 +79,8 @@
         self.parent = None
 
 helpDialog = HelpDialog()  # singleton instance
+def _Help_dialog(parent):  # wrapper for htest
+    helpDialog.show_dialog(parent)
 
 
 class EditorWindow(object):
@@ -1064,7 +1066,7 @@
         try:
             try:
                 mod = importlib.import_module('.' + name, package=__package__)
-            except ImportError:
+            except (ImportError, TypeError):
                 mod = importlib.import_module(name)
         except ImportError:
             print("\nFailed to import extension: ", name)
@@ -1700,19 +1702,21 @@
     tk.call('set', 'tcl_nonwordchars', '[^a-zA-Z0-9_]')
 
 
-def test():
-    root = Tk()
+def _Editor_window(parent):
+    root = parent
     fixwordbreaks(root)
     root.withdraw()
     if sys.argv[1:]:
         filename = sys.argv[1]
     else:
         filename = None
+    macosxSupport.setupApp(root, None)
     edit = EditorWindow(root=root, filename=filename)
     edit.set_close_hook(root.quit)
     edit.text.bind("<<close-all-windows>>", edit.close_event)
-    root.mainloop()
-    root.destroy()
 
 if __name__ == '__main__':
-    test()
+    from idlelib.idle_test.htest import run
+    if len(sys.argv) <= 1:
+        run(_Help_dialog)
+    run(_Editor_window)
diff --git a/Lib/idlelib/aboutDialog.py b/Lib/idlelib/aboutDialog.py
--- a/Lib/idlelib/aboutDialog.py
+++ b/Lib/idlelib/aboutDialog.py
@@ -12,7 +12,7 @@
     """Modal about dialog for idle
 
     """
-    def __init__(self,parent,title):
+    def __init__(self, parent, title):
         Toplevel.__init__(self, parent)
         self.configure(borderwidth=5)
         self.geometry("+%d+%d" % (parent.winfo_rootx()+30,
@@ -136,10 +136,5 @@
         self.destroy()
 
 if __name__ == '__main__':
-    # test the dialog
-    root = Tk()
-    def run():
-        from idlelib import aboutDialog
-        aboutDialog.AboutDialog(root, 'About')
-    Button(root, text='Dialog', command=run).pack()
-    root.mainloop()
+    from idlelib.idle_test.htest import run
+    run(AboutDialog)
diff --git a/Lib/idlelib/configSectionNameDialog.py b/Lib/idlelib/configSectionNameDialog.py
--- a/Lib/idlelib/configSectionNameDialog.py
+++ b/Lib/idlelib/configSectionNameDialog.py
@@ -8,10 +8,11 @@
 import tkinter.messagebox as tkMessageBox
 
 class GetCfgSectionNameDialog(Toplevel):
-    def __init__(self, parent, title, message, used_names):
+    def __init__(self, parent, title, message, used_names, _htest=False):
         """
         message - string, informational message to display
         used_names - string collection, names already in use for validity check
+        _htest - bool, change box location when running htest
         """
         Toplevel.__init__(self, parent)
         self.configure(borderwidth=5)
@@ -30,11 +31,12 @@
         self.messageInfo.config(width=self.frameMain.winfo_reqwidth())
         self.geometry(
                 "+%d+%d" % (
-                parent.winfo_rootx() +
-                (parent.winfo_width()/2 - self.winfo_reqwidth()/2),
-                parent.winfo_rooty() +
-                (parent.winfo_height()/2 - self.winfo_reqheight()/2)
-                ) )  #centre dialog over parent
+                    parent.winfo_rootx() +
+                    (parent.winfo_width()/2 - self.winfo_reqwidth()/2),
+                    parent.winfo_rooty() +
+                    ((parent.winfo_height()/2 - self.winfo_reqheight()/2)
+                    if not _htest else 100)
+                ) )  #centre dialog over parent (or below htest box)
         self.deiconify()  #geometry set, unhide
         self.wait_window()
 
@@ -92,15 +94,5 @@
     import unittest
     unittest.main('idlelib.idle_test.test_config_name', verbosity=2, exit=False)
 
-    # also human test the dialog
-    root = Tk()
-    def run():
-        dlg=GetCfgSectionNameDialog(root,'Get Name',
-                "After the text entered with [Ok] is stripped, <nothing>, "
-                "'abc', or more that 30 chars are errors. "
-                "Close with a valid entry (printed), [Cancel], or [X]",
-                {'abc'})
-        print(dlg.result)
-    Message(root, text='').pack()  # will be needed for oher dialog tests
-    Button(root, text='Click to begin dialog test', command=run).pack()
-    root.mainloop()
+    from idlelib.idle_test.htest import run
+    run(GetCfgSectionNameDialog)
diff --git a/Lib/idlelib/idle_test/htest.py b/Lib/idlelib/idle_test/htest.py
new file mode 100644
--- /dev/null
+++ b/Lib/idlelib/idle_test/htest.py
@@ -0,0 +1,91 @@
+'''Run a human test of Idle wndow, dialog, and other widget classes.
+
+run(klass) runs a test for one class.
+runall() runs all the defined tests
+
+The file wih the widget class should end with
+if __name__ == '__main__':
+    <unittest, if there is one>
+    from idlelib.idle_test.htest import run
+    run(X)
+where X is a global object of the module. X must be a callable with a
+.__name__ attribute that accepts a 'parent' attribute. X will usually be
+a widget class, but a callable instance with .__name__ or a wrapper
+function also work. The name of wrapper functions, like _Editor_Window,
+should start with '_'.
+
+This file must then contain an instance of this template.
+_spec = {
+    'file': '',
+    'kwds': {'title': ''},
+    'msg': ""
+    }
+with X.__name__ prepended to _spec.
+File (no .py) is used in runall() to import the file and get the class.
+Kwds is passed to X (**kwds) after 'parent' is added, to initialize X.
+Msg. displayed is a window with a start button. hint as to how the user
+might test the widget. Closing The box skips or ends the test.
+'''
+from importlib import import_module
+import tkinter as tk
+
+# Template for class_spec dicts, copy and uncomment
+
+_Editor_window_spec = {
+    'file': 'EditorWindow',
+    'kwds': {},
+    'msg': "Test editor functions of interest"
+    }
+
+_Help_dialog_spec = {
+    'file': 'EditorWindow',
+    'kwds': {},
+    'msg': "If the help text displays, this works"
+    }
+
+AboutDialog_spec = {
+    'file': 'aboutDialog',
+    'kwds': {'title': 'About test'},
+    'msg': "Try each button"
+    }
+
+
+GetCfgSectionNameDialog_spec = {
+    'file': 'configSectionNameDialog',
+    'kwds': {'title':'Get Name',
+                 'message':'Enter something',
+                 'used_names': {'abc'},
+                 '_htest': True},
+    'msg': "After the text entered with [Ok] is stripped, <nothing>, "
+              "'abc', or more that 30 chars are errors.\n"
+              "Close 'Get Name' with a valid entry (printed to Shell), [Cancel], or [X]",
+    }
+
+def run(klas):
+    "Test the widget class klas using _spec dict"
+    root = tk.Tk()
+    klas_spec = globals()[klas.__name__+'_spec']
+    klas_kwds = klas_spec['kwds']
+    klas_kwds['parent'] = root
+    # This presumes that Idle consistently uses 'parent'
+    def run_klas():
+        widget = klas(**klas_kwds)
+        try:
+            print(widget.result)
+        except AttributeError:
+            pass
+    tk.Label(root, text=klas_spec['msg'], justify='left').pack()
+    tk.Button(root, text='Test ' + klas.__name__, command=run_klas).pack()
+    root.mainloop()
+
+def runall():
+    'Run all tests. Quick and dirty version.'
+    for k, d in globals().items():
+        if k.endswith('_spec'):
+            mod = import_module('idlelib.' + d['file'])
+            klas = getattr(mod, k[:-5])
+            run(klas)
+
+if __name__ == '__main__':
+    runall()
+

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 05:37:54 2014
From: python-checkins at python.org (terry.reedy)
Date: Mon, 12 May 2014 05:37:54 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?q?=29=3A_Merge_with_3=2E4?=
Message-ID: <3gRnsZ4Y71z7Ljy@mail.python.org>

http://hg.python.org/cpython/rev/d03a267b59da
changeset:   90653:d03a267b59da
parent:      90650:a6fc2405fdbc
parent:      90652:617656f7b538
user:        Terry Jan Reedy <tjreedy at udel.edu>
date:        Sun May 11 23:32:58 2014 -0400
summary:
  Merge with 3.4

files:
  Lib/idlelib/EditorWindow.py            |  16 +-
  Lib/idlelib/aboutDialog.py             |  11 +-
  Lib/idlelib/configSectionNameDialog.py |  28 +--
  Lib/idlelib/idle_test/htest.py         |  91 ++++++++++++++
  4 files changed, 114 insertions(+), 32 deletions(-)


diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py
--- a/Lib/idlelib/EditorWindow.py
+++ b/Lib/idlelib/EditorWindow.py
@@ -79,6 +79,8 @@
         self.parent = None
 
 helpDialog = HelpDialog()  # singleton instance
+def _Help_dialog(parent):  # wrapper for htest
+    helpDialog.show_dialog(parent)
 
 
 class EditorWindow(object):
@@ -1064,7 +1066,7 @@
         try:
             try:
                 mod = importlib.import_module('.' + name, package=__package__)
-            except ImportError:
+            except (ImportError, TypeError):
                 mod = importlib.import_module(name)
         except ImportError:
             print("\nFailed to import extension: ", name)
@@ -1700,19 +1702,21 @@
     tk.call('set', 'tcl_nonwordchars', '[^a-zA-Z0-9_]')
 
 
-def test():
-    root = Tk()
+def _Editor_window(parent):
+    root = parent
     fixwordbreaks(root)
     root.withdraw()
     if sys.argv[1:]:
         filename = sys.argv[1]
     else:
         filename = None
+    macosxSupport.setupApp(root, None)
     edit = EditorWindow(root=root, filename=filename)
     edit.set_close_hook(root.quit)
     edit.text.bind("<<close-all-windows>>", edit.close_event)
-    root.mainloop()
-    root.destroy()
 
 if __name__ == '__main__':
-    test()
+    from idlelib.idle_test.htest import run
+    if len(sys.argv) <= 1:
+        run(_Help_dialog)
+    run(_Editor_window)
diff --git a/Lib/idlelib/aboutDialog.py b/Lib/idlelib/aboutDialog.py
--- a/Lib/idlelib/aboutDialog.py
+++ b/Lib/idlelib/aboutDialog.py
@@ -12,7 +12,7 @@
     """Modal about dialog for idle
 
     """
-    def __init__(self,parent,title):
+    def __init__(self, parent, title):
         Toplevel.__init__(self, parent)
         self.configure(borderwidth=5)
         self.geometry("+%d+%d" % (parent.winfo_rootx()+30,
@@ -136,10 +136,5 @@
         self.destroy()
 
 if __name__ == '__main__':
-    # test the dialog
-    root = Tk()
-    def run():
-        from idlelib import aboutDialog
-        aboutDialog.AboutDialog(root, 'About')
-    Button(root, text='Dialog', command=run).pack()
-    root.mainloop()
+    from idlelib.idle_test.htest import run
+    run(AboutDialog)
diff --git a/Lib/idlelib/configSectionNameDialog.py b/Lib/idlelib/configSectionNameDialog.py
--- a/Lib/idlelib/configSectionNameDialog.py
+++ b/Lib/idlelib/configSectionNameDialog.py
@@ -8,10 +8,11 @@
 import tkinter.messagebox as tkMessageBox
 
 class GetCfgSectionNameDialog(Toplevel):
-    def __init__(self, parent, title, message, used_names):
+    def __init__(self, parent, title, message, used_names, _htest=False):
         """
         message - string, informational message to display
         used_names - string collection, names already in use for validity check
+        _htest - bool, change box location when running htest
         """
         Toplevel.__init__(self, parent)
         self.configure(borderwidth=5)
@@ -30,11 +31,12 @@
         self.messageInfo.config(width=self.frameMain.winfo_reqwidth())
         self.geometry(
                 "+%d+%d" % (
-                parent.winfo_rootx() +
-                (parent.winfo_width()/2 - self.winfo_reqwidth()/2),
-                parent.winfo_rooty() +
-                (parent.winfo_height()/2 - self.winfo_reqheight()/2)
-                ) )  #centre dialog over parent
+                    parent.winfo_rootx() +
+                    (parent.winfo_width()/2 - self.winfo_reqwidth()/2),
+                    parent.winfo_rooty() +
+                    ((parent.winfo_height()/2 - self.winfo_reqheight()/2)
+                    if not _htest else 100)
+                ) )  #centre dialog over parent (or below htest box)
         self.deiconify()  #geometry set, unhide
         self.wait_window()
 
@@ -92,15 +94,5 @@
     import unittest
     unittest.main('idlelib.idle_test.test_config_name', verbosity=2, exit=False)
 
-    # also human test the dialog
-    root = Tk()
-    def run():
-        dlg=GetCfgSectionNameDialog(root,'Get Name',
-                "After the text entered with [Ok] is stripped, <nothing>, "
-                "'abc', or more that 30 chars are errors. "
-                "Close with a valid entry (printed), [Cancel], or [X]",
-                {'abc'})
-        print(dlg.result)
-    Message(root, text='').pack()  # will be needed for oher dialog tests
-    Button(root, text='Click to begin dialog test', command=run).pack()
-    root.mainloop()
+    from idlelib.idle_test.htest import run
+    run(GetCfgSectionNameDialog)
diff --git a/Lib/idlelib/idle_test/htest.py b/Lib/idlelib/idle_test/htest.py
new file mode 100644
--- /dev/null
+++ b/Lib/idlelib/idle_test/htest.py
@@ -0,0 +1,91 @@
+'''Run a human test of Idle wndow, dialog, and other widget classes.
+
+run(klass) runs a test for one class.
+runall() runs all the defined tests
+
+The file wih the widget class should end with
+if __name__ == '__main__':
+    <unittest, if there is one>
+    from idlelib.idle_test.htest import run
+    run(X)
+where X is a global object of the module. X must be a callable with a
+.__name__ attribute that accepts a 'parent' attribute. X will usually be
+a widget class, but a callable instance with .__name__ or a wrapper
+function also work. The name of wrapper functions, like _Editor_Window,
+should start with '_'.
+
+This file must then contain an instance of this template.
+_spec = {
+    'file': '',
+    'kwds': {'title': ''},
+    'msg': ""
+    }
+with X.__name__ prepended to _spec.
+File (no .py) is used in runall() to import the file and get the class.
+Kwds is passed to X (**kwds) after 'parent' is added, to initialize X.
+Msg. displayed is a window with a start button. hint as to how the user
+might test the widget. Closing The box skips or ends the test.
+'''
+from importlib import import_module
+import tkinter as tk
+
+# Template for class_spec dicts, copy and uncomment
+
+_Editor_window_spec = {
+    'file': 'EditorWindow',
+    'kwds': {},
+    'msg': "Test editor functions of interest"
+    }
+
+_Help_dialog_spec = {
+    'file': 'EditorWindow',
+    'kwds': {},
+    'msg': "If the help text displays, this works"
+    }
+
+AboutDialog_spec = {
+    'file': 'aboutDialog',
+    'kwds': {'title': 'About test'},
+    'msg': "Try each button"
+    }
+
+
+GetCfgSectionNameDialog_spec = {
+    'file': 'configSectionNameDialog',
+    'kwds': {'title':'Get Name',
+                 'message':'Enter something',
+                 'used_names': {'abc'},
+                 '_htest': True},
+    'msg': "After the text entered with [Ok] is stripped, <nothing>, "
+              "'abc', or more that 30 chars are errors.\n"
+              "Close 'Get Name' with a valid entry (printed to Shell), [Cancel], or [X]",
+    }
+
+def run(klas):
+    "Test the widget class klas using _spec dict"
+    root = tk.Tk()
+    klas_spec = globals()[klas.__name__+'_spec']
+    klas_kwds = klas_spec['kwds']
+    klas_kwds['parent'] = root
+    # This presumes that Idle consistently uses 'parent'
+    def run_klas():
+        widget = klas(**klas_kwds)
+        try:
+            print(widget.result)
+        except AttributeError:
+            pass
+    tk.Label(root, text=klas_spec['msg'], justify='left').pack()
+    tk.Button(root, text='Test ' + klas.__name__, command=run_klas).pack()
+    root.mainloop()
+
+def runall():
+    'Run all tests. Quick and dirty version.'
+    for k, d in globals().items():
+        if k.endswith('_spec'):
+            mod = import_module('idlelib.' + d['file'])
+            klas = getattr(mod, k[:-5])
+            run(klas)
+
+if __name__ == '__main__':
+    runall()
+

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 05:37:56 2014
From: python-checkins at python.org (terry.reedy)
Date: Mon, 12 May 2014 05:37:56 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_whitespace?=
Message-ID: <3gRnsc6qhDz7Lk1@mail.python.org>

http://hg.python.org/cpython/rev/7d5aac1650af
changeset:   90654:7d5aac1650af
branch:      3.4
parent:      90652:617656f7b538
user:        Terry Jan Reedy <tjreedy at udel.edu>
date:        Sun May 11 23:35:09 2014 -0400
summary:
  whitespace

files:
  Lib/idlelib/idle_test/htest.py |  1 -
  1 files changed, 0 insertions(+), 1 deletions(-)


diff --git a/Lib/idlelib/idle_test/htest.py b/Lib/idlelib/idle_test/htest.py
--- a/Lib/idlelib/idle_test/htest.py
+++ b/Lib/idlelib/idle_test/htest.py
@@ -88,4 +88,3 @@
 
 if __name__ == '__main__':
     runall()
-

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 05:37:58 2014
From: python-checkins at python.org (terry.reedy)
Date: Mon, 12 May 2014 05:37:58 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?q?=29=3A_Merge_with_3=2E4?=
Message-ID: <3gRnsf4X0Hz7LkF@mail.python.org>

http://hg.python.org/cpython/rev/8d43e5c1f8b4
changeset:   90655:8d43e5c1f8b4
parent:      90653:d03a267b59da
parent:      90654:7d5aac1650af
user:        Terry Jan Reedy <tjreedy at udel.edu>
date:        Sun May 11 23:35:35 2014 -0400
summary:
  Merge with 3.4

files:
  Lib/idlelib/idle_test/htest.py |  1 -
  1 files changed, 0 insertions(+), 1 deletions(-)


diff --git a/Lib/idlelib/idle_test/htest.py b/Lib/idlelib/idle_test/htest.py
--- a/Lib/idlelib/idle_test/htest.py
+++ b/Lib/idlelib/idle_test/htest.py
@@ -88,4 +88,3 @@
 
 if __name__ == '__main__':
     runall()
-

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 05:37:59 2014
From: python-checkins at python.org (terry.reedy)
Date: Mon, 12 May 2014 05:37:59 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_whitespace?=
Message-ID: <3gRnsg5wZzz7Ljn@mail.python.org>

http://hg.python.org/cpython/rev/670fb496f1f6
changeset:   90656:670fb496f1f6
branch:      2.7
parent:      90651:460203eaf731
user:        Terry Jan Reedy <tjreedy at udel.edu>
date:        Sun May 11 23:37:26 2014 -0400
summary:
  whitespace

files:
  Lib/idlelib/idle_test/htest.py |  1 -
  1 files changed, 0 insertions(+), 1 deletions(-)


diff --git a/Lib/idlelib/idle_test/htest.py b/Lib/idlelib/idle_test/htest.py
--- a/Lib/idlelib/idle_test/htest.py
+++ b/Lib/idlelib/idle_test/htest.py
@@ -88,4 +88,3 @@
 
 if __name__ == '__main__':
     runall()
-

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 05:47:53 2014
From: python-checkins at python.org (terry.reedy)
Date: Mon, 12 May 2014 05:47:53 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE4MTA0?=
 =?utf-8?q?=3A_News_for_3=2E4_=28which_will_not_merge_forward=29=2E?=
Message-ID: <3gRp554YMZz7Ljc@mail.python.org>

http://hg.python.org/cpython/rev/aea4f427902f
changeset:   90657:aea4f427902f
branch:      3.4
parent:      90654:7d5aac1650af
user:        Terry Jan Reedy <tjreedy at udel.edu>
date:        Sun May 11 23:42:43 2014 -0400
summary:
  Issue #18104: News for 3.4 (which will not merge forward).

files:
  Misc/NEWS |  7 +++++++
  1 files changed, 7 insertions(+), 0 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -28,6 +28,13 @@
 
 - Issue #17752: Fix distutils tests when run from the installed location.
 
+IDLE
+----
+
+- Issue #18104: Add idlelib/idle_test/htest.py with a few sample tests to begin
+  consolidating and improving human-validated tests of Idle. Change other files
+  as needed to work with htest.  Running the module as __main__ runs all tests.
+
 
 What's New in Python 3.4.1rc1?
 ==============================

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 05:47:54 2014
From: python-checkins at python.org (terry.reedy)
Date: Mon, 12 May 2014 05:47:54 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_=2318104=3A_null_merge_of_3=2E4_News_entry?=
Message-ID: <3gRp566Bh4z7Ljn@mail.python.org>

http://hg.python.org/cpython/rev/b7bc43e96041
changeset:   90658:b7bc43e96041
parent:      90655:8d43e5c1f8b4
parent:      90657:aea4f427902f
user:        Terry Jan Reedy <tjreedy at udel.edu>
date:        Sun May 11 23:46:22 2014 -0400
summary:
  #18104: null merge of 3.4 News entry

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 05:47:56 2014
From: python-checkins at python.org (terry.reedy)
Date: Mon, 12 May 2014 05:47:56 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2318104=3A_News_for?=
 =?utf-8?q?_3=2E4=2C_in_proper_place=2E?=
Message-ID: <3gRp5823F5z7Ljx@mail.python.org>

http://hg.python.org/cpython/rev/7cdb38fa191e
changeset:   90659:7cdb38fa191e
user:        Terry Jan Reedy <tjreedy at udel.edu>
date:        Sun May 11 23:47:31 2014 -0400
summary:
  Issue #18104: News for 3.4, in proper place.

files:
  Misc/NEWS |  4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -358,6 +358,10 @@
 IDLE
 ----
 
+- Issue #18104: Add idlelib/idle_test/htest.py with a few sample tests to begin
+  consolidating and improving human-validated tests of Idle. Change other files
+  as needed to work with htest.  Running the module as __main__ runs all tests.
+
 - Issue #21139: Change default paragraph width to 72, the PEP 8 recommendation.
 
 - Issue #21284: Paragraph reformat test passes after user changes reformat width.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 06:47:54 2014
From: python-checkins at python.org (ned.deily)
Date: Mon, 12 May 2014 06:47:54 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMzgz?=
 =?utf-8?q?=3A_Allow_=22make_touch=22_to_work_when_building_outside_of_the?=
Message-ID: <3gRqQL4ydwz7Ljc@mail.python.org>

http://hg.python.org/cpython/rev/bc160f985b7b
changeset:   90660:bc160f985b7b
branch:      3.4
parent:      90657:aea4f427902f
user:        Ned Deily <nad at acm.org>
date:        Sun May 11 21:45:13 2014 -0700
summary:
  Issue #21383: Allow "make touch" to work when building outside of the
source directory.

files:
  Makefile.pre.in |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Makefile.pre.in b/Makefile.pre.in
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -1487,6 +1487,7 @@
 
 # Touch generated files
 touch:
+	cd $(srcdir); \
 	hg --config extensions.touch=Tools/hg/hgtouch.py touch -v
 
 # Sanitation targets -- clean leaves libraries, executables and tags

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 06:47:55 2014
From: python-checkins at python.org (ned.deily)
Date: Mon, 12 May 2014 06:47:55 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Issue_21383=3A_merge_from_3=2E4?=
Message-ID: <3gRqQM6cVqz7Ljf@mail.python.org>

http://hg.python.org/cpython/rev/9493fdad2a75
changeset:   90661:9493fdad2a75
parent:      90659:7cdb38fa191e
parent:      90660:bc160f985b7b
user:        Ned Deily <nad at acm.org>
date:        Sun May 11 21:47:21 2014 -0700
summary:
  Issue 21383: merge from 3.4

files:
  Makefile.pre.in |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Makefile.pre.in b/Makefile.pre.in
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -1497,6 +1497,7 @@
 
 # Touch generated files
 touch:
+	cd $(srcdir); \
 	hg --config extensions.touch=Tools/hg/hgtouch.py touch -v
 
 # Sanitation targets -- clean leaves libraries, executables and tags

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 12:55:24 2014
From: python-checkins at python.org (nick.coghlan)
Date: Mon, 12 May 2014 12:55:24 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_466=3A_restore_some_ratio?=
	=?utf-8?q?nale_lost_in_final_edits?=
Message-ID: <3gRzZN1FL5z7Ljc@mail.python.org>

http://hg.python.org/peps/rev/260c06fc4882
changeset:   5473:260c06fc4882
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Mon May 12 20:50:35 2014 +1000
summary:
  PEP 466: restore some rationale lost in final edits

files:
  pep-0466.txt |  41 ++++++++++++++++++++++++++++++++++++++++
  1 files changed, 41 insertions(+), 0 deletions(-)


diff --git a/pep-0466.txt b/pep-0466.txt
--- a/pep-0466.txt
+++ b/pep-0466.txt
@@ -322,6 +322,47 @@
 Enterprise Linux and its downstream derivatives.
 
 
+Why these particular changes?
+-----------------------------
+
+The key requirement for a feature to be considered for inclusion in this
+proposal was that it must have security implications *beyond* the specific
+application that is written in Python and the system that application is
+running on. Thus the focus on network security protocols, password storage
+and related cryptographic infrastructure - Python is a popular choice for
+the development of web services and clients, and thus the capabilities of
+widely used Python versions have implications for the security design of
+other services that may themselves be using newer versions of Python or
+other development languages, but need to interoperate with clients or
+servers written using older versions of Python.
+
+The intent behind this requirement was to minimise any impact that the
+introduction of this policy may have on the stability and compatibility of
+maintenance releases, while still addressing some key security concerns
+relating to the particular aspects of Python 2.7. It would be thoroughly
+counterproductive if end users became as cautious about updating to new
+Python 2.7 maintenance releases as they are about updating to new feature
+releases within the same release series.
+
+The ``ssl`` module changes are included in this proposal to bring the
+Python 2 series up to date with the past 4 years of evolution in network
+security standards, and make it easier for those standards to be broadly
+adopted in both servers and clients. Similarly the hash algorithm
+availability indicators in ``hashlib`` are included to make it easier for
+applications to detect and employ appropriate hash definitions across both
+Python 2 and 3.
+
+The ``hmac.compare_digest()`` and ``hashlib.pbkdf2_hmac()`` are included to
+help lower the barriers to secure password storage and checking in Python 2
+server applications.
+
+The os.urandom change has been included in this proposal to help encourage
+users to leave the task of providing high quality random numbers for
+cryptographic use case to operating system vendors (as this is a genuinely
+hard problem, and operating system developers have more tools available to
+deal with it than Python application runtimes)
+
+
 Rejected alternative: just advise developers to migrate to Python 3
 -------------------------------------------------------------------
 

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Mon May 12 12:56:00 2014
From: python-checkins at python.org (nick.coghlan)
Date: Mon, 12 May 2014 12:56:00 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_466=3A_tweak_wording_of_o?=
 =?utf-8?q?s=2Eurandom=28=29_rationale?=
Message-ID: <3gRzb46G59z7Ljc@mail.python.org>

http://hg.python.org/peps/rev/7d8fa3101fb8
changeset:   5474:7d8fa3101fb8
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Mon May 12 20:55:52 2014 +1000
summary:
  PEP 466: tweak wording of os.urandom() rationale

files:
  pep-0466.txt |  12 +++++++-----
  1 files changed, 7 insertions(+), 5 deletions(-)


diff --git a/pep-0466.txt b/pep-0466.txt
--- a/pep-0466.txt
+++ b/pep-0466.txt
@@ -356,11 +356,13 @@
 help lower the barriers to secure password storage and checking in Python 2
 server applications.
 
-The os.urandom change has been included in this proposal to help encourage
-users to leave the task of providing high quality random numbers for
-cryptographic use case to operating system vendors (as this is a genuinely
-hard problem, and operating system developers have more tools available to
-deal with it than Python application runtimes)
+The ``os.urandom()`` change has been included in this proposal to further
+encourage users to leave the task of providing high quality random numbers
+for cryptographic use cases to operating system vendors. The use of
+insufficiently random numbers has the potential to compromise *any*
+cryptographic system, and operating system developers have more tools
+available to address that problem adequately than the typical Python
+application runtime.
 
 
 Rejected alternative: just advise developers to migrate to Python 3

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Mon May 12 19:05:18 2014
From: python-checkins at python.org (guido.van.rossum)
Date: Mon, 12 May 2014 19:05:18 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogRml4?=
 =?utf-8?q?_upstream_issue_168=3A_StreamReader=2Eread=28-1=29_from_pipe_ma?=
 =?utf-8?q?y_hang_if?=
Message-ID: <3gS7nB0dSJz7LjN@mail.python.org>

http://hg.python.org/cpython/rev/909ea8cc86bb
changeset:   90662:909ea8cc86bb
branch:      3.4
parent:      90660:bc160f985b7b
user:        Guido van Rossum <guido at python.org>
date:        Mon May 12 10:04:37 2014 -0700
summary:
  asyncio: Fix upstream issue 168: StreamReader.read(-1) from pipe may hang if data exceeds buffer limit.

files:
  Lib/asyncio/streams.py                |  17 ++++--
  Lib/test/test_asyncio/test_streams.py |  36 +++++++++++++++
  2 files changed, 47 insertions(+), 6 deletions(-)


diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py
--- a/Lib/asyncio/streams.py
+++ b/Lib/asyncio/streams.py
@@ -419,12 +419,17 @@
             return b''
 
         if n < 0:
-            while not self._eof:
-                self._waiter = self._create_waiter('read')
-                try:
-                    yield from self._waiter
-                finally:
-                    self._waiter = None
+            # This used to just loop creating a new waiter hoping to
+            # collect everything in self._buffer, but that would
+            # deadlock if the subprocess sends more than self.limit
+            # bytes.  So just call self.read(self._limit) until EOF.
+            blocks = []
+            while True:
+                block = yield from self.read(self._limit)
+                if not block:
+                    break
+                blocks.append(block)
+            return b''.join(blocks)
         else:
             if not self._buffer and not self._eof:
                 self._waiter = self._create_waiter('read')
diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py
--- a/Lib/test/test_asyncio/test_streams.py
+++ b/Lib/test/test_asyncio/test_streams.py
@@ -1,7 +1,9 @@
 """Tests for streams.py."""
 
 import gc
+import os
 import socket
+import sys
 import unittest
 from unittest import mock
 try:
@@ -583,6 +585,40 @@
             server.stop()
             self.assertEqual(msg, b"hello world!\n")
 
+    @unittest.skipIf(sys.platform == 'win32', "Don't have pipes")
+    def test_read_all_from_pipe_reader(self):
+        # See Tulip issue 168.  This test is derived from the example
+        # subprocess_attach_read_pipe.py, but we configure the
+        # StreamReader's limit so that twice it is less than the size
+        # of the data writter.  Also we must explicitly attach a child
+        # watcher to the event loop.
+
+        watcher = asyncio.get_child_watcher()
+        watcher.attach_loop(self.loop)
+
+        code = """\
+import os, sys
+fd = int(sys.argv[1])
+os.write(fd, b'data')
+os.close(fd)
+"""
+        rfd, wfd = os.pipe()
+        args = [sys.executable, '-c', code, str(wfd)]
+
+        pipe = open(rfd, 'rb', 0)
+        reader = asyncio.StreamReader(loop=self.loop, limit=1)
+        protocol = asyncio.StreamReaderProtocol(reader, loop=self.loop)
+        transport, _ = self.loop.run_until_complete(
+            self.loop.connect_read_pipe(lambda: protocol, pipe))
+
+        proc = self.loop.run_until_complete(
+            asyncio.create_subprocess_exec(*args, pass_fds={wfd}, loop=self.loop))
+        self.loop.run_until_complete(proc.wait())
+
+        os.close(wfd)
+        data = self.loop.run_until_complete(reader.read(-1))
+        self.assertEqual(data, b'data')
+
 
 if __name__ == '__main__':
     unittest.main()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 19:05:19 2014
From: python-checkins at python.org (guido.van.rossum)
Date: Mon, 12 May 2014 19:05:19 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge_3=2E4-=3Edefault=3A_asyncio=3A_Fix_upstream_issue_?=
 =?utf-8?q?168=3A_StreamReader=2Eread=28-1=29_from?=
Message-ID: <3gS7nC2qJXz7LjP@mail.python.org>

http://hg.python.org/cpython/rev/2af5a52b9b87
changeset:   90663:2af5a52b9b87
parent:      90661:9493fdad2a75
parent:      90662:909ea8cc86bb
user:        Guido van Rossum <guido at python.org>
date:        Mon May 12 10:05:04 2014 -0700
summary:
  Merge 3.4->default: asyncio: Fix upstream issue 168: StreamReader.read(-1) from pipe may hang if data exceeds buffer limit.

files:
  Lib/asyncio/streams.py                |  17 ++++--
  Lib/test/test_asyncio/test_streams.py |  36 +++++++++++++++
  2 files changed, 47 insertions(+), 6 deletions(-)


diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py
--- a/Lib/asyncio/streams.py
+++ b/Lib/asyncio/streams.py
@@ -419,12 +419,17 @@
             return b''
 
         if n < 0:
-            while not self._eof:
-                self._waiter = self._create_waiter('read')
-                try:
-                    yield from self._waiter
-                finally:
-                    self._waiter = None
+            # This used to just loop creating a new waiter hoping to
+            # collect everything in self._buffer, but that would
+            # deadlock if the subprocess sends more than self.limit
+            # bytes.  So just call self.read(self._limit) until EOF.
+            blocks = []
+            while True:
+                block = yield from self.read(self._limit)
+                if not block:
+                    break
+                blocks.append(block)
+            return b''.join(blocks)
         else:
             if not self._buffer and not self._eof:
                 self._waiter = self._create_waiter('read')
diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py
--- a/Lib/test/test_asyncio/test_streams.py
+++ b/Lib/test/test_asyncio/test_streams.py
@@ -1,7 +1,9 @@
 """Tests for streams.py."""
 
 import gc
+import os
 import socket
+import sys
 import unittest
 from unittest import mock
 try:
@@ -583,6 +585,40 @@
             server.stop()
             self.assertEqual(msg, b"hello world!\n")
 
+    @unittest.skipIf(sys.platform == 'win32', "Don't have pipes")
+    def test_read_all_from_pipe_reader(self):
+        # See Tulip issue 168.  This test is derived from the example
+        # subprocess_attach_read_pipe.py, but we configure the
+        # StreamReader's limit so that twice it is less than the size
+        # of the data writter.  Also we must explicitly attach a child
+        # watcher to the event loop.
+
+        watcher = asyncio.get_child_watcher()
+        watcher.attach_loop(self.loop)
+
+        code = """\
+import os, sys
+fd = int(sys.argv[1])
+os.write(fd, b'data')
+os.close(fd)
+"""
+        rfd, wfd = os.pipe()
+        args = [sys.executable, '-c', code, str(wfd)]
+
+        pipe = open(rfd, 'rb', 0)
+        reader = asyncio.StreamReader(loop=self.loop, limit=1)
+        protocol = asyncio.StreamReaderProtocol(reader, loop=self.loop)
+        transport, _ = self.loop.run_until_complete(
+            self.loop.connect_read_pipe(lambda: protocol, pipe))
+
+        proc = self.loop.run_until_complete(
+            asyncio.create_subprocess_exec(*args, pass_fds={wfd}, loop=self.loop))
+        self.loop.run_until_complete(proc.wait())
+
+        os.close(wfd)
+        data = self.loop.run_until_complete(reader.read(-1))
+        self.assertEqual(data, b'data')
+
 
 if __name__ == '__main__':
     unittest.main()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 20:36:54 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Mon, 12 May 2014 20:36:54 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2316531=3A_ipaddres?=
 =?utf-8?q?s=2EIPv4Network_and_ipaddress=2EIPv6Network_now_accept_an?=
Message-ID: <3gS9pt0Hbnz7Ljd@mail.python.org>

http://hg.python.org/cpython/rev/4e33c343a264
changeset:   90664:4e33c343a264
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Mon May 12 20:36:46 2014 +0200
summary:
  Issue #16531: ipaddress.IPv4Network and ipaddress.IPv6Network now accept an (address, netmask) tuple argument, so as to easily construct network objects from existing addresses.

files:
  Doc/library/ipaddress.rst  |   19 ++++
  Lib/ipaddress.py           |   95 ++++++++++++++++----
  Lib/test/test_ipaddress.py |  113 +++++++++++++++++++++++++
  Misc/NEWS                  |    4 +
  4 files changed, 210 insertions(+), 21 deletions(-)


diff --git a/Doc/library/ipaddress.rst b/Doc/library/ipaddress.rst
--- a/Doc/library/ipaddress.rst
+++ b/Doc/library/ipaddress.rst
@@ -392,6 +392,12 @@
    3. An integer packed into a :class:`bytes` object of length 4, big-endian.
       The interpretation is similar to an integer *address*.
 
+   4. A two-tuple of an address description and a netmask, where the address
+      description is either a string, a 32-bits integer, a 4-bytes packed
+      integer, or an existing IPv4Address object; and the netmask is either
+      an integer representing the prefix length (e.g. ``24``) or a string
+      representing the prefix mask (e.g. ``255.255.255.0``).
+
    An :exc:`AddressValueError` is raised if *address* is not a valid IPv4
    address.  A :exc:`NetmaskValueError` is raised if the mask is not valid for
    an IPv4 address.
@@ -404,6 +410,10 @@
    objects will raise :exc:`TypeError` if the argument's IP version is
    incompatible to ``self``
 
+   .. versionchanged:: 3.5
+
+      Added the two-tuple form for the *address* constructor parameter.
+
    .. attribute:: version
    .. attribute:: max_prefixlen
 
@@ -568,6 +578,11 @@
    3. An integer packed into a :class:`bytes` object of length 16, bit-endian.
       The interpretation is similar to an integer *address*.
 
+   4. A two-tuple of an address description and a netmask, where the address
+      description is either a string, a 128-bits integer, a 16-bytes packed
+      integer, or an existing IPv4Address object; and the netmask is an
+      integer representing the prefix length.
+
    An :exc:`AddressValueError` is raised if *address* is not a valid IPv6
    address.  A :exc:`NetmaskValueError` is raised if the mask is not valid for
    an IPv6 address.
@@ -576,6 +591,10 @@
    then :exc:`ValueError` is raised.  Otherwise, the host bits are masked out
    to determine the appropriate network address.
 
+   .. versionchanged:: 3.5
+
+      Added the two-tuple form for the *address* constructor parameter.
+
    .. attribute:: version
    .. attribute:: max_prefixlen
    .. attribute:: is_multicast
diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py
--- a/Lib/ipaddress.py
+++ b/Lib/ipaddress.py
@@ -991,15 +991,15 @@
                 raise ValueError('cannot set prefixlen_diff and new_prefix')
             prefixlen_diff = self._prefixlen - new_prefix
 
-        if self.prefixlen - prefixlen_diff < 0:
+        new_prefixlen = self.prefixlen - prefixlen_diff
+        if new_prefixlen < 0:
             raise ValueError(
                 'current prefixlen is %d, cannot have a prefixlen_diff of %d' %
                 (self.prefixlen, prefixlen_diff))
-        # TODO (pmoody): optimize this.
-        t = self.__class__('%s/%d' % (self.network_address,
-                                      self.prefixlen - prefixlen_diff),
-                                     strict=False)
-        return t.__class__('%s/%d' % (t.network_address, t.prefixlen))
+        return self.__class__((
+            int(self.network_address) & (int(self.netmask) << prefixlen_diff),
+            new_prefixlen
+            ))
 
     @property
     def is_multicast(self):
@@ -1389,6 +1389,18 @@
             self._prefixlen = self._max_prefixlen
             return
 
+        if isinstance(address, tuple):
+            IPv4Address.__init__(self, address[0])
+            if len(address) > 1:
+                self._prefixlen = int(address[1])
+            else:
+                self._prefixlen = self._max_prefixlen
+
+            self.network = IPv4Network(address, strict=False)
+            self.netmask = self.network.netmask
+            self.hostmask = self.network.hostmask
+            return
+
         addr = _split_optional_netmask(address)
         IPv4Address.__init__(self, addr[0])
 
@@ -1504,22 +1516,42 @@
         _BaseV4.__init__(self, address)
         _BaseNetwork.__init__(self, address)
 
-        # Constructing from a packed address
-        if isinstance(address, bytes):
+        # Constructing from a packed address or integer
+        if isinstance(address, (int, bytes)):
             self.network_address = IPv4Address(address)
             self._prefixlen = self._max_prefixlen
             self.netmask = IPv4Address(self._ALL_ONES)
             #fixme: address/network test here
             return
 
-        # Efficient constructor from integer.
-        if isinstance(address, int):
-            self.network_address = IPv4Address(address)
-            self._prefixlen = self._max_prefixlen
-            self.netmask = IPv4Address(self._ALL_ONES)
-            #fixme: address/network test here.
+        if isinstance(address, tuple):
+            if len(address) > 1:
+                # If address[1] is a string, treat it like a netmask.
+                if isinstance(address[1], str):
+                    self.netmask = IPv4Address(address[1])
+                    self._prefixlen = self._prefix_from_ip_int(
+                        int(self.netmask))
+                # address[1] should be an int.
+                else:
+                    self._prefixlen = int(address[1])
+                    self.netmask = IPv4Address(self._ip_int_from_prefix(
+                            self._prefixlen))
+            # We weren't given an address[1].
+            else:
+                self._prefixlen = self._max_prefixlen
+                self.netmask = IPv4Address(self._ip_int_from_prefix(
+                        self._prefixlen))
+            self.network_address = IPv4Address(address[0])
+            packed = int(self.network_address)
+            if packed & int(self.netmask) != packed:
+                if strict:
+                    raise ValueError('%s has host bits set' % self)
+                else:
+                    self.network_address = IPv4Address(packed &
+                                                       int(self.netmask))
             return
 
+
         # Assume input argument to be string or any object representation
         # which converts into a formatted IP prefix string.
         addr = _split_optional_netmask(address)
@@ -2030,6 +2062,16 @@
             self.network = IPv6Network(self._ip)
             self._prefixlen = self._max_prefixlen
             return
+        if isinstance(address, tuple):
+            IPv6Address.__init__(self, address[0])
+            if len(address) > 1:
+                self._prefixlen = int(address[1])
+            else:
+                self._prefixlen = self._max_prefixlen
+            self.network = IPv6Network(address, strict=False)
+            self.netmask = self.network.netmask
+            self.hostmask = self.network.hostmask
+            return
 
         addr = _split_optional_netmask(address)
         IPv6Address.__init__(self, addr[0])
@@ -2147,18 +2189,29 @@
         _BaseV6.__init__(self, address)
         _BaseNetwork.__init__(self, address)
 
-        # Efficient constructor from integer.
-        if isinstance(address, int):
+        # Efficient constructor from integer or packed address
+        if isinstance(address, (bytes, int)):
             self.network_address = IPv6Address(address)
             self._prefixlen = self._max_prefixlen
             self.netmask = IPv6Address(self._ALL_ONES)
             return
 
-        # Constructing from a packed address
-        if isinstance(address, bytes):
-            self.network_address = IPv6Address(address)
-            self._prefixlen = self._max_prefixlen
-            self.netmask = IPv6Address(self._ALL_ONES)
+        if isinstance(address, tuple):
+            self.network_address = IPv6Address(address[0])
+            if len(address) > 1:
+                self._prefixlen = int(address[1])
+            else:
+                self._prefixlen = self._max_prefixlen
+            self.netmask = IPv6Address(self._ip_int_from_prefix(
+                    self._prefixlen))
+            self.network_address = IPv6Address(address[0])
+            packed = int(self.network_address)
+            if packed & int(self.netmask) != packed:
+                if strict:
+                    raise ValueError('%s has host bits set' % self)
+                else:
+                    self.network_address = IPv6Address(packed &
+                                                       int(self.netmask))
             return
 
         # Assume input argument to be string or any object representation
diff --git a/Lib/test/test_ipaddress.py b/Lib/test/test_ipaddress.py
--- a/Lib/test/test_ipaddress.py
+++ b/Lib/test/test_ipaddress.py
@@ -628,6 +628,119 @@
         self.assertEqual("IPv6Interface('::1/128')",
                          repr(ipaddress.IPv6Interface('::1')))
 
+    # issue #16531: constructing IPv4Network from a (address, mask) tuple
+    def testIPv4Tuple(self):
+        # /32
+        ip = ipaddress.IPv4Address('192.0.2.1')
+        net = ipaddress.IPv4Network('192.0.2.1/32')
+        self.assertEqual(ipaddress.IPv4Network(('192.0.2.1', 32)), net)
+        self.assertEqual(ipaddress.IPv4Network((ip, 32)), net)
+        self.assertEqual(ipaddress.IPv4Network((3221225985, 32)), net)
+        self.assertEqual(ipaddress.IPv4Network(('192.0.2.1',
+                                                '255.255.255.255')), net)
+        self.assertEqual(ipaddress.IPv4Network((ip,
+                                                '255.255.255.255')), net)
+        self.assertEqual(ipaddress.IPv4Network((3221225985,
+                                                '255.255.255.255')), net)
+        # strict=True and host bits set
+        with self.assertRaises(ValueError):
+            ipaddress.IPv4Network(('192.0.2.1', 24))
+        with self.assertRaises(ValueError):
+            ipaddress.IPv4Network((ip, 24))
+        with self.assertRaises(ValueError):
+            ipaddress.IPv4Network((3221225985, 24))
+        with self.assertRaises(ValueError):
+            ipaddress.IPv4Network(('192.0.2.1', '255.255.255.0'))
+        with self.assertRaises(ValueError):
+            ipaddress.IPv4Network((ip, '255.255.255.0'))
+        with self.assertRaises(ValueError):
+            ipaddress.IPv4Network((3221225985, '255.255.255.0'))
+        # strict=False and host bits set
+        net = ipaddress.IPv4Network('192.0.2.0/24')
+        self.assertEqual(ipaddress.IPv4Network(('192.0.2.1', 24),
+                                               strict=False), net)
+        self.assertEqual(ipaddress.IPv4Network((ip, 24),
+                                               strict=False), net)
+        self.assertEqual(ipaddress.IPv4Network((3221225985, 24),
+                                               strict=False), net)
+        self.assertEqual(ipaddress.IPv4Network(('192.0.2.1',
+                                                '255.255.255.0'),
+                                               strict=False), net)
+        self.assertEqual(ipaddress.IPv4Network((ip,
+                                                '255.255.255.0'),
+                                               strict=False), net)
+        self.assertEqual(ipaddress.IPv4Network((3221225985,
+                                                '255.255.255.0'),
+                                               strict=False), net)
+
+        # /24
+        ip = ipaddress.IPv4Address('192.0.2.0')
+        net = ipaddress.IPv4Network('192.0.2.0/24')
+        self.assertEqual(ipaddress.IPv4Network(('192.0.2.0',
+                                                '255.255.255.0')), net)
+        self.assertEqual(ipaddress.IPv4Network((ip,
+                                                '255.255.255.0')), net)
+        self.assertEqual(ipaddress.IPv4Network((3221225984,
+                                                '255.255.255.0')), net)
+        self.assertEqual(ipaddress.IPv4Network(('192.0.2.0', 24)), net)
+        self.assertEqual(ipaddress.IPv4Network((ip, 24)), net)
+        self.assertEqual(ipaddress.IPv4Network((3221225984, 24)), net)
+
+        self.assertEqual(ipaddress.IPv4Interface(('192.0.2.1', 24)),
+                         ipaddress.IPv4Interface('192.0.2.1/24'))
+        self.assertEqual(ipaddress.IPv4Interface((3221225985, 24)),
+                         ipaddress.IPv4Interface('192.0.2.1/24'))
+
+    # issue #16531: constructing IPv6Network from a (address, mask) tuple
+    def testIPv6Tuple(self):
+        # /128
+        ip = ipaddress.IPv6Address('2001:db8::')
+        net = ipaddress.IPv6Network('2001:db8::/128')
+        self.assertEqual(ipaddress.IPv6Network(('2001:db8::', '128')),
+                         net)
+        self.assertEqual(ipaddress.IPv6Network(
+                (42540766411282592856903984951653826560, 128)),
+                         net)
+        self.assertEqual(ipaddress.IPv6Network((ip, '128')),
+                         net)
+        ip = ipaddress.IPv6Address('2001:db8::')
+        net = ipaddress.IPv6Network('2001:db8::/96')
+        self.assertEqual(ipaddress.IPv6Network(('2001:db8::', '96')),
+                         net)
+        self.assertEqual(ipaddress.IPv6Network(
+                (42540766411282592856903984951653826560, 96)),
+                         net)
+        self.assertEqual(ipaddress.IPv6Network((ip, '96')),
+                         net)
+
+        # strict=True and host bits set
+        ip = ipaddress.IPv6Address('2001:db8::1')
+        with self.assertRaises(ValueError):
+            ipaddress.IPv6Network(('2001:db8::1', 96))
+        with self.assertRaises(ValueError):
+            ipaddress.IPv6Network((
+                42540766411282592856903984951653826561, 96))
+        with self.assertRaises(ValueError):
+            ipaddress.IPv6Network((ip, 96))
+        # strict=False and host bits set
+        net = ipaddress.IPv6Network('2001:db8::/96')
+        self.assertEqual(ipaddress.IPv6Network(('2001:db8::1', 96),
+                                               strict=False),
+                         net)
+        self.assertEqual(ipaddress.IPv6Network(
+                             (42540766411282592856903984951653826561, 96),
+                             strict=False),
+                         net)
+        self.assertEqual(ipaddress.IPv6Network((ip, 96), strict=False),
+                         net)
+
+        # /96
+        self.assertEqual(ipaddress.IPv6Interface(('2001:db8::1', '96')),
+                         ipaddress.IPv6Interface('2001:db8::1/96'))
+        self.assertEqual(ipaddress.IPv6Interface(
+                (42540766411282592856903984951653826561, '96')),
+                         ipaddress.IPv6Interface('2001:db8::1/96'))
+
     # issue57
     def testAddressIntMath(self):
         self.assertEqual(ipaddress.IPv4Address('1.1.1.1') + 255,
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -81,6 +81,10 @@
 Library
 -------
 
+- Issue #16531: ipaddress.IPv4Network and ipaddress.IPv6Network now accept
+  an (address, netmask) tuple argument, so as to easily construct network
+  objects from existing addresses.
+
 - Issue #21156: importlib.abc.InspectLoader.source_to_code() is now a
   staticmethod.
 

-- 
Repository URL: http://hg.python.org/cpython

From rdmurray at bitdance.com  Mon May 12 20:43:22 2014
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 12 May 2014 14:43:22 -0400
Subject: [Python-checkins]
	=?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?q?=29=3A_Merge_3=2E4-=3Edefault=3A_asyncio=3A_Fix_upstream?=
	=?utf-8?q?_issue_168=3A_StreamReader=2Eread=28-1=29_from?=
In-Reply-To: <3gS7nC2qJXz7LjP@mail.python.org>
References: <3gS7nC2qJXz7LjP@mail.python.org>
Message-ID: <20140512184322.A9D01250D4E@webabinitio.net>

These changes appear to have caused several builbot failures, and
there doesn't appear to be a bugs.python.org issue to report it to.
One failure example:

    http://buildbot.python.org/all/builders/PPC64%20PowerLinux%203.4/builds/119

test_asyncio fails similarly for me on tip.

On Mon, 12 May 2014 19:05:19 +0200, guido.van.rossum <python-checkins at python.org> wrote:
> http://hg.python.org/cpython/rev/2af5a52b9b87
> changeset:   90663:2af5a52b9b87
> parent:      90661:9493fdad2a75
> parent:      90662:909ea8cc86bb
> user:        Guido van Rossum <guido at python.org>
> date:        Mon May 12 10:05:04 2014 -0700
> summary:
>   Merge 3.4->default: asyncio: Fix upstream issue 168: StreamReader.read(-1) from pipe may hang if data exceeds buffer limit.
> 
> files:
>   Lib/asyncio/streams.py                |  17 ++++--
>   Lib/test/test_asyncio/test_streams.py |  36 +++++++++++++++
>   2 files changed, 47 insertions(+), 6 deletions(-)
> 
> 
> diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py
> --- a/Lib/asyncio/streams.py
> +++ b/Lib/asyncio/streams.py
> @@ -419,12 +419,17 @@
>              return b''
>  
>          if n < 0:
> -            while not self._eof:
> -                self._waiter = self._create_waiter('read')
> -                try:
> -                    yield from self._waiter
> -                finally:
> -                    self._waiter = None
> +            # This used to just loop creating a new waiter hoping to
> +            # collect everything in self._buffer, but that would
> +            # deadlock if the subprocess sends more than self.limit
> +            # bytes.  So just call self.read(self._limit) until EOF.
> +            blocks = []
> +            while True:
> +                block = yield from self.read(self._limit)
> +                if not block:
> +                    break
> +                blocks.append(block)
> +            return b''.join(blocks)
>          else:
>              if not self._buffer and not self._eof:
>                  self._waiter = self._create_waiter('read')
> diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py
> --- a/Lib/test/test_asyncio/test_streams.py
> +++ b/Lib/test/test_asyncio/test_streams.py
> @@ -1,7 +1,9 @@
>  """Tests for streams.py."""
>  
>  import gc
> +import os
>  import socket
> +import sys
>  import unittest
>  from unittest import mock
>  try:
> @@ -583,6 +585,40 @@
>              server.stop()
>              self.assertEqual(msg, b"hello world!\n")
>  
> +    @unittest.skipIf(sys.platform == 'win32', "Don't have pipes")
> +    def test_read_all_from_pipe_reader(self):
> +        # See Tulip issue 168.  This test is derived from the example
> +        # subprocess_attach_read_pipe.py, but we configure the
> +        # StreamReader's limit so that twice it is less than the size
> +        # of the data writter.  Also we must explicitly attach a child
> +        # watcher to the event loop.
> +
> +        watcher = asyncio.get_child_watcher()
> +        watcher.attach_loop(self.loop)
> +
> +        code = """\
> +import os, sys
> +fd = int(sys.argv[1])
> +os.write(fd, b'data')
> +os.close(fd)
> +"""
> +        rfd, wfd = os.pipe()
> +        args = [sys.executable, '-c', code, str(wfd)]
> +
> +        pipe = open(rfd, 'rb', 0)
> +        reader = asyncio.StreamReader(loop=self.loop, limit=1)
> +        protocol = asyncio.StreamReaderProtocol(reader, loop=self.loop)
> +        transport, _ = self.loop.run_until_complete(
> +            self.loop.connect_read_pipe(lambda: protocol, pipe))
> +
> +        proc = self.loop.run_until_complete(
> +            asyncio.create_subprocess_exec(*args, pass_fds={wfd}, loop=self.loop))
> +        self.loop.run_until_complete(proc.wait())
> +
> +        os.close(wfd)
> +        data = self.loop.run_until_complete(reader.read(-1))
> +        self.assertEqual(data, b'data')
> +
>  
>  if __name__ == '__main__':
>      unittest.main()
> 
> -- 
> Repository URL: http://hg.python.org/cpython
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> https://mail.python.org/mailman/listinfo/python-checkins

From python-checkins at python.org  Mon May 12 22:43:30 2014
From: python-checkins at python.org (victor.stinner)
Date: Mon, 12 May 2014 22:43:30 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxNDIy?=
 =?utf-8?q?=3A_Add_a_test_to_check_that_bool_=3C=3C_int_and_bool_=3E=3E_in?=
 =?utf-8?q?t_return_an_int?=
Message-ID: <3gSDcy20bTz7Ljf@mail.python.org>

http://hg.python.org/cpython/rev/ef49aaad3812
changeset:   90665:ef49aaad3812
branch:      3.4
parent:      90662:909ea8cc86bb
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Mon May 12 22:35:40 2014 +0200
summary:
  Issue #21422: Add a test to check that bool << int and bool >> int return an int

files:
  Lib/test/test_long.py |  7 +++++++
  1 files changed, 7 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_long.py b/Lib/test/test_long.py
--- a/Lib/test/test_long.py
+++ b/Lib/test/test_long.py
@@ -1235,6 +1235,13 @@
         for n in map(int, integers):
             self.assertEqual(n, 0)
 
+    def test_shift_bool(self):
+        # Issue #21422: ensure that bool << int and bool >> int return int
+        for value in (True, False):
+            for shift in (0, 2):
+                self.assertEqual(type(value << shift), int)
+                self.assertEqual(type(value >> shift), int)
+
 
 def test_main():
     support.run_unittest(LongTest)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 22:43:31 2014
From: python-checkins at python.org (victor.stinner)
Date: Mon, 12 May 2014 22:43:31 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_=28Merge_3=2E4=29_Issue_=2321422=3A_Add_a_test_to_check_?=
 =?utf-8?q?that_bool_=3C=3C_int_and_bool_=3E=3E_int?=
Message-ID: <3gSDcz3nMPz7Ljf@mail.python.org>

http://hg.python.org/cpython/rev/3da4aed1d18a
changeset:   90666:3da4aed1d18a
parent:      90664:4e33c343a264
parent:      90665:ef49aaad3812
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Mon May 12 22:43:07 2014 +0200
summary:
  (Merge 3.4) Issue #21422: Add a test to check that bool << int and bool >> int
return an int

files:
  Lib/test/test_long.py |  7 +++++++
  1 files changed, 7 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_long.py b/Lib/test/test_long.py
--- a/Lib/test/test_long.py
+++ b/Lib/test/test_long.py
@@ -1235,6 +1235,13 @@
         for n in map(int, integers):
             self.assertEqual(n, 0)
 
+    def test_shift_bool(self):
+        # Issue #21422: ensure that bool << int and bool >> int return int
+        for value in (True, False):
+            for shift in (0, 2):
+                self.assertEqual(type(value << shift), int)
+                self.assertEqual(type(value >> shift), int)
+
 
 def test_main():
     support.run_unittest(LongTest)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 23:25:35 2014
From: python-checkins at python.org (victor.stinner)
Date: Mon, 12 May 2014 23:25:35 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxNDg1?=
 =?utf-8?q?=3A_remove_unnecesary_=2Eflush=28=29_calls_in_the_asyncio_subpr?=
 =?utf-8?q?ocess_code?=
Message-ID: <3gSFYW62bHz7LjP@mail.python.org>

http://hg.python.org/cpython/rev/c0404f0da01a
changeset:   90667:c0404f0da01a
branch:      3.4
parent:      90665:ef49aaad3812
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Mon May 12 23:25:09 2014 +0200
summary:
  Issue #21485: remove unnecesary .flush() calls in the asyncio subprocess code
example

files:
  Doc/library/asyncio-subprocess.rst |  4 +---
  1 files changed, 1 insertions(+), 3 deletions(-)


diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst
--- a/Doc/library/asyncio-subprocess.rst
+++ b/Doc/library/asyncio-subprocess.rst
@@ -262,9 +262,7 @@
         stdout = stdout.decode('ascii').rstrip()
         print("Platform: %s" % stdout)
     else:
-        print("Python failed with exit code %s:" % exitcode)
-        sys.stdout.flush()
-        sys.stdout.buffer.flush()
+        print("Python failed with exit code %s:" % exitcode, flush=True)
         sys.stdout.buffer.write(stdout)
         sys.stdout.buffer.flush()
     loop.close()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon May 12 23:25:37 2014
From: python-checkins at python.org (victor.stinner)
Date: Mon, 12 May 2014 23:25:37 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_=28Merge_3=2E4=29_Issue_=2321485=3A_remove_unnecesary_?=
 =?utf-8?q?=2Eflush=28=29_calls_in_the_asyncio?=
Message-ID: <3gSFYY0bkXz7Ljt@mail.python.org>

http://hg.python.org/cpython/rev/3c26389d741c
changeset:   90668:3c26389d741c
parent:      90666:3da4aed1d18a
parent:      90667:c0404f0da01a
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Mon May 12 23:25:25 2014 +0200
summary:
  (Merge 3.4) Issue #21485: remove unnecesary .flush() calls in the asyncio
subprocess code example

files:
  Doc/library/asyncio-subprocess.rst |  4 +---
  1 files changed, 1 insertions(+), 3 deletions(-)


diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst
--- a/Doc/library/asyncio-subprocess.rst
+++ b/Doc/library/asyncio-subprocess.rst
@@ -262,9 +262,7 @@
         stdout = stdout.decode('ascii').rstrip()
         print("Platform: %s" % stdout)
     else:
-        print("Python failed with exit code %s:" % exitcode)
-        sys.stdout.flush()
-        sys.stdout.buffer.flush()
+        print("Python failed with exit code %s:" % exitcode, flush=True)
         sys.stdout.buffer.write(stdout)
         sys.stdout.buffer.flush()
     loop.close()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 01:33:18 2014
From: python-checkins at python.org (victor.stinner)
Date: Tue, 13 May 2014 01:33:18 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxNDE4?=
 =?utf-8?q?=3A_Fix_a_crash_in_the_builtin_function_super=28=29_when_called?=
 =?utf-8?q?_without?=
Message-ID: <3gSJNt2jprz7LjN@mail.python.org>

http://hg.python.org/cpython/rev/cee528d44b1e
changeset:   90669:cee528d44b1e
branch:      3.4
parent:      90667:c0404f0da01a
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Tue May 13 01:32:36 2014 +0200
summary:
  Issue #21418: Fix a crash in the builtin function super() when called without
argument and without current frame (ex: embedded Python).

files:
  Misc/NEWS            |   3 +++
  Objects/typeobject.c |  11 +++++++++--
  2 files changed, 12 insertions(+), 2 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #21418: Fix a crash in the builtin function super() when called without
+  argument and without current frame (ex: embedded Python).
+
 - Issue #21425: Fix flushing of standard streams in the interactive
   interpreter.
 
diff --git a/Objects/typeobject.c b/Objects/typeobject.c
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -6919,9 +6919,16 @@
     if (type == NULL) {
         /* Call super(), without args -- fill in from __class__
            and first local variable on the stack. */
-        PyFrameObject *f = PyThreadState_GET()->frame;
-        PyCodeObject *co = f->f_code;
+        PyFrameObject *f;
+        PyCodeObject *co;
         Py_ssize_t i, n;
+        f = PyThreadState_GET()->frame;
+        if (f == NULL) {
+            PyErr_SetString(PyExc_RuntimeError,
+                            "super(): no current frame");
+            return -1;
+        }
+        co = f->f_code;
         if (co == NULL) {
             PyErr_SetString(PyExc_RuntimeError,
                             "super(): no code object");

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 01:33:19 2014
From: python-checkins at python.org (victor.stinner)
Date: Tue, 13 May 2014 01:33:19 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_=28Merge_3=2E4=29_Issue_=2321418=3A_Fix_a_crash_in_the_b?=
 =?utf-8?q?uiltin_function_super=28=29_when?=
Message-ID: <3gSJNv4d46z7Ljg@mail.python.org>

http://hg.python.org/cpython/rev/53cf343c4fff
changeset:   90670:53cf343c4fff
parent:      90668:3c26389d741c
parent:      90669:cee528d44b1e
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Tue May 13 01:32:54 2014 +0200
summary:
  (Merge 3.4) Issue #21418: Fix a crash in the builtin function super() when
called without argument and without current frame (ex: embedded Python).

files:
  Misc/NEWS            |   3 +++
  Objects/typeobject.c |  11 +++++++++--
  2 files changed, 12 insertions(+), 2 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #21418: Fix a crash in the builtin function super() when called without
+  argument and without current frame (ex: embedded Python).
+
 - Issue #21425: Fix flushing of standard streams in the interactive
   interpreter.
 
diff --git a/Objects/typeobject.c b/Objects/typeobject.c
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -6929,9 +6929,16 @@
     if (type == NULL) {
         /* Call super(), without args -- fill in from __class__
            and first local variable on the stack. */
-        PyFrameObject *f = PyThreadState_GET()->frame;
-        PyCodeObject *co = f->f_code;
+        PyFrameObject *f;
+        PyCodeObject *co;
         Py_ssize_t i, n;
+        f = PyThreadState_GET()->frame;
+        if (f == NULL) {
+            PyErr_SetString(PyExc_RuntimeError,
+                            "super(): no current frame");
+            return -1;
+        }
+        co = f->f_code;
         if (co == NULL) {
             PyErr_SetString(PyExc_RuntimeError,
                             "super(): no code object");

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 02:06:43 2014
From: python-checkins at python.org (victor.stinner)
Date: Tue, 13 May 2014 02:06:43 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMzk4?=
 =?utf-8?q?=3A_Fix_an_unicode_error_in_the_pydoc_pager_when_the_documentat?=
 =?utf-8?q?ion?=
Message-ID: <3gSK7R6Vdcz7LjZ@mail.python.org>

http://hg.python.org/cpython/rev/89a29e92416f
changeset:   90671:89a29e92416f
branch:      3.4
parent:      90669:cee528d44b1e
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Tue May 13 02:05:35 2014 +0200
summary:
  Issue #21398: Fix an unicode error in the pydoc pager when the documentation
contains characters not encodable to the stdout encoding.

files:
  Lib/pydoc.py |  3 +++
  Misc/NEWS    |  3 +++
  2 files changed, 6 insertions(+), 0 deletions(-)


diff --git a/Lib/pydoc.py b/Lib/pydoc.py
--- a/Lib/pydoc.py
+++ b/Lib/pydoc.py
@@ -1404,6 +1404,9 @@
 def pager(text):
     """The first time this is called, determine what kind of pager to use."""
     global pager
+    # Escape non-encodable characters to avoid encoding errors later
+    encoding = sys.getfilesystemencoding()
+    text = text.encode(encoding, 'backslashreplace').decode(encoding)
     pager = getpager()
     pager(text)
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -26,6 +26,9 @@
 - Issue #21396: Fix TextIOWrapper(..., write_through=True) to not force a
   flush() on the underlying binary stream.  Patch by akira.
 
+- Issue #21398: Fix an unicode error in the pydoc pager when the documentation
+  contains characters not encodable to the stdout encoding.
+
 Tests
 -----
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 02:06:45 2014
From: python-checkins at python.org (victor.stinner)
Date: Tue, 13 May 2014 02:06:45 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_=28Merge_3=2E4=29_Issue_=2321398=3A_Fix_an_unicode_error?=
 =?utf-8?q?_in_the_pydoc_pager_when_the?=
Message-ID: <3gSK7T0wB3z7Lk4@mail.python.org>

http://hg.python.org/cpython/rev/3424d65ad5ce
changeset:   90672:3424d65ad5ce
parent:      90670:53cf343c4fff
parent:      90671:89a29e92416f
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Tue May 13 02:06:33 2014 +0200
summary:
  (Merge 3.4) Issue #21398: Fix an unicode error in the pydoc pager when the
documentation contains characters not encodable to the stdout encoding.

files:
  Lib/pydoc.py |  3 +++
  Misc/NEWS    |  3 +++
  2 files changed, 6 insertions(+), 0 deletions(-)


diff --git a/Lib/pydoc.py b/Lib/pydoc.py
--- a/Lib/pydoc.py
+++ b/Lib/pydoc.py
@@ -1404,6 +1404,9 @@
 def pager(text):
     """The first time this is called, determine what kind of pager to use."""
     global pager
+    # Escape non-encodable characters to avoid encoding errors later
+    encoding = sys.getfilesystemencoding()
+    text = text.encode(encoding, 'backslashreplace').decode(encoding)
     pager = getpager()
     pager(text)
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -84,6 +84,9 @@
 Library
 -------
 
+- Issue #21398: Fix an unicode error in the pydoc pager when the documentation
+  contains characters not encodable to the stdout encoding.
+
 - Issue #16531: ipaddress.IPv4Network and ipaddress.IPv6Network now accept
   an (address, netmask) tuple argument, so as to easily construct network
   objects from existing addresses.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 02:27:33 2014
From: python-checkins at python.org (eric.snow)
Date: Tue, 13 May 2014 02:27:33 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMjI2?=
 =?utf-8?q?=3A_Set_all_attrs_in_PyImport=5FExecCodeModuleObject=2E?=
Message-ID: <3gSKbT3Tsfz7LkC@mail.python.org>

http://hg.python.org/cpython/rev/7d20e30bd540
changeset:   90673:7d20e30bd540
branch:      3.4
parent:      90671:89a29e92416f
user:        Eric Snow <ericsnowcurrently at gmail.com>
date:        Mon May 12 17:54:55 2014 -0600
summary:
  Issue #21226: Set all attrs in PyImport_ExecCodeModuleObject.

files:
  Doc/c-api/import.rst        |     8 +-
  Lib/importlib/_bootstrap.py |    23 +
  Misc/NEWS                   |     2 +
  Python/import.c             |    32 +-
  Python/importlib.h          |  8555 +++++++++++-----------
  5 files changed, 4335 insertions(+), 4285 deletions(-)


diff --git a/Doc/c-api/import.rst b/Doc/c-api/import.rst
--- a/Doc/c-api/import.rst
+++ b/Doc/c-api/import.rst
@@ -132,8 +132,14 @@
    such modules have no way to know that the module object is an unknown (and
    probably damaged with respect to the module author's intents) state.
 
+   The module's :attr:`__spec__` and :attr:`__loader__` will be set, if
+   not set already, with the appropriate values.  The spec's loader will
+   be set to the module's ``__loader__`` (if set) and to an instance of
+   :class:`SourceFileLoader` otherwise.
+
    The module's :attr:`__file__` attribute will be set to the code object's
-   :c:member:`co_filename`.
+   :c:member:`co_filename`.  If applicable, :attr:`__cached__` will also
+   be set.
 
    This function will reload the module if it was already imported.  See
    :c:func:`PyImport_ReloadModule` for the intended way to reload a module.
diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py
--- a/Lib/importlib/_bootstrap.py
+++ b/Lib/importlib/_bootstrap.py
@@ -1220,6 +1220,29 @@
             return self._load_unlocked()
 
 
+def _fix_up_module(ns, name, pathname, cpathname=None):
+    # This function is used by PyImport_ExecCodeModuleObject().
+    loader = ns.get('__loader__')
+    spec = ns.get('__spec__')
+    if not loader:
+        if spec:
+            loader = spec.loader
+        elif pathname == cpathname:
+            loader = SourcelessFileLoader(name, pathname)
+        else:
+            loader = SourceFileLoader(name, pathname)
+    if not spec:
+        spec = spec_from_file_location(name, pathname, loader=loader)
+    try:
+        ns['__spec__'] = spec
+        ns['__loader__'] = loader
+        ns['__file__'] = pathname
+        ns['__cached__'] = cpathname
+    except Exception:
+        # Not important enough to report.
+        pass
+
+
 # Loaders #####################################################################
 
 class BuiltinImporter:
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -247,6 +247,8 @@
 -----------------
 
 - Issue #21276: posixmodule: Don't define USE_XATTRS on KFreeBSD and the Hurd.
+- Issue #21226: Set up modules properly in PyImport_ExecCodeModuleObject
+  (and friends).
 
 IDLE
 ----
diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -856,7 +856,7 @@
         }
     }
 
-    return d;
+    return d;  /* Return a borrowed reference. */
 }
 
 static PyObject *
@@ -888,33 +888,25 @@
 PyImport_ExecCodeModuleObject(PyObject *name, PyObject *co, PyObject *pathname,
                               PyObject *cpathname)
 {
-    PyObject *d, *v;
+    PyObject *d, *res;
+    PyInterpreterState *interp = PyThreadState_GET()->interp;
+    _Py_IDENTIFIER(_fix_up_module);
 
     d = module_dict_for_exec(name);
     if (d == NULL) {
         return NULL;
     }
 
-    if (pathname != NULL) {
-        v = pathname;
+    if (pathname == NULL) {
+        pathname = ((PyCodeObject *)co)->co_filename;
     }
-    else {
-        v = ((PyCodeObject *)co)->co_filename;
+    res = _PyObject_CallMethodIdObjArgs(interp->importlib,
+                                        &PyId__fix_up_module,
+                                        d, name, pathname, cpathname, NULL);
+    if (res != NULL) {
+        res = exec_code_in_module(name, d, co);
     }
-    Py_INCREF(v);
-    if (PyDict_SetItemString(d, "__file__", v) != 0)
-        PyErr_Clear(); /* Not important enough to report */
-    Py_DECREF(v);
-
-    /* Remember the pyc path name as the __cached__ attribute. */
-    if (cpathname != NULL)
-        v = cpathname;
-    else
-        v = Py_None;
-    if (PyDict_SetItemString(d, "__cached__", v) != 0)
-        PyErr_Clear(); /* Not important enough to report */
-
-    return exec_code_in_module(name, d, co);
+    return res;
 }
 
 
diff --git a/Python/importlib.h b/Python/importlib.h
--- a/Python/importlib.h
+++ b/Python/importlib.h
[stripped]

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 02:27:34 2014
From: python-checkins at python.org (eric.snow)
Date: Tue, 13 May 2014 02:27:34 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?b?KTogTWVyZ2UgZnJvbSAzLjQgKGZvciAjMjEyMjYpLg==?=
Message-ID: <3gSKbV6cC5z7LkF@mail.python.org>

http://hg.python.org/cpython/rev/bc324a49d0fc
changeset:   90674:bc324a49d0fc
parent:      90672:3424d65ad5ce
parent:      90673:7d20e30bd540
user:        Eric Snow <ericsnowcurrently at gmail.com>
date:        Mon May 12 18:25:00 2014 -0600
summary:
  Merge from 3.4 (for #21226).

files:
  Doc/c-api/import.rst        |     8 +-
  Lib/importlib/_bootstrap.py |    23 +
  Misc/NEWS                   |     2 +
  Python/import.c             |    32 +-
  Python/importlib.h          |  8555 +++++++++++-----------
  5 files changed, 4335 insertions(+), 4285 deletions(-)


diff --git a/Doc/c-api/import.rst b/Doc/c-api/import.rst
--- a/Doc/c-api/import.rst
+++ b/Doc/c-api/import.rst
@@ -132,8 +132,14 @@
    such modules have no way to know that the module object is an unknown (and
    probably damaged with respect to the module author's intents) state.
 
+   The module's :attr:`__spec__` and :attr:`__loader__` will be set, if
+   not set already, with the appropriate values.  The spec's loader will
+   be set to the module's ``__loader__`` (if set) and to an instance of
+   :class:`SourceFileLoader` otherwise.
+
    The module's :attr:`__file__` attribute will be set to the code object's
-   :c:member:`co_filename`.
+   :c:member:`co_filename`.  If applicable, :attr:`__cached__` will also
+   be set.
 
    This function will reload the module if it was already imported.  See
    :c:func:`PyImport_ReloadModule` for the intended way to reload a module.
diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py
--- a/Lib/importlib/_bootstrap.py
+++ b/Lib/importlib/_bootstrap.py
@@ -1221,6 +1221,29 @@
             return self._load_unlocked()
 
 
+def _fix_up_module(ns, name, pathname, cpathname=None):
+    # This function is used by PyImport_ExecCodeModuleObject().
+    loader = ns.get('__loader__')
+    spec = ns.get('__spec__')
+    if not loader:
+        if spec:
+            loader = spec.loader
+        elif pathname == cpathname:
+            loader = SourcelessFileLoader(name, pathname)
+        else:
+            loader = SourceFileLoader(name, pathname)
+    if not spec:
+        spec = spec_from_file_location(name, pathname, loader=loader)
+    try:
+        ns['__spec__'] = spec
+        ns['__loader__'] = loader
+        ns['__file__'] = pathname
+        ns['__cached__'] = cpathname
+    except Exception:
+        # Not important enough to report.
+        pass
+
+
 # Loaders #####################################################################
 
 class BuiltinImporter:
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -364,6 +364,8 @@
 - Issue #21407: _decimal: The module now supports function signatures.
 
 - Issue #21276: posixmodule: Don't define USE_XATTRS on KFreeBSD and the Hurd.
+- Issue #21226: Set up modules properly in PyImport_ExecCodeModuleObject
+  (and friends).
 
 IDLE
 ----
diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -856,7 +856,7 @@
         }
     }
 
-    return d;
+    return d;  /* Return a borrowed reference. */
 }
 
 static PyObject *
@@ -888,33 +888,25 @@
 PyImport_ExecCodeModuleObject(PyObject *name, PyObject *co, PyObject *pathname,
                               PyObject *cpathname)
 {
-    PyObject *d, *v;
+    PyObject *d, *res;
+    PyInterpreterState *interp = PyThreadState_GET()->interp;
+    _Py_IDENTIFIER(_fix_up_module);
 
     d = module_dict_for_exec(name);
     if (d == NULL) {
         return NULL;
     }
 
-    if (pathname != NULL) {
-        v = pathname;
+    if (pathname == NULL) {
+        pathname = ((PyCodeObject *)co)->co_filename;
     }
-    else {
-        v = ((PyCodeObject *)co)->co_filename;
+    res = _PyObject_CallMethodIdObjArgs(interp->importlib,
+                                        &PyId__fix_up_module,
+                                        d, name, pathname, cpathname, NULL);
+    if (res != NULL) {
+        res = exec_code_in_module(name, d, co);
     }
-    Py_INCREF(v);
-    if (PyDict_SetItemString(d, "__file__", v) != 0)
-        PyErr_Clear(); /* Not important enough to report */
-    Py_DECREF(v);
-
-    /* Remember the pyc path name as the __cached__ attribute. */
-    if (cpathname != NULL)
-        v = cpathname;
-    else
-        v = Py_None;
-    if (PyDict_SetItemString(d, "__cached__", v) != 0)
-        PyErr_Clear(); /* Not important enough to report */
-
-    return exec_code_in_module(name, d, co);
+    return res;
 }
 
 
diff --git a/Python/importlib.h b/Python/importlib.h
--- a/Python/importlib.h
+++ b/Python/importlib.h
[stripped]

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 04:42:07 2014
From: python-checkins at python.org (jason.coombs)
Date: Tue, 13 May 2014 04:42:07 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Update_docs_to?=
 =?utf-8?q?_reflect_resurrection_of_Setuptools_over_Distribute?=
Message-ID: <3gSNZl5D1Gz7Ljd@mail.python.org>

http://hg.python.org/cpython/rev/1a3f2d0ced35
changeset:   90675:1a3f2d0ced35
branch:      3.3
parent:      90586:ab5e2b0fba15
user:        Jason R. Coombs <jaraco at jaraco.com>
date:        Mon May 12 22:40:49 2014 -0400
summary:
  Update docs to reflect resurrection of Setuptools over Distribute

files:
  Doc/library/venv.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst
--- a/Doc/library/venv.rst
+++ b/Doc/library/venv.rst
@@ -40,11 +40,11 @@
    A venv is a directory tree which contains Python executable files and
    other files which indicate that it is a venv.
 
-   Common installation tools such as ``Distribute`` and ``pip`` work as
+   Common installation tools such as ``Setuptools`` and ``pip`` work as
    expected with venvs - i.e. when a venv is active, they install Python
    packages into the venv without needing to be told to do so explicitly.
    Of course, you need to install them into the venv first: this could be
-   done by running ``distribute_setup.py`` with the venv activated,
+   done by running ``ez_setup.py`` with the venv activated,
    followed by running ``easy_install pip``. Alternatively, you could download
    the source tarballs and run ``python setup.py install`` after unpacking,
    with the venv activated.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 04:42:09 2014
From: python-checkins at python.org (jason.coombs)
Date: Tue, 13 May 2014 04:42:09 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuNCk6?=
 =?utf-8?q?_Merge_doc_change_from_3=2E3?=
Message-ID: <3gSNZn00fsz7Ljd@mail.python.org>

http://hg.python.org/cpython/rev/fa5779a62b5d
changeset:   90676:fa5779a62b5d
branch:      3.4
parent:      90673:7d20e30bd540
parent:      90675:1a3f2d0ced35
user:        Jason R. Coombs <jaraco at jaraco.com>
date:        Mon May 12 22:41:15 2014 -0400
summary:
  Merge doc change from 3.3

files:
  Doc/library/venv.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst
--- a/Doc/library/venv.rst
+++ b/Doc/library/venv.rst
@@ -40,11 +40,11 @@
    A venv is a directory tree which contains Python executable files and
    other files which indicate that it is a venv.
 
-   Common installation tools such as ``Distribute`` and ``pip`` work as
+   Common installation tools such as ``Setuptools`` and ``pip`` work as
    expected with venvs - i.e. when a venv is active, they install Python
    packages into the venv without needing to be told to do so explicitly.
    Of course, you need to install them into the venv first: this could be
-   done by running ``distribute_setup.py`` with the venv activated,
+   done by running ``ez_setup.py`` with the venv activated,
    followed by running ``easy_install pip``. Alternatively, you could download
    the source tarballs and run ``python setup.py install`` after unpacking,
    with the venv activated.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 04:42:10 2014
From: python-checkins at python.org (jason.coombs)
Date: Tue, 13 May 2014 04:42:10 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge_doc_change_from_3=2E3?=
Message-ID: <3gSNZp3kw5z7Ljs@mail.python.org>

http://hg.python.org/cpython/rev/7e1b3f804279
changeset:   90677:7e1b3f804279
parent:      90674:bc324a49d0fc
parent:      90676:fa5779a62b5d
user:        Jason R. Coombs <jaraco at jaraco.com>
date:        Mon May 12 22:41:49 2014 -0400
summary:
  Merge doc change from 3.3

files:
  Doc/library/venv.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst
--- a/Doc/library/venv.rst
+++ b/Doc/library/venv.rst
@@ -40,11 +40,11 @@
    A venv is a directory tree which contains Python executable files and
    other files which indicate that it is a venv.
 
-   Common installation tools such as ``Distribute`` and ``pip`` work as
+   Common installation tools such as ``Setuptools`` and ``pip`` work as
    expected with venvs - i.e. when a venv is active, they install Python
    packages into the venv without needing to be told to do so explicitly.
    Of course, you need to install them into the venv first: this could be
-   done by running ``distribute_setup.py`` with the venv activated,
+   done by running ``ez_setup.py`` with the venv activated,
    followed by running ``easy_install pip``. Alternatively, you could download
    the source tarballs and run ``python setup.py install`` after unpacking,
    with the venv activated.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 05:36:34 2014
From: python-checkins at python.org (nick.coghlan)
Date: Tue, 13 May 2014 05:36:34 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_8=3A_tweak_some_awkward_w?=
 =?utf-8?q?ording_pointed_out_in_private_email?=
Message-ID: <3gSPnZ1mgjz7Ljg@mail.python.org>

http://hg.python.org/peps/rev/b93d501ba313
changeset:   5475:b93d501ba313
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Tue May 13 13:36:23 2014 +1000
summary:
  PEP 8: tweak some awkward wording pointed out in private email

files:
  pep-0008.txt |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/pep-0008.txt b/pep-0008.txt
--- a/pep-0008.txt
+++ b/pep-0008.txt
@@ -526,7 +526,7 @@
 
 You should use two spaces after a sentence-ending period.
 
-When writing English, Strunk and White apply.
+When writing English, follow Strunk and White.
 
 Python coders from non-English speaking countries: please write your
 comments in English, unless you are 120% sure that the code will never

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Tue May 13 06:57:27 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Tue, 13 May 2014 06:57:27 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgMjE0Njk6?=
 =?utf-8?q?__Mitigate_risk_of_false_positives_with_robotparser=2E?=
Message-ID: <3gSRZv4gWKz7LjN@mail.python.org>

http://hg.python.org/cpython/rev/4ea86cd87f95
changeset:   90678:4ea86cd87f95
branch:      3.4
parent:      90676:fa5779a62b5d
user:        Raymond Hettinger <python at rcn.com>
date:        Mon May 12 21:56:33 2014 -0700
summary:
  Issue 21469:  Mitigate risk of false positives with robotparser.

* Repair the broken link to norobots-rfc.txt.

* HTTP response codes >= 500 treated as a failed read rather than as a not
found.  Not found means that we can assume the entire site is allowed.  A 5xx
server error tells us nothing.

* A successful read() or parse() updates the mtime (which is defined to be "the
  time the robots.txt file was last fetched").

* The can_fetch() method returns False unless we've had a read() with a 2xx or
4xx response.  This avoids false positives in the case where a user calls
can_fetch() before calling read().

* I don't see any easy way to test this patch without hitting internet
resources that might change or without use of mock objects that wouldn't
provide must reassurance.

files:
  Lib/urllib/robotparser.py |  11 +++++++++--
  1 files changed, 9 insertions(+), 2 deletions(-)


diff --git a/Lib/urllib/robotparser.py b/Lib/urllib/robotparser.py
--- a/Lib/urllib/robotparser.py
+++ b/Lib/urllib/robotparser.py
@@ -7,7 +7,7 @@
     2) PSF license for Python 2.2
 
     The robots.txt Exclusion Protocol is implemented as specified in
-    http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html
+    http://www.robotstxt.org/norobots-rfc.txt
 """
 
 import urllib.parse, urllib.request
@@ -57,7 +57,7 @@
         except urllib.error.HTTPError as err:
             if err.code in (401, 403):
                 self.disallow_all = True
-            elif err.code >= 400:
+            elif err.code >= 400 and err.code < 500:
                 self.allow_all = True
         else:
             raw = f.read()
@@ -85,6 +85,7 @@
         state = 0
         entry = Entry()
 
+        self.modified()
         for line in lines:
             if not line:
                 if state == 1:
@@ -129,6 +130,12 @@
             return False
         if self.allow_all:
             return True
+        # Until the robots.txt file has been read or found not
+        # to exist, we must assume that no url is allowable.
+        # This prevents false positives when a user erronenously
+        # calls can_fetch() before calling read().
+        if not self.last_checked:
+            return False
         # search for given user agent matches
         # the first match counts
         parsed_url = urllib.parse.urlparse(urllib.parse.unquote(url))

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 06:57:28 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Tue, 13 May 2014 06:57:28 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?q?=29=3A_merge?=
Message-ID: <3gSRZw62Gqz7Ljk@mail.python.org>

http://hg.python.org/cpython/rev/ce6187f97a51
changeset:   90679:ce6187f97a51
parent:      90677:7e1b3f804279
parent:      90678:4ea86cd87f95
user:        Raymond Hettinger <python at rcn.com>
date:        Mon May 12 21:57:19 2014 -0700
summary:
  merge

files:
  Lib/urllib/robotparser.py |  11 +++++++++--
  1 files changed, 9 insertions(+), 2 deletions(-)


diff --git a/Lib/urllib/robotparser.py b/Lib/urllib/robotparser.py
--- a/Lib/urllib/robotparser.py
+++ b/Lib/urllib/robotparser.py
@@ -7,7 +7,7 @@
     2) PSF license for Python 2.2
 
     The robots.txt Exclusion Protocol is implemented as specified in
-    http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html
+    http://www.robotstxt.org/norobots-rfc.txt
 """
 
 import urllib.parse, urllib.request
@@ -57,7 +57,7 @@
         except urllib.error.HTTPError as err:
             if err.code in (401, 403):
                 self.disallow_all = True
-            elif err.code >= 400:
+            elif err.code >= 400 and err.code < 500:
                 self.allow_all = True
         else:
             raw = f.read()
@@ -85,6 +85,7 @@
         state = 0
         entry = Entry()
 
+        self.modified()
         for line in lines:
             if not line:
                 if state == 1:
@@ -129,6 +130,12 @@
             return False
         if self.allow_all:
             return True
+        # Until the robots.txt file has been read or found not
+        # to exist, we must assume that no url is allowable.
+        # This prevents false positives when a user erronenously
+        # calls can_fetch() before calling read().
+        if not self.last_checked:
+            return False
         # search for given user agent matches
         # the first match counts
         parsed_url = urllib.parse.urlparse(urllib.parse.unquote(url))

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 07:05:32 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Tue, 13 May 2014 07:05:32 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgMjE0Njk6?=
	=?utf-8?q?__Add_missing_news_item?=
Message-ID: <3gSRmD6GnNz7LjQ@mail.python.org>

http://hg.python.org/cpython/rev/f67cf5747a26
changeset:   90680:f67cf5747a26
branch:      3.4
parent:      90678:4ea86cd87f95
user:        Raymond Hettinger <python at rcn.com>
date:        Mon May 12 22:01:46 2014 -0700
summary:
  Issue 21469:  Add missing news item

files:
  Misc/NEWS |  4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -82,6 +82,10 @@
 - Issue #21088: Bugfix for curses.window.addch() regression in 3.4.0.
   In porting to Argument Clinic, the first two arguments were reversed.
 
+- Issue #21469:  Reduced the risk of false positives in robotparser by
+  checking to make sure that robots.txt has been read or does not exist
+  prior to returning True in can_fetch().
+
 - Issue #21321: itertools.islice() now releases the reference to the source
   iterator when the slice is exhausted.  Patch by Anton Afanasyev.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 07:05:34 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Tue, 13 May 2014 07:05:34 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?q?=29=3A_merge?=
Message-ID: <3gSRmG1S2Bz7Ljr@mail.python.org>

http://hg.python.org/cpython/rev/0b90962e5f19
changeset:   90681:0b90962e5f19
parent:      90679:ce6187f97a51
parent:      90680:f67cf5747a26
user:        Raymond Hettinger <python at rcn.com>
date:        Mon May 12 22:05:09 2014 -0700
summary:
  merge

files:
  Misc/NEWS |  4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -109,6 +109,10 @@
   Decimal.quantize() method in the Python version.  It had never been
   present in the C version.
 
+- Issue #21469:  Reduced the risk of false positives in robotparser by
+  checking to make sure that robots.txt has been read or does not exist
+  prior to returning True in can_fetch().
+
 - Issue #19414: Have the OrderedDict mark deleted links as unusable.
   This gives an early failure if the link is deleted during iteration.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 07:19:01 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Tue, 13 May 2014 07:19:01 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgMjE0Njk6?=
 =?utf-8?q?__Mitigate_risk_of_false_positives_with_robotparser=2E?=
Message-ID: <3gSS3n1xx5z7LjS@mail.python.org>

http://hg.python.org/cpython/rev/d4fd55278cec
changeset:   90682:d4fd55278cec
branch:      2.7
parent:      90656:670fb496f1f6
user:        Raymond Hettinger <python at rcn.com>
date:        Mon May 12 22:18:50 2014 -0700
summary:
  Issue 21469:  Mitigate risk of false positives with robotparser.

* Repair the broken link to norobots-rfc.txt.

* HTTP response codes >= 500 treated as a failed read rather than as a not
found.  Not found means that we can assume the entire site is allowed.  A 5xx
server error tells us nothing.

* A successful read() or parse() updates the mtime (which is defined to be "the
  time the robots.txt file was last fetched").

* The can_fetch() method returns False unless we've had a read() with a 2xx or
4xx response.  This avoids false positives in the case where a user calls
can_fetch() before calling read().

* I don't see any easy way to test this patch without hitting internet
resources that might change or without use of mock objects that wouldn't
provide must reassurance.

files:
  Lib/robotparser.py |  14 ++++++++++++--
  Misc/NEWS          |   4 ++++
  2 files changed, 16 insertions(+), 2 deletions(-)


diff --git a/Lib/robotparser.py b/Lib/robotparser.py
--- a/Lib/robotparser.py
+++ b/Lib/robotparser.py
@@ -7,7 +7,8 @@
     2) PSF license for Python 2.2
 
     The robots.txt Exclusion Protocol is implemented as specified in
-    http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html
+    http://www.robotstxt.org/norobots-rfc.txt
+
 """
 import urlparse
 import urllib
@@ -60,7 +61,7 @@
         self.errcode = opener.errcode
         if self.errcode in (401, 403):
             self.disallow_all = True
-        elif self.errcode >= 400:
+        elif self.errcode >= 400 and self.errcode < 500:
             self.allow_all = True
         elif self.errcode == 200 and lines:
             self.parse(lines)
@@ -86,6 +87,7 @@
         linenumber = 0
         entry = Entry()
 
+        self.modified()
         for line in lines:
             linenumber += 1
             if not line:
@@ -131,6 +133,14 @@
             return False
         if self.allow_all:
             return True
+
+        # Until the robots.txt file has been read or found not
+        # to exist, we must assume that no url is allowable.
+        # This prevents false positives when a user erronenously
+        # calls can_fetch() before calling read().
+        if not self.last_checked:
+            return False
+
         # search for given user agent matches
         # the first match counts
         parsed_url = urlparse.urlparse(urllib.unquote(url))
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -52,6 +52,10 @@
 - Issue #21306: Backport hmac.compare_digest from Python 3. This is part of PEP
   466.
 
+- Issue #21469:  Reduced the risk of false positives in robotparser by
+  checking to make sure that robots.txt has been read or does not exist
+  prior to returning True in can_fetch().
+
 - Issue #21321: itertools.islice() now releases the reference to the source
   iterator when the slice is exhausted.  Patch by Anton Afanasyev.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 07:22:58 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Tue, 13 May 2014 07:22:58 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_21469=3A__Minor_code?=
 =?utf-8?q?_modernization_=28convert_and/or_expression_to_an_if/else?=
Message-ID: <3gSS8L02Tvz7LjS@mail.python.org>

http://hg.python.org/cpython/rev/560320c10564
changeset:   90683:560320c10564
parent:      90681:0b90962e5f19
user:        Raymond Hettinger <python at rcn.com>
date:        Mon May 12 22:22:46 2014 -0700
summary:
  Issue 21469:  Minor code modernization (convert and/or expression to an if/else expression).

Suggested by: Tal Einat

files:
  Lib/urllib/robotparser.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/urllib/robotparser.py b/Lib/urllib/robotparser.py
--- a/Lib/urllib/robotparser.py
+++ b/Lib/urllib/robotparser.py
@@ -172,7 +172,7 @@
         return self.path == "*" or filename.startswith(self.path)
 
     def __str__(self):
-        return (self.allowance and "Allow" or "Disallow") + ": " + self.path
+        return ("Allow" if self.allowance else "Disallow") + ": " + self.path
 
 
 class Entry:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 10:50:39 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Tue, 13 May 2014 10:50:39 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319775=3A_Add_a_sa?=
 =?utf-8?q?mefile=28=29_method_to_pathlib_Path_objects=2E?=
Message-ID: <3gSXlz5bQNz7LjT@mail.python.org>

http://hg.python.org/cpython/rev/197ac5d79456
changeset:   90684:197ac5d79456
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Tue May 13 10:50:15 2014 +0200
summary:
  Issue #19775: Add a samefile() method to pathlib Path objects.

Initial patch by Vajrasky Kok.

files:
  Doc/library/pathlib.rst  |  19 +++++++++++++++++++
  Lib/pathlib.py           |  11 +++++++++++
  Lib/test/test_pathlib.py |  20 ++++++++++++++++++++
  Misc/NEWS                |   3 +++
  4 files changed, 53 insertions(+), 0 deletions(-)


diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst
--- a/Doc/library/pathlib.rst
+++ b/Doc/library/pathlib.rst
@@ -884,6 +884,25 @@
    Remove this directory.  The directory must be empty.
 
 
+.. method:: Path.samefile(other_path)
+
+   Return whether this path points to the same file as *other_path*, which
+   can be either a Path object, or a string.  The semantics are similar
+   to :func:`os.path.samefile` and :func:`os.path.samestat`.
+
+   An :exc:`OSError` can be raised if either file cannot be accessed for some
+   reason.
+
+      >>> p = Path('spam')
+      >>> q = Path('eggs')
+      >>> p.samefile(q)
+      False
+      >>> p.samefile('spam')
+      True
+
+   .. versionadded:: 3.5
+
+
 .. method:: Path.symlink_to(target, target_is_directory=False)
 
    Make this path a symbolic link to *target*.  Under Windows,
diff --git a/Lib/pathlib.py b/Lib/pathlib.py
--- a/Lib/pathlib.py
+++ b/Lib/pathlib.py
@@ -961,6 +961,17 @@
         """
         return cls(os.getcwd())
 
+    def samefile(self, other_path):
+        """Return whether `other_file` is the same or not as this file.
+        (as returned by os.path.samefile(file, other_file)).
+        """
+        st = self.stat()
+        try:
+            other_st = other_path.stat()
+        except AttributeError:
+            other_st = os.stat(other_path)
+        return os.path.samestat(st, other_st)
+
     def iterdir(self):
         """Iterate over the files in this directory.  Does not yield any
         result for the special paths '.' and '..'.
diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py
--- a/Lib/test/test_pathlib.py
+++ b/Lib/test/test_pathlib.py
@@ -1251,6 +1251,26 @@
         p = self.cls.cwd()
         self._test_cwd(p)
 
+    def test_samefile(self):
+        fileA_path = os.path.join(BASE, 'fileA')
+        fileB_path = os.path.join(BASE, 'dirB', 'fileB')
+        p = self.cls(fileA_path)
+        pp = self.cls(fileA_path)
+        q = self.cls(fileB_path)
+        self.assertTrue(p.samefile(fileA_path))
+        self.assertTrue(p.samefile(pp))
+        self.assertFalse(p.samefile(fileB_path))
+        self.assertFalse(p.samefile(q))
+        # Test the non-existent file case
+        non_existent = os.path.join(BASE, 'foo')
+        r = self.cls(non_existent)
+        self.assertRaises(FileNotFoundError, p.samefile, r)
+        self.assertRaises(FileNotFoundError, p.samefile, non_existent)
+        self.assertRaises(FileNotFoundError, r.samefile, p)
+        self.assertRaises(FileNotFoundError, r.samefile, non_existent)
+        self.assertRaises(FileNotFoundError, r.samefile, r)
+        self.assertRaises(FileNotFoundError, r.samefile, non_existent)
+
     def test_empty_path(self):
         # The empty path points to '.'
         p = self.cls('')
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -84,6 +84,9 @@
 Library
 -------
 
+- Issue #19775: Add a samefile() method to pathlib Path objects.  Initial
+  patch by Vajrasky Kok.
+
 - Issue #21398: Fix an unicode error in the pydoc pager when the documentation
   contains characters not encodable to the stdout encoding.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 11:30:45 2014
From: python-checkins at python.org (matthias.klose)
Date: Tue, 13 May 2014 11:30:45 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogLSBJc3N1ZSAjMTc3?=
 =?utf-8?q?56=3A_Fix_test=5Fcode_test_when_run_from_the_installed_location?=
 =?utf-8?q?=2E?=
Message-ID: <3gSYfF741Wz7Ljf@mail.python.org>

http://hg.python.org/cpython/rev/20db5e9086d4
changeset:   90685:20db5e9086d4
branch:      3.4
parent:      90680:f67cf5747a26
user:        doko at ubuntu.com
date:        Tue May 13 11:28:12 2014 +0200
summary:
  - Issue #17756: Fix test_code test when run from the installed location.

files:
  Lib/test/test_code_module.py |  2 +-
  Misc/NEWS                    |  2 ++
  2 files changed, 3 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_code_module.py b/Lib/test/test_code_module.py
--- a/Lib/test/test_code_module.py
+++ b/Lib/test/test_code_module.py
@@ -51,7 +51,7 @@
         self.infunc.side_effect = ["undefined", EOFError('Finished')]
         self.console.interact()
         for call in self.stderr.method_calls:
-            if 'NameError:' in ''.join(call[1]):
+            if 'NameError' in ''.join(call[1]):
                 break
         else:
             raise AssertionError("No syntax error from console")
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -32,6 +32,8 @@
 Tests
 -----
 
+- Issue #17756: Fix test_code test when run from the installed location.
+
 - Issue #17752: Fix distutils tests when run from the installed location.
 
 IDLE

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 11:30:47 2014
From: python-checkins at python.org (matthias.klose)
Date: Tue, 13 May 2014 11:30:47 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?b?KTogTWVyZ2UgZnJvbSAzLjQ6?=
Message-ID: <3gSYfH1tfLz7Ljf@mail.python.org>

http://hg.python.org/cpython/rev/8885fc2e92b3
changeset:   90686:8885fc2e92b3
parent:      90684:197ac5d79456
parent:      90685:20db5e9086d4
user:        doko at ubuntu.com
date:        Tue May 13 11:30:17 2014 +0200
summary:
  Merge from 3.4:
- Issue #17756: Fix test_code test when run from the installed location.

files:
  Lib/test/test_code_module.py |  2 +-
  Misc/NEWS                    |  2 ++
  2 files changed, 3 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_code_module.py b/Lib/test/test_code_module.py
--- a/Lib/test/test_code_module.py
+++ b/Lib/test/test_code_module.py
@@ -51,7 +51,7 @@
         self.infunc.side_effect = ["undefined", EOFError('Finished')]
         self.console.interact()
         for call in self.stderr.method_calls:
-            if 'NameError:' in ''.join(call[1]):
+            if 'NameError' in ''.join(call[1]):
                 break
         else:
             raise AssertionError("No syntax error from console")
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -456,6 +456,8 @@
 Tests
 -----
 
+- Issue #17756: Fix test_code test when run from the installed location.
+
 - Issue #17752: Fix distutils tests when run from the installed location.
 
 - Issue #18604: Consolidated checks for GUI availability.  All platforms now

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 18:21:42 2014
From: python-checkins at python.org (guido.van.rossum)
Date: Tue, 13 May 2014 18:21:42 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Fix_test-order?=
 =?utf-8?q?-dependend_asyncio_test_failure_caused_by_rev?=
Message-ID: <3gSkmQ5mN6z7Ljy@mail.python.org>

http://hg.python.org/cpython/rev/16d0bcce10de
changeset:   90687:16d0bcce10de
branch:      3.4
parent:      90685:20db5e9086d4
user:        Guido van Rossum <guido at python.org>
date:        Tue May 13 09:19:39 2014 -0700
summary:
  Fix test-order-dependend asyncio test failure caused by rev 909ea8cc86bbab92dbb6231668f403b7360f30fa.

files:
  Lib/test/test_asyncio/test_streams.py |  15 +++++++++------
  1 files changed, 9 insertions(+), 6 deletions(-)


diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py
--- a/Lib/test/test_asyncio/test_streams.py
+++ b/Lib/test/test_asyncio/test_streams.py
@@ -593,9 +593,6 @@
         # of the data writter.  Also we must explicitly attach a child
         # watcher to the event loop.
 
-        watcher = asyncio.get_child_watcher()
-        watcher.attach_loop(self.loop)
-
         code = """\
 import os, sys
 fd = int(sys.argv[1])
@@ -611,9 +608,15 @@
         transport, _ = self.loop.run_until_complete(
             self.loop.connect_read_pipe(lambda: protocol, pipe))
 
-        proc = self.loop.run_until_complete(
-            asyncio.create_subprocess_exec(*args, pass_fds={wfd}, loop=self.loop))
-        self.loop.run_until_complete(proc.wait())
+        watcher = asyncio.SafeChildWatcher()
+        watcher.attach_loop(self.loop)
+        try:
+            asyncio.set_child_watcher(watcher)
+            proc = self.loop.run_until_complete(
+                asyncio.create_subprocess_exec(*args, pass_fds={wfd}, loop=self.loop))
+            self.loop.run_until_complete(proc.wait())
+        finally:
+            asyncio.set_child_watcher(None)
 
         os.close(wfd)
         data = self.loop.run_until_complete(reader.read(-1))

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 18:21:44 2014
From: python-checkins at python.org (guido.van.rossum)
Date: Tue, 13 May 2014 18:21:44 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge_3=2E4-=3Edefault=3A_Fix_test-order-dependend_async?=
 =?utf-8?q?io_test_failure_caused_by_rev?=
Message-ID: <3gSkmS0SMkz7Ljy@mail.python.org>

http://hg.python.org/cpython/rev/0bb92c9fdb35
changeset:   90688:0bb92c9fdb35
parent:      90686:8885fc2e92b3
parent:      90687:16d0bcce10de
user:        Guido van Rossum <guido at python.org>
date:        Tue May 13 09:21:33 2014 -0700
summary:
  Merge 3.4->default: Fix test-order-dependend asyncio test failure caused by rev 2af5a52b9b87 (in this branch).

files:
  Lib/test/test_asyncio/test_streams.py |  15 +++++++++------
  1 files changed, 9 insertions(+), 6 deletions(-)


diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py
--- a/Lib/test/test_asyncio/test_streams.py
+++ b/Lib/test/test_asyncio/test_streams.py
@@ -593,9 +593,6 @@
         # of the data writter.  Also we must explicitly attach a child
         # watcher to the event loop.
 
-        watcher = asyncio.get_child_watcher()
-        watcher.attach_loop(self.loop)
-
         code = """\
 import os, sys
 fd = int(sys.argv[1])
@@ -611,9 +608,15 @@
         transport, _ = self.loop.run_until_complete(
             self.loop.connect_read_pipe(lambda: protocol, pipe))
 
-        proc = self.loop.run_until_complete(
-            asyncio.create_subprocess_exec(*args, pass_fds={wfd}, loop=self.loop))
-        self.loop.run_until_complete(proc.wait())
+        watcher = asyncio.SafeChildWatcher()
+        watcher.attach_loop(self.loop)
+        try:
+            asyncio.set_child_watcher(watcher)
+            proc = self.loop.run_until_complete(
+                asyncio.create_subprocess_exec(*args, pass_fds={wfd}, loop=self.loop))
+            self.loop.run_until_complete(proc.wait())
+        finally:
+            asyncio.set_child_watcher(None)
 
         os.close(wfd)
         data = self.loop.run_until_complete(reader.read(-1))

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 20:19:48 2014
From: python-checkins at python.org (eric.snow)
Date: Tue, 13 May 2014 20:19:48 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxNDk5?=
 =?utf-8?q?=3A_Ignore_=5F=5Fbuiltins=5F=5F_in_several_test=5Fimportlib=2Et?=
 =?utf-8?q?est=5Fapi_tests=2E?=
Message-ID: <3gSnNh1Xq0z7Ljj@mail.python.org>

http://hg.python.org/cpython/rev/16d26391ec36
changeset:   90689:16d26391ec36
branch:      3.4
parent:      90687:16d0bcce10de
user:        Eric Snow <ericsnowcurrently at gmail.com>
date:        Tue May 13 12:15:42 2014 -0600
summary:
  Issue #21499: Ignore __builtins__ in several test_importlib.test_api tests.

files:
  Lib/test/test_importlib/test_api.py |  15 ++++++++-------
  Misc/NEWS                           |   2 ++
  2 files changed, 10 insertions(+), 7 deletions(-)


diff --git a/Lib/test/test_importlib/test_api.py b/Lib/test/test_importlib/test_api.py
--- a/Lib/test/test_importlib/test_api.py
+++ b/Lib/test/test_importlib/test_api.py
@@ -241,13 +241,13 @@
                                 '__file__': path,
                                 '__cached__': cached,
                                 '__doc__': None,
-                                '__builtins__': __builtins__,
                                 }
                     support.create_empty_file(path)
                     module = self.init.import_module(name)
-                    ns = vars(module)
+                    ns = vars(module).copy()
                     loader = ns.pop('__loader__')
                     spec = ns.pop('__spec__')
+                    ns.pop('__builtins__', None)  # An implementation detail.
                     self.assertEqual(spec.name, name)
                     self.assertEqual(spec.loader, loader)
                     self.assertEqual(loader.path, path)
@@ -263,14 +263,14 @@
                                 '__cached__': cached,
                                 '__path__': [os.path.dirname(init_path)],
                                 '__doc__': None,
-                                '__builtins__': __builtins__,
                                 }
                     os.mkdir(name)
                     os.rename(path, init_path)
                     reloaded = self.init.reload(module)
-                    ns = vars(reloaded)
+                    ns = vars(reloaded).copy()
                     loader = ns.pop('__loader__')
                     spec = ns.pop('__spec__')
+                    ns.pop('__builtins__', None)  # An implementation detail.
                     self.assertEqual(spec.name, name)
                     self.assertEqual(spec.loader, loader)
                     self.assertIs(reloaded, module)
@@ -295,10 +295,11 @@
                     with open(bad_path, 'w') as init_file:
                         init_file.write('eggs = None')
                     module = self.init.import_module(name)
-                    ns = vars(module)
+                    ns = vars(module).copy()
                     loader = ns.pop('__loader__')
                     path = ns.pop('__path__')
                     spec = ns.pop('__spec__')
+                    ns.pop('__builtins__', None)  # An implementation detail.
                     self.assertEqual(spec.name, name)
                     self.assertIs(spec.loader, None)
                     self.assertIsNot(loader, None)
@@ -319,14 +320,14 @@
                                 '__cached__': cached,
                                 '__path__': [os.path.dirname(init_path)],
                                 '__doc__': None,
-                                '__builtins__': __builtins__,
                                 'eggs': None,
                                 }
                     os.rename(bad_path, init_path)
                     reloaded = self.init.reload(module)
-                    ns = vars(reloaded)
+                    ns = vars(reloaded).copy()
                     loader = ns.pop('__loader__')
                     spec = ns.pop('__spec__')
+                    ns.pop('__builtins__', None)  # An implementation detail.
                     self.assertEqual(spec.name, name)
                     self.assertEqual(spec.loader, loader)
                     self.assertIs(reloaded, module)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -187,6 +187,8 @@
 
 - Issue #20884: Don't assume that __file__ is defined on importlib.__init__.
 
+- Issue #21499: Ignore __builtins__ in several test_importlib.test_api tests.
+
 - Issue #20879: Delay the initialization of encoding and decoding tables for
   base32, ascii85 and base85 codecs in the base64 module, and delay the
   initialization of the unquote_to_bytes() table of the urllib.parse module, to

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue May 13 20:19:49 2014
From: python-checkins at python.org (eric.snow)
Date: Tue, 13 May 2014 20:19:49 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?b?KTogTWVyZ2UgZnJvbSAzLjQgKGZvciAjMjE0OTkpLg==?=
Message-ID: <3gSnNj5lDKz7Lk7@mail.python.org>

http://hg.python.org/cpython/rev/bdf94b2c0639
changeset:   90690:bdf94b2c0639
parent:      90688:0bb92c9fdb35
parent:      90689:16d26391ec36
user:        Eric Snow <ericsnowcurrently at gmail.com>
date:        Tue May 13 12:18:07 2014 -0600
summary:
  Merge from 3.4 (for #21499).

files:
  Lib/test/test_importlib/test_api.py |  15 ++++++++-------
  Misc/NEWS                           |   2 ++
  2 files changed, 10 insertions(+), 7 deletions(-)


diff --git a/Lib/test/test_importlib/test_api.py b/Lib/test/test_importlib/test_api.py
--- a/Lib/test/test_importlib/test_api.py
+++ b/Lib/test/test_importlib/test_api.py
@@ -241,13 +241,13 @@
                                 '__file__': path,
                                 '__cached__': cached,
                                 '__doc__': None,
-                                '__builtins__': __builtins__,
                                 }
                     support.create_empty_file(path)
                     module = self.init.import_module(name)
-                    ns = vars(module)
+                    ns = vars(module).copy()
                     loader = ns.pop('__loader__')
                     spec = ns.pop('__spec__')
+                    ns.pop('__builtins__', None)  # An implementation detail.
                     self.assertEqual(spec.name, name)
                     self.assertEqual(spec.loader, loader)
                     self.assertEqual(loader.path, path)
@@ -263,14 +263,14 @@
                                 '__cached__': cached,
                                 '__path__': [os.path.dirname(init_path)],
                                 '__doc__': None,
-                                '__builtins__': __builtins__,
                                 }
                     os.mkdir(name)
                     os.rename(path, init_path)
                     reloaded = self.init.reload(module)
-                    ns = vars(reloaded)
+                    ns = vars(reloaded).copy()
                     loader = ns.pop('__loader__')
                     spec = ns.pop('__spec__')
+                    ns.pop('__builtins__', None)  # An implementation detail.
                     self.assertEqual(spec.name, name)
                     self.assertEqual(spec.loader, loader)
                     self.assertIs(reloaded, module)
@@ -295,10 +295,11 @@
                     with open(bad_path, 'w') as init_file:
                         init_file.write('eggs = None')
                     module = self.init.import_module(name)
-                    ns = vars(module)
+                    ns = vars(module).copy()
                     loader = ns.pop('__loader__')
                     path = ns.pop('__path__')
                     spec = ns.pop('__spec__')
+                    ns.pop('__builtins__', None)  # An implementation detail.
                     self.assertEqual(spec.name, name)
                     self.assertIs(spec.loader, None)
                     self.assertIsNot(loader, None)
@@ -319,14 +320,14 @@
                                 '__cached__': cached,
                                 '__path__': [os.path.dirname(init_path)],
                                 '__doc__': None,
-                                '__builtins__': __builtins__,
                                 'eggs': None,
                                 }
                     os.rename(bad_path, init_path)
                     reloaded = self.init.reload(module)
-                    ns = vars(reloaded)
+                    ns = vars(reloaded).copy()
                     loader = ns.pop('__loader__')
                     spec = ns.pop('__spec__')
+                    ns.pop('__builtins__', None)  # An implementation detail.
                     self.assertEqual(spec.name, name)
                     self.assertEqual(spec.loader, loader)
                     self.assertIs(reloaded, module)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -272,6 +272,8 @@
 
 - Issue #20884: Don't assume that __file__ is defined on importlib.__init__.
 
+- Issue #21499: Ignore __builtins__ in several test_importlib.test_api tests.
+
 - Issue #20627: xmlrpc.client.ServerProxy is now a context manager.
 
 - Issue #19165: The formatter module now raises DeprecationWarning instead of

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 07:09:44 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Wed, 14 May 2014 07:09:44 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIxNDcw?=
 =?utf-8?q?=3A_Do_a_better_job_seeding_the_random_number_generator?=
Message-ID: <3gT3pc3NjXz7Lkf@mail.python.org>

http://hg.python.org/cpython/rev/7b5265752942
changeset:   90691:7b5265752942
branch:      2.7
parent:      90682:d4fd55278cec
user:        Raymond Hettinger <python at rcn.com>
date:        Tue May 13 22:09:23 2014 -0700
summary:
  Issue #21470: Do a better job seeding the random number generator
to fully cover its state space.

files:
  Lib/random.py |  4 +++-
  Misc/NEWS     |  3 +++
  2 files changed, 6 insertions(+), 1 deletions(-)


diff --git a/Lib/random.py b/Lib/random.py
--- a/Lib/random.py
+++ b/Lib/random.py
@@ -108,7 +108,9 @@
 
         if a is None:
             try:
-                a = long(_hexlify(_urandom(32)), 16)
+                # Seed with enough bytes to span the 19937 bit
+                # state space for the Mersenne Twister
+                a = long(_hexlify(_urandom(2500)), 16)
             except NotImplementedError:
                 import time
                 a = long(time.time() * 256) # use fractional seconds
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -52,6 +52,9 @@
 - Issue #21306: Backport hmac.compare_digest from Python 3. This is part of PEP
   466.
 
+- Issue #21470: Do a better job seeding the random number generator by
+  using enough bytes to span the full state space of the Mersenne Twister.
+
 - Issue #21469:  Reduced the risk of false positives in robotparser by
   checking to make sure that robots.txt has been read or does not exist
   prior to returning True in can_fetch().

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 07:21:36 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Wed, 14 May 2014 07:21:36 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxNDcw?=
 =?utf-8?q?=3A_Do_a_better_job_seeding_the_random_number_generator?=
Message-ID: <3gT44J1md3z7Ljc@mail.python.org>

http://hg.python.org/cpython/rev/c203df907092
changeset:   90692:c203df907092
branch:      3.4
parent:      90689:16d26391ec36
user:        Raymond Hettinger <python at rcn.com>
date:        Tue May 13 22:13:40 2014 -0700
summary:
  Issue #21470: Do a better job seeding the random number generator
to fully cover its state space.

files:
  Lib/random.py |  4 +++-
  Misc/NEWS     |  3 +++
  2 files changed, 6 insertions(+), 1 deletions(-)


diff --git a/Lib/random.py b/Lib/random.py
--- a/Lib/random.py
+++ b/Lib/random.py
@@ -105,7 +105,9 @@
 
         if a is None:
             try:
-                a = int.from_bytes(_urandom(32), 'big')
+                # Seed with enough bytes to span the 19937 bit
+                # state space for the Mersenne Twister
+                a = int.from_bytes(_urandom(2500), 'big')
             except NotImplementedError:
                 import time
                 a = int(time.time() * 256) # use fractional seconds
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -26,6 +26,9 @@
 - Issue #21396: Fix TextIOWrapper(..., write_through=True) to not force a
   flush() on the underlying binary stream.  Patch by akira.
 
+- Issue #21470: Do a better job seeding the random number generator by
+  using enough bytes to span the full state space of the Mersenne Twister.
+
 - Issue #21398: Fix an unicode error in the pydoc pager when the documentation
   contains characters not encodable to the stdout encoding.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 07:21:37 2014
From: python-checkins at python.org (raymond.hettinger)
Date: Wed, 14 May 2014 07:21:37 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?q?=29=3A_merge?=
Message-ID: <3gT44K4Cm7z7LlM@mail.python.org>

http://hg.python.org/cpython/rev/c5464268aead
changeset:   90693:c5464268aead
parent:      90690:bdf94b2c0639
parent:      90692:c203df907092
user:        Raymond Hettinger <python at rcn.com>
date:        Tue May 13 22:21:04 2014 -0700
summary:
  merge

files:
  Lib/random.py |  4 +++-
  1 files changed, 3 insertions(+), 1 deletions(-)


diff --git a/Lib/random.py b/Lib/random.py
--- a/Lib/random.py
+++ b/Lib/random.py
@@ -105,7 +105,9 @@
 
         if a is None:
             try:
-                a = int.from_bytes(_urandom(32), 'big')
+                # Seed with enough bytes to span the 19937 bit
+                # state space for the Mersenne Twister
+                a = int.from_bytes(_urandom(2500), 'big')
             except NotImplementedError:
                 import time
                 a = int(time.time() * 256) # use fractional seconds

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 14:10:36 2014
From: python-checkins at python.org (nick.coghlan)
Date: Wed, 14 May 2014 14:10:36 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_470=3A_use_external_index?=
	=?utf-8?q?es_over_link_spidering?=
Message-ID: <3gTF8D69Nyz7Llc@mail.python.org>

http://hg.python.org/peps/rev/12990e9fa4a9
changeset:   5476:12990e9fa4a9
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Wed May 14 22:10:25 2014 +1000
summary:
  PEP 470: use external indexes over link spidering

files:
  pep-0470.txt |  377 +++++++++++++++++++++++++++++++++++++++
  1 files changed, 377 insertions(+), 0 deletions(-)


diff --git a/pep-0470.txt b/pep-0470.txt
new file mode 100644
--- /dev/null
+++ b/pep-0470.txt
@@ -0,0 +1,377 @@
+PEP: 470
+Title: Using Multi Index Support for External to PyPI Package File Hosting
+Version: $Revision$
+Last-Modified: $Date$
+Author: Donald Stufft <donald at stufft.io>,
+BDFL-Delegate: Richard Jones <richard at python.org>
+Discussions-To: distutils-sig at python.org
+Status: Draft
+Type: Process
+Content-Type: text/x-rst
+Created: 12-May-2014
+Post-History: 14-May-2014
+
+
+Abstract
+========
+
+This PEP proposes that the official means of having an installer locate and
+find package files which are hosted externally to PyPI become the use of
+multi index support instead of the practice of using external links on the
+simple installer API.
+
+It is important to remember that this is **not** about forcing anyone to host
+their files on PyPI. If someone does not wish to do so they will never be under
+any obligation too. They can still list their project in PyPI as an index, and
+the tooling will still allow them to host it elsewhere.
+
+
+Rationale
+=========
+
+There is a long history documented in PEP 438 that explains why externally
+hosted files exist today in the state that they do on PyPI. For the sake of
+brevity I will not duplicate that and instead urge readers to first take a look
+at PEP 438 for background.
+
+There are currently two primary ways for a project to make itself available
+without directly hosting the package files on PyPI. They can either include
+links to the package files in the simpler installer API or they can publish
+a custom package index which contains their project.
+
+
+Custom Additional Index
+-----------------------
+
+Each installer which speaks to PyPI offers a mechanism for the user invoking
+that installer to provide additional custom locations to search for files
+during the dependency resolution phase. For pip these locations can be
+configured per invocation, per shell environment, per requirements file, per
+virtual environment, and per user.
+
+The use of additional indexes instead of external links on the simple
+installer API provides a simple clean interface which is consistent with the
+way most Linux package systems work (apt-get, yum, etc). More importantly it
+works the same even for projects which are commercial or otherwise have their
+access restricted in some form (private networks, password, IP ACLs etc)
+while the external links method only realistically works for projects which
+do not have their access restricted.
+
+Compared to the complex rules which a project must be aware of to prevent
+themselves from being considered unsafely hosted setting up an index is fairly
+trivial and in the simplest case does not require anything more than a
+filesystem and a standard web server such as Nginx. Even if using simple
+static hosting without autoindexing support, it is still straightforward
+to generate appropriate index pages as static HTML.
+
+Example Index with Nginx
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Create a root directory for your index, for the purposes of the example
+   I'll assume you've chosen ``/var/www/index.example.com/``.
+2. Inside of this root directory, create a directory for each project such
+   as ``mkdir -p /var/www/index.example.com/{foo,bar,other}/``.
+3. Place the package files for each project in their respective folder,
+   creating paths like ``/var/www/index.example.com/foo/foo-1.0.tar.gz``.
+4. Configure nginx to serve the root directory, ideally with TLS, with the
+   autoindex directive enable (see below for example configuration).
+
+    ::
+
+        server {
+            listen 443 ssl;
+            server_name index.example.com;
+
+            ssl_certificate     /etc/pki/tls/certs/index.example.com.crt;
+            ssl_certificate_key /etc/pki/tls/certs/index.example.com.key;
+
+            root /var/www/index.example.com;
+
+            autoindex on;
+        }
+
+
+Examples of Additional indexes with pip
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+**Invocation:**
+
+    ::
+        $ pip install --extra-index-url https://pypi.example.com/ foobar
+
+**Shell Environment:**
+
+    ::
+        $ export PIP_EXTRA_INDEX_URL=https://pypi.example.com/
+        $ pip install foobar
+
+**Requirements File:**
+
+    ::
+        $ echo "--extra-index-url https://pypi.example.com/\nfoobar" > requirements.txt
+        $ pip install -r requirements.txt
+
+**Virtual Environment:**
+
+    ::
+        $ python -m venv myvenv
+        $ echo "[global]\nextra-index-url = https://pypi.exmaple.com/" > myvenv/pip.conf
+        $ myvenv/bin/pip install foobar
+
+**User:**
+
+    ::
+        $ echo "[global]\nextra-index-url = https://pypi.exmaple.com/" >~/.pip/pip.conf
+        $ pip install foobar
+
+
+External Links on the Simple Installer API
+------------------------------------------
+
+PEP438 proposed a system of classifying file links as either internal,
+external, or unsafe. It recommended that by default only internal links would
+be installed by an installer however users could opt into external links on
+either a global or a per package basis. Additionally they could also opt into
+unsafe links on a per package basis.
+
+This system has turned out to be *extremely* unfriendly towards the end users
+and it is the position of this PEP that the situation has become untenable. The
+situation as provided by PEP438 requires an end user to be aware not only of
+the difference between internal, external, and unsafe, but also to be aware of
+what hosting mode the package they are trying to install is in, what links are
+available on that project's /simple/ page, whether or not those links have
+a properly formatted hash fragment, and what links are available from pages
+linked to from that project's /simple/ page.
+
+There are a number of common confusion/pain points with this system that I
+have witnessed:
+
+* Users unaware what the simple installer api is at all or how an installer
+  locates installable files.
+* Users unaware that even if the simple api links to a file, if it does
+  not include a ``#md5=...`` fragment that it will be counted as unsafe.
+* Users unaware that an installer can look at pages linked from the
+  simple api to determine additional links, or that any links found in this
+  fashion are considered unsafe.
+* Users are unaware and often surprised that PyPI supports hosting your files
+  someplace other than PyPI at all.
+
+In addition to that, the information that an installer is able to provide
+when an installation fails is pretty minimal. We are able to detect if there
+are externally hosted files directly linked from the simple installer api,
+however we cannot detect if there are files hosted on a linked page without
+fetching that page and doing so would cause a massive performance hit just to
+see if there might be a file there so that a better error message could be
+provided.
+
+Finally very few projects have properly linked to their external files so that
+they can be safely downloaded and verified. At the time of this writing there
+are a total of 65 projects which have files that are only available externally
+and are safely hosted.
+
+The end result of all of this, is that with PEP 438, when a user attempts to
+install a file that is not hosted on PyPI typically the steps they follow are:
+
+1. First, they attempt to install it normally, using ``pip install foobar``.
+   This fails because the file is not hosted on PyPI and PEP 438 has us default
+   to only hosted on PyPI. If pip detected any externally hosted files or other
+   pages that we *could* have attempted to find other files at it will give an
+   error message suggesting that they try ``--allow-external foobar``.
+2. They then attempt to install their package using
+   ``pip install --allow-external foobar foobar``. If they are lucky foobar is
+   one of the packages which is hosted externally and safely and this will
+   succeed. If they are unlucky they will get a different error message
+   suggesting that they *also* try ``--allow-unverified foobar``.
+3. They then attempt to install their package using
+   ``pip install --allow-external foobar --allow-unverified foobar foobar``
+   and this finally works.
+
+This is the same basic steps that practically everyone goes through every time
+they try to install something that is not hosted on PyPI. If they are lucky it'll
+only take them two steps, but typically it requires three steps. Worse there is
+no real indication to these people why one package might install after two
+but most require three. Even worse than that most of them will never get an
+externally hosted package that does not take three steps, so they will be
+increasingly annoyed and frustrated at the intermediate step and will likely
+eventually just start skipping it.
+
+
+External Index Discovery
+========================
+
+One of the problems with using an additional index is one of discovery. Users
+will not generally be aware that an additional index is required at all much
+less where that index can be found. Projects can attempt to convey this
+information using their description on the PyPI page however that excludes
+people who discover their project organically through ``pip search``.
+
+To support projects that wish to externally host their files and to enable
+users to easily discover what additional indexes are required, PyPI will gain
+the ability for projects to register external index URLs and additionally an
+associated comment for each. These URLs will be made available on the simple
+page however they will not be linked or provided in a form that older
+installers will automatically search them.
+
+When an installer fetches the simple page for a project, if it finds this
+additional meta-data and it cannot find any files for that project in it's
+configured URLs then it should use this data to tell the user how to add one
+or more of the additional URLs to search in. This message should include any
+comments that the project has included to enable them to communicate to the
+user and provide hints as to which URL they might want if some are only
+useful or compatible with certain platforms or situations.
+
+This feature *must* be added to PyPI prior to starting the deprecation and
+removal process for link spidering.
+
+
+Deprecation and Removal of Link Spidering
+=========================================
+
+A new hosting mode will be added to PyPI. This hosting mode will be called
+``pypi-only`` and will be in addition to the three that PEP438 has already given
+us which are ``pypi-explicit``, ``pypi-scrape``, ``pypi-scrape-crawl``. This
+new hosting mode will modify a project's simple api page so that it only lists
+the files which are directly hosted on PyPI and will not link to anything else.
+
+Upon acceptance of this PEP and the addition of the ``pypi-only`` mode, all new
+projects will by defaulted to the PyPI only mode and they will be locked to
+this mode and unable to change this particular setting. ``pypi-only`` projects
+will still be able to register external index URLs as described above - the
+"pypi-only" refers only to the download links that are published directly on
+PyPI.
+
+An email will then be sent out to all of the projects which are hosted only on
+PyPI informing them that in one month their project will be automatically
+converted to the ``pypi-only`` mode. A month after these emails have been sent
+any of those projects which were emailed, which still are hosted only on PyPI
+will have their mode set to ``pypi-only``.
+
+After that switch, an email will be sent to projects which rely on hosting
+external to PyPI. This email will warn these projects that externally hosted
+files have been deprecated on PyPI and that in 6 months from the time of that
+email that all external links will be removed from the installer APIs. This
+email *must* include instructions for converting their projects to be hosted
+on PyPI and *must* include links to a script or package that will enable them
+to enter their PyPI credentials and package name and have it automatically
+download and re-host all of their files on PyPI. This email *must also*
+include instructions for setting up their own index page and registering that
+with PyPI.
+
+Five months after the initial email, another email must be sent to any projects
+still relying on external hosting. This email will include all of the same
+information that the first email contained, except that the removal date will
+be one month away instead of six.
+
+Finally a month later all projects will be switched to the ``pypa-only`` mode
+and PyPI will be modified to remove the externally linked files functionality.
+
+
+Impact
+======
+
+============ ======= ========== =======
+\             PyPI    External   Total
+============ ======= ========== =======
+ **Safe**     37779   65         37844
+ **Unsafe**   0       2974       2974
+ **Total**    37779   3039
+============ ======= ========== =======
+
+
+Rejected Proposals
+==================
+
+Keep the current classification system but adjust the options
+-------------------------------------------------------------
+
+This PEP rejects several related proposals which attempt to fix some of the
+usability problems with the current system but while still keeping the
+general gist of PEP 438.
+
+This includes:
+
+* Default to allowing safely externally hosted files, but disallow unsafely
+  hosted.
+* Default to disallowing safely externally hosted files with only a global
+  flag to enable them, but disallow unsafely hosted.
+
+These proposals are rejected because:
+
+* The classification "system" is complex, hard to explain, and requires an
+  intimate knowledge of how the simple API works in order to be able to reason
+  about which classification is required. This is reflected in the fact that
+  the code to implement it is complicated and hard to understand as well.
+
+* People are generally surprised that PyPI allows externally linking to files
+  and doesn't require people to host on PyPI. In contrast most of them are
+  familiar with the concept of multiple software repositories such as is in
+  use by many OSs.
+
+* PyPI is fronted by a globally distributed CDN which has improved the
+  reliability and speed for end users. It is unlikely that any particular
+  external host has something comparable. This can lead to extremely bad
+  performance for end users when the external host is located in different
+  parts of the world or does not generally have good connectivity.
+
+  As a data point, many users reported sub DSL speeds and latency when
+  accessing PyPI from parts of Europe and Asia prior to the use of the CDN.
+
+* PyPI has monitoring and an on-call rotation of sysadmins whom can respond to
+  downtime quickly, thus enabling a quicker response to downtime. Again it is
+  unlikely that any particular external host will have this. This can lead
+  to single packages in a dependency chain being un-installable. This will
+  often confuse users, who often times have no idea that this package relies
+  on an external host, and they cannot figure out why PyPI appears to be up
+  but the installer cannot find a package.
+
+* PyPI supports mirroring, both for private organizations and public mirrors.
+  The legal terms of uploading to PyPI ensure that mirror operators, both
+  public and private, have the right to distribute the software found on PyPI.
+  However software that is hosted externally does not have this, causing
+  private organizations to need to investigate each package individually and
+  manually to determine if the license allows them to mirror it.
+
+  For public mirrors this essentially means that these externally hosted
+  packages *cannot* be reasonably mirrored. This is particularly troublesome
+  in countries such as China where the bandwidth to outside of China is
+  highly congested making a mirror within China often times a massively better
+  experience.
+
+* Installers have no method to determine if they should expect any particular
+  URL to be available or not. It is not unusual for the simple API to reference
+  old packages and URLs which have long since stopped working. This causes
+  installers to have to assume that it is OK for any particular URL to not be
+  accessible. This causes problems where an URL is temporarily down or
+  otherwise unavailable (a common cause of this is using a copy of Python
+  linked against a really ancient copy of OpenSSL which is unable to verify
+  the SSL certificate on PyPI) but it *should* be expected to be up. In this
+  case installers will typically silently ignore this URL and later the user
+  will get a confusing error stating that the installer couldn't find any
+  versions instead of getting the real error message indicating that the URL
+  was unavailable.
+
+* In the long run, global opt in flags like ``--allow-all-external`` will
+  become little annoyances that developers cargo cult around in order to make
+  their installer work. When they run into a project that requires it they
+  will most likely simply add it to their configuration file for that installer
+  and continue on with whatever they were actually trying to do. This will
+  continue until they try to install their requirements on another computer
+  or attempt to deploy to a server where their install will fail again until
+  they add the "make it work" flag in their configuration file.
+
+
+Copyright
+=========
+
+This document has been placed in the public domain.
+
+
+

+..
+   Local Variables:
+   mode: indented-text
+   indent-tabs-mode: nil
+   sentence-end-double-space: t
+   fill-column: 70
+   coding: utf-8
+   End:

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Wed May 14 16:10:38 2014
From: python-checkins at python.org (r.david.murray)
Date: Wed, 14 May 2014 16:10:38 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogIzIxMzQ3OiB1c2Ug?=
 =?utf-8?q?string_not_list_in_shell=3DTrue_example=2E?=
Message-ID: <3gTHpk0hz4z7Ljl@mail.python.org>

http://hg.python.org/cpython/rev/5ef9a2c711f5
changeset:   90694:5ef9a2c711f5
branch:      2.7
parent:      90691:7b5265752942
user:        R David Murray <rdmurray at bitdance.com>
date:        Wed May 14 10:09:21 2014 -0400
summary:
  #21347: use string not list in shell=True example.

Patch by Akira.

files:
  Doc/library/subprocess.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst
--- a/Doc/library/subprocess.rst
+++ b/Doc/library/subprocess.rst
@@ -866,7 +866,7 @@
 
    (child_stdout, child_stdin) = popen2.popen2("somestring", bufsize, mode)
    ==>
-   p = Popen(["somestring"], shell=True, bufsize=bufsize,
+   p = Popen("somestring", shell=True, bufsize=bufsize,
              stdin=PIPE, stdout=PIPE, close_fds=True)
    (child_stdout, child_stdin) = (p.stdout, p.stdin)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 16:10:39 2014
From: python-checkins at python.org (r.david.murray)
Date: Wed, 14 May 2014 16:10:39 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogIzIxMzQ3OiB1c2Ug?=
 =?utf-8?q?string_not_list_in_shell=3DTrue_example=2E?=
Message-ID: <3gTHpl2b67z7Lnp@mail.python.org>

http://hg.python.org/cpython/rev/3b27f3acf0c4
changeset:   90695:3b27f3acf0c4
branch:      3.4
parent:      90692:c203df907092
user:        R David Murray <rdmurray at bitdance.com>
date:        Wed May 14 10:09:52 2014 -0400
summary:
  #21347: use string not list in shell=True example.

Patch by Akira.

files:
  Doc/library/subprocess.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst
--- a/Doc/library/subprocess.rst
+++ b/Doc/library/subprocess.rst
@@ -1013,7 +1013,7 @@
 
    (child_stdout, child_stdin) = popen2.popen2("somestring", bufsize, mode)
    ==>
-   p = Popen(["somestring"], shell=True, bufsize=bufsize,
+   p = Popen("somestring", shell=True, bufsize=bufsize,
              stdin=PIPE, stdout=PIPE, close_fds=True)
    (child_stdout, child_stdin) = (p.stdout, p.stdin)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 16:10:40 2014
From: python-checkins at python.org (r.david.murray)
Date: Wed, 14 May 2014 16:10:40 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge=3A_=2321347=3A_use_string_not_list_in_shell=3DTrue?=
 =?utf-8?q?_example=2E?=
Message-ID: <3gTHpm4Wdzz7Lp0@mail.python.org>

http://hg.python.org/cpython/rev/217006c5455f
changeset:   90696:217006c5455f
parent:      90693:c5464268aead
parent:      90695:3b27f3acf0c4
user:        R David Murray <rdmurray at bitdance.com>
date:        Wed May 14 10:10:14 2014 -0400
summary:
  Merge: #21347: use string not list in shell=True example.

files:
  Doc/library/subprocess.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst
--- a/Doc/library/subprocess.rst
+++ b/Doc/library/subprocess.rst
@@ -1014,7 +1014,7 @@
 
    (child_stdout, child_stdin) = popen2.popen2("somestring", bufsize, mode)
    ==>
-   p = Popen(["somestring"], shell=True, bufsize=bufsize,
+   p = Popen("somestring", shell=True, bufsize=bufsize,
              stdin=PIPE, stdout=PIPE, close_fds=True)
    (child_stdout, child_stdin) = (p.stdout, p.stdin)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 16:38:09 2014
From: python-checkins at python.org (donald.stufft)
Date: Wed, 14 May 2014 16:38:09 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?peps=3A_Use_Twisted_Web_as_it=27s_a_b?=
 =?utf-8?q?it_simpler_to_configure_and_is_Python?=
Message-ID: <3gTJQT0dNNz7LkT@mail.python.org>

http://hg.python.org/peps/rev/2e8893ae1431
changeset:   5477:2e8893ae1431
user:        Donald Stufft <donald at stufft.io>
date:        Wed May 14 10:37:40 2014 -0400
summary:
  Use Twisted Web as it's a bit simpler to configure and is Python

files:
  pep-0470.txt |  26 +++++++-------------------
  1 files changed, 7 insertions(+), 19 deletions(-)


diff --git a/pep-0470.txt b/pep-0470.txt
--- a/pep-0470.txt
+++ b/pep-0470.txt
@@ -60,12 +60,12 @@
 Compared to the complex rules which a project must be aware of to prevent
 themselves from being considered unsafely hosted setting up an index is fairly
 trivial and in the simplest case does not require anything more than a
-filesystem and a standard web server such as Nginx. Even if using simple
-static hosting without autoindexing support, it is still straightforward
-to generate appropriate index pages as static HTML.
+filesystem and a standard web server such as Nginx or Twisted Web. Even if
+using simple static hosting without autoindexing support, it is still
+straightforward to generate appropriate index pages as static HTML.
 
-Example Index with Nginx
-~~~~~~~~~~~~~~~~~~~~~~~~
+Example Index with Twisted Web
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 1. Create a root directory for your index, for the purposes of the example
    I'll assume you've chosen ``/var/www/index.example.com/``.
@@ -73,22 +73,10 @@
    as ``mkdir -p /var/www/index.example.com/{foo,bar,other}/``.
 3. Place the package files for each project in their respective folder,
    creating paths like ``/var/www/index.example.com/foo/foo-1.0.tar.gz``.
-4. Configure nginx to serve the root directory, ideally with TLS, with the
-   autoindex directive enable (see below for example configuration).
+4. Configure Twisted Web to serve the root directory, ideally with TLS.
 
     ::
-
-        server {
-            listen 443 ssl;
-            server_name index.example.com;
-
-            ssl_certificate     /etc/pki/tls/certs/index.example.com.crt;
-            ssl_certificate_key /etc/pki/tls/certs/index.example.com.key;
-
-            root /var/www/index.example.com;
-
-            autoindex on;
-        }
+        $ twistd -n web --path /var/www/index.example.com/
 
 
 Examples of Additional indexes with pip

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Wed May 14 16:58:31 2014
From: python-checkins at python.org (donald.stufft)
Date: Wed, 14 May 2014 16:58:31 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?peps=3A_Fix_the_literal_blocks?=
Message-ID: <3gTJsz4f67z7Lnc@mail.python.org>

http://hg.python.org/peps/rev/8270223d7c91
changeset:   5478:8270223d7c91
user:        Donald Stufft <donald at stufft.io>
date:        Wed May 14 10:58:03 2014 -0400
summary:
  Fix the literal blocks

files:
  pep-0470.txt |  35 ++++++++++++++++++++---------------
  1 files changed, 20 insertions(+), 15 deletions(-)


diff --git a/pep-0470.txt b/pep-0470.txt
--- a/pep-0470.txt
+++ b/pep-0470.txt
@@ -84,33 +84,38 @@
 
 **Invocation:**
 
-    ::
-        $ pip install --extra-index-url https://pypi.example.com/ foobar
+::
+
+    $ pip install --extra-index-url https://pypi.example.com/ foobar
 
 **Shell Environment:**
 
-    ::
-        $ export PIP_EXTRA_INDEX_URL=https://pypi.example.com/
-        $ pip install foobar
+::
+
+    $ export PIP_EXTRA_INDEX_URL=https://pypi.example.com/
+    $ pip install foobar
 
 **Requirements File:**
 
-    ::
-        $ echo "--extra-index-url https://pypi.example.com/\nfoobar" > requirements.txt
-        $ pip install -r requirements.txt
+::
+
+    $ echo "--extra-index-url https://pypi.example.com/\nfoobar" > requirements.txt
+    $ pip install -r requirements.txt
 
 **Virtual Environment:**
 
-    ::
-        $ python -m venv myvenv
-        $ echo "[global]\nextra-index-url = https://pypi.exmaple.com/" > myvenv/pip.conf
-        $ myvenv/bin/pip install foobar
+::
+
+    $ python -m venv myvenv
+    $ echo "[global]\nextra-index-url = https://pypi.exmaple.com/" > myvenv/pip.conf
+    $ myvenv/bin/pip install foobar
 
 **User:**
 
-    ::
-        $ echo "[global]\nextra-index-url = https://pypi.exmaple.com/" >~/.pip/pip.conf
-        $ pip install foobar
+::
+
+    $ echo "[global]\nextra-index-url = https://pypi.exmaple.com/" >~/.pip/pip.conf
+    $ pip install foobar
 
 
 External Links on the Simple Installer API

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Wed May 14 16:59:06 2014
From: python-checkins at python.org (donald.stufft)
Date: Wed, 14 May 2014 16:59:06 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?peps=3A_Another_missing_literal_block?=
Message-ID: <3gTJtf0yt8z7Lkf@mail.python.org>

http://hg.python.org/peps/rev/ee7e8acf43f9
changeset:   5479:ee7e8acf43f9
user:        Donald Stufft <donald at stufft.io>
date:        Wed May 14 10:58:38 2014 -0400
summary:
  Another missing literal block

files:
  pep-0470.txt |  5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)


diff --git a/pep-0470.txt b/pep-0470.txt
--- a/pep-0470.txt
+++ b/pep-0470.txt
@@ -75,8 +75,9 @@
    creating paths like ``/var/www/index.example.com/foo/foo-1.0.tar.gz``.
 4. Configure Twisted Web to serve the root directory, ideally with TLS.
 
-    ::
-        $ twistd -n web --path /var/www/index.example.com/
+::
+
+    $ twistd -n web --path /var/www/index.example.com/
 
 
 Examples of Additional indexes with pip

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Wed May 14 17:10:01 2014
From: python-checkins at python.org (victor.stinner)
Date: Wed, 14 May 2014 17:10:01 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIxNDg4?=
 =?utf-8?q?=3A_Fix_doc_of_codecs=2Edecode=28=29_and_codecs=2Eencode=28=29?=
 =?utf-8?q?=2C_no_keyword?=
Message-ID: <3gTK7F16l0z7Lny@mail.python.org>

http://hg.python.org/cpython/rev/cc5e3b93c35a
changeset:   90697:cc5e3b93c35a
branch:      2.7
parent:      90694:5ef9a2c711f5
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Wed May 14 17:07:08 2014 +0200
summary:
  Issue #21488: Fix doc of codecs.decode() and codecs.encode(), no keyword support.
Patch written by Brad Aylsworth.

files:
  Doc/library/codecs.rst |  14 ++++++++------
  Misc/ACKS              |   1 +
  2 files changed, 9 insertions(+), 6 deletions(-)


diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst
--- a/Doc/library/codecs.rst
+++ b/Doc/library/codecs.rst
@@ -23,24 +23,26 @@
 
 It defines the following functions:
 
-.. function:: encode(obj, encoding='ascii', errors='strict')
+.. function:: encode(obj, [encoding[, errors]])
 
-   Encodes *obj* using the codec registered for *encoding*.
+   Encodes *obj* using the codec registered for *encoding*. The default
+   encoding is ``'ascii'``.
 
    *Errors* may be given to set the desired error handling scheme. The
-   default error handler is ``strict`` meaning that encoding errors raise
+   default error handler is ``'strict'`` meaning that encoding errors raise
    :exc:`ValueError` (or a more codec specific subclass, such as
    :exc:`UnicodeEncodeError`). Refer to :ref:`codec-base-classes` for more
    information on codec error handling.
 
    .. versionadded:: 2.4
 
-.. function:: decode(obj, encoding='ascii', errors='strict')
+.. function:: decode(obj, [encoding[, errors]])
 
-   Decodes *obj* using the codec registered for *encoding*.
+   Decodes *obj* using the codec registered for *encoding*. The default
+   encoding is ``'ascii'``.
 
    *Errors* may be given to set the desired error handling scheme. The
-   default error handler is ``strict`` meaning that decoding errors raise
+   default error handler is ``'strict'`` meaning that decoding errors raise
    :exc:`ValueError` (or a more codec specific subclass, such as
    :exc:`UnicodeDecodeError`). Refer to :ref:`codec-base-classes` for more
    information on codec error handling.
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -56,6 +56,7 @@
 Chris AtLee
 Aymeric Augustin
 John Aycock
+Brad Aylsworth
 Donovan Baarda
 Arne Babenhauserheide
 Attila Babo

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 17:10:04 2014
From: python-checkins at python.org (victor.stinner)
Date: Wed, 14 May 2014 17:10:04 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxNDg4?=
 =?utf-8?q?=3A_Fix_doc_of_codecs=2Edecode=28=29_and_codecs=2Eencode=28=29?=
 =?utf-8?q?=2C_no_keyword?=
Message-ID: <3gTK7J59dBz7Lm7@mail.python.org>

http://hg.python.org/cpython/rev/2e116176a81f
changeset:   90698:2e116176a81f
branch:      3.4
parent:      90695:3b27f3acf0c4
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Wed May 14 17:08:45 2014 +0200
summary:
  Issue #21488: Fix doc of codecs.decode() and codecs.encode(), no keyword support.
Patch written by Brad Aylsworth.

files:
  Doc/library/codecs.rst |  10 ++++++----
  Misc/ACKS              |   1 +
  2 files changed, 7 insertions(+), 4 deletions(-)


diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst
--- a/Doc/library/codecs.rst
+++ b/Doc/library/codecs.rst
@@ -22,9 +22,10 @@
 
 It defines the following functions:
 
-.. function:: encode(obj, encoding='utf-8', errors='strict')
+.. function:: encode(obj, [encoding[, errors]])
 
-   Encodes *obj* using the codec registered for *encoding*.
+   Encodes *obj* using the codec registered for *encoding*. The default
+   encoding is ``utf-8``.
 
    *Errors* may be given to set the desired error handling scheme. The
    default error handler is ``strict`` meaning that encoding errors raise
@@ -32,9 +33,10 @@
    :exc:`UnicodeEncodeError`). Refer to :ref:`codec-base-classes` for more
    information on codec error handling.
 
-.. function:: decode(obj, encoding='utf-8', errors='strict')
+.. function:: decode(obj, [encoding[, errors]])
 
-   Decodes *obj* using the codec registered for *encoding*.
+   Decodes *obj* using the codec registered for *encoding*. The default
+   encoding is ``utf-8``.
 
    *Errors* may be given to set the desired error handling scheme. The
    default error handler is ``strict`` meaning that decoding errors raise
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -57,6 +57,7 @@
 Chris AtLee
 Aymeric Augustin
 John Aycock
+Brad Aylsworth
 Donovan Baarda
 Arne Babenhauserheide
 Attila Babo

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 17:10:05 2014
From: python-checkins at python.org (victor.stinner)
Date: Wed, 14 May 2014 17:10:05 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge_3=2E4=3A_ignore_change_specific_to_3=2E4_for_=2321?=
 =?utf-8?q?488=2C_I_had_a_different_patch?=
Message-ID: <3gTK7K6j0wz7Lnr@mail.python.org>

http://hg.python.org/cpython/rev/889896471498
changeset:   90699:889896471498
parent:      90696:217006c5455f
parent:      90698:2e116176a81f
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Wed May 14 17:09:39 2014 +0200
summary:
  Merge 3.4: ignore change specific to 3.4 for #21488, I had a different patch for Python 3.5

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 17:11:00 2014
From: python-checkins at python.org (victor.stinner)
Date: Wed, 14 May 2014 17:11:00 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxNDg4?=
 =?utf-8?q?=3A_Add_support_of_keyword_arguments_for_codecs=2Eencode_and?=
Message-ID: <3gTK8N4kqVz7LjV@mail.python.org>

http://hg.python.org/cpython/rev/6ceedbd88b5f
changeset:   90700:6ceedbd88b5f
branch:      3.4
parent:      90698:2e116176a81f
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Wed May 14 17:10:45 2014 +0200
summary:
  Issue #21488: Add support of keyword arguments for codecs.encode and codecs.decode

files:
  Lib/test/test_codecs.py |  12 ++++++++++++
  Modules/_codecsmodule.c |  16 ++++++++++------
  2 files changed, 22 insertions(+), 6 deletions(-)


diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py
--- a/Lib/test/test_codecs.py
+++ b/Lib/test/test_codecs.py
@@ -1600,6 +1600,12 @@
         self.assertEqual(codecs.decode(b'abc'), 'abc')
         self.assertRaises(UnicodeDecodeError, codecs.decode, b'\xff', 'ascii')
 
+        # test keywords
+        self.assertEqual(codecs.decode(obj=b'\xe4\xf6\xfc', encoding='latin-1'),
+                         '\xe4\xf6\xfc')
+        self.assertEqual(codecs.decode(b'[\xff]', 'ascii', errors='ignore'),
+                         '[]')
+
     def test_encode(self):
         self.assertEqual(codecs.encode('\xe4\xf6\xfc', 'latin-1'),
                          b'\xe4\xf6\xfc')
@@ -1608,6 +1614,12 @@
         self.assertEqual(codecs.encode('abc'), b'abc')
         self.assertRaises(UnicodeEncodeError, codecs.encode, '\xffff', 'ascii')
 
+        # test keywords
+        self.assertEqual(codecs.encode(obj='\xe4\xf6\xfc', encoding='latin-1'),
+                         b'\xe4\xf6\xfc')
+        self.assertEqual(codecs.encode('[\xff]', 'ascii', errors='ignore'),
+                         b'[]')
+
     def test_register(self):
         self.assertRaises(TypeError, codecs.register)
         self.assertRaises(TypeError, codecs.register, 42)
diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c
--- a/Modules/_codecsmodule.c
+++ b/Modules/_codecsmodule.c
@@ -89,13 +89,15 @@
 codecs.register_error that can handle ValueErrors.");
 
 static PyObject *
-codec_encode(PyObject *self, PyObject *args)
+codec_encode(PyObject *self, PyObject *args, PyObject *kwargs)
 {
+    static char *kwlist[] = {"obj", "encoding", "errors", NULL};
     const char *encoding = NULL;
     const char *errors = NULL;
     PyObject *v;
 
-    if (!PyArg_ParseTuple(args, "O|ss:encode", &v, &encoding, &errors))
+    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O|ss:encode", kwlist,
+                                     &v, &encoding, &errors))
         return NULL;
 
     if (encoding == NULL)
@@ -116,13 +118,15 @@
 able to handle ValueErrors.");
 
 static PyObject *
-codec_decode(PyObject *self, PyObject *args)
+codec_decode(PyObject *self, PyObject *args, PyObject *kwargs)
 {
+    static char *kwlist[] = {"obj", "encoding", "errors", NULL};
     const char *encoding = NULL;
     const char *errors = NULL;
     PyObject *v;
 
-    if (!PyArg_ParseTuple(args, "O|ss:decode", &v, &encoding, &errors))
+    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O|ss:decode", kwlist,
+                                     &v, &encoding, &errors))
         return NULL;
 
     if (encoding == NULL)
@@ -1120,9 +1124,9 @@
         register__doc__},
     {"lookup",                  codec_lookup,                   METH_VARARGS,
         lookup__doc__},
-    {"encode",                  codec_encode,                   METH_VARARGS,
+    {"encode",     (PyCFunction)codec_encode,     METH_VARARGS|METH_KEYWORDS,
         encode__doc__},
-    {"decode",                  codec_decode,                   METH_VARARGS,
+    {"decode",     (PyCFunction)codec_decode,     METH_VARARGS|METH_KEYWORDS,
         decode__doc__},
     {"escape_encode",           escape_encode,                  METH_VARARGS},
     {"escape_decode",           escape_decode,                  METH_VARARGS},

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 17:13:57 2014
From: python-checkins at python.org (victor.stinner)
Date: Wed, 14 May 2014 17:13:57 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Backed_out_cha?=
	=?utf-8?q?ngeset_6ceedbd88b5f?=
Message-ID: <3gTKCn463fz7LjV@mail.python.org>

http://hg.python.org/cpython/rev/36bc19c78270
changeset:   90701:36bc19c78270
branch:      3.4
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Wed May 14 17:12:27 2014 +0200
summary:
  Backed out changeset 6ceedbd88b5f

files:
  Lib/test/test_codecs.py |  12 ------------
  Modules/_codecsmodule.c |  16 ++++++----------
  2 files changed, 6 insertions(+), 22 deletions(-)


diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py
--- a/Lib/test/test_codecs.py
+++ b/Lib/test/test_codecs.py
@@ -1600,12 +1600,6 @@
         self.assertEqual(codecs.decode(b'abc'), 'abc')
         self.assertRaises(UnicodeDecodeError, codecs.decode, b'\xff', 'ascii')
 
-        # test keywords
-        self.assertEqual(codecs.decode(obj=b'\xe4\xf6\xfc', encoding='latin-1'),
-                         '\xe4\xf6\xfc')
-        self.assertEqual(codecs.decode(b'[\xff]', 'ascii', errors='ignore'),
-                         '[]')
-
     def test_encode(self):
         self.assertEqual(codecs.encode('\xe4\xf6\xfc', 'latin-1'),
                          b'\xe4\xf6\xfc')
@@ -1614,12 +1608,6 @@
         self.assertEqual(codecs.encode('abc'), b'abc')
         self.assertRaises(UnicodeEncodeError, codecs.encode, '\xffff', 'ascii')
 
-        # test keywords
-        self.assertEqual(codecs.encode(obj='\xe4\xf6\xfc', encoding='latin-1'),
-                         b'\xe4\xf6\xfc')
-        self.assertEqual(codecs.encode('[\xff]', 'ascii', errors='ignore'),
-                         b'[]')
-
     def test_register(self):
         self.assertRaises(TypeError, codecs.register)
         self.assertRaises(TypeError, codecs.register, 42)
diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c
--- a/Modules/_codecsmodule.c
+++ b/Modules/_codecsmodule.c
@@ -89,15 +89,13 @@
 codecs.register_error that can handle ValueErrors.");
 
 static PyObject *
-codec_encode(PyObject *self, PyObject *args, PyObject *kwargs)
+codec_encode(PyObject *self, PyObject *args)
 {
-    static char *kwlist[] = {"obj", "encoding", "errors", NULL};
     const char *encoding = NULL;
     const char *errors = NULL;
     PyObject *v;
 
-    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O|ss:encode", kwlist,
-                                     &v, &encoding, &errors))
+    if (!PyArg_ParseTuple(args, "O|ss:encode", &v, &encoding, &errors))
         return NULL;
 
     if (encoding == NULL)
@@ -118,15 +116,13 @@
 able to handle ValueErrors.");
 
 static PyObject *
-codec_decode(PyObject *self, PyObject *args, PyObject *kwargs)
+codec_decode(PyObject *self, PyObject *args)
 {
-    static char *kwlist[] = {"obj", "encoding", "errors", NULL};
     const char *encoding = NULL;
     const char *errors = NULL;
     PyObject *v;
 
-    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O|ss:decode", kwlist,
-                                     &v, &encoding, &errors))
+    if (!PyArg_ParseTuple(args, "O|ss:decode", &v, &encoding, &errors))
         return NULL;
 
     if (encoding == NULL)
@@ -1124,9 +1120,9 @@
         register__doc__},
     {"lookup",                  codec_lookup,                   METH_VARARGS,
         lookup__doc__},
-    {"encode",     (PyCFunction)codec_encode,     METH_VARARGS|METH_KEYWORDS,
+    {"encode",                  codec_encode,                   METH_VARARGS,
         encode__doc__},
-    {"decode",     (PyCFunction)codec_decode,     METH_VARARGS|METH_KEYWORDS,
+    {"decode",                  codec_decode,                   METH_VARARGS,
         decode__doc__},
     {"escape_encode",           escape_encode,                  METH_VARARGS},
     {"escape_decode",           escape_decode,                  METH_VARARGS},

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 17:13:58 2014
From: python-checkins at python.org (victor.stinner)
Date: Wed, 14 May 2014 17:13:58 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge_3=2E4_=28I_commited_a_patch_to_the_wrong_branch_an?=
 =?utf-8?q?d_then_used_hg_backout=29?=
Message-ID: <3gTKCp64Hyz7LmV@mail.python.org>

http://hg.python.org/cpython/rev/294f8ad35c3a
changeset:   90702:294f8ad35c3a
parent:      90699:889896471498
parent:      90701:36bc19c78270
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Wed May 14 17:13:02 2014 +0200
summary:
  Merge 3.4 (I commited a patch to the wrong branch and then used hg backout)

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 17:14:00 2014
From: python-checkins at python.org (victor.stinner)
Date: Wed, 14 May 2014 17:14:00 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321488=3A_Add_supp?=
 =?utf-8?q?ort_of_keyword_arguments_for_codecs=2Eencode_and?=
Message-ID: <3gTKCr4C19z7LnL@mail.python.org>

http://hg.python.org/cpython/rev/0d38044c0b02
changeset:   90703:0d38044c0b02
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Wed May 14 17:13:14 2014 +0200
summary:
  Issue #21488: Add support of keyword arguments for codecs.encode and codecs.decode

files:
  Lib/test/test_codecs.py |  12 ++++++++++++
  Modules/_codecsmodule.c |  16 ++++++++++------
  2 files changed, 22 insertions(+), 6 deletions(-)


diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py
--- a/Lib/test/test_codecs.py
+++ b/Lib/test/test_codecs.py
@@ -1596,6 +1596,12 @@
         self.assertEqual(codecs.decode(b'abc'), 'abc')
         self.assertRaises(UnicodeDecodeError, codecs.decode, b'\xff', 'ascii')
 
+        # test keywords
+        self.assertEqual(codecs.decode(obj=b'\xe4\xf6\xfc', encoding='latin-1'),
+                         '\xe4\xf6\xfc')
+        self.assertEqual(codecs.decode(b'[\xff]', 'ascii', errors='ignore'),
+                         '[]')
+
     def test_encode(self):
         self.assertEqual(codecs.encode('\xe4\xf6\xfc', 'latin-1'),
                          b'\xe4\xf6\xfc')
@@ -1604,6 +1610,12 @@
         self.assertEqual(codecs.encode('abc'), b'abc')
         self.assertRaises(UnicodeEncodeError, codecs.encode, '\xffff', 'ascii')
 
+        # test keywords
+        self.assertEqual(codecs.encode(obj='\xe4\xf6\xfc', encoding='latin-1'),
+                         b'\xe4\xf6\xfc')
+        self.assertEqual(codecs.encode('[\xff]', 'ascii', errors='ignore'),
+                         b'[]')
+
     def test_register(self):
         self.assertRaises(TypeError, codecs.register)
         self.assertRaises(TypeError, codecs.register, 42)
diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c
--- a/Modules/_codecsmodule.c
+++ b/Modules/_codecsmodule.c
@@ -89,13 +89,15 @@
 codecs.register_error that can handle ValueErrors.");
 
 static PyObject *
-codec_encode(PyObject *self, PyObject *args)
+codec_encode(PyObject *self, PyObject *args, PyObject *kwargs)
 {
+    static char *kwlist[] = {"obj", "encoding", "errors", NULL};
     const char *encoding = NULL;
     const char *errors = NULL;
     PyObject *v;
 
-    if (!PyArg_ParseTuple(args, "O|ss:encode", &v, &encoding, &errors))
+    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O|ss:encode", kwlist,
+                                     &v, &encoding, &errors))
         return NULL;
 
     if (encoding == NULL)
@@ -116,13 +118,15 @@
 able to handle ValueErrors.");
 
 static PyObject *
-codec_decode(PyObject *self, PyObject *args)
+codec_decode(PyObject *self, PyObject *args, PyObject *kwargs)
 {
+    static char *kwlist[] = {"obj", "encoding", "errors", NULL};
     const char *encoding = NULL;
     const char *errors = NULL;
     PyObject *v;
 
-    if (!PyArg_ParseTuple(args, "O|ss:decode", &v, &encoding, &errors))
+    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O|ss:decode", kwlist,
+                                     &v, &encoding, &errors))
         return NULL;
 
     if (encoding == NULL)
@@ -1120,9 +1124,9 @@
         register__doc__},
     {"lookup",                  codec_lookup,                   METH_VARARGS,
         lookup__doc__},
-    {"encode",                  codec_encode,                   METH_VARARGS,
+    {"encode",     (PyCFunction)codec_encode,     METH_VARARGS|METH_KEYWORDS,
         encode__doc__},
-    {"decode",                  codec_decode,                   METH_VARARGS,
+    {"decode",     (PyCFunction)codec_decode,     METH_VARARGS|METH_KEYWORDS,
         decode__doc__},
     {"escape_encode",           escape_encode,                  METH_VARARGS},
     {"escape_decode",           escape_decode,                  METH_VARARGS},

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 17:17:16 2014
From: python-checkins at python.org (victor.stinner)
Date: Wed, 14 May 2014 17:17:16 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxNDk3?=
 =?utf-8?q?=3A_faulthandler_functions_now_raise_a_better_error_if_sys=2Est?=
 =?utf-8?q?derr_is?=
Message-ID: <3gTKHc2nn0z7Llb@mail.python.org>

http://hg.python.org/cpython/rev/ddd7db7cf036
changeset:   90704:ddd7db7cf036
branch:      3.4
parent:      90701:36bc19c78270
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Wed May 14 17:15:50 2014 +0200
summary:
  Issue #21497: faulthandler functions now raise a better error if sys.stderr is
None: RuntimeError("sys.stderr is None") instead of AttributeError("'NoneType'
object has no attribute 'fileno'").

files:
  Lib/test/test_faulthandler.py |  25 +++++++++++++++++++++++
  Modules/faulthandler.c        |   4 +++
  2 files changed, 29 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_faulthandler.py b/Lib/test/test_faulthandler.py
--- a/Lib/test/test_faulthandler.py
+++ b/Lib/test/test_faulthandler.py
@@ -591,6 +591,31 @@
     def test_register_chain(self):
         self.check_register(chain=True)
 
+    @contextmanager
+    def check_stderr_none(self):
+        stderr = sys.stderr
+        try:
+            sys.stderr = None
+            with self.assertRaises(RuntimeError) as cm:
+                yield
+            self.assertEqual(str(cm.exception), "sys.stderr is None")
+        finally:
+            sys.stderr = stderr
+
+    def test_stderr_None(self):
+        # Issue #21497: provide an helpful error if sys.stderr is None,
+        # instead of just an attribute error: "None has no attribute fileno".
+        with self.check_stderr_none():
+            faulthandler.enable()
+        with self.check_stderr_none():
+            faulthandler.dump_traceback()
+        if hasattr(faulthandler, 'dump_traceback_later'):
+            with self.check_stderr_none():
+                faulthandler.dump_traceback_later(1e-3)
+        if hasattr(faulthandler, "register"):
+            with self.check_stderr_none():
+                faulthandler.register(signal.SIGUSR1)
+
 
 if __name__ == "__main__":
     unittest.main()
diff --git a/Modules/faulthandler.c b/Modules/faulthandler.c
--- a/Modules/faulthandler.c
+++ b/Modules/faulthandler.c
@@ -144,6 +144,10 @@
             PyErr_SetString(PyExc_RuntimeError, "unable to get sys.stderr");
             return NULL;
         }
+        if (file == Py_None) {
+            PyErr_SetString(PyExc_RuntimeError, "sys.stderr is None");
+            return NULL;
+        }
     }
 
     result = _PyObject_CallMethodId(file, &PyId_fileno, "");

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 17:17:17 2014
From: python-checkins at python.org (victor.stinner)
Date: Wed, 14 May 2014 17:17:17 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_=28Merge_3=2E4=29_Issue_=2321497=3A_faulthandler_functio?=
 =?utf-8?q?ns_now_raise_a_better_error_if?=
Message-ID: <3gTKHd4X4hz7Llt@mail.python.org>

http://hg.python.org/cpython/rev/cb26887f9140
changeset:   90705:cb26887f9140
parent:      90703:0d38044c0b02
parent:      90704:ddd7db7cf036
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Wed May 14 17:16:58 2014 +0200
summary:
  (Merge 3.4) Issue #21497: faulthandler functions now raise a better error if
sys.stderr is None: RuntimeError("sys.stderr is None") instead of
AttributeError("'NoneType' object has no attribute 'fileno'").

files:
  Lib/test/test_faulthandler.py |  25 +++++++++++++++++++++++
  Modules/faulthandler.c        |   4 +++
  2 files changed, 29 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_faulthandler.py b/Lib/test/test_faulthandler.py
--- a/Lib/test/test_faulthandler.py
+++ b/Lib/test/test_faulthandler.py
@@ -591,6 +591,31 @@
     def test_register_chain(self):
         self.check_register(chain=True)
 
+    @contextmanager
+    def check_stderr_none(self):
+        stderr = sys.stderr
+        try:
+            sys.stderr = None
+            with self.assertRaises(RuntimeError) as cm:
+                yield
+            self.assertEqual(str(cm.exception), "sys.stderr is None")
+        finally:
+            sys.stderr = stderr
+
+    def test_stderr_None(self):
+        # Issue #21497: provide an helpful error if sys.stderr is None,
+        # instead of just an attribute error: "None has no attribute fileno".
+        with self.check_stderr_none():
+            faulthandler.enable()
+        with self.check_stderr_none():
+            faulthandler.dump_traceback()
+        if hasattr(faulthandler, 'dump_traceback_later'):
+            with self.check_stderr_none():
+                faulthandler.dump_traceback_later(1e-3)
+        if hasattr(faulthandler, "register"):
+            with self.check_stderr_none():
+                faulthandler.register(signal.SIGUSR1)
+
 
 if __name__ == "__main__":
     unittest.main()
diff --git a/Modules/faulthandler.c b/Modules/faulthandler.c
--- a/Modules/faulthandler.c
+++ b/Modules/faulthandler.c
@@ -144,6 +144,10 @@
             PyErr_SetString(PyExc_RuntimeError, "unable to get sys.stderr");
             return NULL;
         }
+        if (file == Py_None) {
+            PyErr_SetString(PyExc_RuntimeError, "sys.stderr is None");
+            return NULL;
+        }
     }
 
     result = _PyObject_CallMethodId(file, &PyId_fileno, "");

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 17:18:53 2014
From: python-checkins at python.org (donald.stufft)
Date: Wed, 14 May 2014 17:18:53 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?peps=3A_Fix_typos?=
Message-ID: <3gTKKT12Bnz7Llb@mail.python.org>

http://hg.python.org/peps/rev/49d18bb47ebc
changeset:   5480:49d18bb47ebc
user:        Donald Stufft <donald at stufft.io>
date:        Wed May 14 11:18:22 2014 -0400
summary:
  Fix typos

files:
  pep-0470.txt |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/pep-0470.txt b/pep-0470.txt
--- a/pep-0470.txt
+++ b/pep-0470.txt
@@ -108,14 +108,14 @@
 ::
 
     $ python -m venv myvenv
-    $ echo "[global]\nextra-index-url = https://pypi.exmaple.com/" > myvenv/pip.conf
+    $ echo "[global]\nextra-index-url = https://pypi.example.com/" > myvenv/pip.conf
     $ myvenv/bin/pip install foobar
 
 **User:**
 
 ::
 
-    $ echo "[global]\nextra-index-url = https://pypi.exmaple.com/" >~/.pip/pip.conf
+    $ echo "[global]\nextra-index-url = https://pypi.example.com/" >~/.pip/pip.conf
     $ pip install foobar
 
 

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Wed May 14 17:30:51 2014
From: python-checkins at python.org (victor.stinner)
Date: Wed, 14 May 2014 17:30:51 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321490=3A_Add_new_?=
 =?utf-8?q?C_macros=3A_Py=5FABS=28=29_and_Py=5FSTRINGIFY=28=29?=
Message-ID: <3gTKbH1kkbz7LnB@mail.python.org>

http://hg.python.org/cpython/rev/849efd365ab4
changeset:   90706:849efd365ab4
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Wed May 14 17:24:35 2014 +0200
summary:
  Issue #21490: Add new C macros: Py_ABS() and Py_STRINGIFY()

Keep _Py_STRINGIZE() in PC/pyconfig.h to not introduce a dependency between
pyconfig.h and pymacros.h.

files:
  Include/pymacro.h      |  15 +++++-
  Modules/_ssl.c         |   4 +-
  Modules/_struct.c      |   9 +--
  Modules/_tracemalloc.c |   3 -
  Objects/longobject.c   |  73 ++++++++++++++---------------
  Objects/memoryobject.c |   7 +--
  Python/marshal.c       |   8 +--
  7 files changed, 60 insertions(+), 59 deletions(-)


diff --git a/Include/pymacro.h b/Include/pymacro.h
--- a/Include/pymacro.h
+++ b/Include/pymacro.h
@@ -1,13 +1,26 @@
 #ifndef Py_PYMACRO_H
 #define Py_PYMACRO_H
 
+/* Minimum value between x and y */
 #define Py_MIN(x, y) (((x) > (y)) ? (y) : (x))
+
+/* Maximum value between x and y */
 #define Py_MAX(x, y) (((x) > (y)) ? (x) : (y))
 
+/* Absolute value of the number x */
+#define Py_ABS(x) ((x) < 0 ? -(x) : (x))
+
+#define _Py_XSTRINGIFY(x) #x
+
+/* Convert the argument to a string. For example, Py_STRINGIFY(123) is replaced
+   with "123" by the preprocessor. Defines are also replaced by their value.
+   For example Py_STRINGIFY(__LINE__) is replaced by the line number, not
+   by "__LINE__". */
+#define Py_STRINGIFY(x) _Py_XSTRINGIFY(x)
+
 /* Argument must be a char or an int in [-128, 127] or [0, 255]. */
 #define Py_CHARMASK(c) ((unsigned char)((c) & 0xff))
 
-
 /* Assert a build-time dependency, as an expression.
 
    Your compile will fail if the condition isn't true, or can't be evaluated
diff --git a/Modules/_ssl.c b/Modules/_ssl.c
--- a/Modules/_ssl.c
+++ b/Modules/_ssl.c
@@ -249,10 +249,8 @@
 } timeout_state;
 
 /* Wrap error strings with filename and line # */
-#define STRINGIFY1(x) #x
-#define STRINGIFY2(x) STRINGIFY1(x)
 #define ERRSTR1(x,y,z) (x ":" y ": " z)
-#define ERRSTR(x) ERRSTR1("_ssl.c", STRINGIFY2(__LINE__), x)
+#define ERRSTR(x) ERRSTR1("_ssl.c", Py_STRINGIFY(__LINE__), x)
 
 
 /*
diff --git a/Modules/_struct.c b/Modules/_struct.c
--- a/Modules/_struct.c
+++ b/Modules/_struct.c
@@ -85,8 +85,6 @@
 #define BOOL_ALIGN 0
 #endif
 
-#define STRINGIFY(x)    #x
-
 #ifdef __powerc
 #pragma options align=reset
 #endif
@@ -546,8 +544,8 @@
         return -1;
     if (x < SHRT_MIN || x > SHRT_MAX){
         PyErr_SetString(StructError,
-                        "short format requires " STRINGIFY(SHRT_MIN)
-                        " <= number <= " STRINGIFY(SHRT_MAX));
+                        "short format requires " Py_STRINGIFY(SHRT_MIN)
+                        " <= number <= " Py_STRINGIFY(SHRT_MAX));
         return -1;
     }
     y = (short)x;
@@ -564,7 +562,8 @@
         return -1;
     if (x < 0 || x > USHRT_MAX){
         PyErr_SetString(StructError,
-                        "ushort format requires 0 <= number <= " STRINGIFY(USHRT_MAX));
+                        "ushort format requires 0 <= number <= "
+                        Py_STRINGIFY(USHRT_MAX));
         return -1;
     }
     y = (unsigned short)x;
diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c
--- a/Modules/_tracemalloc.c
+++ b/Modules/_tracemalloc.c
@@ -16,9 +16,6 @@
 #  define TRACE_DEBUG
 #endif
 
-#define _STR(VAL) #VAL
-#define STR(VAL) _STR(VAL)
-
 /* Protected by the GIL */
 static struct {
     PyMemAllocator mem;
diff --git a/Objects/longobject.c b/Objects/longobject.c
--- a/Objects/longobject.c
+++ b/Objects/longobject.c
@@ -21,7 +21,6 @@
          Py_SIZE(x) < 0 ? -(sdigit)(x)->ob_digit[0] :   \
              (Py_SIZE(x) == 0 ? (sdigit)0 :                             \
               (sdigit)(x)->ob_digit[0]))
-#define ABS(x) ((x) < 0 ? -(x) : (x))
 
 #if NSMALLNEGINTS + NSMALLPOSINTS > 0
 /* Small integers are preallocated in this array so that they
@@ -57,7 +56,7 @@
 static PyLongObject *
 maybe_small_long(PyLongObject *v)
 {
-    if (v && ABS(Py_SIZE(v)) <= 1) {
+    if (v && Py_ABS(Py_SIZE(v)) <= 1) {
         sdigit ival = MEDIUM_VALUE(v);
         if (-NSMALLNEGINTS <= ival && ival < NSMALLPOSINTS) {
             Py_DECREF(v);
@@ -114,7 +113,7 @@
 static PyLongObject *
 long_normalize(PyLongObject *v)
 {
-    Py_ssize_t j = ABS(Py_SIZE(v));
+    Py_ssize_t j = Py_ABS(Py_SIZE(v));
     Py_ssize_t i = j;
 
     while (i > 0 && v->ob_digit[i-1] == 0)
@@ -718,7 +717,7 @@
 
     assert(v != NULL);
     assert(PyLong_Check(v));
-    ndigits = ABS(Py_SIZE(v));
+    ndigits = Py_ABS(Py_SIZE(v));
     assert(ndigits == 0 || v->ob_digit[ndigits - 1] != 0);
     if (ndigits > 0) {
         digit msd = v->ob_digit[ndigits - 1];
@@ -1565,7 +1564,7 @@
 static PyLongObject *
 divrem1(PyLongObject *a, digit n, digit *prem)
 {
-    const Py_ssize_t size = ABS(Py_SIZE(a));
+    const Py_ssize_t size = Py_ABS(Py_SIZE(a));
     PyLongObject *z;
 
     assert(n > 0 && n <= PyLong_MASK);
@@ -1597,7 +1596,7 @@
         PyErr_BadInternalCall();
         return -1;
     }
-    size_a = ABS(Py_SIZE(a));
+    size_a = Py_ABS(Py_SIZE(a));
     negative = Py_SIZE(a) < 0;
 
     /* quick and dirty upper bound for the number of digits
@@ -1766,7 +1765,7 @@
         PyErr_BadInternalCall();
         return -1;
     }
-    size_a = ABS(Py_SIZE(a));
+    size_a = Py_ABS(Py_SIZE(a));
     negative = Py_SIZE(a) < 0;
 
     /* Compute a rough upper bound for the length of the string */
@@ -2380,7 +2379,7 @@
 long_divrem(PyLongObject *a, PyLongObject *b,
             PyLongObject **pdiv, PyLongObject **prem)
 {
-    Py_ssize_t size_a = ABS(Py_SIZE(a)), size_b = ABS(Py_SIZE(b));
+    Py_ssize_t size_a = Py_ABS(Py_SIZE(a)), size_b = Py_ABS(Py_SIZE(b));
     PyLongObject *z;
 
     if (size_b == 0) {
@@ -2439,7 +2438,7 @@
 }
 
 /* Unsigned int division with remainder -- the algorithm.  The arguments v1
-   and w1 should satisfy 2 <= ABS(Py_SIZE(w1)) <= ABS(Py_SIZE(v1)). */
+   and w1 should satisfy 2 <= Py_ABS(Py_SIZE(w1)) <= Py_ABS(Py_SIZE(v1)). */
 
 static PyLongObject *
 x_divrem(PyLongObject *v1, PyLongObject *w1, PyLongObject **prem)
@@ -2459,8 +2458,8 @@
        that won't overflow a digit. */
 
     /* allocate space; w will also be used to hold the final remainder */
-    size_v = ABS(Py_SIZE(v1));
-    size_w = ABS(Py_SIZE(w1));
+    size_v = Py_ABS(Py_SIZE(v1));
+    size_w = Py_ABS(Py_SIZE(w1));
     assert(size_v >= size_w && size_w >= 2); /* Assert checks by div() */
     v = _PyLong_New(size_v+1);
     if (v == NULL) {
@@ -2591,7 +2590,7 @@
        multiple of 4, rounding ties to a multiple of 8. */
     static const int half_even_correction[8] = {0, -1, -2, 1, 0, -1, 2, 1};
 
-    a_size = ABS(Py_SIZE(a));
+    a_size = Py_ABS(Py_SIZE(a));
     if (a_size == 0) {
         /* Special case for 0: significand 0.0, exponent 0. */
         *e = 0;
@@ -2732,7 +2731,7 @@
         sign = Py_SIZE(a) - Py_SIZE(b);
     }
     else {
-        Py_ssize_t i = ABS(Py_SIZE(a));
+        Py_ssize_t i = Py_ABS(Py_SIZE(a));
         while (--i >= 0 && a->ob_digit[i] == b->ob_digit[i])
             ;
         if (i < 0)
@@ -2850,7 +2849,7 @@
 static PyLongObject *
 x_add(PyLongObject *a, PyLongObject *b)
 {
-    Py_ssize_t size_a = ABS(Py_SIZE(a)), size_b = ABS(Py_SIZE(b));
+    Py_ssize_t size_a = Py_ABS(Py_SIZE(a)), size_b = Py_ABS(Py_SIZE(b));
     PyLongObject *z;
     Py_ssize_t i;
     digit carry = 0;
@@ -2884,7 +2883,7 @@
 static PyLongObject *
 x_sub(PyLongObject *a, PyLongObject *b)
 {
-    Py_ssize_t size_a = ABS(Py_SIZE(a)), size_b = ABS(Py_SIZE(b));
+    Py_ssize_t size_a = Py_ABS(Py_SIZE(a)), size_b = Py_ABS(Py_SIZE(b));
     PyLongObject *z;
     Py_ssize_t i;
     int sign = 1;
@@ -2944,7 +2943,7 @@
 
     CHECK_BINOP(a, b);
 
-    if (ABS(Py_SIZE(a)) <= 1 && ABS(Py_SIZE(b)) <= 1) {
+    if (Py_ABS(Py_SIZE(a)) <= 1 && Py_ABS(Py_SIZE(b)) <= 1) {
         PyObject *result = PyLong_FromLong(MEDIUM_VALUE(a) +
                                           MEDIUM_VALUE(b));
         return result;
@@ -2974,7 +2973,7 @@
 
     CHECK_BINOP(a, b);
 
-    if (ABS(Py_SIZE(a)) <= 1 && ABS(Py_SIZE(b)) <= 1) {
+    if (Py_ABS(Py_SIZE(a)) <= 1 && Py_ABS(Py_SIZE(b)) <= 1) {
         PyObject* r;
         r = PyLong_FromLong(MEDIUM_VALUE(a)-MEDIUM_VALUE(b));
         return r;
@@ -3003,8 +3002,8 @@
 x_mul(PyLongObject *a, PyLongObject *b)
 {
     PyLongObject *z;
-    Py_ssize_t size_a = ABS(Py_SIZE(a));
-    Py_ssize_t size_b = ABS(Py_SIZE(b));
+    Py_ssize_t size_a = Py_ABS(Py_SIZE(a));
+    Py_ssize_t size_b = Py_ABS(Py_SIZE(b));
     Py_ssize_t i;
 
     z = _PyLong_New(size_a + size_b);
@@ -3098,7 +3097,7 @@
 {
     PyLongObject *hi, *lo;
     Py_ssize_t size_lo, size_hi;
-    const Py_ssize_t size_n = ABS(Py_SIZE(n));
+    const Py_ssize_t size_n = Py_ABS(Py_SIZE(n));
 
     size_lo = Py_MIN(size_n, size);
     size_hi = size_n - size_lo;
@@ -3127,8 +3126,8 @@
 static PyLongObject *
 k_mul(PyLongObject *a, PyLongObject *b)
 {
-    Py_ssize_t asize = ABS(Py_SIZE(a));
-    Py_ssize_t bsize = ABS(Py_SIZE(b));
+    Py_ssize_t asize = Py_ABS(Py_SIZE(a));
+    Py_ssize_t bsize = Py_ABS(Py_SIZE(b));
     PyLongObject *ah = NULL;
     PyLongObject *al = NULL;
     PyLongObject *bh = NULL;
@@ -3348,8 +3347,8 @@
 static PyLongObject *
 k_lopsided_mul(PyLongObject *a, PyLongObject *b)
 {
-    const Py_ssize_t asize = ABS(Py_SIZE(a));
-    Py_ssize_t bsize = ABS(Py_SIZE(b));
+    const Py_ssize_t asize = Py_ABS(Py_SIZE(a));
+    Py_ssize_t bsize = Py_ABS(Py_SIZE(b));
     Py_ssize_t nbdone;          /* # of b digits already multiplied */
     PyLongObject *ret;
     PyLongObject *bslice = NULL;
@@ -3407,7 +3406,7 @@
     CHECK_BINOP(a, b);
 
     /* fast path for single-digit multiplication */
-    if (ABS(Py_SIZE(a)) <= 1 && ABS(Py_SIZE(b)) <= 1) {
+    if (Py_ABS(Py_SIZE(a)) <= 1 && Py_ABS(Py_SIZE(b)) <= 1) {
         stwodigits v = (stwodigits)(MEDIUM_VALUE(a)) * MEDIUM_VALUE(b);
 #ifdef HAVE_LONG_LONG
         return PyLong_FromLongLong((PY_LONG_LONG)v);
@@ -3614,8 +3613,8 @@
     */
 
     /* Reduce to case where a and b are both positive. */
-    a_size = ABS(Py_SIZE(a));
-    b_size = ABS(Py_SIZE(b));
+    a_size = Py_ABS(Py_SIZE(a));
+    b_size = Py_ABS(Py_SIZE(b));
     negate = (Py_SIZE(a) < 0) ^ (Py_SIZE(b) < 0);
     if (b_size == 0) {
         PyErr_SetString(PyExc_ZeroDivisionError,
@@ -3731,7 +3730,7 @@
             inexact = 1;
         Py_DECREF(rem);
     }
-    x_size = ABS(Py_SIZE(x));
+    x_size = Py_ABS(Py_SIZE(x));
     assert(x_size > 0); /* result of division is never zero */
     x_bits = (x_size-1)*PyLong_SHIFT+bits_in_digit(x->ob_digit[x_size-1]);
 
@@ -4003,7 +4002,7 @@
     /* Implement ~x as -(x+1) */
     PyLongObject *x;
     PyLongObject *w;
-    if (ABS(Py_SIZE(v)) <=1)
+    if (Py_ABS(Py_SIZE(v)) <=1)
         return PyLong_FromLong(-(MEDIUM_VALUE(v)+1));
     w = (PyLongObject *)PyLong_FromLong(1L);
     if (w == NULL)
@@ -4020,7 +4019,7 @@
 long_neg(PyLongObject *v)
 {
     PyLongObject *z;
-    if (ABS(Py_SIZE(v)) <= 1)
+    if (Py_ABS(Py_SIZE(v)) <= 1)
         return PyLong_FromLong(-MEDIUM_VALUE(v));
     z = (PyLongObject *)_PyLong_Copy(v);
     if (z != NULL)
@@ -4075,7 +4074,7 @@
             goto rshift_error;
         }
         wordshift = shiftby / PyLong_SHIFT;
-        newsize = ABS(Py_SIZE(a)) - wordshift;
+        newsize = Py_ABS(Py_SIZE(a)) - wordshift;
         if (newsize <= 0)
             return PyLong_FromLong(0);
         loshift = shiftby % PyLong_SHIFT;
@@ -4122,7 +4121,7 @@
     wordshift = shiftby / PyLong_SHIFT;
     remshift  = shiftby - wordshift * PyLong_SHIFT;
 
-    oldsize = ABS(Py_SIZE(a));
+    oldsize = Py_ABS(Py_SIZE(a));
     newsize = oldsize + wordshift;
     if (remshift)
         ++newsize;
@@ -4183,7 +4182,7 @@
        result back to sign-magnitude at the end. */
 
     /* If a is negative, replace it by its two's complement. */
-    size_a = ABS(Py_SIZE(a));
+    size_a = Py_ABS(Py_SIZE(a));
     nega = Py_SIZE(a) < 0;
     if (nega) {
         z = _PyLong_New(size_a);
@@ -4197,7 +4196,7 @@
         Py_INCREF(a);
 
     /* Same for b. */
-    size_b = ABS(Py_SIZE(b));
+    size_b = Py_ABS(Py_SIZE(b));
     negb = Py_SIZE(b) < 0;
     if (negb) {
         z = _PyLong_New(size_b);
@@ -4630,7 +4629,7 @@
 {
     Py_ssize_t res;
 
-    res = offsetof(PyLongObject, ob_digit) + ABS(Py_SIZE(v))*sizeof(digit);
+    res = offsetof(PyLongObject, ob_digit) + Py_ABS(Py_SIZE(v))*sizeof(digit);
     return PyLong_FromSsize_t(res);
 }
 
@@ -4644,7 +4643,7 @@
     assert(v != NULL);
     assert(PyLong_Check(v));
 
-    ndigits = ABS(Py_SIZE(v));
+    ndigits = Py_ABS(Py_SIZE(v));
     if (ndigits == 0)
         return PyLong_FromLong(0);
 
@@ -4849,7 +4848,7 @@
     if (type != &PyLong_Type && PyType_IsSubtype(type, &PyLong_Type)) {
         PyLongObject *newobj;
         int i;
-        Py_ssize_t n = ABS(Py_SIZE(long_obj));
+        Py_ssize_t n = Py_ABS(Py_SIZE(long_obj));
 
         newobj = (PyLongObject *)type->tp_alloc(type, n);
         if (newobj == NULL) {
diff --git a/Objects/memoryobject.c b/Objects/memoryobject.c
--- a/Objects/memoryobject.c
+++ b/Objects/memoryobject.c
@@ -48,9 +48,6 @@
 */
 
 
-#define XSTRINGIZE(v) #v
-#define STRINGIZE(v) XSTRINGIZE(v)
-
 #define CHECK_MBUF_RELEASED(mbuf) \
     if (((_PyManagedBufferObject *)mbuf)->flags&_Py_MANAGED_BUFFER_RELEASED) { \
         PyErr_SetString(PyExc_ValueError,                                      \
@@ -660,7 +657,7 @@
     if (src->ndim > PyBUF_MAX_NDIM) {
         PyErr_SetString(PyExc_ValueError,
             "memoryview: number of dimensions must not exceed "
-            STRINGIZE(PyBUF_MAX_NDIM));
+            Py_STRINGIFY(PyBUF_MAX_NDIM));
         return NULL;
     }
 
@@ -1341,7 +1338,7 @@
         if (ndim > PyBUF_MAX_NDIM) {
             PyErr_SetString(PyExc_ValueError,
                 "memoryview: number of dimensions must not exceed "
-                STRINGIZE(PyBUF_MAX_NDIM));
+                Py_STRINGIFY(PyBUF_MAX_NDIM));
             return NULL;
         }
         if (self->view.ndim != 1 && ndim != 1) {
diff --git a/Python/marshal.c b/Python/marshal.c
--- a/Python/marshal.c
+++ b/Python/marshal.c
@@ -13,8 +13,6 @@
 #include "code.h"
 #include "marshal.h"
 
-#define ABS(x) ((x) < 0 ? -(x) : (x))
-
 /* High water mark to determine when the marshalled object is dangerously deep
  * and risks coring the interpreter.  When the object stack gets this deep,
  * raise an exception instead of continuing.
@@ -192,7 +190,7 @@
     }
 
     /* set l to number of base PyLong_MARSHAL_BASE digits */
-    n = ABS(Py_SIZE(ob));
+    n = Py_ABS(Py_SIZE(ob));
     l = (n-1) * PyLong_MARSHAL_RATIO;
     d = ob->ob_digit[n-1];
     assert(d != 0); /* a PyLong is always normalized */
@@ -727,8 +725,8 @@
         return NULL;
     }
 
-    size = 1 + (ABS(n) - 1) / PyLong_MARSHAL_RATIO;
-    shorts_in_top_digit = 1 + (ABS(n) - 1) % PyLong_MARSHAL_RATIO;
+    size = 1 + (Py_ABS(n) - 1) / PyLong_MARSHAL_RATIO;
+    shorts_in_top_digit = 1 + (Py_ABS(n) - 1) % PyLong_MARSHAL_RATIO;
     ob = _PyLong_New(size);
     if (ob == NULL)
         return NULL;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 20:12:35 2014
From: python-checkins at python.org (serhiy.storchaka)
Date: Wed, 14 May 2014 20:12:35 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMDc1?=
 =?utf-8?q?=3A_fileinput=2EFileInput_now_reads_bytes_from_standard_stream_?=
 =?utf-8?q?if?=
Message-ID: <3gTP9v0VFZz7LjR@mail.python.org>

http://hg.python.org/cpython/rev/7e640fefc9c1
changeset:   90707:7e640fefc9c1
branch:      3.4
parent:      90704:ddd7db7cf036
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Wed May 14 21:08:33 2014 +0300
summary:
  Issue #21075: fileinput.FileInput now reads bytes from standard stream if
binary mode is specified.  Patch by Sam Kimbrel.

files:
  Lib/fileinput.py           |   5 ++++-
  Lib/test/test_fileinput.py |  10 +++++++++-
  Misc/ACKS                  |   1 +
  Misc/NEWS                  |   3 +++
  4 files changed, 17 insertions(+), 2 deletions(-)


diff --git a/Lib/fileinput.py b/Lib/fileinput.py
--- a/Lib/fileinput.py
+++ b/Lib/fileinput.py
@@ -320,7 +320,10 @@
             self._backupfilename = 0
             if self._filename == '-':
                 self._filename = '<stdin>'
-                self._file = sys.stdin
+                if 'b' in self._mode:
+                    self._file = sys.stdin.buffer
+                else:
+                    self._file = sys.stdin
                 self._isstdin = True
             else:
                 if self._inplace:
diff --git a/Lib/test/test_fileinput.py b/Lib/test/test_fileinput.py
--- a/Lib/test/test_fileinput.py
+++ b/Lib/test/test_fileinput.py
@@ -19,11 +19,12 @@
 except ImportError:
     gzip = None
 
-from io import StringIO
+from io import BytesIO, StringIO
 from fileinput import FileInput, hook_encoded
 
 from test.support import verbose, TESTFN, run_unittest, check_warnings
 from test.support import unlink as safe_unlink
+from unittest import mock
 
 
 # The fileinput module has 2 interfaces: the FileInput class which does
@@ -232,6 +233,13 @@
         finally:
             remove_tempfiles(t1)
 
+    def test_stdin_binary_mode(self):
+        with mock.patch('sys.stdin') as m_stdin:
+            m_stdin.buffer = BytesIO(b'spam, bacon, sausage, and spam')
+            fi = FileInput(files=['-'], mode='rb')
+            lines = list(fi)
+            self.assertEqual(lines, [b'spam, bacon, sausage, and spam'])
+
     def test_file_opening_hook(self):
         try:
             # cannot use openhook and inplace mode
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -674,6 +674,7 @@
 Jason Killen
 Jan Kim
 Taek Joo Kim
+Sam Kimbrel
 W. Trevor King
 Paul Kippes
 Steve Kirsch
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -23,6 +23,9 @@
 Library
 -------
 
+- Issue #21075: fileinput.FileInput now reads bytes from standard stream if
+  binary mode is specified.  Patch by Sam Kimbrel.
+
 - Issue #21396: Fix TextIOWrapper(..., write_through=True) to not force a
   flush() on the underlying binary stream.  Patch by akira.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 20:12:36 2014
From: python-checkins at python.org (serhiy.storchaka)
Date: Wed, 14 May 2014 20:12:36 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Issue_=2321075=3A_fileinput=2EFileInput_now_reads_bytes_?=
 =?utf-8?q?from_standard_stream_if?=
Message-ID: <3gTP9w2LgGz7LkD@mail.python.org>

http://hg.python.org/cpython/rev/4041d4077a85
changeset:   90708:4041d4077a85
parent:      90706:849efd365ab4
parent:      90707:7e640fefc9c1
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Wed May 14 21:11:08 2014 +0300
summary:
  Issue #21075: fileinput.FileInput now reads bytes from standard stream if
binary mode is specified.  Patch by Sam Kimbrel.

files:
  Lib/fileinput.py           |   5 ++++-
  Lib/test/test_fileinput.py |  10 +++++++++-
  Misc/ACKS                  |   1 +
  Misc/NEWS                  |   3 +++
  4 files changed, 17 insertions(+), 2 deletions(-)


diff --git a/Lib/fileinput.py b/Lib/fileinput.py
--- a/Lib/fileinput.py
+++ b/Lib/fileinput.py
@@ -320,7 +320,10 @@
             self._backupfilename = 0
             if self._filename == '-':
                 self._filename = '<stdin>'
-                self._file = sys.stdin
+                if 'b' in self._mode:
+                    self._file = sys.stdin.buffer
+                else:
+                    self._file = sys.stdin
                 self._isstdin = True
             else:
                 if self._inplace:
diff --git a/Lib/test/test_fileinput.py b/Lib/test/test_fileinput.py
--- a/Lib/test/test_fileinput.py
+++ b/Lib/test/test_fileinput.py
@@ -19,11 +19,12 @@
 except ImportError:
     gzip = None
 
-from io import StringIO
+from io import BytesIO, StringIO
 from fileinput import FileInput, hook_encoded
 
 from test.support import verbose, TESTFN, run_unittest, check_warnings
 from test.support import unlink as safe_unlink
+from unittest import mock
 
 
 # The fileinput module has 2 interfaces: the FileInput class which does
@@ -232,6 +233,13 @@
         finally:
             remove_tempfiles(t1)
 
+    def test_stdin_binary_mode(self):
+        with mock.patch('sys.stdin') as m_stdin:
+            m_stdin.buffer = BytesIO(b'spam, bacon, sausage, and spam')
+            fi = FileInput(files=['-'], mode='rb')
+            lines = list(fi)
+            self.assertEqual(lines, [b'spam, bacon, sausage, and spam'])
+
     def test_file_opening_hook(self):
         try:
             # cannot use openhook and inplace mode
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -676,6 +676,7 @@
 Jason Killen
 Jan Kim
 Taek Joo Kim
+Sam Kimbrel
 W. Trevor King
 Paul Kippes
 Steve Kirsch
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -84,6 +84,9 @@
 Library
 -------
 
+- Issue #21075: fileinput.FileInput now reads bytes from standard stream if
+  binary mode is specified.  Patch by Sam Kimbrel.
+
 - Issue #19775: Add a samefile() method to pathlib Path objects.  Initial
   patch by Vajrasky Kok.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 20:52:08 2014
From: python-checkins at python.org (serhiy.storchaka)
Date: Wed, 14 May 2014 20:52:08 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIwOTk4?=
 =?utf-8?q?=3A_Fixed_re=2Efullmatch=28=29_of_repeated_single_character_pat?=
 =?utf-8?q?tern?=
Message-ID: <3gTQ3X3R75z7LlW@mail.python.org>

http://hg.python.org/cpython/rev/6267428afbdb
changeset:   90709:6267428afbdb
branch:      3.4
parent:      90707:7e640fefc9c1
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Wed May 14 21:48:17 2014 +0300
summary:
  Issue #20998: Fixed re.fullmatch() of repeated single character pattern
with ignore case.  Original patch by Matthew Barnett.

files:
  Lib/test/test_re.py |   5 +++++
  Misc/NEWS           |   3 +++
  Modules/_sre.c      |  15 +++++++--------
  Modules/sre.h       |   1 -
  Modules/sre_lib.h   |  20 ++++++++++----------
  5 files changed, 25 insertions(+), 19 deletions(-)


diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py
--- a/Lib/test/test_re.py
+++ b/Lib/test/test_re.py
@@ -1223,6 +1223,11 @@
             pat.scanner(string='abracadabra', pos=3, endpos=10).search().span(),
             (7, 9))
 
+    def test_bug_20998(self):
+        # Issue #20998: Fullmatch of repeated single character pattern
+        # with ignore case.
+        self.assertEqual(re.fullmatch('[a-c]+', 'ABC', re.I).span(), (0, 3))
+
 
 class PatternReprTests(unittest.TestCase):
     def check(self, pattern, expected):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -23,6 +23,9 @@
 Library
 -------
 
+- Issue #20998: Fixed re.fullmatch() of repeated single character pattern
+  with ignore case.  Original patch by Matthew Barnett.
+
 - Issue #21075: fileinput.FileInput now reads bytes from standard stream if
   binary mode is specified.  Patch by Sam Kimbrel.
 
diff --git a/Modules/_sre.c b/Modules/_sre.c
--- a/Modules/_sre.c
+++ b/Modules/_sre.c
@@ -505,14 +505,14 @@
 }
 
 LOCAL(Py_ssize_t)
-sre_match(SRE_STATE* state, SRE_CODE* pattern)
+sre_match(SRE_STATE* state, SRE_CODE* pattern, int match_all)
 {
     if (state->charsize == 1)
-        return sre_ucs1_match(state, pattern);
+        return sre_ucs1_match(state, pattern, match_all);
     if (state->charsize == 2)
-        return sre_ucs2_match(state, pattern);
+        return sre_ucs2_match(state, pattern, match_all);
     assert(state->charsize == 4);
-    return sre_ucs4_match(state, pattern);
+    return sre_ucs4_match(state, pattern, match_all);
 }
 
 LOCAL(Py_ssize_t)
@@ -576,7 +576,7 @@
 
     TRACE(("|%p|%p|MATCH\n", PatternObject_GetCode(self), state.ptr));
 
-    status = sre_match(&state, PatternObject_GetCode(self));
+    status = sre_match(&state, PatternObject_GetCode(self), 0);
 
     TRACE(("|%p|%p|END\n", PatternObject_GetCode(self), state.ptr));
     if (PyErr_Occurred())
@@ -609,12 +609,11 @@
     if (!string)
         return NULL;
 
-    state.match_all = 1;
     state.ptr = state.start;
 
     TRACE(("|%p|%p|FULLMATCH\n", PatternObject_GetCode(self), state.ptr));
 
-    status = sre_match(&state, PatternObject_GetCode(self));
+    status = sre_match(&state, PatternObject_GetCode(self), 1);
 
     TRACE(("|%p|%p|END\n", PatternObject_GetCode(self), state.ptr));
     if (PyErr_Occurred())
@@ -2572,7 +2571,7 @@
 
     state->ptr = state->start;
 
-    status = sre_match(state, PatternObject_GetCode(self->pattern));
+    status = sre_match(state, PatternObject_GetCode(self->pattern), 0);
     if (PyErr_Occurred())
         return NULL;
 
diff --git a/Modules/sre.h b/Modules/sre.h
--- a/Modules/sre.h
+++ b/Modules/sre.h
@@ -86,7 +86,6 @@
     SRE_REPEAT *repeat;
     /* hooks */
     SRE_TOLOWER_HOOK lower;
-    int match_all;
 } SRE_STATE;
 
 typedef struct {
diff --git a/Modules/sre_lib.h b/Modules/sre_lib.h
--- a/Modules/sre_lib.h
+++ b/Modules/sre_lib.h
@@ -173,7 +173,7 @@
     }
 }
 
-LOCAL(Py_ssize_t) SRE(match)(SRE_STATE* state, SRE_CODE* pattern);
+LOCAL(Py_ssize_t) SRE(match)(SRE_STATE* state, SRE_CODE* pattern, int match_all);
 
 LOCAL(Py_ssize_t)
 SRE(count)(SRE_STATE* state, SRE_CODE* pattern, Py_ssize_t maxcount)
@@ -259,7 +259,7 @@
         /* repeated single character pattern */
         TRACE(("|%p|%p|COUNT SUBPATTERN\n", pattern, ptr));
         while ((SRE_CHAR*) state->ptr < end) {
-            i = SRE(match)(state, pattern);
+            i = SRE(match)(state, pattern, 0);
             if (i < 0)
                 return i;
             if (!i)
@@ -490,7 +490,7 @@
 /* check if string matches the given pattern.  returns <0 for
    error, 0 for failure, and 1 for success */
 LOCAL(Py_ssize_t)
-SRE(match)(SRE_STATE* state, SRE_CODE* pattern)
+SRE(match)(SRE_STATE* state, SRE_CODE* pattern, int match_all)
 {
     SRE_CHAR* end = (SRE_CHAR *)state->end;
     Py_ssize_t alloc_pos, ctx_pos = -1;
@@ -507,7 +507,7 @@
     ctx->last_ctx_pos = -1;
     ctx->jump = JUMP_NONE;
     ctx->pattern = pattern;
-    ctx->match_all = state->match_all;
+    ctx->match_all = match_all;
     ctx_pos = alloc_pos;
 
 entrance:
@@ -739,7 +739,7 @@
                 RETURN_FAILURE;
 
             if (ctx->pattern[ctx->pattern[0]] == SRE_OP_SUCCESS &&
-                (!ctx->match_all || ctx->ptr == state->end)) {
+                ctx->ptr == state->end) {
                 /* tail is empty.  we're finished */
                 state->ptr = ctx->ptr;
                 RETURN_SUCCESS;
@@ -824,7 +824,7 @@
             }
 
             if (ctx->pattern[ctx->pattern[0]] == SRE_OP_SUCCESS &&
-                (!ctx->match_all || ctx->ptr == state->end)) {
+                (!match_all || ctx->ptr == state->end)) {
                 /* tail is empty.  we're finished */
                 state->ptr = ctx->ptr;
                 RETURN_SUCCESS;
@@ -1269,7 +1269,7 @@
                     state->ptr = ptr - (prefix_len - prefix_skip - 1);
                     if (flags & SRE_INFO_LITERAL)
                         return 1; /* we got all of it */
-                    status = SRE(match)(state, pattern + 2*prefix_skip);
+                    status = SRE(match)(state, pattern + 2*prefix_skip, 0);
                     if (status != 0)
                         return status;
                     /* close but no cigar -- try again */
@@ -1302,7 +1302,7 @@
             state->ptr = ++ptr;
             if (flags & SRE_INFO_LITERAL)
                 return 1; /* we got all of it */
-            status = SRE(match)(state, pattern + 2);
+            status = SRE(match)(state, pattern + 2, 0);
             if (status != 0)
                 break;
         }
@@ -1317,7 +1317,7 @@
             TRACE(("|%p|%p|SEARCH CHARSET\n", pattern, ptr));
             state->start = ptr;
             state->ptr = ptr;
-            status = SRE(match)(state, pattern);
+            status = SRE(match)(state, pattern, 0);
             if (status != 0)
                 break;
             ptr++;
@@ -1327,7 +1327,7 @@
         while (ptr <= end) {
             TRACE(("|%p|%p|SEARCH\n", pattern, ptr));
             state->start = state->ptr = ptr++;
-            status = SRE(match)(state, pattern);
+            status = SRE(match)(state, pattern, 0);
             if (status != 0)
                 break;
         }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 20:52:09 2014
From: python-checkins at python.org (serhiy.storchaka)
Date: Wed, 14 May 2014 20:52:09 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Issue_=2320998=3A_Fixed_re=2Efullmatch=28=29_of_repeated?=
 =?utf-8?q?_single_character_pattern?=
Message-ID: <3gTQ3Y6QjVz7Lks@mail.python.org>

http://hg.python.org/cpython/rev/bcf64c1c92f6
changeset:   90710:bcf64c1c92f6
parent:      90708:4041d4077a85
parent:      90709:6267428afbdb
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Wed May 14 21:51:37 2014 +0300
summary:
  Issue #20998: Fixed re.fullmatch() of repeated single character pattern
with ignore case.  Original patch by Matthew Barnett.

files:
  Lib/test/test_re.py |   5 +++++
  Misc/NEWS           |   3 +++
  Modules/_sre.c      |  15 +++++++--------
  Modules/sre.h       |   1 -
  Modules/sre_lib.h   |  20 ++++++++++----------
  5 files changed, 25 insertions(+), 19 deletions(-)


diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py
--- a/Lib/test/test_re.py
+++ b/Lib/test/test_re.py
@@ -1223,6 +1223,11 @@
             pat.scanner(string='abracadabra', pos=3, endpos=10).search().span(),
             (7, 9))
 
+    def test_bug_20998(self):
+        # Issue #20998: Fullmatch of repeated single character pattern
+        # with ignore case.
+        self.assertEqual(re.fullmatch('[a-c]+', 'ABC', re.I).span(), (0, 3))
+
 
 class PatternReprTests(unittest.TestCase):
     def check(self, pattern, expected):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -84,6 +84,9 @@
 Library
 -------
 
+- Issue #20998: Fixed re.fullmatch() of repeated single character pattern
+  with ignore case.  Original patch by Matthew Barnett.
+
 - Issue #21075: fileinput.FileInput now reads bytes from standard stream if
   binary mode is specified.  Patch by Sam Kimbrel.
 
diff --git a/Modules/_sre.c b/Modules/_sre.c
--- a/Modules/_sre.c
+++ b/Modules/_sre.c
@@ -505,14 +505,14 @@
 }
 
 LOCAL(Py_ssize_t)
-sre_match(SRE_STATE* state, SRE_CODE* pattern)
+sre_match(SRE_STATE* state, SRE_CODE* pattern, int match_all)
 {
     if (state->charsize == 1)
-        return sre_ucs1_match(state, pattern);
+        return sre_ucs1_match(state, pattern, match_all);
     if (state->charsize == 2)
-        return sre_ucs2_match(state, pattern);
+        return sre_ucs2_match(state, pattern, match_all);
     assert(state->charsize == 4);
-    return sre_ucs4_match(state, pattern);
+    return sre_ucs4_match(state, pattern, match_all);
 }
 
 LOCAL(Py_ssize_t)
@@ -576,7 +576,7 @@
 
     TRACE(("|%p|%p|MATCH\n", PatternObject_GetCode(self), state.ptr));
 
-    status = sre_match(&state, PatternObject_GetCode(self));
+    status = sre_match(&state, PatternObject_GetCode(self), 0);
 
     TRACE(("|%p|%p|END\n", PatternObject_GetCode(self), state.ptr));
     if (PyErr_Occurred())
@@ -609,12 +609,11 @@
     if (!string)
         return NULL;
 
-    state.match_all = 1;
     state.ptr = state.start;
 
     TRACE(("|%p|%p|FULLMATCH\n", PatternObject_GetCode(self), state.ptr));
 
-    status = sre_match(&state, PatternObject_GetCode(self));
+    status = sre_match(&state, PatternObject_GetCode(self), 1);
 
     TRACE(("|%p|%p|END\n", PatternObject_GetCode(self), state.ptr));
     if (PyErr_Occurred())
@@ -2572,7 +2571,7 @@
 
     state->ptr = state->start;
 
-    status = sre_match(state, PatternObject_GetCode(self->pattern));
+    status = sre_match(state, PatternObject_GetCode(self->pattern), 0);
     if (PyErr_Occurred())
         return NULL;
 
diff --git a/Modules/sre.h b/Modules/sre.h
--- a/Modules/sre.h
+++ b/Modules/sre.h
@@ -86,7 +86,6 @@
     SRE_REPEAT *repeat;
     /* hooks */
     SRE_TOLOWER_HOOK lower;
-    int match_all;
 } SRE_STATE;
 
 typedef struct {
diff --git a/Modules/sre_lib.h b/Modules/sre_lib.h
--- a/Modules/sre_lib.h
+++ b/Modules/sre_lib.h
@@ -173,7 +173,7 @@
     }
 }
 
-LOCAL(Py_ssize_t) SRE(match)(SRE_STATE* state, SRE_CODE* pattern);
+LOCAL(Py_ssize_t) SRE(match)(SRE_STATE* state, SRE_CODE* pattern, int match_all);
 
 LOCAL(Py_ssize_t)
 SRE(count)(SRE_STATE* state, SRE_CODE* pattern, Py_ssize_t maxcount)
@@ -259,7 +259,7 @@
         /* repeated single character pattern */
         TRACE(("|%p|%p|COUNT SUBPATTERN\n", pattern, ptr));
         while ((SRE_CHAR*) state->ptr < end) {
-            i = SRE(match)(state, pattern);
+            i = SRE(match)(state, pattern, 0);
             if (i < 0)
                 return i;
             if (!i)
@@ -490,7 +490,7 @@
 /* check if string matches the given pattern.  returns <0 for
    error, 0 for failure, and 1 for success */
 LOCAL(Py_ssize_t)
-SRE(match)(SRE_STATE* state, SRE_CODE* pattern)
+SRE(match)(SRE_STATE* state, SRE_CODE* pattern, int match_all)
 {
     SRE_CHAR* end = (SRE_CHAR *)state->end;
     Py_ssize_t alloc_pos, ctx_pos = -1;
@@ -507,7 +507,7 @@
     ctx->last_ctx_pos = -1;
     ctx->jump = JUMP_NONE;
     ctx->pattern = pattern;
-    ctx->match_all = state->match_all;
+    ctx->match_all = match_all;
     ctx_pos = alloc_pos;
 
 entrance:
@@ -739,7 +739,7 @@
                 RETURN_FAILURE;
 
             if (ctx->pattern[ctx->pattern[0]] == SRE_OP_SUCCESS &&
-                (!ctx->match_all || ctx->ptr == state->end)) {
+                ctx->ptr == state->end) {
                 /* tail is empty.  we're finished */
                 state->ptr = ctx->ptr;
                 RETURN_SUCCESS;
@@ -824,7 +824,7 @@
             }
 
             if (ctx->pattern[ctx->pattern[0]] == SRE_OP_SUCCESS &&
-                (!ctx->match_all || ctx->ptr == state->end)) {
+                (!match_all || ctx->ptr == state->end)) {
                 /* tail is empty.  we're finished */
                 state->ptr = ctx->ptr;
                 RETURN_SUCCESS;
@@ -1269,7 +1269,7 @@
                     state->ptr = ptr - (prefix_len - prefix_skip - 1);
                     if (flags & SRE_INFO_LITERAL)
                         return 1; /* we got all of it */
-                    status = SRE(match)(state, pattern + 2*prefix_skip);
+                    status = SRE(match)(state, pattern + 2*prefix_skip, 0);
                     if (status != 0)
                         return status;
                     /* close but no cigar -- try again */
@@ -1302,7 +1302,7 @@
             state->ptr = ++ptr;
             if (flags & SRE_INFO_LITERAL)
                 return 1; /* we got all of it */
-            status = SRE(match)(state, pattern + 2);
+            status = SRE(match)(state, pattern + 2, 0);
             if (status != 0)
                 break;
         }
@@ -1317,7 +1317,7 @@
             TRACE(("|%p|%p|SEARCH CHARSET\n", pattern, ptr));
             state->start = ptr;
             state->ptr = ptr;
-            status = SRE(match)(state, pattern);
+            status = SRE(match)(state, pattern, 0);
             if (status != 0)
                 break;
             ptr++;
@@ -1327,7 +1327,7 @@
         while (ptr <= end) {
             TRACE(("|%p|%p|SEARCH\n", pattern, ptr));
             state->start = state->ptr = ptr++;
-            status = SRE(match)(state, pattern);
+            status = SRE(match)(state, pattern, 0);
             if (status != 0)
                 break;
         }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 23:37:36 2014
From: python-checkins at python.org (victor.stinner)
Date: Wed, 14 May 2014 23:37:36 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIxNDg4?=
 =?utf-8?q?=3A_Oops=2C_the_patch_for_codecs=2Eencode/decode_doc_was_writte?=
 =?utf-8?q?n_by?=
Message-ID: <3gTTkS1hF0z7LlJ@mail.python.org>

http://hg.python.org/cpython/rev/0a6f0aaeb96a
changeset:   90711:0a6f0aaeb96a
branch:      2.7
parent:      90697:cc5e3b93c35a
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Wed May 14 23:28:48 2014 +0200
summary:
  Issue #21488: Oops, the patch for codecs.encode/decode doc was written by
Berker Peksag (already present in Misc/ACKS). The issue was reported by Brad
Aylsworth.

files:
  Misc/ACKS |  1 -
  1 files changed, 0 insertions(+), 1 deletions(-)


diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -56,7 +56,6 @@
 Chris AtLee
 Aymeric Augustin
 John Aycock
-Brad Aylsworth
 Donovan Baarda
 Arne Babenhauserheide
 Attila Babo

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 23:37:37 2014
From: python-checkins at python.org (victor.stinner)
Date: Wed, 14 May 2014 23:37:37 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxNDg4?=
 =?utf-8?q?=3A_Oops=2C_the_patch_for_codecs=2Eencode/decode_doc_was_writte?=
 =?utf-8?q?n_by?=
Message-ID: <3gTTkT3T70z7Lkb@mail.python.org>

http://hg.python.org/cpython/rev/91dca6b9ef0f
changeset:   90712:91dca6b9ef0f
branch:      3.4
parent:      90709:6267428afbdb
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Wed May 14 23:29:38 2014 +0200
summary:
  Issue #21488: Oops, the patch for codecs.encode/decode doc was written by
Berker Peksag (already present in Misc/ACKS). The issue was reported by Brad
Aylsworth.

files:
  Misc/ACKS |  1 -
  1 files changed, 0 insertions(+), 1 deletions(-)


diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -57,7 +57,6 @@
 Chris AtLee
 Aymeric Augustin
 John Aycock
-Brad Aylsworth
 Donovan Baarda
 Arne Babenhauserheide
 Attila Babo

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed May 14 23:37:38 2014
From: python-checkins at python.org (victor.stinner)
Date: Wed, 14 May 2014 23:37:38 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?b?KTogTWVyZ2UgMy40?=
Message-ID: <3gTTkV5XDPz7LkL@mail.python.org>

http://hg.python.org/cpython/rev/785e29d8ce13
changeset:   90713:785e29d8ce13
parent:      90710:bcf64c1c92f6
parent:      90712:91dca6b9ef0f
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Wed May 14 23:37:14 2014 +0200
summary:
  Merge 3.4

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu May 15 13:37:55 2014
From: python-checkins at python.org (serhiy.storchaka)
Date: Thu, 15 May 2014 13:37:55 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2313916=3A_Disallow?=
 =?utf-8?q?ed_the_surrogatepass_error_handler_for_non_UTF-*?=
Message-ID: <3gTrN33L7Bz7Lkw@mail.python.org>

http://hg.python.org/cpython/rev/5e98a50e0f55
changeset:   90714:5e98a50e0f55
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Thu May 15 14:37:42 2014 +0300
summary:
  Issue #13916: Disallowed the surrogatepass error handler for non UTF-*
encodings.

files:
  Lib/test/test_codecs.py |  13 +++++++++++++
  Misc/NEWS               |   3 +++
  Python/codecs.c         |  23 +++++++++++++++++++----
  3 files changed, 35 insertions(+), 4 deletions(-)


diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py
--- a/Lib/test/test_codecs.py
+++ b/Lib/test/test_codecs.py
@@ -2807,6 +2807,9 @@
             ('[\u20ac]', 'replace', b'[?]'),
             ('[\xff]', 'backslashreplace', b'[\\xff]'),
             ('[\xff]', 'xmlcharrefreplace', b'[&#255;]'),
+            ('\udcff', 'strict', None),
+            ('[\udcff]', 'surrogateescape', b'[\xff]'),
+            ('[\udcff]', 'surrogatepass', None),
         ))
         self.check_decode(932, (
             (b'abc', 'strict', 'abc'),
@@ -2816,6 +2819,7 @@
             (b'[\xff]', 'ignore', '[]'),
             (b'[\xff]', 'replace', '[\ufffd]'),
             (b'[\xff]', 'surrogateescape', '[\udcff]'),
+            (b'[\xff]', 'surrogatepass', None),
             (b'\x81\x00abc', 'strict', None),
             (b'\x81\x00abc', 'ignore', '\x00abc'),
             (b'\x81\x00abc', 'replace', '\ufffd\x00abc'),
@@ -2826,14 +2830,23 @@
             ('abc', 'strict', b'abc'),
             ('\xe9\u20ac', 'strict',  b'\xe9\x80'),
             ('\xff', 'strict', b'\xff'),
+            # test error handlers
             ('\u0141', 'strict', None),
             ('\u0141', 'ignore', b''),
             ('\u0141', 'replace', b'L'),
+            ('\udc98', 'surrogateescape', b'\x98'),
+            ('\udc98', 'surrogatepass', None),
         ))
         self.check_decode(1252, (
             (b'abc', 'strict', 'abc'),
             (b'\xe9\x80', 'strict', '\xe9\u20ac'),
             (b'\xff', 'strict', '\xff'),
+            # invalid bytes
+            (b'[\x98]', 'strict', None),
+            (b'[\x98]', 'ignore', '[]'),
+            (b'[\x98]', 'replace', '[\ufffd]'),
+            (b'[\x98]', 'surrogateescape', '[\udc98]'),
+            (b'[\x98]', 'surrogatepass', None),
         ))
 
     def test_cp_utf7(self):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -84,6 +84,9 @@
 Library
 -------
 
+- Issue #13916: Disallowed the surrogatepass error handler for non UTF-*
+  encodings.
+
 - Issue #20998: Fixed re.fullmatch() of repeated single character pattern
   with ignore case.  Original patch by Matthew Barnett.
 
diff --git a/Python/codecs.c b/Python/codecs.c
--- a/Python/codecs.c
+++ b/Python/codecs.c
@@ -901,6 +901,7 @@
     }
 }
 
+#define ENC_UNKNOWN     -1
 #define ENC_UTF8        0
 #define ENC_UTF16BE     1
 #define ENC_UTF16LE     2
@@ -916,7 +917,11 @@
         encoding += 3;
         if (*encoding == '-' || *encoding == '_' )
             encoding++;
-        if (encoding[0] == '1' && encoding[1] == '6') {
+        if (encoding[0] == '8' && encoding[1] == '\0') {
+            *bytelength = 3;
+            return ENC_UTF8;
+        }
+        else if (encoding[0] == '1' && encoding[1] == '6') {
             encoding += 2;
             *bytelength = 2;
             if (*encoding == '\0') {
@@ -955,9 +960,7 @@
             }
         }
     }
-    /* utf-8 */
-    *bytelength = 3;
-    return ENC_UTF8;
+    return ENC_UNKNOWN;
 }
 
 /* This handler is declared static until someone demonstrates
@@ -994,6 +997,12 @@
         }
         code = get_standard_encoding(encoding, &bytelength);
         Py_DECREF(encode);
+        if (code == ENC_UNKNOWN) {
+            /* Not supported, fail with original exception */
+            PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
+            Py_DECREF(object);
+            return NULL;
+        }
 
         res = PyBytes_FromStringAndSize(NULL, bytelength*(end-start));
         if (!res) {
@@ -1068,6 +1077,12 @@
         }
         code = get_standard_encoding(encoding, &bytelength);
         Py_DECREF(encode);
+        if (code == ENC_UNKNOWN) {
+            /* Not supported, fail with original exception */
+            PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
+            Py_DECREF(object);
+            return NULL;
+        }
 
         /* Try decoding a single surrogate character. If
            there are more, let the codec call us again. */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu May 15 20:19:51 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 15 May 2014 20:19:51 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321486=3A_Optimize?=
 =?utf-8?q?_parsing_of_netmasks_in_ipaddress=2EIPv4Network_and?=
Message-ID: <3gV1Hq547Vz7Lkp@mail.python.org>

http://hg.python.org/cpython/rev/2158614e1607
changeset:   90715:2158614e1607
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu May 15 20:18:41 2014 +0200
summary:
  Issue #21486: Optimize parsing of netmasks in ipaddress.IPv4Network and ipaddress.IPv6Network.

files:
  Lib/ipaddress.py |  206 ++++++++++++++++++++--------------
  Misc/NEWS        |    3 +
  2 files changed, 126 insertions(+), 83 deletions(-)


diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py
--- a/Lib/ipaddress.py
+++ b/Lib/ipaddress.py
@@ -467,7 +467,8 @@
             raise AddressValueError(msg % (address, address_len,
                                            expected_len, self._version))
 
-    def _ip_int_from_prefix(self, prefixlen):
+    @classmethod
+    def _ip_int_from_prefix(cls, prefixlen):
         """Turn the prefix length into a bitwise netmask
 
         Args:
@@ -477,9 +478,10 @@
             An integer.
 
         """
-        return self._ALL_ONES ^ (self._ALL_ONES >> prefixlen)
-
-    def _prefix_from_ip_int(self, ip_int):
+        return cls._ALL_ONES ^ (cls._ALL_ONES >> prefixlen)
+
+    @classmethod
+    def _prefix_from_ip_int(cls, ip_int):
         """Return prefix length from the bitwise netmask.
 
         Args:
@@ -492,22 +494,24 @@
             ValueError: If the input intermingles zeroes & ones
         """
         trailing_zeroes = _count_righthand_zero_bits(ip_int,
-                                                     self._max_prefixlen)
-        prefixlen = self._max_prefixlen - trailing_zeroes
+                                                     cls._max_prefixlen)
+        prefixlen = cls._max_prefixlen - trailing_zeroes
         leading_ones = ip_int >> trailing_zeroes
         all_ones = (1 << prefixlen) - 1
         if leading_ones != all_ones:
-            byteslen = self._max_prefixlen // 8
+            byteslen = cls._max_prefixlen // 8
             details = ip_int.to_bytes(byteslen, 'big')
             msg = 'Netmask pattern %r mixes zeroes & ones'
             raise ValueError(msg % details)
         return prefixlen
 
-    def _report_invalid_netmask(self, netmask_str):
+    @classmethod
+    def _report_invalid_netmask(cls, netmask_str):
         msg = '%r is not a valid netmask' % netmask_str
         raise NetmaskValueError(msg) from None
 
-    def _prefix_from_prefix_string(self, prefixlen_str):
+    @classmethod
+    def _prefix_from_prefix_string(cls, prefixlen_str):
         """Return prefix length from a numeric string
 
         Args:
@@ -522,16 +526,17 @@
         # int allows a leading +/- as well as surrounding whitespace,
         # so we ensure that isn't the case
         if not _BaseV4._DECIMAL_DIGITS.issuperset(prefixlen_str):
-            self._report_invalid_netmask(prefixlen_str)
+            cls._report_invalid_netmask(prefixlen_str)
         try:
             prefixlen = int(prefixlen_str)
         except ValueError:
-            self._report_invalid_netmask(prefixlen_str)
-        if not (0 <= prefixlen <= self._max_prefixlen):
-            self._report_invalid_netmask(prefixlen_str)
+            cls._report_invalid_netmask(prefixlen_str)
+        if not (0 <= prefixlen <= cls._max_prefixlen):
+            cls._report_invalid_netmask(prefixlen_str)
         return prefixlen
 
-    def _prefix_from_ip_string(self, ip_str):
+    @classmethod
+    def _prefix_from_ip_string(cls, ip_str):
         """Turn a netmask/hostmask string into a prefix length
 
         Args:
@@ -545,24 +550,24 @@
         """
         # Parse the netmask/hostmask like an IP address.
         try:
-            ip_int = self._ip_int_from_string(ip_str)
+            ip_int = cls._ip_int_from_string(ip_str)
         except AddressValueError:
-            self._report_invalid_netmask(ip_str)
+            cls._report_invalid_netmask(ip_str)
 
         # Try matching a netmask (this would be /1*0*/ as a bitwise regexp).
         # Note that the two ambiguous cases (all-ones and all-zeroes) are
         # treated as netmasks.
         try:
-            return self._prefix_from_ip_int(ip_int)
+            return cls._prefix_from_ip_int(ip_int)
         except ValueError:
             pass
 
         # Invert the bits, and try matching a /0+1+/ hostmask instead.
-        ip_int ^= self._ALL_ONES
+        ip_int ^= cls._ALL_ONES
         try:
-            return self._prefix_from_ip_int(ip_int)
+            return cls._prefix_from_ip_int(ip_int)
         except ValueError:
-            self._report_invalid_netmask(ip_str)
+            cls._report_invalid_netmask(ip_str)
 
 
 class _BaseAddress(_IPAddressBase):
@@ -1100,14 +1105,43 @@
     # the valid octets for host and netmasks. only useful for IPv4.
     _valid_mask_octets = frozenset((255, 254, 252, 248, 240, 224, 192, 128, 0))
 
+    _max_prefixlen = IPV4LENGTH
+    # There are only a handful of valid v4 netmasks, so we cache them all
+    # when constructed (see _make_netmask()).
+    _netmask_cache = {}
+
     def __init__(self, address):
         self._version = 4
-        self._max_prefixlen = IPV4LENGTH
 
     def _explode_shorthand_ip_string(self):
         return str(self)
 
-    def _ip_int_from_string(self, ip_str):
+    @classmethod
+    def _make_netmask(cls, arg):
+        """Make a (netmask, prefix_len) tuple from the given argument.
+
+        Argument can be:
+        - an integer (the prefix length)
+        - a string representing the prefix length (e.g. "24")
+        - a string representing the prefix netmask (e.g. "255.255.255.0")
+        """
+        if arg not in cls._netmask_cache:
+            if isinstance(arg, int):
+                prefixlen = arg
+            else:
+                try:
+                    # Check for a netmask in prefix length form
+                    prefixlen = cls._prefix_from_prefix_string(arg)
+                except NetmaskValueError:
+                    # Check for a netmask or hostmask in dotted-quad form.
+                    # This may raise NetmaskValueError.
+                    prefixlen = cls._prefix_from_ip_string(arg)
+            netmask = IPv4Address(cls._ip_int_from_prefix(prefixlen))
+            cls._netmask_cache[arg] = netmask, prefixlen
+        return cls._netmask_cache[arg]
+
+    @classmethod
+    def _ip_int_from_string(cls, ip_str):
         """Turn the given IP string into an integer for comparison.
 
         Args:
@@ -1128,11 +1162,12 @@
             raise AddressValueError("Expected 4 octets in %r" % ip_str)
 
         try:
-            return int.from_bytes(map(self._parse_octet, octets), 'big')
+            return int.from_bytes(map(cls._parse_octet, octets), 'big')
         except ValueError as exc:
             raise AddressValueError("%s in %r" % (exc, ip_str)) from None
 
-    def _parse_octet(self, octet_str):
+    @classmethod
+    def _parse_octet(cls, octet_str):
         """Convert a decimal octet into an integer.
 
         Args:
@@ -1148,7 +1183,7 @@
         if not octet_str:
             raise ValueError("Empty octet not permitted")
         # Whitelist the characters, since int() allows a lot of bizarre stuff.
-        if not self._DECIMAL_DIGITS.issuperset(octet_str):
+        if not cls._DECIMAL_DIGITS.issuperset(octet_str):
             msg = "Only decimal digits permitted in %r"
             raise ValueError(msg % octet_str)
         # We do the length check second, since the invalid character error
@@ -1168,7 +1203,8 @@
             raise ValueError("Octet %d (> 255) not permitted" % octet_int)
         return octet_int
 
-    def _string_from_ip_int(self, ip_int):
+    @classmethod
+    def _string_from_ip_int(cls, ip_int):
         """Turns a 32-bit integer into dotted decimal notation.
 
         Args:
@@ -1519,29 +1555,18 @@
         # Constructing from a packed address or integer
         if isinstance(address, (int, bytes)):
             self.network_address = IPv4Address(address)
-            self._prefixlen = self._max_prefixlen
-            self.netmask = IPv4Address(self._ALL_ONES)
-            #fixme: address/network test here
+            self.netmask, self._prefixlen = self._make_netmask(self._max_prefixlen)
+            #fixme: address/network test here.
             return
 
         if isinstance(address, tuple):
             if len(address) > 1:
-                # If address[1] is a string, treat it like a netmask.
-                if isinstance(address[1], str):
-                    self.netmask = IPv4Address(address[1])
-                    self._prefixlen = self._prefix_from_ip_int(
-                        int(self.netmask))
-                # address[1] should be an int.
-                else:
-                    self._prefixlen = int(address[1])
-                    self.netmask = IPv4Address(self._ip_int_from_prefix(
-                            self._prefixlen))
-            # We weren't given an address[1].
+                arg = address[1]
             else:
-                self._prefixlen = self._max_prefixlen
-                self.netmask = IPv4Address(self._ip_int_from_prefix(
-                        self._prefixlen))
+                # We weren't given an address[1]
+                arg = self._max_prefixlen
             self.network_address = IPv4Address(address[0])
+            self.netmask, self._prefixlen = self._make_netmask(arg)
             packed = int(self.network_address)
             if packed & int(self.netmask) != packed:
                 if strict:
@@ -1551,23 +1576,16 @@
                                                        int(self.netmask))
             return
 
-
         # Assume input argument to be string or any object representation
         # which converts into a formatted IP prefix string.
         addr = _split_optional_netmask(address)
         self.network_address = IPv4Address(self._ip_int_from_string(addr[0]))
 
         if len(addr) == 2:
-            try:
-                # Check for a netmask in prefix length form
-                self._prefixlen = self._prefix_from_prefix_string(addr[1])
-            except NetmaskValueError:
-                # Check for a netmask or hostmask in dotted-quad form.
-                # This may raise NetmaskValueError.
-                self._prefixlen = self._prefix_from_ip_string(addr[1])
+            arg = addr[1]
         else:
-            self._prefixlen = self._max_prefixlen
-        self.netmask = IPv4Address(self._ip_int_from_prefix(self._prefixlen))
+            arg = self._max_prefixlen
+        self.netmask, self._prefixlen = self._make_netmask(arg)
 
         if strict:
             if (IPv4Address(int(self.network_address) & int(self.netmask)) !=
@@ -1607,12 +1625,35 @@
     _ALL_ONES = (2**IPV6LENGTH) - 1
     _HEXTET_COUNT = 8
     _HEX_DIGITS = frozenset('0123456789ABCDEFabcdef')
+    _max_prefixlen = IPV6LENGTH
+
+    # There are only a bunch of valid v6 netmasks, so we cache them all
+    # when constructed (see _make_netmask()).
+    _netmask_cache = {}
 
     def __init__(self, address):
         self._version = 6
-        self._max_prefixlen = IPV6LENGTH
-
-    def _ip_int_from_string(self, ip_str):
+
+    @classmethod
+    def _make_netmask(cls, arg):
+        """Make a (netmask, prefix_len) tuple from the given argument.
+
+        Argument can be:
+        - an integer (the prefix length)
+        - a string representing the prefix length (e.g. "24")
+        - a string representing the prefix netmask (e.g. "255.255.255.0")
+        """
+        if arg not in cls._netmask_cache:
+            if isinstance(arg, int):
+                prefixlen = arg
+            else:
+                prefixlen = cls._prefix_from_prefix_string(arg)
+            netmask = IPv6Address(cls._ip_int_from_prefix(prefixlen))
+            cls._netmask_cache[arg] = netmask, prefixlen
+        return cls._netmask_cache[arg]
+
+    @classmethod
+    def _ip_int_from_string(cls, ip_str):
         """Turn an IPv6 ip_str into an integer.
 
         Args:
@@ -1648,7 +1689,7 @@
         # An IPv6 address can't have more than 8 colons (9 parts).
         # The extra colon comes from using the "::" notation for a single
         # leading or trailing zero part.
-        _max_parts = self._HEXTET_COUNT + 1
+        _max_parts = cls._HEXTET_COUNT + 1
         if len(parts) > _max_parts:
             msg = "At most %d colons permitted in %r" % (_max_parts-1, ip_str)
             raise AddressValueError(msg)
@@ -1680,17 +1721,17 @@
                 if parts_lo:
                     msg = "Trailing ':' only permitted as part of '::' in %r"
                     raise AddressValueError(msg % ip_str)  # :$ requires ::$
-            parts_skipped = self._HEXTET_COUNT - (parts_hi + parts_lo)
+            parts_skipped = cls._HEXTET_COUNT - (parts_hi + parts_lo)
             if parts_skipped < 1:
                 msg = "Expected at most %d other parts with '::' in %r"
-                raise AddressValueError(msg % (self._HEXTET_COUNT-1, ip_str))
+                raise AddressValueError(msg % (cls._HEXTET_COUNT-1, ip_str))
         else:
             # Otherwise, allocate the entire address to parts_hi.  The
             # endpoints could still be empty, but _parse_hextet() will check
             # for that.
-            if len(parts) != self._HEXTET_COUNT:
+            if len(parts) != cls._HEXTET_COUNT:
                 msg = "Exactly %d parts expected without '::' in %r"
-                raise AddressValueError(msg % (self._HEXTET_COUNT, ip_str))
+                raise AddressValueError(msg % (cls._HEXTET_COUNT, ip_str))
             if not parts[0]:
                 msg = "Leading ':' only permitted as part of '::' in %r"
                 raise AddressValueError(msg % ip_str)  # ^: requires ^::
@@ -1706,16 +1747,17 @@
             ip_int = 0
             for i in range(parts_hi):
                 ip_int <<= 16
-                ip_int |= self._parse_hextet(parts[i])
+                ip_int |= cls._parse_hextet(parts[i])
             ip_int <<= 16 * parts_skipped
             for i in range(-parts_lo, 0):
                 ip_int <<= 16
-                ip_int |= self._parse_hextet(parts[i])
+                ip_int |= cls._parse_hextet(parts[i])
             return ip_int
         except ValueError as exc:
             raise AddressValueError("%s in %r" % (exc, ip_str)) from None
 
-    def _parse_hextet(self, hextet_str):
+    @classmethod
+    def _parse_hextet(cls, hextet_str):
         """Convert an IPv6 hextet string into an integer.
 
         Args:
@@ -1730,7 +1772,7 @@
 
         """
         # Whitelist the characters, since int() allows a lot of bizarre stuff.
-        if not self._HEX_DIGITS.issuperset(hextet_str):
+        if not cls._HEX_DIGITS.issuperset(hextet_str):
             raise ValueError("Only hex digits permitted in %r" % hextet_str)
         # We do the length check second, since the invalid character error
         # is likely to be more informative for the user
@@ -1740,7 +1782,8 @@
         # Length check means we can skip checking the integer value
         return int(hextet_str, 16)
 
-    def _compress_hextets(self, hextets):
+    @classmethod
+    def _compress_hextets(cls, hextets):
         """Compresses a list of hextets.
 
         Compresses a list of strings, replacing the longest continuous
@@ -1787,7 +1830,8 @@
 
         return hextets
 
-    def _string_from_ip_int(self, ip_int=None):
+    @classmethod
+    def _string_from_ip_int(cls, ip_int=None):
         """Turns a 128-bit integer into hexadecimal notation.
 
         Args:
@@ -1801,15 +1845,15 @@
 
         """
         if ip_int is None:
-            ip_int = int(self._ip)
-
-        if ip_int > self._ALL_ONES:
+            ip_int = int(cls._ip)
+
+        if ip_int > cls._ALL_ONES:
             raise ValueError('IPv6 address is too large')
 
         hex_str = '%032x' % ip_int
         hextets = ['%x' % int(hex_str[x:x+4], 16) for x in range(0, 32, 4)]
 
-        hextets = self._compress_hextets(hextets)
+        hextets = cls._compress_hextets(hextets)
         return ':'.join(hextets)
 
     def _explode_shorthand_ip_string(self):
@@ -2192,18 +2236,15 @@
         # Efficient constructor from integer or packed address
         if isinstance(address, (bytes, int)):
             self.network_address = IPv6Address(address)
-            self._prefixlen = self._max_prefixlen
-            self.netmask = IPv6Address(self._ALL_ONES)
+            self.netmask, self._prefixlen = self._make_netmask(self._max_prefixlen)
             return
 
         if isinstance(address, tuple):
-            self.network_address = IPv6Address(address[0])
             if len(address) > 1:
-                self._prefixlen = int(address[1])
+                arg = address[1]
             else:
-                self._prefixlen = self._max_prefixlen
-            self.netmask = IPv6Address(self._ip_int_from_prefix(
-                    self._prefixlen))
+                arg = self._max_prefixlen
+            self.netmask, self._prefixlen = self._make_netmask(arg)
             self.network_address = IPv6Address(address[0])
             packed = int(self.network_address)
             if packed & int(self.netmask) != packed:
@@ -2221,12 +2262,11 @@
         self.network_address = IPv6Address(self._ip_int_from_string(addr[0]))
 
         if len(addr) == 2:
-            # This may raise NetmaskValueError
-            self._prefixlen = self._prefix_from_prefix_string(addr[1])
+            arg = addr[1]
         else:
-            self._prefixlen = self._max_prefixlen
-
-        self.netmask = IPv6Address(self._ip_int_from_prefix(self._prefixlen))
+            arg = self._max_prefixlen
+        self.netmask, self._prefixlen = self._make_netmask(arg)
+
         if strict:
             if (IPv6Address(int(self.network_address) & int(self.netmask)) !=
                 self.network_address):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -84,6 +84,9 @@
 Library
 -------
 
+- Issue #21486: Optimize parsing of netmasks in ipaddress.IPv4Network and
+  ipaddress.IPv6Network.
+
 - Issue #13916: Disallowed the surrogatepass error handler for non UTF-*
   encodings.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu May 15 20:21:58 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 15 May 2014 20:21:58 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321487=3A_Optimize?=
 =?utf-8?q?_ipaddress=2Esummarize=5Faddress=5Frange=28=29_and?=
Message-ID: <3gV1LG3yRzz7LjM@mail.python.org>

http://hg.python.org/cpython/rev/2711677cf874
changeset:   90716:2711677cf874
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu May 15 20:21:48 2014 +0200
summary:
  Issue #21487: Optimize ipaddress.summarize_address_range() and ipaddress.{IPv4Network,IPv6Network}.subnets().

files:
  Lib/ipaddress.py |  28 +++++++---------------------
  Misc/NEWS        |   3 +++
  2 files changed, 10 insertions(+), 21 deletions(-)


diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py
--- a/Lib/ipaddress.py
+++ b/Lib/ipaddress.py
@@ -195,11 +195,7 @@
     """
     if number == 0:
         return bits
-    for i in range(bits):
-        if (number >> i) & 1:
-            return i
-    # All bits of interest were zero, even if there are more in the number
-    return bits
+    return min(bits, (~number & (number-1)).bit_length())
 
 
 def summarize_address_range(first, last):
@@ -250,12 +246,11 @@
     while first_int <= last_int:
         nbits = min(_count_righthand_zero_bits(first_int, ip_bits),
                     (last_int - first_int + 1).bit_length() - 1)
-        net = ip('%s/%d' % (first, ip_bits - nbits))
+        net = ip((first_int, ip_bits - nbits))
         yield net
         first_int += 1 << nbits
         if first_int - 1 == ip._ALL_ONES:
             break
-        first = first.__class__(first_int)
 
 
 def _collapse_addresses_recursive(addresses):
@@ -949,20 +944,11 @@
                 'prefix length diff %d is invalid for netblock %s' % (
                     new_prefixlen, self))
 
-        first = self.__class__('%s/%s' %
-                                 (self.network_address,
-                                  self._prefixlen + prefixlen_diff))
-
-        yield first
-        current = first
-        while True:
-            broadcast = current.broadcast_address
-            if broadcast == self.broadcast_address:
-                return
-            new_addr = self._address_class(int(broadcast) + 1)
-            current = self.__class__('%s/%s' % (new_addr,
-                                                new_prefixlen))
-
+        start = int(self.network_address)
+        end = int(self.broadcast_address)
+        step = (int(self.hostmask) + 1) >> prefixlen_diff
+        for new_addr in range(start, end, step):
+            current = self.__class__((new_addr, new_prefixlen))
             yield current
 
     def supernet(self, prefixlen_diff=1, new_prefix=None):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -84,6 +84,9 @@
 Library
 -------
 
+- Issue #21487: Optimize ipaddress.summarize_address_range() and
+  ipaddress.{IPv4Network,IPv6Network}.subnets().
+
 - Issue #21486: Optimize parsing of netmasks in ipaddress.IPv4Network and
   ipaddress.IPv6Network.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu May 15 20:41:00 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 15 May 2014 20:41:00 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320826=3A_Optimize?=
 =?utf-8?q?_ipaddress=2Ecollapse=5Faddresses=28=29=2E?=
Message-ID: <3gV1mD1Y7Dz7LjM@mail.python.org>

http://hg.python.org/cpython/rev/8867874a2b7d
changeset:   90717:8867874a2b7d
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu May 15 20:40:53 2014 +0200
summary:
  Issue #20826: Optimize ipaddress.collapse_addresses().

files:
  Lib/ipaddress.py |  53 +++++++++++++++++------------------
  Misc/NEWS        |   2 +
  2 files changed, 28 insertions(+), 27 deletions(-)


diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py
--- a/Lib/ipaddress.py
+++ b/Lib/ipaddress.py
@@ -253,7 +253,7 @@
             break
 
 
-def _collapse_addresses_recursive(addresses):
+def _collapse_addresses_internal(addresses):
     """Loops through the addresses, collapsing concurrent netblocks.
 
     Example:
@@ -263,7 +263,7 @@
         ip3 = IPv4Network('192.0.2.128/26')
         ip4 = IPv4Network('192.0.2.192/26')
 
-        _collapse_addresses_recursive([ip1, ip2, ip3, ip4]) ->
+        _collapse_addresses_internal([ip1, ip2, ip3, ip4]) ->
           [IPv4Network('192.0.2.0/24')]
 
         This shouldn't be called directly; it is called via
@@ -277,28 +277,29 @@
         passed.
 
     """
-    while True:
-        last_addr = None
-        ret_array = []
-        optimized = False
-
-        for cur_addr in addresses:
-            if not ret_array:
-                last_addr = cur_addr
-                ret_array.append(cur_addr)
-            elif (cur_addr.network_address >= last_addr.network_address and
-                cur_addr.broadcast_address <= last_addr.broadcast_address):
-                optimized = True
-            elif cur_addr == list(last_addr.supernet().subnets())[1]:
-                ret_array[-1] = last_addr = last_addr.supernet()
-                optimized = True
-            else:
-                last_addr = cur_addr
-                ret_array.append(cur_addr)
-
-        addresses = ret_array
-        if not optimized:
-            return addresses
+    # First merge
+    to_merge = list(addresses)
+    subnets = {}
+    while to_merge:
+        net = to_merge.pop()
+        supernet = net.supernet()
+        existing = subnets.get(supernet)
+        if existing is None:
+            subnets[supernet] = net
+        elif existing != net:
+            # Merge consecutive subnets
+            del subnets[supernet]
+            to_merge.append(supernet)
+    # Then iterate over resulting networks, skipping subsumed subnets
+    last = None
+    for net in sorted(subnets.values()):
+        if last is not None:
+            # Since they are sorted, last.network_address <= net.network_address
+            # is a given.
+            if last.broadcast_address >= net.broadcast_address:
+                continue
+        yield net
+        last = net
 
 
 def collapse_addresses(addresses):
@@ -347,15 +348,13 @@
 
     # sort and dedup
     ips = sorted(set(ips))
-    nets = sorted(set(nets))
 
     while i < len(ips):
         (first, last) = _find_address_range(ips[i:])
         i = ips.index(last) + 1
         addrs.extend(summarize_address_range(first, last))
 
-    return iter(_collapse_addresses_recursive(sorted(
-        addrs + nets, key=_BaseNetwork._get_networks_key)))
+    return _collapse_addresses_internal(addrs + nets)
 
 
 def get_mixed_type_key(obj):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -84,6 +84,8 @@
 Library
 -------
 
+- Issue #20826: Optimize ipaddress.collapse_addresses().
+
 - Issue #21487: Optimize ipaddress.summarize_address_range() and
   ipaddress.{IPv4Network,IPv6Network}.subnets().
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu May 15 22:39:48 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 15 May 2014 22:39:48 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIxMzY0?=
 =?utf-8?q?=3A_remove_recommendation_of_broken_pattern=2E?=
Message-ID: <3gV4PJ1p47z7LjR@mail.python.org>

http://hg.python.org/cpython/rev/4621bb82ceec
changeset:   90718:4621bb82ceec
branch:      3.4
parent:      90712:91dca6b9ef0f
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu May 15 22:38:56 2014 +0200
summary:
  Issue #21364: remove recommendation of broken pattern.

files:
  Doc/library/sys.rst |  27 ++++++++++++---------------
  1 files changed, 12 insertions(+), 15 deletions(-)


diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -1066,8 +1066,9 @@
      statements and for the prompts of :func:`input`;
    * The interpreter's own prompts and its error messages go to ``stderr``.
 
-   By default, these streams are regular text streams as returned by the
-   :func:`open` function.  Their parameters are chosen as follows:
+   These streams are regular :term:`text files <text file>` like those
+   returned by the :func:`open` function.  Their parameters are chosen as
+   follows:
 
    * The character encoding is platform-dependent.  Under Windows, if the stream
      is interactive (that is, if its :meth:`isatty` method returns ``True``), the
@@ -1075,26 +1076,22 @@
      platforms, the locale encoding is used (see :meth:`locale.getpreferredencoding`).
 
      Under all platforms though, you can override this value by setting the
-     :envvar:`PYTHONIOENCODING` environment variable.
+     :envvar:`PYTHONIOENCODING` environment variable before starting Python.
 
    * When interactive, standard streams are line-buffered.  Otherwise, they
      are block-buffered like regular text files.  You can override this
      value with the :option:`-u` command-line option.
 
-   To write or read binary data from/to the standard streams, use the
-   underlying binary :data:`~io.TextIOBase.buffer`.  For example, to write
-   bytes to :data:`stdout`, use ``sys.stdout.buffer.write(b'abc')``.  Using
-   :meth:`io.TextIOBase.detach`, streams can be made binary by default.  This
-   function sets :data:`stdin` and :data:`stdout` to binary::
+   .. note::
 
-      def make_streams_binary():
-          sys.stdin = sys.stdin.detach()
-          sys.stdout = sys.stdout.detach()
+      To write or read binary data from/to the standard streams, use the
+      underlying binary :data:`~io.TextIOBase.buffer` object.  For example, to
+      write bytes to :data:`stdout`, use ``sys.stdout.buffer.write(b'abc')``.
 
-   Note that the streams may be replaced with objects (like :class:`io.StringIO`)
-   that do not support the :attr:`~io.BufferedIOBase.buffer` attribute or the
-   :meth:`~io.BufferedIOBase.detach` method and can raise :exc:`AttributeError`
-   or :exc:`io.UnsupportedOperation`.
+      However, if you are writing a library (and do not control in which
+      context its code will be executed), be aware that the standard streams
+      may be replaced with file-like objects like :class:`io.StringIO` which
+      do not support the :attr:`~io.BufferedIOBase.buffer` attribute.
 
 
 .. data:: __stdin__

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu May 15 22:39:49 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 15 May 2014 22:39:49 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Issue_=2321364=3A_remove_recommendation_of_broken_patter?=
 =?utf-8?q?n=2E?=
Message-ID: <3gV4PK3d2qz7Ljp@mail.python.org>

http://hg.python.org/cpython/rev/dbf728f9a2f0
changeset:   90719:dbf728f9a2f0
parent:      90717:8867874a2b7d
parent:      90718:4621bb82ceec
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu May 15 22:39:41 2014 +0200
summary:
  Issue #21364: remove recommendation of broken pattern.

files:
  Doc/library/sys.rst |  27 ++++++++++++---------------
  1 files changed, 12 insertions(+), 15 deletions(-)


diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -1066,8 +1066,9 @@
      statements and for the prompts of :func:`input`;
    * The interpreter's own prompts and its error messages go to ``stderr``.
 
-   By default, these streams are regular text streams as returned by the
-   :func:`open` function.  Their parameters are chosen as follows:
+   These streams are regular :term:`text files <text file>` like those
+   returned by the :func:`open` function.  Their parameters are chosen as
+   follows:
 
    * The character encoding is platform-dependent.  Under Windows, if the stream
      is interactive (that is, if its :meth:`isatty` method returns ``True``), the
@@ -1075,26 +1076,22 @@
      platforms, the locale encoding is used (see :meth:`locale.getpreferredencoding`).
 
      Under all platforms though, you can override this value by setting the
-     :envvar:`PYTHONIOENCODING` environment variable.
+     :envvar:`PYTHONIOENCODING` environment variable before starting Python.
 
    * When interactive, standard streams are line-buffered.  Otherwise, they
      are block-buffered like regular text files.  You can override this
      value with the :option:`-u` command-line option.
 
-   To write or read binary data from/to the standard streams, use the
-   underlying binary :data:`~io.TextIOBase.buffer`.  For example, to write
-   bytes to :data:`stdout`, use ``sys.stdout.buffer.write(b'abc')``.  Using
-   :meth:`io.TextIOBase.detach`, streams can be made binary by default.  This
-   function sets :data:`stdin` and :data:`stdout` to binary::
+   .. note::
 
-      def make_streams_binary():
-          sys.stdin = sys.stdin.detach()
-          sys.stdout = sys.stdout.detach()
+      To write or read binary data from/to the standard streams, use the
+      underlying binary :data:`~io.TextIOBase.buffer` object.  For example, to
+      write bytes to :data:`stdout`, use ``sys.stdout.buffer.write(b'abc')``.
 
-   Note that the streams may be replaced with objects (like :class:`io.StringIO`)
-   that do not support the :attr:`~io.BufferedIOBase.buffer` attribute or the
-   :meth:`~io.BufferedIOBase.detach` method and can raise :exc:`AttributeError`
-   or :exc:`io.UnsupportedOperation`.
+      However, if you are writing a library (and do not control in which
+      context its code will be executed), be aware that the standard streams
+      may be replaced with file-like objects like :class:`io.StringIO` which
+      do not support the :attr:`~io.BufferedIOBase.buffer` attribute.
 
 
 .. data:: __stdin__

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu May 15 22:47:38 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 15 May 2014 22:47:38 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Better_organization_of_the?=
	=?utf-8?q?_what=27s_new?=
Message-ID: <3gV4ZL6sS5z7LjP@mail.python.org>

http://hg.python.org/cpython/rev/d21af0d27322
changeset:   90720:d21af0d27322
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu May 15 22:47:33 2014 +0200
summary:
  Better organization of the what's new

files:
  Doc/whatsnew/3.5.rst |  49 ++++++++++++++++++++-----------
  1 files changed, 32 insertions(+), 17 deletions(-)


diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst
--- a/Doc/whatsnew/3.5.rst
+++ b/Doc/whatsnew/3.5.rst
@@ -134,13 +134,27 @@
 Improved Modules
 ================
 
-* Different constants of :mod:`signal` module are now enumeration values using
-  the :mod:`enum` module. This allows meaningful names to be printed during
-  debugging, instead of integer ?magic numbers?. (contribute by Giampaolo
-  Rodola' in :issue:`21076`)
+doctest
+-------
 
-* :class:`xmlrpc.client.ServerProxy` is now a :term:`context manager`
-  (contributed by Claudiu Popa in :issue:`20627`).
+* :func:`doctest.DocTestSuite` returns an empty :class:`unittest.TestSuite` if
+  *module* contains no docstrings instead of raising :exc:`ValueError`
+  (contributed by Glenn Jones in :issue:`15916`).
+
+importlib
+---------
+
+* :class:`importlib.util.LazyLoader` allows for the lazy loading of modules in
+  applications where startup time is paramount (contributed by Brett Cannon in
+  :issue:`17621`).
+
+* :func:`importlib.abc.InspectLoader.source_to_code` is now a
+  static method to make it easier to work with source code in a string.
+  With a module object that you want to initialize you can then use
+  ``exec(code, module.__dict__)`` to execute the code in the module.
+
+inspect
+-------
 
 * :class:`inspect.Signature` and :class:`inspect.Parameter` are now
   picklable and hashable (contributed by Yury Selivanov in :issue:`20726`
@@ -150,24 +164,25 @@
   subclassing of :class:`~inspect.Signature` easier (contributed
   by Yury Selivanov and Eric Snow in :issue:`17373`).
 
-* :class:`importlib.util.LazyLoader` allows for the lazy loading of modules in
-  applications where startup time is paramount (contributed by Brett Cannon in
-  :issue:`17621`).
+signal
+------
 
-* :func:`doctest.DocTestSuite` returns an empty :class:`unittest.TestSuite` if
-  *module* contains no docstrings instead of raising :exc:`ValueError`
-  (contributed by Glenn Jones in :issue:`15916`).
+* Different constants of :mod:`signal` module are now enumeration values using
+  the :mod:`enum` module. This allows meaningful names to be printed during
+  debugging, instead of integer ?magic numbers?. (contribute by Giampaolo
+  Rodola' in :issue:`21076`)
 
-* :func:`importlib.abc.InspectLoader.source_to_code` is now a
-  static method to make it easier to work with source code in a string.
-  With a module object that you want to initialize you can then use
-  ``exec(code, module.__dict__)`` to execute the code in the module.
+xmlrpc
+------
+
+* :class:`xmlrpc.client.ServerProxy` is now a :term:`context manager`
+  (contributed by Claudiu Popa in :issue:`20627`).
 
 
 Optimizations
 =============
 
-Major performance enhancements have been added:
+The following performance enhancements have been added:
 
 * Construction of ``bytes(int)`` and ``bytearray(int)`` (filled by zero bytes)
   is faster and use less memory (until the bytearray buffer is filled with

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu May 15 22:55:47 2014
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 15 May 2014 22:55:47 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Mention_ipaddress_improvem?=
	=?utf-8?q?ents=2E?=
Message-ID: <3gV4ll1sY8z7LjR@mail.python.org>

http://hg.python.org/cpython/rev/758323c8556a
changeset:   90721:758323c8556a
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu May 15 22:55:40 2014 +0200
summary:
  Mention ipaddress improvements.

files:
  Doc/whatsnew/3.5.rst |  15 +++++++++++++++
  1 files changed, 15 insertions(+), 0 deletions(-)


diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst
--- a/Doc/whatsnew/3.5.rst
+++ b/Doc/whatsnew/3.5.rst
@@ -164,6 +164,14 @@
   subclassing of :class:`~inspect.Signature` easier (contributed
   by Yury Selivanov and Eric Snow in :issue:`17373`).
 
+ipaddress
+---------
+
+* :class:`ipaddress.IPv4Network` and :class:`ipaddress.IPv6Network` now
+  accept an ``(address, netmask)`` tuple argument, so as to easily construct
+  network objects from existing addresses (contributed by Peter Moody
+  and Antoine Pitrou in :issue:`16531`).
+
 signal
 ------
 
@@ -189,6 +197,13 @@
   data) for large objects. ``calloc()`` is used instead of ``malloc()`` to
   allocate memory for these objects.
 
+* Some operations on :class:`~ipaddress.IPv4Network` and
+  :class:`~ipaddress.IPv6Network` have been massively sped up, such as
+  :meth:`~ipaddress.IPv4Network.subnets`, :meth:`~ipaddress.IPv4Network.supernet`,
+  :func:`~ipaddress.summarize_address_range`, :func:`~ipaddress.collapse_addresses`.
+  The speed up can range from 3x to 15x.
+  (:issue:`21486`, :issue:`21487`, :issue:`20826`)
+
 
 Build and C API Changes
 =======================

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May 16 02:51:01 2014
From: python-checkins at python.org (terry.reedy)
Date: Fri, 16 May 2014 02:51:01 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE4MTA0?=
 =?utf-8?q?=3A_revise_docstrings=2C_remove_obsolete_comments=2E?=
Message-ID: <3gV9z90X4wz7Ljd@mail.python.org>

http://hg.python.org/cpython/rev/0a6d51ccff54
changeset:   90722:0a6d51ccff54
branch:      2.7
parent:      90711:0a6f0aaeb96a
user:        Terry Jan Reedy <tjreedy at udel.edu>
date:        Thu May 15 20:49:57 2014 -0400
summary:
  Issue #18104: revise docstrings, remove obsolete comments.

files:
  Lib/idlelib/idle_test/htest.py |  61 +++++++++++----------
  1 files changed, 32 insertions(+), 29 deletions(-)


diff --git a/Lib/idlelib/idle_test/htest.py b/Lib/idlelib/idle_test/htest.py
--- a/Lib/idlelib/idle_test/htest.py
+++ b/Lib/idlelib/idle_test/htest.py
@@ -1,35 +1,38 @@
-'''Run a human test of Idle wndow, dialog, and other widget classes.
+'''Run human tests of Idle's window, dialog, and popup widgets.
 
-run(klass) runs a test for one class.
-runall() runs all the defined tests
+run(test): run *test*, a callable that causes a widget to be displayed.
+runall(): run all tests defined in this file.
 
-The file wih the widget class should end with
+Let X be a global name bound to a widget callable. End the module with
+
 if __name__ == '__main__':
     <unittest, if there is one>
     from idlelib.idle_test.htest import run
     run(X)
-where X is a global object of the module. X must be a callable with a
-.__name__ attribute that accepts a 'parent' attribute. X will usually be
-a widget class, but a callable instance with .__name__ or a wrapper
-function also work. The name of wrapper functions, like _Editor_Window,
-should start with '_'.
 
-This file must then contain an instance of this template.
+The X object must have a .__name__ attribute and a 'parent' parameter.
+X will often be a widget class, but a callable instance with .__name__
+or a wrapper function also work. The name of wrapper functions, like
+'_Editor_Window', should start with '_'.
+
+This file must contain a matching instance of the folling template,
+with X.__name__ prepended, as in '_Editor_window_spec ...'.
+
 _spec = {
     'file': '',
     'kwds': {'title': ''},
     'msg': ""
     }
-with X.__name__ prepended to _spec.
-File (no .py) is used in runall() to import the file and get the class.
-Kwds is passed to X (**kwds) after 'parent' is added, to initialize X.
-Msg. displayed is a window with a start button. hint as to how the user
-might test the widget. Closing The box skips or ends the test.
+
+file (no .py): used in runall() to import the file and get X.
+kwds: passed to X (**kwds), after 'parent' is added, to initialize X.
+title: an example; used for some widgets, delete if not.
+msg: displayed in a master window. Hints as to how the user might
+  test the widget. Close the window to skip or end the test.
 '''
 from importlib import import_module
 import Tkinter as tk
 
-# Template for class_spec dicts, copy and uncomment
 
 _Editor_window_spec = {
     'file': 'EditorWindow',
@@ -61,30 +64,30 @@
               "Close 'Get Name' with a valid entry (printed to Shell), [Cancel], or [X]",
     }
 
-def run(klas):
-    "Test the widget class klas using _spec dict"
+def run(test):
+    "Display a widget with callable *test* using a _spec dict"
     root = tk.Tk()
-    klas_spec = globals()[klas.__name__+'_spec']
-    klas_kwds = klas_spec['kwds']
-    klas_kwds['parent'] = root
-    # This presumes that Idle consistently uses 'parent'
-    def run_klas():
-        widget = klas(**klas_kwds)
+    test_spec = globals()[test.__name__ + '_spec']
+    test_kwds = test_spec['kwds']
+    test_kwds['parent'] = root
+
+    def run_test():
+        widget = test(**test_kwds)
         try:
             print(widget.result)
         except AttributeError:
             pass
-    tk.Label(root, text=klas_spec['msg'], justify='left').pack()
-    tk.Button(root, text='Test ' + klas.__name__, command=run_klas).pack()
+    tk.Label(root, text=test_spec['msg'], justify='left').pack()
+    tk.Button(root, text='Test ' + test.__name__, command=run_test).pack()
     root.mainloop()
 
 def runall():
-    'Run all tests. Quick and dirty version.'
+    "Run all tests. Quick and dirty version."
     for k, d in globals().items():
         if k.endswith('_spec'):
             mod = import_module('idlelib.' + d['file'])
-            klas = getattr(mod, k[:-5])
-            run(klas)
+            test = getattr(mod, k[:-5])
+            run(test)
 
 if __name__ == '__main__':
     runall()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May 16 02:51:02 2014
From: python-checkins at python.org (terry.reedy)
Date: Fri, 16 May 2014 02:51:02 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE4MTA0?=
 =?utf-8?q?=3A_revise_docstrings=2C_remove_obsolete_comments=2E?=
Message-ID: <3gV9zB388pz7Ljs@mail.python.org>

http://hg.python.org/cpython/rev/6d2982ff441f
changeset:   90723:6d2982ff441f
branch:      3.4
parent:      90718:4621bb82ceec
user:        Terry Jan Reedy <tjreedy at udel.edu>
date:        Thu May 15 20:50:10 2014 -0400
summary:
  Issue #18104: revise docstrings, remove obsolete comments.

files:
  Lib/idlelib/idle_test/htest.py |  61 +++++++++++----------
  1 files changed, 32 insertions(+), 29 deletions(-)


diff --git a/Lib/idlelib/idle_test/htest.py b/Lib/idlelib/idle_test/htest.py
--- a/Lib/idlelib/idle_test/htest.py
+++ b/Lib/idlelib/idle_test/htest.py
@@ -1,35 +1,38 @@
-'''Run a human test of Idle wndow, dialog, and other widget classes.
+'''Run human tests of Idle's window, dialog, and popup widgets.
 
-run(klass) runs a test for one class.
-runall() runs all the defined tests
+run(test): run *test*, a callable that causes a widget to be displayed.
+runall(): run all tests defined in this file.
 
-The file wih the widget class should end with
+Let X be a global name bound to a widget callable. End the module with
+
 if __name__ == '__main__':
     <unittest, if there is one>
     from idlelib.idle_test.htest import run
     run(X)
-where X is a global object of the module. X must be a callable with a
-.__name__ attribute that accepts a 'parent' attribute. X will usually be
-a widget class, but a callable instance with .__name__ or a wrapper
-function also work. The name of wrapper functions, like _Editor_Window,
-should start with '_'.
 
-This file must then contain an instance of this template.
+The X object must have a .__name__ attribute and a 'parent' parameter.
+X will often be a widget class, but a callable instance with .__name__
+or a wrapper function also work. The name of wrapper functions, like
+'_Editor_Window', should start with '_'.
+
+This file must contain a matching instance of the folling template,
+with X.__name__ prepended, as in '_Editor_window_spec ...'.
+
 _spec = {
     'file': '',
     'kwds': {'title': ''},
     'msg': ""
     }
-with X.__name__ prepended to _spec.
-File (no .py) is used in runall() to import the file and get the class.
-Kwds is passed to X (**kwds) after 'parent' is added, to initialize X.
-Msg. displayed is a window with a start button. hint as to how the user
-might test the widget. Closing The box skips or ends the test.
+
+file (no .py): used in runall() to import the file and get X.
+kwds: passed to X (**kwds), after 'parent' is added, to initialize X.
+title: an example; used for some widgets, delete if not.
+msg: displayed in a master window. Hints as to how the user might
+  test the widget. Close the window to skip or end the test.
 '''
 from importlib import import_module
 import tkinter as tk
 
-# Template for class_spec dicts, copy and uncomment
 
 _Editor_window_spec = {
     'file': 'EditorWindow',
@@ -61,30 +64,30 @@
               "Close 'Get Name' with a valid entry (printed to Shell), [Cancel], or [X]",
     }
 
-def run(klas):
-    "Test the widget class klas using _spec dict"
+def run(test):
+    "Display a widget with callable *test* using a _spec dict"
     root = tk.Tk()
-    klas_spec = globals()[klas.__name__+'_spec']
-    klas_kwds = klas_spec['kwds']
-    klas_kwds['parent'] = root
-    # This presumes that Idle consistently uses 'parent'
-    def run_klas():
-        widget = klas(**klas_kwds)
+    test_spec = globals()[test.__name__ + '_spec']
+    test_kwds = test_spec['kwds']
+    test_kwds['parent'] = root
+
+    def run_test():
+        widget = test(**test_kwds)
         try:
             print(widget.result)
         except AttributeError:
             pass
-    tk.Label(root, text=klas_spec['msg'], justify='left').pack()
-    tk.Button(root, text='Test ' + klas.__name__, command=run_klas).pack()
+    tk.Label(root, text=test_spec['msg'], justify='left').pack()
+    tk.Button(root, text='Test ' + test.__name__, command=run_test).pack()
     root.mainloop()
 
 def runall():
-    'Run all tests. Quick and dirty version.'
+    "Run all tests. Quick and dirty version."
     for k, d in globals().items():
         if k.endswith('_spec'):
             mod = import_module('idlelib.' + d['file'])
-            klas = getattr(mod, k[:-5])
-            run(klas)
+            test = getattr(mod, k[:-5])
+            run(test)
 
 if __name__ == '__main__':
     runall()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May 16 02:51:03 2014
From: python-checkins at python.org (terry.reedy)
Date: Fri, 16 May 2014 02:51:03 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?q?=29=3A_Merge_with_3=2E4?=
Message-ID: <3gV9zC62kDz7Ljd@mail.python.org>

http://hg.python.org/cpython/rev/b6c372147db4
changeset:   90724:b6c372147db4
parent:      90721:758323c8556a
parent:      90723:6d2982ff441f
user:        Terry Jan Reedy <tjreedy at udel.edu>
date:        Thu May 15 20:50:30 2014 -0400
summary:
  Merge with 3.4

files:
  Lib/idlelib/idle_test/htest.py |  61 +++++++++++----------
  1 files changed, 32 insertions(+), 29 deletions(-)


diff --git a/Lib/idlelib/idle_test/htest.py b/Lib/idlelib/idle_test/htest.py
--- a/Lib/idlelib/idle_test/htest.py
+++ b/Lib/idlelib/idle_test/htest.py
@@ -1,35 +1,38 @@
-'''Run a human test of Idle wndow, dialog, and other widget classes.
+'''Run human tests of Idle's window, dialog, and popup widgets.
 
-run(klass) runs a test for one class.
-runall() runs all the defined tests
+run(test): run *test*, a callable that causes a widget to be displayed.
+runall(): run all tests defined in this file.
 
-The file wih the widget class should end with
+Let X be a global name bound to a widget callable. End the module with
+
 if __name__ == '__main__':
     <unittest, if there is one>
     from idlelib.idle_test.htest import run
     run(X)
-where X is a global object of the module. X must be a callable with a
-.__name__ attribute that accepts a 'parent' attribute. X will usually be
-a widget class, but a callable instance with .__name__ or a wrapper
-function also work. The name of wrapper functions, like _Editor_Window,
-should start with '_'.
 
-This file must then contain an instance of this template.
+The X object must have a .__name__ attribute and a 'parent' parameter.
+X will often be a widget class, but a callable instance with .__name__
+or a wrapper function also work. The name of wrapper functions, like
+'_Editor_Window', should start with '_'.
+
+This file must contain a matching instance of the folling template,
+with X.__name__ prepended, as in '_Editor_window_spec ...'.
+
 _spec = {
     'file': '',
     'kwds': {'title': ''},
     'msg': ""
     }
-with X.__name__ prepended to _spec.
-File (no .py) is used in runall() to import the file and get the class.
-Kwds is passed to X (**kwds) after 'parent' is added, to initialize X.
-Msg. displayed is a window with a start button. hint as to how the user
-might test the widget. Closing The box skips or ends the test.
+
+file (no .py): used in runall() to import the file and get X.
+kwds: passed to X (**kwds), after 'parent' is added, to initialize X.
+title: an example; used for some widgets, delete if not.
+msg: displayed in a master window. Hints as to how the user might
+  test the widget. Close the window to skip or end the test.
 '''
 from importlib import import_module
 import tkinter as tk
 
-# Template for class_spec dicts, copy and uncomment
 
 _Editor_window_spec = {
     'file': 'EditorWindow',
@@ -61,30 +64,30 @@
               "Close 'Get Name' with a valid entry (printed to Shell), [Cancel], or [X]",
     }
 
-def run(klas):
-    "Test the widget class klas using _spec dict"
+def run(test):
+    "Display a widget with callable *test* using a _spec dict"
     root = tk.Tk()
-    klas_spec = globals()[klas.__name__+'_spec']
-    klas_kwds = klas_spec['kwds']
-    klas_kwds['parent'] = root
-    # This presumes that Idle consistently uses 'parent'
-    def run_klas():
-        widget = klas(**klas_kwds)
+    test_spec = globals()[test.__name__ + '_spec']
+    test_kwds = test_spec['kwds']
+    test_kwds['parent'] = root
+
+    def run_test():
+        widget = test(**test_kwds)
         try:
             print(widget.result)
         except AttributeError:
             pass
-    tk.Label(root, text=klas_spec['msg'], justify='left').pack()
-    tk.Button(root, text='Test ' + klas.__name__, command=run_klas).pack()
+    tk.Label(root, text=test_spec['msg'], justify='left').pack()
+    tk.Button(root, text='Test ' + test.__name__, command=run_test).pack()
     root.mainloop()
 
 def runall():
-    'Run all tests. Quick and dirty version.'
+    "Run all tests. Quick and dirty version."
     for k, d in globals().items():
         if k.endswith('_spec'):
             mod = import_module('idlelib.' + d['file'])
-            klas = getattr(mod, k[:-5])
-            run(klas)
+            test = getattr(mod, k[:-5])
+            run(test)
 
 if __name__ == '__main__':
     runall()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May 16 14:48:59 2014
From: python-checkins at python.org (victor.stinner)
Date: Fri, 16 May 2014 14:48:59 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2313916=3A_Fix_surr?=
 =?utf-8?q?ogatepass_error_handler_on_Windows?=
Message-ID: <3gVTvb3rMgz7LjV@mail.python.org>

http://hg.python.org/cpython/rev/8ee2b73cda7a
changeset:   90725:8ee2b73cda7a
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Fri May 16 14:46:20 2014 +0200
summary:
  Issue #13916: Fix surrogatepass error handler on Windows

files:
  Lib/test/test_codecs.py |  6 ------
  Python/codecs.c         |  4 ++++
  2 files changed, 4 insertions(+), 6 deletions(-)


diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py
--- a/Lib/test/test_codecs.py
+++ b/Lib/test/test_codecs.py
@@ -2841,12 +2841,6 @@
             (b'abc', 'strict', 'abc'),
             (b'\xe9\x80', 'strict', '\xe9\u20ac'),
             (b'\xff', 'strict', '\xff'),
-            # invalid bytes
-            (b'[\x98]', 'strict', None),
-            (b'[\x98]', 'ignore', '[]'),
-            (b'[\x98]', 'replace', '[\ufffd]'),
-            (b'[\x98]', 'surrogateescape', '[\udc98]'),
-            (b'[\x98]', 'surrogatepass', None),
         ))
 
     def test_cp_utf7(self):
diff --git a/Python/codecs.c b/Python/codecs.c
--- a/Python/codecs.c
+++ b/Python/codecs.c
@@ -960,6 +960,10 @@
             }
         }
     }
+    else if (strcmp(encoding, "CP_UTF8") == 0) {
+        *bytelength = 3;
+        return ENC_UTF8;
+    }
     return ENC_UNKNOWN;
 }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May 16 19:45:13 2014
From: python-checkins at python.org (eric.snow)
Date: Fri, 16 May 2014 19:45:13 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_a_small_typo_in_test?=
	=?utf-8?q?=5Fimportlib=2E?=
Message-ID: <3gVcTP2MVqz7LjV@mail.python.org>

http://hg.python.org/cpython/rev/6127881954b5
changeset:   90726:6127881954b5
user:        Eric Snow <ericsnowcurrently at gmail.com>
date:        Fri May 16 11:22:05 2014 -0600
summary:
  Fix a small typo in test_importlib.

files:
  Lib/test/test_importlib/import_/test___package__.py |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_importlib/import_/test___package__.py b/Lib/test/test_importlib/import_/test___package__.py
--- a/Lib/test/test_importlib/import_/test___package__.py
+++ b/Lib/test/test_importlib/import_/test___package__.py
@@ -75,11 +75,11 @@
 Frozen_UsingPackagePEP302, Source_UsingPackagePEP302 = util.test_both(
         Using__package__PEP302, __import__=util.__import__)
 
-class Using__package__PEP302(Using__package__):
+class Using__package__PEP451(Using__package__):
     mock_modules = util.mock_spec
 
 Frozen_UsingPackagePEP451, Source_UsingPackagePEP451 = util.test_both(
-        Using__package__PEP302, __import__=util.__import__)
+        Using__package__PEP451, __import__=util.__import__)
 
 
 class Setting__package__:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri May 16 19:45:15 2014
From: python-checkins at python.org (eric.snow)
Date: Fri, 16 May 2014 19:45:15 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321503=3A_Use_test?=
 =?utf-8?q?=5Fboth=28=29_consistently_in_test=5Fimportlib=2E?=
Message-ID: <3gVcTR6pbxz7Ljg@mail.python.org>

http://hg.python.org/cpython/rev/34d65746d5ca
changeset:   90727:34d65746d5ca
user:        Eric Snow <ericsnowcurrently at gmail.com>
date:        Fri May 16 11:40:40 2014 -0600
summary:
  Issue #21503: Use test_both() consistently in test_importlib.

files:
  Lib/test/test_importlib/builtin/test_finder.py             |   14 +-
  Lib/test/test_importlib/builtin/test_loader.py             |   14 +-
  Lib/test/test_importlib/extension/test_case_sensitivity.py |    9 +-
  Lib/test/test_importlib/extension/test_finder.py           |    6 +-
  Lib/test/test_importlib/extension/test_loader.py           |    5 +-
  Lib/test/test_importlib/extension/test_path_hook.py        |    6 +-
  Lib/test/test_importlib/frozen/test_finder.py              |   12 +-
  Lib/test/test_importlib/frozen/test_loader.py              |   17 +-
  Lib/test/test_importlib/import_/test___loader__.py         |   11 +-
  Lib/test/test_importlib/import_/test___package__.py        |   16 +-
  Lib/test/test_importlib/import_/test_api.py                |   12 +-
  Lib/test/test_importlib/import_/test_caching.py            |    8 +-
  Lib/test/test_importlib/import_/test_fromlist.py           |   12 +-
  Lib/test/test_importlib/import_/test_meta_path.py          |   20 +-
  Lib/test/test_importlib/import_/test_packages.py           |    6 +-
  Lib/test/test_importlib/import_/test_path.py               |   12 +-
  Lib/test/test_importlib/import_/test_relative_imports.py   |    6 +-
  Lib/test/test_importlib/source/test_case_sensitivity.py    |   16 +-
  Lib/test/test_importlib/source/test_file_loader.py         |   50 +-
  Lib/test/test_importlib/source/test_finder.py              |   17 +-
  Lib/test/test_importlib/source/test_path_hook.py           |    5 +-
  Lib/test/test_importlib/source/test_source_encoding.py     |   28 +-
  Lib/test/test_importlib/test_abc.py                        |  290 ++++-----
  Lib/test/test_importlib/test_api.py                        |   99 +-
  Lib/test/test_importlib/test_locks.py                      |   47 +-
  Lib/test/test_importlib/test_spec.py                       |   72 +-
  Lib/test/test_importlib/test_util.py                       |   85 +-
  Lib/test/test_importlib/test_windows.py                    |   15 +-
  Lib/test/test_importlib/util.py                            |   42 +-
  Misc/NEWS                                                  |    2 +
  30 files changed, 526 insertions(+), 428 deletions(-)


diff --git a/Lib/test/test_importlib/builtin/test_finder.py b/Lib/test/test_importlib/builtin/test_finder.py
--- a/Lib/test/test_importlib/builtin/test_finder.py
+++ b/Lib/test/test_importlib/builtin/test_finder.py
@@ -1,7 +1,7 @@
 from .. import abc
 from .. import util
 
-frozen_machinery, source_machinery = util.import_importlib('importlib.machinery')
+machinery = util.import_importlib('importlib.machinery')
 
 import sys
 import unittest
@@ -44,8 +44,10 @@
                                                             ['pkg'])
             self.assertIsNone(spec)
 
-Frozen_FindSpecTests, Source_FindSpecTests = util.test_both(FindSpecTests,
-        machinery=[frozen_machinery, source_machinery])
+
+(Frozen_FindSpecTests,
+ Source_FindSpecTests
+ ) = util.test_both(FindSpecTests, machinery=machinery)
 
 
 @unittest.skipIf(util.BUILTINS.good_name is None, 'no reasonable builtin module')
@@ -78,8 +80,10 @@
                                                             ['pkg'])
             self.assertIsNone(loader)
 
-Frozen_FinderTests, Source_FinderTests = util.test_both(FinderTests,
-        machinery=[frozen_machinery, source_machinery])
+
+(Frozen_FinderTests,
+ Source_FinderTests
+ ) = util.test_both(FinderTests, machinery=machinery)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/builtin/test_loader.py b/Lib/test/test_importlib/builtin/test_loader.py
--- a/Lib/test/test_importlib/builtin/test_loader.py
+++ b/Lib/test/test_importlib/builtin/test_loader.py
@@ -1,7 +1,7 @@
 from .. import abc
 from .. import util
 
-frozen_machinery, source_machinery = util.import_importlib('importlib.machinery')
+machinery = util.import_importlib('importlib.machinery')
 
 import sys
 import types
@@ -65,8 +65,9 @@
         self.assertEqual(cm.exception.name, module_name)
 
 
-Frozen_LoaderTests, Source_LoaderTests = util.test_both(LoaderTests,
-        machinery=[frozen_machinery, source_machinery])
+(Frozen_LoaderTests,
+ Source_LoaderTests
+ ) = util.test_both(LoaderTests, machinery=machinery)
 
 
 @unittest.skipIf(util.BUILTINS.good_name is None, 'no reasonable builtin module')
@@ -98,9 +99,10 @@
             method(util.BUILTINS.bad_name)
         self.assertRaises(util.BUILTINS.bad_name)
 
-Frozen_InspectLoaderTests, Source_InspectLoaderTests = util.test_both(
-        InspectLoaderTests,
-        machinery=[frozen_machinery, source_machinery])
+
+(Frozen_InspectLoaderTests,
+ Source_InspectLoaderTests
+ ) = util.test_both(InspectLoaderTests, machinery=machinery)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/extension/test_case_sensitivity.py b/Lib/test/test_importlib/extension/test_case_sensitivity.py
--- a/Lib/test/test_importlib/extension/test_case_sensitivity.py
+++ b/Lib/test/test_importlib/extension/test_case_sensitivity.py
@@ -5,7 +5,7 @@
 
 from .. import util
 
-frozen_machinery, source_machinery = util.import_importlib('importlib.machinery')
+machinery = util.import_importlib('importlib.machinery')
 
 
 # XXX find_spec tests
@@ -41,9 +41,10 @@
             loader = self.find_module()
             self.assertTrue(hasattr(loader, 'load_module'))
 
-Frozen_ExtensionCaseSensitivity, Source_ExtensionCaseSensitivity = util.test_both(
-        ExtensionModuleCaseSensitivityTest,
-        machinery=[frozen_machinery, source_machinery])
+
+(Frozen_ExtensionCaseSensitivity,
+ Source_ExtensionCaseSensitivity
+ ) = util.test_both(ExtensionModuleCaseSensitivityTest, machinery=machinery)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/extension/test_finder.py b/Lib/test/test_importlib/extension/test_finder.py
--- a/Lib/test/test_importlib/extension/test_finder.py
+++ b/Lib/test/test_importlib/extension/test_finder.py
@@ -35,8 +35,10 @@
     def test_failure(self):
         self.assertIsNone(self.find_module('asdfjkl;'))
 
-Frozen_FinderTests, Source_FinderTests = util.test_both(
-        FinderTests, machinery=machinery)
+
+(Frozen_FinderTests,
+ Source_FinderTests
+ ) = util.test_both(FinderTests, machinery=machinery)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/extension/test_loader.py b/Lib/test/test_importlib/extension/test_loader.py
--- a/Lib/test/test_importlib/extension/test_loader.py
+++ b/Lib/test/test_importlib/extension/test_loader.py
@@ -76,8 +76,9 @@
             loader = self.machinery.ExtensionFileLoader('pkg', path)
             self.assertTrue(loader.is_package('pkg'))
 
-Frozen_LoaderTests, Source_LoaderTests = util.test_both(
-        LoaderTests, machinery=machinery)
+(Frozen_LoaderTests,
+ Source_LoaderTests
+ ) = util.test_both(LoaderTests, machinery=machinery)
 
 
 
diff --git a/Lib/test/test_importlib/extension/test_path_hook.py b/Lib/test/test_importlib/extension/test_path_hook.py
--- a/Lib/test/test_importlib/extension/test_path_hook.py
+++ b/Lib/test/test_importlib/extension/test_path_hook.py
@@ -23,8 +23,10 @@
         # exists.
         self.assertTrue(hasattr(self.hook(util.EXTENSIONS.path), 'find_module'))
 
-Frozen_PathHooksTests, Source_PathHooksTests = util.test_both(
-        PathHookTests, machinery=machinery)
+
+(Frozen_PathHooksTests,
+ Source_PathHooksTests
+ ) = util.test_both(PathHookTests, machinery=machinery)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/frozen/test_finder.py b/Lib/test/test_importlib/frozen/test_finder.py
--- a/Lib/test/test_importlib/frozen/test_finder.py
+++ b/Lib/test/test_importlib/frozen/test_finder.py
@@ -37,8 +37,10 @@
         spec = self.find('<not real>')
         self.assertIsNone(spec)
 
-Frozen_FindSpecTests, Source_FindSpecTests = util.test_both(FindSpecTests,
-                                                            machinery=machinery)
+
+(Frozen_FindSpecTests,
+ Source_FindSpecTests
+ ) = util.test_both(FindSpecTests, machinery=machinery)
 
 
 class FinderTests(abc.FinderTests):
@@ -72,8 +74,10 @@
         loader = self.find('<not real>')
         self.assertIsNone(loader)
 
-Frozen_FinderTests, Source_FinderTests = util.test_both(FinderTests,
-                                                        machinery=machinery)
+
+(Frozen_FinderTests,
+ Source_FinderTests
+ ) = util.test_both(FinderTests, machinery=machinery)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/frozen/test_loader.py b/Lib/test/test_importlib/frozen/test_loader.py
--- a/Lib/test/test_importlib/frozen/test_loader.py
+++ b/Lib/test/test_importlib/frozen/test_loader.py
@@ -85,8 +85,10 @@
             self.exec_module('_not_real')
         self.assertEqual(cm.exception.name, '_not_real')
 
-Frozen_ExecModuleTests, Source_ExecModuleTests = util.test_both(ExecModuleTests,
-                                                        machinery=machinery)
+
+(Frozen_ExecModuleTests,
+ Source_ExecModuleTests
+ ) = util.test_both(ExecModuleTests, machinery=machinery)
 
 
 class LoaderTests(abc.LoaderTests):
@@ -175,8 +177,10 @@
             self.machinery.FrozenImporter.load_module('_not_real')
         self.assertEqual(cm.exception.name, '_not_real')
 
-Frozen_LoaderTests, Source_LoaderTests = util.test_both(LoaderTests,
-                                                        machinery=machinery)
+
+(Frozen_LoaderTests,
+ Source_LoaderTests
+ ) = util.test_both(LoaderTests, machinery=machinery)
 
 
 class InspectLoaderTests:
@@ -214,8 +218,9 @@
                 method('importlib')
             self.assertEqual(cm.exception.name, 'importlib')
 
-Frozen_ILTests, Source_ILTests = util.test_both(InspectLoaderTests,
-                                                machinery=machinery)
+(Frozen_ILTests,
+ Source_ILTests
+ ) = util.test_both(InspectLoaderTests, machinery=machinery)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/import_/test___loader__.py b/Lib/test/test_importlib/import_/test___loader__.py
--- a/Lib/test/test_importlib/import_/test___loader__.py
+++ b/Lib/test/test_importlib/import_/test___loader__.py
@@ -23,8 +23,10 @@
             module = self.__import__('blah')
         self.assertEqual(loader, module.__loader__)
 
-Frozen_SpecTests, Source_SpecTests = util.test_both(
-        SpecLoaderAttributeTests, __import__=util.__import__)
+
+(Frozen_SpecTests,
+ Source_SpecTests
+ ) = util.test_both(SpecLoaderAttributeTests, __import__=util.__import__)
 
 
 class LoaderMock:
@@ -61,8 +63,9 @@
         self.assertEqual(loader, module.__loader__)
 
 
-Frozen_Tests, Source_Tests = util.test_both(LoaderAttributeTests,
-                                            __import__=util.__import__)
+(Frozen_Tests,
+ Source_Tests
+ ) = util.test_both(LoaderAttributeTests, __import__=util.__import__)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/import_/test___package__.py b/Lib/test/test_importlib/import_/test___package__.py
--- a/Lib/test/test_importlib/import_/test___package__.py
+++ b/Lib/test/test_importlib/import_/test___package__.py
@@ -69,17 +69,23 @@
         with self.assertRaises(TypeError):
             self.__import__('', globals, {}, ['relimport'], 1)
 
+
 class Using__package__PEP302(Using__package__):
     mock_modules = util.mock_modules
 
-Frozen_UsingPackagePEP302, Source_UsingPackagePEP302 = util.test_both(
-        Using__package__PEP302, __import__=util.__import__)
+
+(Frozen_UsingPackagePEP302,
+ Source_UsingPackagePEP302
+ ) = util.test_both(Using__package__PEP302, __import__=util.__import__)
+
 
 class Using__package__PEP451(Using__package__):
     mock_modules = util.mock_spec
 
-Frozen_UsingPackagePEP451, Source_UsingPackagePEP451 = util.test_both(
-        Using__package__PEP451, __import__=util.__import__)
+
+(Frozen_UsingPackagePEP451,
+ Source_UsingPackagePEP451
+ ) = util.test_both(Using__package__PEP451, __import__=util.__import__)
 
 
 class Setting__package__:
@@ -94,7 +100,7 @@
 
     """
 
-    __import__ = util.__import__[1]
+    __import__ = util.__import__['Source']
 
     # [top-level]
     def test_top_level(self):
diff --git a/Lib/test/test_importlib/import_/test_api.py b/Lib/test/test_importlib/import_/test_api.py
--- a/Lib/test/test_importlib/import_/test_api.py
+++ b/Lib/test/test_importlib/import_/test_api.py
@@ -78,15 +78,19 @@
 class OldAPITests(APITest):
     bad_finder_loader = BadLoaderFinder
 
-Frozen_OldAPITests, Source_OldAPITests = util.test_both(
-        OldAPITests, __import__=util.__import__)
+
+(Frozen_OldAPITests,
+ Source_OldAPITests
+ ) = util.test_both(OldAPITests, __import__=util.__import__)
 
 
 class SpecAPITests(APITest):
     bad_finder_loader = BadSpecFinderLoader
 
-Frozen_SpecAPITests, Source_SpecAPITests = util.test_both(
-        SpecAPITests, __import__=util.__import__)
+
+(Frozen_SpecAPITests,
+ Source_SpecAPITests
+ ) = util.test_both(SpecAPITests, __import__=util.__import__)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/import_/test_caching.py b/Lib/test/test_importlib/import_/test_caching.py
--- a/Lib/test/test_importlib/import_/test_caching.py
+++ b/Lib/test/test_importlib/import_/test_caching.py
@@ -38,15 +38,17 @@
                 self.__import__(name)
             self.assertEqual(cm.exception.name, name)
 
-Frozen_UseCache, Source_UseCache = util.test_both(
-        UseCache, __import__=util.__import__)
+
+(Frozen_UseCache,
+ Source_UseCache
+ ) = util.test_both(UseCache, __import__=util.__import__)
 
 
 class ImportlibUseCache(UseCache, unittest.TestCase):
 
     # Pertinent only to PEP 302; exec_module() doesn't return a module.
 
-    __import__ = util.__import__[1]
+    __import__ = util.__import__['Source']
 
     def create_mock(self, *names, return_=None):
         mock = util.mock_modules(*names)
diff --git a/Lib/test/test_importlib/import_/test_fromlist.py b/Lib/test/test_importlib/import_/test_fromlist.py
--- a/Lib/test/test_importlib/import_/test_fromlist.py
+++ b/Lib/test/test_importlib/import_/test_fromlist.py
@@ -28,8 +28,10 @@
                 module = self.__import__('pkg.module', fromlist=['attr'])
                 self.assertEqual(module.__name__, 'pkg.module')
 
-Frozen_ReturnValue, Source_ReturnValue = util.test_both(
-        ReturnValue, __import__=util.__import__)
+
+(Frozen_ReturnValue,
+ Source_ReturnValue
+ ) = util.test_both(ReturnValue, __import__=util.__import__)
 
 
 class HandlingFromlist:
@@ -120,8 +122,10 @@
                 self.assertEqual(module.module1.__name__, 'pkg.module1')
                 self.assertEqual(module.module2.__name__, 'pkg.module2')
 
-Frozen_FromList, Source_FromList = util.test_both(
-        HandlingFromlist, __import__=util.__import__)
+
+(Frozen_FromList,
+ Source_FromList
+ ) = util.test_both(HandlingFromlist, __import__=util.__import__)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/import_/test_meta_path.py b/Lib/test/test_importlib/import_/test_meta_path.py
--- a/Lib/test/test_importlib/import_/test_meta_path.py
+++ b/Lib/test/test_importlib/import_/test_meta_path.py
@@ -45,8 +45,10 @@
                 self.assertEqual(len(w), 1)
                 self.assertTrue(issubclass(w[-1].category, ImportWarning))
 
-Frozen_CallingOrder, Source_CallingOrder = util.test_both(
-        CallingOrder, __import__=util.__import__)
+
+(Frozen_CallingOrder,
+ Source_CallingOrder
+ ) = util.test_both(CallingOrder, __import__=util.__import__)
 
 
 class CallSignature:
@@ -99,19 +101,25 @@
                 self.assertEqual(args[0], mod_name)
                 self.assertIs(args[1], path)
 
+
 class CallSignaturePEP302(CallSignature):
     mock_modules = util.mock_modules
     finder_name = 'find_module'
 
-Frozen_CallSignaturePEP302, Source_CallSignaturePEP302 = util.test_both(
-        CallSignaturePEP302, __import__=util.__import__)
+
+(Frozen_CallSignaturePEP302,
+ Source_CallSignaturePEP302
+ ) = util.test_both(CallSignaturePEP302, __import__=util.__import__)
+
 
 class CallSignaturePEP451(CallSignature):
     mock_modules = util.mock_spec
     finder_name = 'find_spec'
 
-Frozen_CallSignaturePEP451, Source_CallSignaturePEP451 = util.test_both(
-        CallSignaturePEP451, __import__=util.__import__)
+
+(Frozen_CallSignaturePEP451,
+ Source_CallSignaturePEP451
+ ) = util.test_both(CallSignaturePEP451, __import__=util.__import__)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/import_/test_packages.py b/Lib/test/test_importlib/import_/test_packages.py
--- a/Lib/test/test_importlib/import_/test_packages.py
+++ b/Lib/test/test_importlib/import_/test_packages.py
@@ -101,8 +101,10 @@
                 finally:
                     support.unload(subname)
 
-Frozen_ParentTests, Source_ParentTests = util.test_both(
-        ParentModuleTests, __import__=util.__import__)
+
+(Frozen_ParentTests,
+ Source_ParentTests
+ ) = util.test_both(ParentModuleTests, __import__=util.__import__)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/import_/test_path.py b/Lib/test/test_importlib/import_/test_path.py
--- a/Lib/test/test_importlib/import_/test_path.py
+++ b/Lib/test/test_importlib/import_/test_path.py
@@ -158,8 +158,10 @@
             got = self.machinery.PathFinder.find_spec('whatever', [path])
         self.assertEqual(got, success_finder.spec)
 
-Frozen_FinderTests, Source_FinderTests = util.test_both(
-        FinderTests, importlib=importlib, machinery=machinery)
+
+(Frozen_FinderTests,
+ Source_FinderTests
+ ) = util.test_both(FinderTests, importlib=importlib, machinery=machinery)
 
 
 class PathEntryFinderTests:
@@ -182,8 +184,10 @@
                                path_hooks=[Finder]):
             self.machinery.PathFinder.find_spec('importlib')
 
-Frozen_PEFTests, Source_PEFTests = util.test_both(
-        PathEntryFinderTests, machinery=machinery)
+
+(Frozen_PEFTests,
+ Source_PEFTests
+ ) = util.test_both(PathEntryFinderTests, machinery=machinery)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/import_/test_relative_imports.py b/Lib/test/test_importlib/import_/test_relative_imports.py
--- a/Lib/test/test_importlib/import_/test_relative_imports.py
+++ b/Lib/test/test_importlib/import_/test_relative_imports.py
@@ -207,8 +207,10 @@
         with self.assertRaises(KeyError):
             self.__import__('sys', level=1)
 
-Frozen_RelativeImports, Source_RelativeImports = util.test_both(
-        RelativeImports, __import__=util.__import__)
+
+(Frozen_RelativeImports,
+ Source_RelativeImports
+ ) = util.test_both(RelativeImports, __import__=util.__import__)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/source/test_case_sensitivity.py b/Lib/test/test_importlib/source/test_case_sensitivity.py
--- a/Lib/test/test_importlib/source/test_case_sensitivity.py
+++ b/Lib/test/test_importlib/source/test_case_sensitivity.py
@@ -62,20 +62,28 @@
             self.assertIsNotNone(insensitive)
             self.assertIn(self.name, insensitive.get_filename(self.name))
 
+
 class CaseSensitivityTestPEP302(CaseSensitivityTest):
     def find(self, finder):
         return finder.find_module(self.name)
 
-Frozen_CaseSensitivityTestPEP302, Source_CaseSensitivityTestPEP302 = util.test_both(
-    CaseSensitivityTestPEP302, importlib=importlib, machinery=machinery)
+
+(Frozen_CaseSensitivityTestPEP302,
+ Source_CaseSensitivityTestPEP302
+ ) = util.test_both(CaseSensitivityTestPEP302, importlib=importlib,
+                    machinery=machinery)
+
 
 class CaseSensitivityTestPEP451(CaseSensitivityTest):
     def find(self, finder):
         found = finder.find_spec(self.name)
         return found.loader if found is not None else found
 
-Frozen_CaseSensitivityTestPEP451, Source_CaseSensitivityTestPEP451 = util.test_both(
-    CaseSensitivityTestPEP451, importlib=importlib, machinery=machinery)
+
+(Frozen_CaseSensitivityTestPEP451,
+ Source_CaseSensitivityTestPEP451
+ ) = util.test_both(CaseSensitivityTestPEP451, importlib=importlib,
+                    machinery=machinery)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/source/test_file_loader.py b/Lib/test/test_importlib/source/test_file_loader.py
--- a/Lib/test/test_importlib/source/test_file_loader.py
+++ b/Lib/test/test_importlib/source/test_file_loader.py
@@ -235,9 +235,11 @@
                 warnings.simplefilter('ignore', DeprecationWarning)
                 loader.load_module('bad name')
 
-Frozen_SimpleTest, Source_SimpleTest = util.test_both(
-        SimpleTest, importlib=importlib, machinery=machinery, abc=importlib_abc,
-        util=importlib_util)
+
+(Frozen_SimpleTest,
+ Source_SimpleTest
+ ) = util.test_both(SimpleTest, importlib=importlib, machinery=machinery,
+                    abc=importlib_abc, util=importlib_util)
 
 
 class BadBytecodeTest:
@@ -346,6 +348,7 @@
                                     lambda bc: b'\x00\x00\x00\x00' + bc[4:])
             test('_temp', mapping, bc_path)
 
+
 class BadBytecodeTestPEP451(BadBytecodeTest):
 
     def import_(self, file, module_name):
@@ -354,6 +357,7 @@
         module.__spec__ = self.util.spec_from_loader(module_name, loader)
         loader.exec_module(module)
 
+
 class BadBytecodeTestPEP302(BadBytecodeTest):
 
     def import_(self, file, module_name):
@@ -490,21 +494,29 @@
                 # Make writable for eventual clean-up.
                 os.chmod(bytecode_path, stat.S_IWUSR)
 
+
 class SourceLoaderBadBytecodeTestPEP451(
         SourceLoaderBadBytecodeTest, BadBytecodeTestPEP451):
     pass
 
-Frozen_SourceBadBytecodePEP451, Source_SourceBadBytecodePEP451 = util.test_both(
-        SourceLoaderBadBytecodeTestPEP451, importlib=importlib, machinery=machinery,
-        abc=importlib_abc, util=importlib_util)
+
+(Frozen_SourceBadBytecodePEP451,
+ Source_SourceBadBytecodePEP451
+ ) = util.test_both(SourceLoaderBadBytecodeTestPEP451, importlib=importlib,
+                    machinery=machinery, abc=importlib_abc,
+                    util=importlib_util)
+
 
 class SourceLoaderBadBytecodeTestPEP302(
         SourceLoaderBadBytecodeTest, BadBytecodeTestPEP302):
     pass
 
-Frozen_SourceBadBytecodePEP302, Source_SourceBadBytecodePEP302 = util.test_both(
-        SourceLoaderBadBytecodeTestPEP302, importlib=importlib, machinery=machinery,
-        abc=importlib_abc, util=importlib_util)
+
+(Frozen_SourceBadBytecodePEP302,
+ Source_SourceBadBytecodePEP302
+ ) = util.test_both(SourceLoaderBadBytecodeTestPEP302, importlib=importlib,
+                    machinery=machinery, abc=importlib_abc,
+                    util=importlib_util)
 
 
 class SourcelessLoaderBadBytecodeTest:
@@ -566,21 +578,29 @@
     def test_non_code_marshal(self):
         self._test_non_code_marshal(del_source=True)
 
+
 class SourcelessLoaderBadBytecodeTestPEP451(SourcelessLoaderBadBytecodeTest,
         BadBytecodeTestPEP451):
     pass
 
-Frozen_SourcelessBadBytecodePEP451, Source_SourcelessBadBytecodePEP451 = util.test_both(
-        SourcelessLoaderBadBytecodeTestPEP451, importlib=importlib,
-        machinery=machinery, abc=importlib_abc, util=importlib_util)
+
+(Frozen_SourcelessBadBytecodePEP451,
+ Source_SourcelessBadBytecodePEP451
+ ) = util.test_both(SourcelessLoaderBadBytecodeTestPEP451, importlib=importlib,
+                    machinery=machinery, abc=importlib_abc,
+                    util=importlib_util)
+
 
 class SourcelessLoaderBadBytecodeTestPEP302(SourcelessLoaderBadBytecodeTest,
         BadBytecodeTestPEP302):
     pass
 
-Frozen_SourcelessBadBytecodePEP302, Source_SourcelessBadBytecodePEP302 = util.test_both(
-        SourcelessLoaderBadBytecodeTestPEP302, importlib=importlib,
-        machinery=machinery, abc=importlib_abc, util=importlib_util)
+
+(Frozen_SourcelessBadBytecodePEP302,
+ Source_SourcelessBadBytecodePEP302
+ ) = util.test_both(SourcelessLoaderBadBytecodeTestPEP302, importlib=importlib,
+                    machinery=machinery, abc=importlib_abc,
+                    util=importlib_util)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/source/test_finder.py b/Lib/test/test_importlib/source/test_finder.py
--- a/Lib/test/test_importlib/source/test_finder.py
+++ b/Lib/test/test_importlib/source/test_finder.py
@@ -195,8 +195,10 @@
         spec = finder.find_spec(name)
         return spec.loader if spec is not None else spec
 
-Frozen_FinderTestsPEP451, Source_FinderTestsPEP451 = util.test_both(
-        FinderTestsPEP451, machinery=machinery)
+
+(Frozen_FinderTestsPEP451,
+ Source_FinderTestsPEP451
+ ) = util.test_both(FinderTestsPEP451, machinery=machinery)
 
 
 class FinderTestsPEP420(FinderTests):
@@ -209,8 +211,10 @@
             loader_portions = finder.find_loader(name)
             return loader_portions[0] if loader_only else loader_portions
 
-Frozen_FinderTestsPEP420, Source_FinderTestsPEP420 = util.test_both(
-        FinderTestsPEP420, machinery=machinery)
+
+(Frozen_FinderTestsPEP420,
+ Source_FinderTestsPEP420
+ ) = util.test_both(FinderTestsPEP420, machinery=machinery)
 
 
 class FinderTestsPEP302(FinderTests):
@@ -222,9 +226,10 @@
             warnings.simplefilter("ignore", DeprecationWarning)
             return finder.find_module(name)
 
-Frozen_FinderTestsPEP302, Source_FinderTestsPEP302 = util.test_both(
-        FinderTestsPEP302, machinery=machinery)
 
+(Frozen_FinderTestsPEP302,
+ Source_FinderTestsPEP302
+ ) = util.test_both(FinderTestsPEP302, machinery=machinery)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/source/test_path_hook.py b/Lib/test/test_importlib/source/test_path_hook.py
--- a/Lib/test/test_importlib/source/test_path_hook.py
+++ b/Lib/test/test_importlib/source/test_path_hook.py
@@ -22,7 +22,10 @@
         # The empty string represents the cwd.
         self.assertTrue(hasattr(self.path_hook()(''), 'find_module'))
 
-Frozen_PathHookTest, Source_PathHooktest = util.test_both(PathHookTest, machinery=machinery)
+
+(Frozen_PathHookTest,
+ Source_PathHooktest
+ ) = util.test_both(PathHookTest, machinery=machinery)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/source/test_source_encoding.py b/Lib/test/test_importlib/source/test_source_encoding.py
--- a/Lib/test/test_importlib/source/test_source_encoding.py
+++ b/Lib/test/test_importlib/source/test_source_encoding.py
@@ -88,6 +88,7 @@
         with self.assertRaises(SyntaxError):
             self.run_test(source)
 
+
 class EncodingTestPEP451(EncodingTest):
 
     def load(self, loader):
@@ -96,8 +97,11 @@
         loader.exec_module(module)
         return module
 
-Frozen_EncodingTestPEP451, Source_EncodingTestPEP451 = util.test_both(
-        EncodingTestPEP451, machinery=machinery)
+
+(Frozen_EncodingTestPEP451,
+ Source_EncodingTestPEP451
+ ) = util.test_both(EncodingTestPEP451, machinery=machinery)
+
 
 class EncodingTestPEP302(EncodingTest):
 
@@ -106,8 +110,10 @@
             warnings.simplefilter('ignore', DeprecationWarning)
             return loader.load_module(self.module_name)
 
-Frozen_EncodingTestPEP302, Source_EncodingTestPEP302 = util.test_both(
-        EncodingTestPEP302, machinery=machinery)
+
+(Frozen_EncodingTestPEP302,
+ Source_EncodingTestPEP302
+ ) = util.test_both(EncodingTestPEP302, machinery=machinery)
 
 
 class LineEndingTest:
@@ -138,6 +144,7 @@
     def test_lf(self):
         self.run_test(b'\n')
 
+
 class LineEndingTestPEP451(LineEndingTest):
 
     def load(self, loader, module_name):
@@ -146,8 +153,11 @@
         loader.exec_module(module)
         return module
 
-Frozen_LineEndingTestPEP451, Source_LineEndingTestPEP451 = util.test_both(
-        LineEndingTestPEP451, machinery=machinery)
+
+(Frozen_LineEndingTestPEP451,
+ Source_LineEndingTestPEP451
+ ) = util.test_both(LineEndingTestPEP451, machinery=machinery)
+
 
 class LineEndingTestPEP302(LineEndingTest):
 
@@ -156,8 +166,10 @@
             warnings.simplefilter('ignore', DeprecationWarning)
             return loader.load_module(module_name)
 
-Frozen_LineEndingTestPEP302, Source_LineEndingTestPEP302 = util.test_both(
-        LineEndingTestPEP302, machinery=machinery)
+
+(Frozen_LineEndingTestPEP302,
+ Source_LineEndingTestPEP302
+ ) = util.test_both(LineEndingTestPEP302, machinery=machinery)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/test_abc.py b/Lib/test/test_importlib/test_abc.py
--- a/Lib/test/test_importlib/test_abc.py
+++ b/Lib/test/test_importlib/test_abc.py
@@ -10,12 +10,13 @@
 from unittest import mock
 import warnings
 
-from . import util
+from . import util as test_util
 
-frozen_init, source_init = util.import_importlib('importlib')
-frozen_abc, source_abc = util.import_importlib('importlib.abc')
-machinery = util.import_importlib('importlib.machinery')
-frozen_util, source_util = util.import_importlib('importlib.util')
+init = test_util.import_importlib('importlib')
+abc = test_util.import_importlib('importlib.abc')
+machinery = test_util.import_importlib('importlib.machinery')
+util = test_util.import_importlib('importlib.util')
+
 
 ##### Inheritance ##############################################################
 class InheritanceTests:
@@ -26,8 +27,7 @@
     subclasses = []
     superclasses = []
 
-    def __init__(self, *args, **kwargs):
-        super().__init__(*args, **kwargs)
+    def setUp(self):
         self.superclasses = [getattr(self.abc, class_name)
                              for class_name in self.superclass_names]
         if hasattr(self, 'subclass_names'):
@@ -36,11 +36,11 @@
             # checking across module boundaries (i.e. the _bootstrap in abc is
             # not the same as the one in machinery). That means stealing one of
             # the modules from the other to make sure the same instance is used.
-            self.subclasses = [getattr(self.abc.machinery, class_name)
-                                for class_name in self.subclass_names]
+            machinery = self.abc.machinery
+            self.subclasses = [getattr(machinery, class_name)
+                               for class_name in self.subclass_names]
         assert self.subclasses or self.superclasses, self.__class__
-        testing = self.__class__.__name__.partition('_')[2]
-        self.__test = getattr(self.abc, testing)
+        self.__test = getattr(self.abc, self._NAME)
 
     def test_subclasses(self):
         # Test that the expected subclasses inherit.
@@ -54,94 +54,97 @@
             self.assertTrue(issubclass(self.__test, superclass),
                "{0} is not a superclass of {1}".format(superclass, self.__test))
 
-def create_inheritance_tests(base_class):
-    def set_frozen(ns):
-        ns['abc'] = frozen_abc
-    def set_source(ns):
-        ns['abc'] = source_abc
-
-    classes = []
-    for prefix, ns_set in [('Frozen', set_frozen), ('Source', set_source)]:
-        classes.append(types.new_class('_'.join([prefix, base_class.__name__]),
-                                       (base_class, unittest.TestCase),
-                                       exec_body=ns_set))
-    return classes
-
 
 class MetaPathFinder(InheritanceTests):
     superclass_names = ['Finder']
     subclass_names = ['BuiltinImporter', 'FrozenImporter', 'PathFinder',
                       'WindowsRegistryFinder']
 
-tests = create_inheritance_tests(MetaPathFinder)
-Frozen_MetaPathFinderInheritanceTests, Source_MetaPathFinderInheritanceTests = tests
+
+(Frozen_MetaPathFinderInheritanceTests,
+ Source_MetaPathFinderInheritanceTests
+ ) = test_util.test_both(MetaPathFinder, abc=abc)
 
 
 class PathEntryFinder(InheritanceTests):
     superclass_names = ['Finder']
     subclass_names = ['FileFinder']
 
-tests = create_inheritance_tests(PathEntryFinder)
-Frozen_PathEntryFinderInheritanceTests, Source_PathEntryFinderInheritanceTests = tests
+
+(Frozen_PathEntryFinderInheritanceTests,
+ Source_PathEntryFinderInheritanceTests
+ ) = test_util.test_both(PathEntryFinder, abc=abc)
 
 
 class ResourceLoader(InheritanceTests):
     superclass_names = ['Loader']
 
-tests = create_inheritance_tests(ResourceLoader)
-Frozen_ResourceLoaderInheritanceTests, Source_ResourceLoaderInheritanceTests = tests
+
+(Frozen_ResourceLoaderInheritanceTests,
+ Source_ResourceLoaderInheritanceTests
+ ) = test_util.test_both(ResourceLoader, abc=abc)
 
 
 class InspectLoader(InheritanceTests):
     superclass_names = ['Loader']
     subclass_names = ['BuiltinImporter', 'FrozenImporter', 'ExtensionFileLoader']
 
-tests = create_inheritance_tests(InspectLoader)
-Frozen_InspectLoaderInheritanceTests, Source_InspectLoaderInheritanceTests = tests
+
+(Frozen_InspectLoaderInheritanceTests,
+ Source_InspectLoaderInheritanceTests
+ ) = test_util.test_both(InspectLoader, abc=abc)
 
 
 class ExecutionLoader(InheritanceTests):
     superclass_names = ['InspectLoader']
     subclass_names = ['ExtensionFileLoader']
 
-tests = create_inheritance_tests(ExecutionLoader)
-Frozen_ExecutionLoaderInheritanceTests, Source_ExecutionLoaderInheritanceTests = tests
+
+(Frozen_ExecutionLoaderInheritanceTests,
+ Source_ExecutionLoaderInheritanceTests
+ ) = test_util.test_both(ExecutionLoader, abc=abc)
 
 
 class FileLoader(InheritanceTests):
     superclass_names = ['ResourceLoader', 'ExecutionLoader']
     subclass_names = ['SourceFileLoader', 'SourcelessFileLoader']
 
-tests = create_inheritance_tests(FileLoader)
-Frozen_FileLoaderInheritanceTests, Source_FileLoaderInheritanceTests = tests
+
+(Frozen_FileLoaderInheritanceTests,
+ Source_FileLoaderInheritanceTests
+ ) = test_util.test_both(FileLoader, abc=abc)
 
 
 class SourceLoader(InheritanceTests):
     superclass_names = ['ResourceLoader', 'ExecutionLoader']
     subclass_names = ['SourceFileLoader']
 
-tests = create_inheritance_tests(SourceLoader)
-Frozen_SourceLoaderInheritanceTests, Source_SourceLoaderInheritanceTests = tests
+
+(Frozen_SourceLoaderInheritanceTests,
+ Source_SourceLoaderInheritanceTests
+ ) = test_util.test_both(SourceLoader, abc=abc)
+
 
 ##### Default return values ####################################################
-def make_abc_subclasses(base_class):
-    classes = []
-    for kind, abc in [('Frozen', frozen_abc), ('Source', source_abc)]:
-        name = '_'.join([kind, base_class.__name__])
-        base_classes = base_class, getattr(abc, base_class.__name__)
-        classes.append(types.new_class(name, base_classes))
-    return classes
 
-def make_return_value_tests(base_class, test_class):
-    frozen_class, source_class = make_abc_subclasses(base_class)
-    tests = []
-    for prefix, class_in_test in [('Frozen', frozen_class), ('Source', source_class)]:
-        def set_ns(ns):
-            ns['ins'] = class_in_test()
-        tests.append(types.new_class('_'.join([prefix, test_class.__name__]),
-                                     (test_class, unittest.TestCase),
-                                     exec_body=set_ns))
-    return tests
+def make_abc_subclasses(base_class, name=None, inst=False, **kwargs):
+    if name is None:
+        name = base_class.__name__
+    base = {kind: getattr(splitabc, name)
+            for kind, splitabc in abc.items()}
+    return {cls._KIND: cls() if inst else cls
+            for cls in test_util.split_frozen(base_class, base, **kwargs)}
+
+
+class ABCTestHarness:
+
+    @property
+    def ins(self):
+        # Lazily set ins on the class.
+        cls = self.SPLIT[self._KIND]
+        ins = cls()
+        self.__class__.ins = ins
+        return ins
 
 
 class MetaPathFinder:
@@ -149,10 +152,10 @@
     def find_module(self, fullname, path):
         return super().find_module(fullname, path)
 
-Frozen_MPF, Source_MPF = make_abc_subclasses(MetaPathFinder)
 
+class MetaPathFinderDefaultsTests(ABCTestHarness):
 
-class MetaPathFinderDefaultsTests:
+    SPLIT = make_abc_subclasses(MetaPathFinder)
 
     def test_find_module(self):
         # Default should return None.
@@ -163,8 +166,9 @@
         self.ins.invalidate_caches()
 
 
-tests = make_return_value_tests(MetaPathFinder, MetaPathFinderDefaultsTests)
-Frozen_MPFDefaultTests, Source_MPFDefaultTests = tests
+(Frozen_MPFDefaultTests,
+ Source_MPFDefaultTests
+ ) = test_util.test_both(MetaPathFinderDefaultsTests)
 
 
 class PathEntryFinder:
@@ -172,10 +176,10 @@
     def find_loader(self, fullname):
         return super().find_loader(fullname)
 
-Frozen_PEF, Source_PEF = make_abc_subclasses(PathEntryFinder)
 
+class PathEntryFinderDefaultsTests(ABCTestHarness):
 
-class PathEntryFinderDefaultsTests:
+    SPLIT = make_abc_subclasses(PathEntryFinder)
 
     def test_find_loader(self):
         self.assertEqual((None, []), self.ins.find_loader('something'))
@@ -188,8 +192,9 @@
         self.ins.invalidate_caches()
 
 
-tests = make_return_value_tests(PathEntryFinder, PathEntryFinderDefaultsTests)
-Frozen_PEFDefaultTests, Source_PEFDefaultTests = tests
+(Frozen_PEFDefaultTests,
+ Source_PEFDefaultTests
+ ) = test_util.test_both(PathEntryFinderDefaultsTests)
 
 
 class Loader:
@@ -198,10 +203,9 @@
         return super().load_module(fullname)
 
 
-Frozen_L, Source_L = make_abc_subclasses(Loader)
+class LoaderDefaultsTests(ABCTestHarness):
 
-
-class LoaderDefaultsTests:
+    SPLIT = make_abc_subclasses(Loader)
 
     def test_load_module(self):
         with self.assertRaises(ImportError):
@@ -217,8 +221,9 @@
         self.assertTrue(repr(mod))
 
 
-tests = make_return_value_tests(Loader, LoaderDefaultsTests)
-Frozen_LDefaultTests, SourceLDefaultTests = tests
+(Frozen_LDefaultTests,
+ SourceLDefaultTests
+ ) = test_util.test_both(LoaderDefaultsTests)
 
 
 class ResourceLoader(Loader):
@@ -227,18 +232,18 @@
         return super().get_data(path)
 
 
-Frozen_RL, Source_RL = make_abc_subclasses(ResourceLoader)
+class ResourceLoaderDefaultsTests(ABCTestHarness):
 
-
-class ResourceLoaderDefaultsTests:
+    SPLIT = make_abc_subclasses(ResourceLoader)
 
     def test_get_data(self):
         with self.assertRaises(IOError):
             self.ins.get_data('/some/path')
 
 
-tests = make_return_value_tests(ResourceLoader, ResourceLoaderDefaultsTests)
-Frozen_RLDefaultTests, Source_RLDefaultTests = tests
+(Frozen_RLDefaultTests,
+ Source_RLDefaultTests
+ ) = test_util.test_both(ResourceLoaderDefaultsTests)
 
 
 class InspectLoader(Loader):
@@ -250,10 +255,12 @@
         return super().get_source(fullname)
 
 
-Frozen_IL, Source_IL = make_abc_subclasses(InspectLoader)
+SPLIT_IL = make_abc_subclasses(InspectLoader)
 
 
-class InspectLoaderDefaultsTests:
+class InspectLoaderDefaultsTests(ABCTestHarness):
+
+    SPLIT = SPLIT_IL
 
     def test_is_package(self):
         with self.assertRaises(ImportError):
@@ -264,8 +271,9 @@
             self.ins.get_source('blah')
 
 
-tests = make_return_value_tests(InspectLoader, InspectLoaderDefaultsTests)
-Frozen_ILDefaultTests, Source_ILDefaultTests = tests
+(Frozen_ILDefaultTests,
+ Source_ILDefaultTests
+ ) = test_util.test_both(InspectLoaderDefaultsTests)
 
 
 class ExecutionLoader(InspectLoader):
@@ -273,21 +281,25 @@
     def get_filename(self, fullname):
         return super().get_filename(fullname)
 
-Frozen_EL, Source_EL = make_abc_subclasses(ExecutionLoader)
 
+SPLIT_EL = make_abc_subclasses(ExecutionLoader)
 
-class ExecutionLoaderDefaultsTests:
+
+class ExecutionLoaderDefaultsTests(ABCTestHarness):
+
+    SPLIT = SPLIT_EL
 
     def test_get_filename(self):
         with self.assertRaises(ImportError):
             self.ins.get_filename('blah')
 
 
-tests = make_return_value_tests(ExecutionLoader, InspectLoaderDefaultsTests)
-Frozen_ELDefaultTests, Source_ELDefaultsTests = tests
+(Frozen_ELDefaultTests,
+ Source_ELDefaultsTests
+ ) = test_util.test_both(InspectLoaderDefaultsTests)
+
 
 ##### MetaPathFinder concrete methods ##########################################
-
 class MetaPathFinderFindModuleTests:
 
     @classmethod
@@ -317,13 +329,12 @@
         self.assertIs(found, spec.loader)
 
 
-Frozen_MPFFindModuleTests, Source_MPFFindModuleTests = util.test_both(
-        MetaPathFinderFindModuleTests,
-        abc=(frozen_abc, source_abc),
-        util=(frozen_util, source_util))
+(Frozen_MPFFindModuleTests,
+ Source_MPFFindModuleTests
+ ) = test_util.test_both(MetaPathFinderFindModuleTests, abc=abc, util=util)
+
 
 ##### PathEntryFinder concrete methods #########################################
-
 class PathEntryFinderFindLoaderTests:
 
     @classmethod
@@ -361,11 +372,10 @@
         self.assertEqual(paths, found[1])
 
 
-Frozen_PEFFindLoaderTests, Source_PEFFindLoaderTests = util.test_both(
-        PathEntryFinderFindLoaderTests,
-        abc=(frozen_abc, source_abc),
-        machinery=machinery,
-        util=(frozen_util, source_util))
+(Frozen_PEFFindLoaderTests,
+ Source_PEFFindLoaderTests
+ ) = test_util.test_both(PathEntryFinderFindLoaderTests, abc=abc, util=util,
+                         machinery=machinery)
 
 
 ##### Loader concrete methods ##################################################
@@ -386,7 +396,7 @@
     def test_fresh(self):
         loader = self.loader()
         name = 'blah'
-        with util.uncache(name):
+        with test_util.uncache(name):
             loader.load_module(name)
             module = loader.found
             self.assertIs(sys.modules[name], module)
@@ -404,7 +414,7 @@
         module = types.ModuleType(name)
         module.__spec__ = self.util.spec_from_loader(name, loader)
         module.__loader__ = loader
-        with util.uncache(name):
+        with test_util.uncache(name):
             sys.modules[name] = module
             loader.load_module(name)
             found = loader.found
@@ -412,10 +422,9 @@
             self.assertIs(module, sys.modules[name])
 
 
-Frozen_LoaderLoadModuleTests, Source_LoaderLoadModuleTests = util.test_both(
-        LoaderLoadModuleTests,
-        abc=(frozen_abc, source_abc),
-        util=(frozen_util, source_util))
+(Frozen_LoaderLoadModuleTests,
+ Source_LoaderLoadModuleTests
+ ) = test_util.test_both(LoaderLoadModuleTests, abc=abc, util=util)
 
 
 ##### InspectLoader concrete methods ###########################################
@@ -461,11 +470,10 @@
         self.assertEqual(code.co_filename, '<string>')
 
 
-class Frozen_ILSourceToCodeTests(InspectLoaderSourceToCodeTests, unittest.TestCase):
-    InspectLoaderSubclass = Frozen_IL
-
-class Source_ILSourceToCodeTests(InspectLoaderSourceToCodeTests, unittest.TestCase):
-    InspectLoaderSubclass = Source_IL
+(Frozen_ILSourceToCodeTests,
+ Source_ILSourceToCodeTests
+ ) = test_util.test_both(InspectLoaderSourceToCodeTests,
+                         InspectLoaderSubclass=SPLIT_IL)
 
 
 class InspectLoaderGetCodeTests:
@@ -495,11 +503,10 @@
             loader.get_code('blah')
 
 
-class Frozen_ILGetCodeTests(InspectLoaderGetCodeTests, unittest.TestCase):
-    InspectLoaderSubclass = Frozen_IL
-
-class Source_ILGetCodeTests(InspectLoaderGetCodeTests, unittest.TestCase):
-    InspectLoaderSubclass = Source_IL
+(Frozen_ILGetCodeTests,
+ Source_ILGetCodeTests
+ ) = test_util.test_both(InspectLoaderGetCodeTests,
+                         InspectLoaderSubclass=SPLIT_IL)
 
 
 class InspectLoaderLoadModuleTests:
@@ -543,11 +550,10 @@
             self.assertEqual(module, sys.modules[self.module_name])
 
 
-class Frozen_ILLoadModuleTests(InspectLoaderLoadModuleTests, unittest.TestCase):
-    InspectLoaderSubclass = Frozen_IL
-
-class Source_ILLoadModuleTests(InspectLoaderLoadModuleTests, unittest.TestCase):
-    InspectLoaderSubclass = Source_IL
+(Frozen_ILLoadModuleTests,
+ Source_ILLoadModuleTests
+ ) = test_util.test_both(InspectLoaderLoadModuleTests,
+                         InspectLoaderSubclass=SPLIT_IL)
 
 
 ##### ExecutionLoader concrete methods #########################################
@@ -608,15 +614,14 @@
         self.assertEqual(module.attr, 42)
 
 
-class Frozen_ELGetCodeTests(ExecutionLoaderGetCodeTests, unittest.TestCase):
-    ExecutionLoaderSubclass = Frozen_EL
-
-class Source_ELGetCodeTests(ExecutionLoaderGetCodeTests, unittest.TestCase):
-    ExecutionLoaderSubclass = Source_EL
+(Frozen_ELGetCodeTests,
+ Source_ELGetCodeTests
+ ) = test_util.test_both(ExecutionLoaderGetCodeTests,
+                         ExecutionLoaderSubclass=SPLIT_EL)
 
 
 ##### SourceLoader concrete methods ############################################
-class SourceLoader:
+class SourceOnlyLoader:
 
     # Globals that should be defined for all modules.
     source = (b"_ = '::'.join([__name__, __file__, __cached__, __package__, "
@@ -637,10 +642,10 @@
         return '<module>'
 
 
-Frozen_SourceOnlyL, Source_SourceOnlyL = make_abc_subclasses(SourceLoader)
+SPLIT_SOL = make_abc_subclasses(SourceOnlyLoader, 'SourceLoader')
 
 
-class SourceLoader(SourceLoader):
+class SourceLoader(SourceOnlyLoader):
 
     source_mtime = 1
 
@@ -677,11 +682,7 @@
         return path == self.bytecode_path
 
 
-Frozen_SL, Source_SL = make_abc_subclasses(SourceLoader)
-Frozen_SL.util = frozen_util
-Source_SL.util = source_util
-Frozen_SL.init = frozen_init
-Source_SL.init = source_init
+SPLIT_SL = make_abc_subclasses(SourceLoader, util=util, init=init)
 
 
 class SourceLoaderTestHarness:
@@ -765,7 +766,7 @@
         # Loading a module should set __name__, __loader__, __package__,
         # __path__ (for packages), __file__, and __cached__.
         # The module should also be put into sys.modules.
-        with util.uncache(self.name):
+        with test_util.uncache(self.name):
             with warnings.catch_warnings():
                 warnings.simplefilter('ignore', DeprecationWarning)
                 module = self.loader.load_module(self.name)
@@ -778,7 +779,7 @@
         # is a package.
         # Testing the values for a package are covered by test_load_module.
         self.setUp(is_package=False)
-        with util.uncache(self.name):
+        with test_util.uncache(self.name):
             with warnings.catch_warnings():
                 warnings.simplefilter('ignore', DeprecationWarning)
                 module = self.loader.load_module(self.name)
@@ -798,13 +799,10 @@
         self.assertEqual(returned_source, source)
 
 
-class Frozen_SourceOnlyLTests(SourceOnlyLoaderTests, unittest.TestCase):
-    loader_mock = Frozen_SourceOnlyL
-    util = frozen_util
-
-class Source_SourceOnlyLTests(SourceOnlyLoaderTests, unittest.TestCase):
-    loader_mock = Source_SourceOnlyL
-    util = source_util
+(Frozen_SourceOnlyLoaderTests,
+ Source_SourceOnlyLoaderTests
+ ) = test_util.test_both(SourceOnlyLoaderTests, util=util,
+                         loader_mock=SPLIT_SOL)
 
 
 @unittest.skipIf(sys.dont_write_bytecode, "sys.dont_write_bytecode is true")
@@ -896,15 +894,10 @@
         self.verify_code(code_object)
 
 
-class Frozen_SLBytecodeTests(SourceLoaderBytecodeTests, unittest.TestCase):
-    loader_mock = Frozen_SL
-    init = frozen_init
-    util = frozen_util
-
-class SourceSLBytecodeTests(SourceLoaderBytecodeTests, unittest.TestCase):
-    loader_mock = Source_SL
-    init = source_init
-    util = source_util
+(Frozen_SLBytecodeTests,
+ SourceSLBytecodeTests
+ ) = test_util.test_both(SourceLoaderBytecodeTests, init=init, util=util,
+                         loader_mock=SPLIT_SL)
 
 
 class SourceLoaderGetSourceTests:
@@ -940,11 +933,10 @@
         self.assertEqual(mock.get_source(name), expect)
 
 
-class Frozen_SourceOnlyLGetSourceTests(SourceLoaderGetSourceTests, unittest.TestCase):
-    SourceOnlyLoaderMock = Frozen_SourceOnlyL
-
-class Source_SourceOnlyLGetSourceTests(SourceLoaderGetSourceTests, unittest.TestCase):
-    SourceOnlyLoaderMock = Source_SourceOnlyL
+(Frozen_SourceOnlyLoaderGetSourceTests,
+ Source_SourceOnlyLoaderGetSourceTests
+ ) = test_util.test_both(SourceLoaderGetSourceTests,
+                         SourceOnlyLoaderMock=SPLIT_SOL)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/test_api.py b/Lib/test/test_importlib/test_api.py
--- a/Lib/test/test_importlib/test_api.py
+++ b/Lib/test/test_importlib/test_api.py
@@ -1,8 +1,8 @@
-from . import util
+from . import util as test_util
 
-frozen_init, source_init = util.import_importlib('importlib')
-frozen_util, source_util = util.import_importlib('importlib.util')
-frozen_machinery, source_machinery = util.import_importlib('importlib.machinery')
+init = test_util.import_importlib('importlib')
+util = test_util.import_importlib('importlib.util')
+machinery = test_util.import_importlib('importlib.machinery')
 
 import os.path
 import sys
@@ -18,8 +18,8 @@
 
     def test_module_import(self):
         # Test importing a top-level module.
-        with util.mock_modules('top_level') as mock:
-            with util.import_state(meta_path=[mock]):
+        with test_util.mock_modules('top_level') as mock:
+            with test_util.import_state(meta_path=[mock]):
                 module = self.init.import_module('top_level')
                 self.assertEqual(module.__name__, 'top_level')
 
@@ -28,8 +28,8 @@
         pkg_name = 'pkg'
         pkg_long_name = '{0}.__init__'.format(pkg_name)
         name = '{0}.mod'.format(pkg_name)
-        with util.mock_modules(pkg_long_name, name) as mock:
-            with util.import_state(meta_path=[mock]):
+        with test_util.mock_modules(pkg_long_name, name) as mock:
+            with test_util.import_state(meta_path=[mock]):
                 module = self.init.import_module(name)
                 self.assertEqual(module.__name__, name)
 
@@ -40,16 +40,16 @@
         module_name = 'mod'
         absolute_name = '{0}.{1}'.format(pkg_name, module_name)
         relative_name = '.{0}'.format(module_name)
-        with util.mock_modules(pkg_long_name, absolute_name) as mock:
-            with util.import_state(meta_path=[mock]):
+        with test_util.mock_modules(pkg_long_name, absolute_name) as mock:
+            with test_util.import_state(meta_path=[mock]):
                 self.init.import_module(pkg_name)
                 module = self.init.import_module(relative_name, pkg_name)
                 self.assertEqual(module.__name__, absolute_name)
 
     def test_deep_relative_package_import(self):
         modules = ['a.__init__', 'a.b.__init__', 'a.c']
-        with util.mock_modules(*modules) as mock:
-            with util.import_state(meta_path=[mock]):
+        with test_util.mock_modules(*modules) as mock:
+            with test_util.import_state(meta_path=[mock]):
                 self.init.import_module('a')
                 self.init.import_module('a.b')
                 module = self.init.import_module('..c', 'a.b')
@@ -61,8 +61,8 @@
         pkg_name = 'pkg'
         pkg_long_name = '{0}.__init__'.format(pkg_name)
         name = '{0}.mod'.format(pkg_name)
-        with util.mock_modules(pkg_long_name, name) as mock:
-            with util.import_state(meta_path=[mock]):
+        with test_util.mock_modules(pkg_long_name, name) as mock:
+            with test_util.import_state(meta_path=[mock]):
                 self.init.import_module(pkg_name)
                 module = self.init.import_module(name, pkg_name)
                 self.assertEqual(module.__name__, name)
@@ -86,16 +86,15 @@
             b_load_count += 1
         code = {'a': load_a, 'a.b': load_b}
         modules = ['a.__init__', 'a.b']
-        with util.mock_modules(*modules, module_code=code) as mock:
-            with util.import_state(meta_path=[mock]):
+        with test_util.mock_modules(*modules, module_code=code) as mock:
+            with test_util.import_state(meta_path=[mock]):
                 self.init.import_module('a.b')
         self.assertEqual(b_load_count, 1)
 
-class Frozen_ImportModuleTests(ImportModuleTests, unittest.TestCase):
-    init = frozen_init
 
-class Source_ImportModuleTests(ImportModuleTests, unittest.TestCase):
-    init = source_init
+(Frozen_ImportModuleTests,
+ Source_ImportModuleTests
+ ) = test_util.test_both(ImportModuleTests, init=init)
 
 
 class FindLoaderTests:
@@ -107,7 +106,7 @@
     def test_sys_modules(self):
         # If a module with __loader__ is in sys.modules, then return it.
         name = 'some_mod'
-        with util.uncache(name):
+        with test_util.uncache(name):
             module = types.ModuleType(name)
             loader = 'a loader!'
             module.__loader__ = loader
@@ -120,7 +119,7 @@
     def test_sys_modules_loader_is_None(self):
         # If sys.modules[name].__loader__ is None, raise ValueError.
         name = 'some_mod'
-        with util.uncache(name):
+        with test_util.uncache(name):
             module = types.ModuleType(name)
             module.__loader__ = None
             sys.modules[name] = module
@@ -133,7 +132,7 @@
         # Should raise ValueError
         # Issue #17099
         name = 'some_mod'
-        with util.uncache(name):
+        with test_util.uncache(name):
             module = types.ModuleType(name)
             try:
                 del module.__loader__
@@ -148,8 +147,8 @@
     def test_success(self):
         # Return the loader found on sys.meta_path.
         name = 'some_mod'
-        with util.uncache(name):
-            with util.import_state(meta_path=[self.FakeMetaFinder]):
+        with test_util.uncache(name):
+            with test_util.import_state(meta_path=[self.FakeMetaFinder]):
                 with warnings.catch_warnings():
                     warnings.simplefilter('ignore', DeprecationWarning)
                     self.assertEqual((name, None), self.init.find_loader(name))
@@ -158,8 +157,8 @@
         # Searching on a path should work.
         name = 'some_mod'
         path = 'path to some place'
-        with util.uncache(name):
-            with util.import_state(meta_path=[self.FakeMetaFinder]):
+        with test_util.uncache(name):
+            with test_util.import_state(meta_path=[self.FakeMetaFinder]):
                 with warnings.catch_warnings():
                     warnings.simplefilter('ignore', DeprecationWarning)
                     self.assertEqual((name, path),
@@ -171,11 +170,10 @@
             warnings.simplefilter('ignore', DeprecationWarning)
             self.assertIsNone(self.init.find_loader('nevergoingtofindthismodule'))
 
-class Frozen_FindLoaderTests(FindLoaderTests, unittest.TestCase):
-    init = frozen_init
 
-class Source_FindLoaderTests(FindLoaderTests, unittest.TestCase):
-    init = source_init
+(Frozen_FindLoaderTests,
+ Source_FindLoaderTests
+ ) = test_util.test_both(FindLoaderTests, init=init)
 
 
 class ReloadTests:
@@ -195,10 +193,10 @@
             module = type(sys)('top_level')
             module.spam = 3
             sys.modules['top_level'] = module
-        mock = util.mock_modules('top_level',
-                                 module_code={'top_level': code})
+        mock = test_util.mock_modules('top_level',
+                                      module_code={'top_level': code})
         with mock:
-            with util.import_state(meta_path=[mock]):
+            with test_util.import_state(meta_path=[mock]):
                 module = self.init.import_module('top_level')
                 reloaded = self.init.reload(module)
                 actual = sys.modules['top_level']
@@ -230,7 +228,7 @@
     def test_reload_location_changed(self):
         name = 'spam'
         with support.temp_cwd(None) as cwd:
-            with util.uncache('spam'):
+            with test_util.uncache('spam'):
                 with support.DirsOnSysPath(cwd):
                     # Start as a plain module.
                     self.init.invalidate_caches()
@@ -281,7 +279,7 @@
     def test_reload_namespace_changed(self):
         name = 'spam'
         with support.temp_cwd(None) as cwd:
-            with util.uncache('spam'):
+            with test_util.uncache('spam'):
                 with support.DirsOnSysPath(cwd):
                     # Start as a namespace package.
                     self.init.invalidate_caches()
@@ -338,20 +336,16 @@
         # See #19851.
         name = 'spam'
         subname = 'ham'
-        with util.temp_module(name, pkg=True) as pkg_dir:
-            fullname, _ = util.submodule(name, subname, pkg_dir)
+        with test_util.temp_module(name, pkg=True) as pkg_dir:
+            fullname, _ = test_util.submodule(name, subname, pkg_dir)
             ham = self.init.import_module(fullname)
             reloaded = self.init.reload(ham)
             self.assertIs(reloaded, ham)
 
 
-class Frozen_ReloadTests(ReloadTests, unittest.TestCase):
-    init = frozen_init
-    util = frozen_util
-
-class Source_ReloadTests(ReloadTests, unittest.TestCase):
-    init = source_init
-    util = source_util
+(Frozen_ReloadTests,
+ Source_ReloadTests
+ ) = test_util.test_both(ReloadTests, init=init, util=util)
 
 
 class InvalidateCacheTests:
@@ -384,11 +378,10 @@
         self.addCleanup(lambda: sys.path_importer_cache.__delitem__(key))
         self.init.invalidate_caches()  # Shouldn't trigger an exception.
 
-class Frozen_InvalidateCacheTests(InvalidateCacheTests, unittest.TestCase):
-    init = frozen_init
 
-class Source_InvalidateCacheTests(InvalidateCacheTests, unittest.TestCase):
-    init = source_init
+(Frozen_InvalidateCacheTests,
+ Source_InvalidateCacheTests
+ ) = test_util.test_both(InvalidateCacheTests, init=init)
 
 
 class FrozenImportlibTests(unittest.TestCase):
@@ -398,6 +391,7 @@
         # Can't do an isinstance() check since separate copies of importlib
         # may have been used for import, so just check the name is not for the
         # frozen loader.
+        source_init = init['Source']
         self.assertNotEqual(source_init.__loader__.__class__.__name__,
                             'FrozenImporter')
 
@@ -426,11 +420,10 @@
                     elif self.machinery.FrozenImporter.find_module(name):
                         self.assertIsNot(module.__spec__, None)
 
-class Frozen_StartupTests(StartupTests, unittest.TestCase):
-    machinery = frozen_machinery
 
-class Source_StartupTests(StartupTests, unittest.TestCase):
-    machinery = source_machinery
+(Frozen_StartupTests,
+ Source_StartupTests
+ ) = test_util.test_both(StartupTests, machinery=machinery)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/test_locks.py b/Lib/test/test_importlib/test_locks.py
--- a/Lib/test/test_importlib/test_locks.py
+++ b/Lib/test/test_importlib/test_locks.py
@@ -1,7 +1,6 @@
-from . import util
-frozen_init, source_init = util.import_importlib('importlib')
-frozen_bootstrap = frozen_init._bootstrap
-source_bootstrap = source_init._bootstrap
+from . import util as test_util
+
+init = test_util.import_importlib('importlib')
 
 import sys
 import time
@@ -33,13 +32,16 @@
         # _release_save() unsupported
         test_release_save_unacquired = None
 
-    class Frozen_ModuleLockAsRLockTests(ModuleLockAsRLockTests, lock_tests.RLockTests):
-        LockType = frozen_bootstrap._ModuleLock
+    LOCK_TYPES = {kind: splitinit._bootstrap._ModuleLock
+                  for kind, splitinit in init.items()}
 
-    class Source_ModuleLockAsRLockTests(ModuleLockAsRLockTests, lock_tests.RLockTests):
-        LockType = source_bootstrap._ModuleLock
+    (Frozen_ModuleLockAsRLockTests,
+     Source_ModuleLockAsRLockTests
+     ) = test_util.test_both(ModuleLockAsRLockTests, lock_tests.RLockTests,
+                             LockType=LOCK_TYPES)
+else:
+    LOCK_TYPES = {}
 
-else:
     class Frozen_ModuleLockAsRLockTests(unittest.TestCase):
         pass
 
@@ -47,6 +49,7 @@
         pass
 
 
+ at unittest.skipUnless(threading, "threads needed for this test")
 class DeadlockAvoidanceTests:
 
     def setUp(self):
@@ -106,19 +109,22 @@
         self.assertEqual(results.count((True, False)), 0)
         self.assertEqual(results.count((True, True)), len(results))
 
- at unittest.skipUnless(threading, "threads needed for this test")
-class Frozen_DeadlockAvoidanceTests(DeadlockAvoidanceTests, unittest.TestCase):
-    LockType = frozen_bootstrap._ModuleLock
-    DeadlockError = frozen_bootstrap._DeadlockError
 
- at unittest.skipUnless(threading, "threads needed for this test")
-class Source_DeadlockAvoidanceTests(DeadlockAvoidanceTests, unittest.TestCase):
-    LockType = source_bootstrap._ModuleLock
-    DeadlockError = source_bootstrap._DeadlockError
+DEADLOCK_ERRORS = {kind: splitinit._bootstrap._DeadlockError
+                   for kind, splitinit in init.items()}
+
+(Frozen_DeadlockAvoidanceTests,
+ Source_DeadlockAvoidanceTests
+ ) = test_util.test_both(DeadlockAvoidanceTests,
+                         LockType=LOCK_TYPES, DeadlockError=DEADLOCK_ERRORS)
 
 
 class LifetimeTests:
 
+    @property
+    def bootstrap(self):
+        return self.init._bootstrap
+
     def test_lock_lifetime(self):
         name = "xyzzy"
         self.assertNotIn(name, self.bootstrap._module_locks)
@@ -135,11 +141,10 @@
         self.assertEqual(0, len(self.bootstrap._module_locks),
                          self.bootstrap._module_locks)
 
-class Frozen_LifetimeTests(LifetimeTests, unittest.TestCase):
-    bootstrap = frozen_bootstrap
 
-class Source_LifetimeTests(LifetimeTests, unittest.TestCase):
-    bootstrap = source_bootstrap
+(Frozen_LifetimeTests,
+ Source_LifetimeTests
+ ) = test_util.test_both(LifetimeTests, init=init)
 
 
 @support.reap_threads
diff --git a/Lib/test/test_importlib/test_spec.py b/Lib/test/test_importlib/test_spec.py
--- a/Lib/test/test_importlib/test_spec.py
+++ b/Lib/test/test_importlib/test_spec.py
@@ -1,10 +1,8 @@
-from . import util
+from . import util as test_util
 
-frozen_init, source_init = util.import_importlib('importlib')
-frozen_bootstrap = frozen_init._bootstrap
-source_bootstrap = source_init._bootstrap
-frozen_machinery, source_machinery = util.import_importlib('importlib.machinery')
-frozen_util, source_util = util.import_importlib('importlib.util')
+init = test_util.import_importlib('importlib')
+machinery = test_util.import_importlib('importlib.machinery')
+util = test_util.import_importlib('importlib.util')
 
 import os.path
 from test.support import CleanImport
@@ -52,6 +50,8 @@
     with warnings.catch_warnings():
         warnings.simplefilter("ignore", DeprecationWarning)
 
+        frozen_util = util['Frozen']
+
         @frozen_util.module_for_loader
         def load_module(self, module):
             module.ham = self.HAM
@@ -221,18 +221,17 @@
         self.assertEqual(self.loc_spec.cached, 'spam.pyc')
 
 
-class Frozen_ModuleSpecTests(ModuleSpecTests, unittest.TestCase):
-    util = frozen_util
-    machinery = frozen_machinery
-
-
-class Source_ModuleSpecTests(ModuleSpecTests, unittest.TestCase):
-    util = source_util
-    machinery = source_machinery
+(Frozen_ModuleSpecTests,
+ Source_ModuleSpecTests
+ ) = test_util.test_both(ModuleSpecTests, util=util, machinery=machinery)
 
 
 class ModuleSpecMethodsTests:
 
+    @property
+    def bootstrap(self):
+        return self.init._bootstrap
+
     def setUp(self):
         self.name = 'spam'
         self.path = 'spam.py'
@@ -528,20 +527,18 @@
         self.assertIs(installed, loaded)
 
 
-class Frozen_ModuleSpecMethodsTests(ModuleSpecMethodsTests, unittest.TestCase):
-    bootstrap = frozen_bootstrap
-    machinery = frozen_machinery
-    util = frozen_util
-
-
-class Source_ModuleSpecMethodsTests(ModuleSpecMethodsTests, unittest.TestCase):
-    bootstrap = source_bootstrap
-    machinery = source_machinery
-    util = source_util
+(Frozen_ModuleSpecMethodsTests,
+ Source_ModuleSpecMethodsTests
+ ) = test_util.test_both(ModuleSpecMethodsTests, init=init, util=util,
+                         machinery=machinery)
 
 
 class ModuleReprTests:
 
+    @property
+    def bootstrap(self):
+        return self.init._bootstrap
+
     def setUp(self):
         self.module = type(os)('spam')
         self.spec = self.machinery.ModuleSpec('spam', TestLoader())
@@ -625,16 +622,10 @@
         self.assertEqual(modrepr, '<module {!r}>'.format('spam'))
 
 
-class Frozen_ModuleReprTests(ModuleReprTests, unittest.TestCase):
-    bootstrap = frozen_bootstrap
-    machinery = frozen_machinery
-    util = frozen_util
-
-
-class Source_ModuleReprTests(ModuleReprTests, unittest.TestCase):
-    bootstrap = source_bootstrap
-    machinery = source_machinery
-    util = source_util
+(Frozen_ModuleReprTests,
+ Source_ModuleReprTests
+ ) = test_util.test_both(ModuleReprTests, init=init, util=util,
+                         machinery=machinery)
 
 
 class FactoryTests:
@@ -787,7 +778,7 @@
     # spec_from_file_location()
 
     def test_spec_from_file_location_default(self):
-        if self.machinery is source_machinery:
+        if self.machinery is machinery['Source']:
             raise unittest.SkipTest('not sure why this is breaking...')
         spec = self.util.spec_from_file_location(self.name, self.path)
 
@@ -947,11 +938,6 @@
         self.assertTrue(spec.has_location)
 
 
-class Frozen_FactoryTests(FactoryTests, unittest.TestCase):
-    util = frozen_util
-    machinery = frozen_machinery
-
-
-class Source_FactoryTests(FactoryTests, unittest.TestCase):
-    util = source_util
-    machinery = source_machinery
+(Frozen_FactoryTests,
+ Source_FactoryTests
+ ) = test_util.test_both(FactoryTests, util=util, machinery=machinery)
diff --git a/Lib/test/test_importlib/test_util.py b/Lib/test/test_importlib/test_util.py
--- a/Lib/test/test_importlib/test_util.py
+++ b/Lib/test/test_importlib/test_util.py
@@ -1,8 +1,8 @@
-from importlib import util
+import importlib.util
 from . import util as test_util
-frozen_init, source_init = test_util.import_importlib('importlib')
-frozen_machinery, source_machinery = test_util.import_importlib('importlib.machinery')
-frozen_util, source_util = test_util.import_importlib('importlib.util')
+init = test_util.import_importlib('importlib')
+machinery = test_util.import_importlib('importlib.machinery')
+util = test_util.import_importlib('importlib.util')
 
 import os
 import sys
@@ -32,8 +32,10 @@
         self.assertEqual(self.util.decode_source(source_bytes),
                          '\n'.join([self.source, self.source]))
 
-Frozen_DecodeSourceBytesTests, Source_DecodeSourceBytesTests = test_util.test_both(
-        DecodeSourceBytesTests, util=[frozen_util, source_util])
+
+(Frozen_DecodeSourceBytesTests,
+ Source_DecodeSourceBytesTests
+ ) = test_util.test_both(DecodeSourceBytesTests, util=util)
 
 
 class ModuleForLoaderTests:
@@ -161,8 +163,10 @@
             self.assertIs(module.__loader__, loader)
             self.assertEqual(module.__package__, name)
 
-Frozen_ModuleForLoaderTests, Source_ModuleForLoaderTests = test_util.test_both(
-        ModuleForLoaderTests, util=[frozen_util, source_util])
+
+(Frozen_ModuleForLoaderTests,
+ Source_ModuleForLoaderTests
+ ) = test_util.test_both(ModuleForLoaderTests, util=util)
 
 
 class SetPackageTests:
@@ -222,18 +226,25 @@
         self.assertEqual(wrapped.__name__, fxn.__name__)
         self.assertEqual(wrapped.__qualname__, fxn.__qualname__)
 
-Frozen_SetPackageTests, Source_SetPackageTests = test_util.test_both(
-        SetPackageTests, util=[frozen_util, source_util])
+
+(Frozen_SetPackageTests,
+ Source_SetPackageTests
+ ) = test_util.test_both(SetPackageTests, util=util)
 
 
 class SetLoaderTests:
 
     """Tests importlib.util.set_loader()."""
 
-    class DummyLoader:
-        @util.set_loader
-        def load_module(self, module):
-            return self.module
+    @property
+    def DummyLoader(self):
+        # Set DummyLoader on the class lazily.
+        class DummyLoader:
+            @self.util.set_loader
+            def load_module(self, module):
+                return self.module
+        self.__class__.DummyLoader = DummyLoader
+        return DummyLoader
 
     def test_no_attribute(self):
         loader = self.DummyLoader()
@@ -262,17 +273,10 @@
             warnings.simplefilter('ignore', DeprecationWarning)
             self.assertEqual(42, loader.load_module('blah').__loader__)
 
-class Frozen_SetLoaderTests(SetLoaderTests, unittest.TestCase):
-    class DummyLoader:
-        @frozen_util.set_loader
-        def load_module(self, module):
-            return self.module
 
-class Source_SetLoaderTests(SetLoaderTests, unittest.TestCase):
-    class DummyLoader:
-        @source_util.set_loader
-        def load_module(self, module):
-            return self.module
+(Frozen_SetLoaderTests,
+ Source_SetLoaderTests
+ ) = test_util.test_both(SetLoaderTests, util=util)
 
 
 class ResolveNameTests:
@@ -307,9 +311,10 @@
         with self.assertRaises(ValueError):
             self.util.resolve_name('..bacon', 'spam')
 
-Frozen_ResolveNameTests, Source_ResolveNameTests = test_util.test_both(
-        ResolveNameTests,
-        util=[frozen_util, source_util])
+
+(Frozen_ResolveNameTests,
+ Source_ResolveNameTests
+ ) = test_util.test_both(ResolveNameTests, util=util)
 
 
 class FindSpecTests:
@@ -446,15 +451,10 @@
             self.assertNotIn(fullname, sorted(sys.modules))
 
 
-class Frozen_FindSpecTests(FindSpecTests, unittest.TestCase):
-    init = frozen_init
-    machinery = frozen_machinery
-    util = frozen_util
-
-class Source_FindSpecTests(FindSpecTests, unittest.TestCase):
-    init = source_init
-    machinery = source_machinery
-    util = source_util
+(Frozen_FindSpecTests,
+ Source_FindSpecTests
+ ) = test_util.test_both(FindSpecTests, init=init, util=util,
+                         machinery=machinery)
 
 
 class MagicNumberTests:
@@ -467,8 +467,10 @@
         # The magic number uses \r\n to come out wrong when splitting on lines.
         self.assertTrue(self.util.MAGIC_NUMBER.endswith(b'\r\n'))
 
-Frozen_MagicNumberTests, Source_MagicNumberTests = test_util.test_both(
-        MagicNumberTests, util=[frozen_util, source_util])
+
+(Frozen_MagicNumberTests,
+ Source_MagicNumberTests
+ ) = test_util.test_both(MagicNumberTests, util=util)
 
 
 class PEP3147Tests:
@@ -583,9 +585,10 @@
             ValueError, self.util.source_from_cache,
             '/foo/bar/foo.cpython-32.foo.pyc')
 
-Frozen_PEP3147Tests, Source_PEP3147Tests = test_util.test_both(
-        PEP3147Tests,
-        util=[frozen_util, source_util])
+
+(Frozen_PEP3147Tests,
+ Source_PEP3147Tests
+ ) = test_util.test_both(PEP3147Tests, util=util)
 
 
 if __name__ == '__main__':
diff --git a/Lib/test/test_importlib/test_windows.py b/Lib/test/test_importlib/test_windows.py
--- a/Lib/test/test_importlib/test_windows.py
+++ b/Lib/test/test_importlib/test_windows.py
@@ -1,5 +1,5 @@
-from . import util
-frozen_machinery, source_machinery = util.import_importlib('importlib.machinery')
+from . import util as test_util
+machinery = test_util.import_importlib('importlib.machinery')
 
 import sys
 import unittest
@@ -19,11 +19,6 @@
         self.assertIs(loader, None)
 
 
-class Frozen_WindowsRegistryFinderTests(WindowsRegistryFinderTests,
-                                        unittest.TestCase):
-    machinery = frozen_machinery
-
-
-class Source_WindowsRegistryFinderTests(WindowsRegistryFinderTests,
-                                        unittest.TestCase):
-    machinery = source_machinery
+(Frozen_WindowsRegistryFinderTests,
+ Source_WindowsRegistryFinderTests
+ ) = test_util.test_both(WindowsRegistryFinderTests, machinery=machinery)
diff --git a/Lib/test/test_importlib/util.py b/Lib/test/test_importlib/util.py
--- a/Lib/test/test_importlib/util.py
+++ b/Lib/test/test_importlib/util.py
@@ -50,19 +50,36 @@
     frozen = support.import_fresh_module(module_name)
     source = support.import_fresh_module(module_name, fresh=fresh,
                                          blocked=('_frozen_importlib',))
+    return {'Frozen': frozen, 'Source': source}
+
+
+def specialize_class(cls, kind, base=None, **kwargs):
+    # XXX Support passing in submodule names--load (and cache) them?
+    # That would clean up the test modules a bit more.
+    if base is None:
+        base = unittest.TestCase
+    elif not isinstance(base, type):
+        base = base[kind]
+    name = '{}_{}'.format(kind, cls.__name__)
+    bases = (cls, base)
+    specialized = types.new_class(name, bases)
+    specialized.__module__ = cls.__module__
+    specialized._NAME = cls.__name__
+    specialized._KIND = kind
+    for attr, values in kwargs.items():
+        value = values[kind]
+        setattr(specialized, attr, value)
+    return specialized
+
+
+def split_frozen(cls, base=None, **kwargs):
+    frozen = specialize_class(cls, 'Frozen', base, **kwargs)
+    source = specialize_class(cls, 'Source', base, **kwargs)
     return frozen, source
 
 
-def test_both(test_class, **kwargs):
-    frozen_tests = types.new_class('Frozen_'+test_class.__name__,
-                                   (test_class, unittest.TestCase))
-    source_tests = types.new_class('Source_'+test_class.__name__,
-                                   (test_class, unittest.TestCase))
-    frozen_tests.__module__ = source_tests.__module__ = test_class.__module__
-    for attr, (frozen_value, source_value) in kwargs.items():
-        setattr(frozen_tests, attr, frozen_value)
-        setattr(source_tests, attr, source_value)
-    return frozen_tests, source_tests
+def test_both(test_class, base=None, **kwargs):
+    return split_frozen(test_class, base, **kwargs)
 
 
 CASE_INSENSITIVE_FS = True
@@ -75,8 +92,9 @@
     if not os.path.exists(changed_name):
         CASE_INSENSITIVE_FS = False
 
-_, source_importlib = import_importlib('importlib')
-__import__ = staticmethod(builtins.__import__), staticmethod(source_importlib.__import__)
+source_importlib = import_importlib('importlib')['Source']
+__import__ = {'Frozen': staticmethod(builtins.__import__),
+              'Source': staticmethod(source_importlib.__import__)}
 
 
 def case_insensitive_tests(test):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -496,6 +496,8 @@
 
 - Issue #21097: Move test_namespace_pkgs into test_importlib.
 
+- Issue #21503: Use test_both() consistently in test_importlib.
+
 - Issue #20939: Avoid various network test failures due to new
   redirect of http://www.python.org/ to https://www.python.org:
   use http://www.example.com instead.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat May 17 03:52:01 2014
From: python-checkins at python.org (senthil.kumaran)
Date: Sat, 17 May 2014 03:52:01 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Backport_Fix_f?=
 =?utf-8?q?or_Issue_=237776=3A_Fix_=60=60Host=3A=27=27_header_and_reconnec?=
 =?utf-8?q?tion_when_using?=
Message-ID: <3gVqH54fYjz7LjZ@mail.python.org>

http://hg.python.org/cpython/rev/568041fd8090
changeset:   90728:568041fd8090
branch:      2.7
parent:      90722:0a6d51ccff54
user:        Senthil Kumaran <senthil at uthcode.com>
date:        Fri May 16 18:51:46 2014 -0700
summary:
  Backport Fix for Issue #7776: Fix ``Host:'' header and reconnection when using http.client.HTTPConnection.set_tunnel().

Patch by Nikolaus Rath.

files:
  Lib/httplib.py           |  54 +++++++++++++++++++--------
  Lib/test/test_httplib.py |  48 +++++++++++++++++++++++-
  Misc/NEWS                |   4 ++
  3 files changed, 88 insertions(+), 18 deletions(-)


diff --git a/Lib/httplib.py b/Lib/httplib.py
--- a/Lib/httplib.py
+++ b/Lib/httplib.py
@@ -700,17 +700,33 @@
         self._tunnel_host = None
         self._tunnel_port = None
         self._tunnel_headers = {}
-
-        self._set_hostport(host, port)
         if strict is not None:
             self.strict = strict
 
+        (self.host, self.port) = self._get_hostport(host, port)
+
+        # This is stored as an instance variable to allow unittests
+        # to replace with a suitable mock
+        self._create_connection = socket.create_connection
+
     def set_tunnel(self, host, port=None, headers=None):
-        """ Sets up the host and the port for the HTTP CONNECT Tunnelling.
+        """ Set up host and port for HTTP CONNECT tunnelling.
+
+        In a connection that uses HTTP Connect tunneling, the host passed to the
+        constructor is used as proxy server that relays all communication to the
+        endpoint passed to set_tunnel. This is done by sending a HTTP CONNECT
+        request to the proxy server when the connection is established.
+
+        This method must be called before the HTML connection has been
+        established.
 
         The headers argument should be a mapping of extra HTTP headers
         to send with the CONNECT request.
         """
+        # Verify if this is required.
+        if self.sock:
+            raise RuntimeError("Can't setup tunnel for established connection.")
+
         self._tunnel_host = host
         self._tunnel_port = port
         if headers:
@@ -718,7 +734,7 @@
         else:
             self._tunnel_headers.clear()
 
-    def _set_hostport(self, host, port):
+    def _get_hostport(self, host, port):
         if port is None:
             i = host.rfind(':')
             j = host.rfind(']')         # ipv6 addresses have [...]
@@ -735,15 +751,14 @@
                 port = self.default_port
             if host and host[0] == '[' and host[-1] == ']':
                 host = host[1:-1]
-        self.host = host
-        self.port = port
+        return (host, port)
 
     def set_debuglevel(self, level):
         self.debuglevel = level
 
     def _tunnel(self):
-        self._set_hostport(self._tunnel_host, self._tunnel_port)
-        self.send("CONNECT %s:%d HTTP/1.0\r\n" % (self.host, self.port))
+        (host, port) = self._get_hostport(self._tunnel_host, self._tunnel_port)
+        self.send("CONNECT %s:%d HTTP/1.0\r\n" % (host, port))
         for header, value in self._tunnel_headers.iteritems():
             self.send("%s: %s\r\n" % (header, value))
         self.send("\r\n")
@@ -768,8 +783,8 @@
 
     def connect(self):
         """Connect to the host and port specified in __init__."""
-        self.sock = socket.create_connection((self.host,self.port),
-                                             self.timeout, self.source_address)
+        self.sock = self._create_connection((self.host,self.port),
+                                           self.timeout, self.source_address)
 
         if self._tunnel_host:
             self._tunnel()
@@ -907,17 +922,24 @@
                         netloc_enc = netloc.encode("idna")
                     self.putheader('Host', netloc_enc)
                 else:
+                    if self._tunnel_host:
+                        host = self._tunnel_host
+                        port = self._tunnel_port
+                    else:
+                        host = self.host
+                        port = self.port
+
                     try:
-                        host_enc = self.host.encode("ascii")
+                        host_enc = host.encode("ascii")
                     except UnicodeEncodeError:
-                        host_enc = self.host.encode("idna")
+                        host_enc = host.encode("idna")
                     # Wrap the IPv6 Host Header with [] (RFC 2732)
                     if host_enc.find(':') >= 0:
                         host_enc = "[" + host_enc + "]"
-                    if self.port == self.default_port:
+                    if port == self.default_port:
                         self.putheader('Host', host_enc)
                     else:
-                        self.putheader('Host', "%s:%s" % (host_enc, self.port))
+                        self.putheader('Host', "%s:%s" % (host_enc, port))
 
             # note: we are assuming that clients will not attempt to set these
             #       headers since *this* library must deal with the
@@ -1168,8 +1190,8 @@
         def connect(self):
             "Connect to a host on a given (SSL) port."
 
-            sock = socket.create_connection((self.host, self.port),
-                                            self.timeout, self.source_address)
+            sock = self._create_connection((self.host, self.port),
+                                          self.timeout, self.source_address)
             if self._tunnel_host:
                 self.sock = sock
                 self._tunnel()
diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py
--- a/Lib/test/test_httplib.py
+++ b/Lib/test/test_httplib.py
@@ -13,10 +13,12 @@
 HOST = test_support.HOST
 
 class FakeSocket:
-    def __init__(self, text, fileclass=StringIO.StringIO):
+    def __init__(self, text, fileclass=StringIO.StringIO, host=None, port=None):
         self.text = text
         self.fileclass = fileclass
         self.data = ''
+        self.host = host
+        self.port = port
 
     def sendall(self, data):
         self.data += ''.join(data)
@@ -26,6 +28,9 @@
             raise httplib.UnimplementedFileMode()
         return self.fileclass(self.text)
 
+    def close(self):
+        pass
+
 class EPipeSocket(FakeSocket):
 
     def __init__(self, text, pipe_trigger):
@@ -526,9 +531,48 @@
                 self.fail("Port incorrectly parsed: %s != %s" % (p, c.host))
 
 
+class TunnelTests(TestCase):
+    def test_connect(self):
+        response_text = (
+            'HTTP/1.0 200 OK\r\n\r\n'   # Reply to CONNECT
+            'HTTP/1.1 200 OK\r\n'       # Reply to HEAD
+            'Content-Length: 42\r\n\r\n'
+        )
+
+        def create_connection(address, timeout=None, source_address=None):
+            return FakeSocket(response_text, host=address[0], port=address[1])
+
+        conn = httplib.HTTPConnection('proxy.com')
+        conn._create_connection = create_connection
+
+        # Once connected, we should not be able to tunnel anymore
+        conn.connect()
+        self.assertRaises(RuntimeError, conn.set_tunnel, 'destination.com')
+
+        # But if close the connection, we are good.
+        conn.close()
+        conn.set_tunnel('destination.com')
+        conn.request('HEAD', '/', '')
+
+        self.assertEqual(conn.sock.host, 'proxy.com')
+        self.assertEqual(conn.sock.port, 80)
+        self.assertTrue('CONNECT destination.com' in conn.sock.data)
+        self.assertTrue('Host: destination.com' in conn.sock.data)
+
+        self.assertTrue('Host: proxy.com' not in conn.sock.data)
+
+        conn.close()
+
+        conn.request('PUT', '/', '')
+        self.assertEqual(conn.sock.host, 'proxy.com')
+        self.assertEqual(conn.sock.port, 80)
+        self.assertTrue('CONNECT destination.com' in conn.sock.data)
+        self.assertTrue('Host: destination.com' in conn.sock.data)
+
+
 def test_main(verbose=None):
     test_support.run_unittest(HeaderTests, OfflineTest, BasicTest, TimeoutTest,
-                              HTTPSTimeoutTest, SourceAddressTest)
+                              HTTPSTimeoutTest, SourceAddressTest, TunnelTests)
 
 if __name__ == '__main__':
     test_main()
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -49,6 +49,10 @@
 Library
 -------
 
+-  Issue #7776: Backport Fix ``Host:'' header and reconnection when using
+   http.client.HTTPConnection.set_tunnel() from Python 3.
+   Patch by Nikolaus Rath.
+
 - Issue #21306: Backport hmac.compare_digest from Python 3. This is part of PEP
   466.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat May 17 05:05:03 2014
From: python-checkins at python.org (donald.stufft)
Date: Sat, 17 May 2014 05:05:03 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Upgrade_setuptools_to_3=2E?=
	=?utf-8?q?6?=
Message-ID: <3gVrvM227sz7LjM@mail.python.org>

http://hg.python.org/cpython/rev/acb5cc616052
changeset:   90729:acb5cc616052
parent:      90727:34d65746d5ca
user:        Donald Stufft <donald at stufft.io>
date:        Fri May 16 23:01:28 2014 -0400
summary:
  Upgrade setuptools to 3.6

files:
  Lib/ensurepip/__init__.py                                  |    2 +-
  Lib/ensurepip/_bundled/setuptools-2.1-py2.py3-none-any.whl |  Bin 
  Lib/ensurepip/_bundled/setuptools-3.6-py2.py3-none-any.whl |  Bin 
  3 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py
--- a/Lib/ensurepip/__init__.py
+++ b/Lib/ensurepip/__init__.py
@@ -8,7 +8,7 @@
 __all__ = ["version", "bootstrap"]
 
 
-_SETUPTOOLS_VERSION = "2.1"
+_SETUPTOOLS_VERSION = "3.6"
 
 _PIP_VERSION = "1.5.4"
 
diff --git a/Lib/ensurepip/_bundled/setuptools-2.1-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-2.1-py2.py3-none-any.whl
deleted file mode 100644
index ed77b59e632f32d09a3ae52adaa7f3e6659d8b48..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
GIT binary patch
[stripped]
diff --git a/Lib/ensurepip/_bundled/setuptools-3.6-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-3.6-py2.py3-none-any.whl
new file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..f0ffcfce5bb385e393a8385413f7a6092c51b33e
GIT binary patch
[stripped]

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat May 17 05:05:04 2014
From: python-checkins at python.org (donald.stufft)
Date: Sat, 17 May 2014 05:05:04 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Upgrade_pip_to_1=2E5=2E6?=
Message-ID: <3gVrvN4wY5z7LjZ@mail.python.org>

http://hg.python.org/cpython/rev/308ff6a5ce67
changeset:   90730:308ff6a5ce67
user:        Donald Stufft <donald at stufft.io>
date:        Fri May 16 23:02:25 2014 -0400
summary:
  Upgrade pip to 1.5.6

files:
  Lib/ensurepip/__init__.py                             |    2 +-
  Lib/ensurepip/_bundled/pip-1.5.4-py2.py3-none-any.whl |  Bin 
  Lib/ensurepip/_bundled/pip-1.5.6-py2.py3-none-any.whl |  Bin 
  3 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py
--- a/Lib/ensurepip/__init__.py
+++ b/Lib/ensurepip/__init__.py
@@ -10,7 +10,7 @@
 
 _SETUPTOOLS_VERSION = "3.6"
 
-_PIP_VERSION = "1.5.4"
+_PIP_VERSION = "1.5.6"
 
 # pip currently requires ssl support, so we try to provide a nicer
 # error message when that is missing (http://bugs.python.org/issue19744)
diff --git a/Lib/ensurepip/_bundled/pip-1.5.4-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-1.5.4-py2.py3-none-any.whl
deleted file mode 100644
index e07d4767a0cc5d6367948bd18149602b2e050fd4..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
GIT binary patch
[stripped]
diff --git a/Lib/ensurepip/_bundled/pip-1.5.6-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-1.5.6-py2.py3-none-any.whl
new file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..097ab43430d4c1302b0be353a8c16407c370693b
GIT binary patch
[stripped]

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat May 17 22:52:36 2014
From: python-checkins at python.org (brian.quinlan)
Date: Sat, 17 May 2014 22:52:36 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2321362=3A_concurre?=
 =?utf-8?q?nt=2Efutures_does_not_validate_that_max=5Fworkers_is_proper?=
Message-ID: <3gWJb80tllz7LjP@mail.python.org>

http://hg.python.org/cpython/rev/3024ad49f00e
changeset:   90731:3024ad49f00e
user:        Brian Quinlan <brian at sweetapp.com>
date:        Sat May 17 13:51:10 2014 -0700
summary:
  Issue #21362: concurrent.futures does not validate that max_workers is proper

files:
  Doc/library/concurrent.futures.rst  |  2 ++
  Lib/concurrent/futures/process.py   |  3 +++
  Lib/concurrent/futures/thread.py    |  3 +++
  Lib/test/test_concurrent_futures.py |  7 +++++++
  4 files changed, 15 insertions(+), 0 deletions(-)


diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst
--- a/Doc/library/concurrent.futures.rst
+++ b/Doc/library/concurrent.futures.rst
@@ -175,6 +175,8 @@
    An :class:`Executor` subclass that executes calls asynchronously using a pool
    of at most *max_workers* processes.  If *max_workers* is ``None`` or not
    given, it will default to the number of processors on the machine.
+   If *max_workers* is lower or equal to ``0``, then a :exc:`ValueError`
+   will be raised.
 
    .. versionchanged:: 3.3
       When one of the worker processes terminates abruptly, a
diff --git a/Lib/concurrent/futures/process.py b/Lib/concurrent/futures/process.py
--- a/Lib/concurrent/futures/process.py
+++ b/Lib/concurrent/futures/process.py
@@ -334,6 +334,9 @@
         if max_workers is None:
             self._max_workers = os.cpu_count() or 1
         else:
+            if max_workers <= 0:
+                raise ValueError("max_workers must be greater than 0")
+
             self._max_workers = max_workers
 
         # Make the call queue slightly larger than the number of processes to
diff --git a/Lib/concurrent/futures/thread.py b/Lib/concurrent/futures/thread.py
--- a/Lib/concurrent/futures/thread.py
+++ b/Lib/concurrent/futures/thread.py
@@ -87,6 +87,9 @@
             max_workers: The maximum number of threads that can be used to
                 execute the given calls.
         """
+        if max_workers <= 0:
+            raise ValueError("max_workers must be greater than 0")
+
         self._max_workers = max_workers
         self._work_queue = queue.Queue()
         self._threads = set()
diff --git a/Lib/test/test_concurrent_futures.py b/Lib/test/test_concurrent_futures.py
--- a/Lib/test/test_concurrent_futures.py
+++ b/Lib/test/test_concurrent_futures.py
@@ -425,6 +425,13 @@
         self.assertTrue(collected,
                         "Stale reference not collected within timeout.")
 
+    def test_max_workers_negative(self):
+        for number in (0, -1):
+            with self.assertRaisesRegexp(ValueError,
+                                         "max_workers must be greater "
+                                         "than 0"):
+                self.executor_type(max_workers=number)
+
 
 class ThreadPoolExecutorTest(ThreadPoolMixin, ExecutorTest, unittest.TestCase):
     def test_map_submits_without_iteration(self):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat May 17 23:32:07 2014
From: python-checkins at python.org (ned.deily)
Date: Sat, 17 May 2014 23:32:07 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Document_chang?=
 =?utf-8?q?es_to_OS_X_installer_configurations_for_2=2E7=2E7=2E?=
Message-ID: <3gWKSl4tf5z7LjV@mail.python.org>

http://hg.python.org/cpython/rev/28ce7dd2070b
changeset:   90732:28ce7dd2070b
branch:      2.7
parent:      90728:568041fd8090
user:        Ned Deily <nad at acm.org>
date:        Sat May 17 14:29:22 2014 -0700
summary:
  Document changes to OS X installer configurations for 2.7.7.
As of 2.7.8, the 32-bit-only installer will support OS X 10.5
and later systems as is currently done for Python 3.x installers.

For 2.7.7 only, we will provide three installers:
the legacy deprecated 10.3+ 32-bit-only format;
the newer 10.5+ 32-bit-only format;
and the unchanged 10.6+ 64-/32-bit format.

Although binary installers will no longer be available from
python.org as of 2.7.8, it will still be possible to build from
source on 10.3.9 and 10.4 systems if necessary.

files:
  Mac/BuildScript/README.txt            |   58 ++-
  Mac/BuildScript/resources/ReadMe.txt  |   26 +
  Mac/BuildScript/resources/Welcome.rtf |   10 +-
  Mac/README                            |  231 +++++++++----
  Misc/NEWS                             |   14 +
  5 files changed, 242 insertions(+), 97 deletions(-)


diff --git a/Mac/BuildScript/README.txt b/Mac/BuildScript/README.txt
--- a/Mac/BuildScript/README.txt
+++ b/Mac/BuildScript/README.txt
@@ -8,11 +8,15 @@
 an Installer package from the installation plus other files in ``resources`` 
 and ``scripts`` and placed that on a ``.dmg`` disk image.
 
-For Python 2.7.x and 3.2.x, PSF practice is to build two installer variants
+For Python 2.7.x and 3.x, PSF practice is to build two installer variants
 for each release.
 
-1.  32-bit-only, i386 and PPC universal, capable on running on all machines
-    supported by Mac OS X 10.3.9 through (at least) 10.8::
+Beginning with Python 2.7.8, we plan to drop binary installer support for
+Mac OS X 10.3.9 and 10.4.x systems.  To ease the transition, for Python 2.7.7
+only there will be three installers provided:
+
+1.  DEPRECATED - 32-bit-only, i386 and PPC universal, capable on running on all
+    machines supported by Mac OS X 10.3.9 through (at least) 10.9::
 
         /usr/bin/python build-installer.py \
             --sdk-path=/Developer/SDKs/MacOSX10.4u.sdk \
@@ -45,8 +49,42 @@
             - need to change ``/System/Library/Frameworks/{Tcl,Tk}.framework/Version/Current`` to ``8.4``
         * Note Xcode 4.* does not support building for PPC so cannot be used for this build
 
+2.  32-bit-only, i386 and PPC universal, capable on running on all machines
+    supported by Mac OS X 10.5 through (at least) 10.9::
 
-2.  64-bit / 32-bit, x86_64 and i386 universal, for OS X 10.6 (and later)::
+        /usr/bin/python  build-installer.py \
+            --sdk-path=/Developer/SDKs/MacOSX10.5.sdk \
+            --universal-archs=32-bit \
+            --dep-target=10.5
+
+    - builds the following third-party libraries
+
+        * NCurses 5.9
+        * SQLite 3.7.13
+        * Oracle Sleepycat DB 4.8 (Python 2.x only)
+
+    - uses system-supplied versions of third-party libraries
+
+        * readline module links with Apple BSD editline (libedit)
+
+    - requires ActiveState ``Tcl/Tk 8.4`` (currently 8.4.20) to be installed for building
+
+    - recommended build environment:
+
+        * Mac OS X 10.5.8 Intel or PPC
+        * Xcode 3.1.4
+        * ``MacOSX10.5`` SDK
+        * ``MACOSX_DEPLOYMENT_TARGET=10.5``
+        * Apple ``gcc-4.2``
+        * system Python 2.5+ for documentation build with Sphinx
+
+    - alternate build environments:
+
+        * Mac OS X 10.6.8 with Xcode 3.2.6
+            - need to change ``/System/Library/Frameworks/{Tcl,Tk}.framework/Version/Current`` to ``8.4``
+        * Note Xcode 4.* does not support building for PPC so cannot be used for this build
+
+3.  64-bit / 32-bit, x86_64 and i386 universal, for OS X 10.6 (and later)::
 
         /usr/bin/python build-installer.py \
             --sdk-path=/Developer/SDKs/MacOSX10.6.sdk \
@@ -57,13 +95,13 @@
 
         * NCurses 5.9 (http://bugs.python.org/issue15037)
         * SQLite 3.7.13
+        * Oracle Sleepycat DB 4.8 (Python 2.x only)
 
     - uses system-supplied versions of third-party libraries
 
         * readline module links with Apple BSD editline (libedit)
-        * builds Oracle Sleepycat DB 4.8 (Python 2.x only)
 
-    - requires ActiveState Tcl/Tk 8.5.9 (or later) to be installed for building
+    - requires ActiveState Tcl/Tk 8.5.15 (or later) to be installed for building
 
     - recommended build environment:
 
@@ -82,10 +120,10 @@
           considered a migration aid by Apple and is not likely to be fixed,
           its use should be avoided.  The other compiler, ``clang``, has been
           undergoing rapid development.  While it appears to have become
-          production-ready in the most recent Xcode 4 releases (Xcode 4.5.x
-          as of this writing), there are still some open issues when
-          building Python and there has not yet been the level of exposure in
-          production environments that the Xcode 3 gcc-4.2 compiler has had.
+          production-ready in the most recent Xcode 5 releases, the versions
+          available on the deprecated Xcode 4.x for 10.6 were early releases
+          and did not receive the level of exposure in production environments
+          that the Xcode 3 gcc-4.2 compiler has had.
 
 
 General Prerequisites
diff --git a/Mac/BuildScript/resources/ReadMe.txt b/Mac/BuildScript/resources/ReadMe.txt
--- a/Mac/BuildScript/resources/ReadMe.txt
+++ b/Mac/BuildScript/resources/ReadMe.txt
@@ -28,6 +28,32 @@
 for current information about supported and recommended versions of
 Tcl/Tk for this version of Python and of Mac OS X.
 
+               **** IMPORTANT ****
+
+Binary installer support for 10.4 and 10.3.9 to be discontinued
+===============================================================
+
+Python 2.7.7 is the last release for which binary installers will be
+released on python.org that support OS X 10.3.9 (Panther) and 10.4.x
+(Tiger) systems.  These systems were last updated by Apple in 2005
+and 2007.  As of 2.7.8, the 32-bit-only installer will support PPC
+and Intel Macs running OS X 10.5 (Leopard) and later.  10.5 was the
+last OS X release for PPC machines (G4 and G5).  (The 64-/32-bit
+installer configuration will remain unchanged.)  This aligns Python
+2.7.x installer configurations with those currently provided with
+Python 3.x.  Some of the reasons for making this change are:
+there were significant additions and compatibility improvements to
+the OS X POSIX system APIs in OS X 10.5 that Python users can now
+take advantage of; it is increasingly difficult to build and test
+on obsolete 10.3 and 10.4 systems and with the 10.3 ABI; and it is
+assumed that most remaining legacy PPC systems have upgraded to 10.5.
+To ease the transition, for Python 2.7.7 only we are providing three
+binary installers: (1) the legacy deprecated 32-bit-only 10.3+
+PPC/Intel format, (2) the newer 32-bit-only 10.5+ PPC/Intel format,
+and (3) the current 64-bit/32-bit 10.6+ Intel-only format.  While
+future releases will not provide the deprecated installer, it will
+still be possible to build Python from source on 10.3.9 and 10.4
+systems if needed.
 
 Using this version of Python on OS X
 ====================================
diff --git a/Mac/BuildScript/resources/Welcome.rtf b/Mac/BuildScript/resources/Welcome.rtf
--- a/Mac/BuildScript/resources/Welcome.rtf
+++ b/Mac/BuildScript/resources/Welcome.rtf
@@ -1,8 +1,8 @@
-{\rtf1\ansi\ansicpg1252\cocoartf1038\cocoasubrtf350
-{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
+{\rtf1\ansi\ansicpg1252\cocoartf1265\cocoasubrtf200
+\cocoascreenfonts1{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
 {\colortbl;\red255\green255\blue255;}
-\paperw11904\paperh16836\margl1440\margr1440\vieww9640\viewh10620\viewkind0
-\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\ql\qnatural
+\paperw11905\paperh16837\margl1440\margr1440\vieww9640\viewh10620\viewkind0
+\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640
 
 \f0\fs24 \cf0 This package will install 
 \b Python $FULL_VERSION
@@ -16,7 +16,7 @@
 \b IDLE
 \b0  and a set of pre-built extension modules that open up specific Macintosh technologies to Python programs.\
 \
-See the ReadMe file and the Python documentation for more information.\
+See the ReadMe file and the Python documentation for important information, including the dropping of support for OS X 10.3.9 and 10.4 in future Python 2.7.x binary installers.\
 \
 
 \b IMPORTANT:
diff --git a/Mac/README b/Mac/README
--- a/Mac/README
+++ b/Mac/README
@@ -1,12 +1,19 @@
-============
-MacOSX Notes
-============
+=========================
+Python on Mac OS X README
+=========================
+
+:Authors:
+    Jack Jansen (2004-07),
+    Ronald Oussoren (2010-04),
+    Ned Deily (2014-05)
+
+:Version: 2.7.7
 
 This document provides a quick overview of some Mac OS X specific features in
 the Python distribution.
 
-Mac-specific arguments to configure
-===================================
+OS X specific arguments to configure
+====================================
 
 * ``--enable-framework[=DIR]``
 
@@ -15,11 +22,11 @@
   _`Building and using a framework-based Python on Mac OS X` for more 
   information on frameworks.
 
-  If the optional directory argument is specified the framework it installed
+  If the optional directory argument is specified the framework is installed
   into that directory. This can be used to install a python framework into
   your home directory::
 
-     $ configure --enable-framework=/Users/ronald/Library/Frameworks
+     $ ./configure --enable-framework=/Users/ronald/Library/Frameworks
      $ make && make install
 
   This will install the framework itself in ``/Users/ronald/Library/Frameworks``,
@@ -36,9 +43,10 @@
   Create a universal binary build of Python. This can be used with both
   regular and framework builds.
 
-  The optional argument specifies which OSX SDK should be used to perform the
-  build. This defaults to ``/Developer/SDKs/MacOSX.10.4u.sdk``, specify 
-  ``/`` when building on a 10.5 system, especially when building 64-bit code.
+  The optional argument specifies which OS X SDK should be used to perform the
+  build. This defaults to ``/Developer/SDKs/MacOSX.10.4u.sdk``.  When building
+  on OS X 10.5 or later, you can specify ``/`` to use the installed system
+  headers rather than an SDK.
 
   See the section _`Building and using a universal binary of Python on Mac OS X`
   for more information.
@@ -56,9 +64,14 @@
 1. What is a universal binary
 -----------------------------
 
-A universal binary build of Python contains object code for both PPC and i386
-and can therefore run at native speed on both classic powerpc based macs and
-the newer intel based macs.
+A universal binary build of Python contains object code for more than one
+CPU architecture.  A universal OS X executable file or library combines the
+architecture-specific code into one file and can therefore run at native
+speed on all supported architectures.  Universal files were introduced in
+OS X 10.4 to add support for Intel-based Macs to the existing PowerPC (PPC)
+machines.  In OS X 10.5 support was extended to 64-bit Intel and 64-bit PPC
+architectures.  It is possible to build Python with various combinations
+of architectures depending on the build tools and OS X version in use.
 
 2. How do I build a universal binary
 ------------------------------------
@@ -71,12 +84,12 @@
   $ make install
 
 This flag can be used with a framework build of python, but also with a classic
-unix build. Either way you will have to build python on Mac OS X 10.4 (or later)
-with Xcode 2.1 (or later). You also have to install the 10.4u SDK when 
-installing Xcode.
+unix build. Universal builds were first supported with OS X 10.4 with Xcode 2.1
+and the 10.4u SDK.  Starting with Xcode 3 and OS X 10.5, more configurations are
+available.
 
 The option ``--enable-universalsdk`` has an optional argument to specify an
-SDK, which defaults to the 10.4u SDK. When you build on OSX 10.5 or later 
+SDK, which defaults to the 10.4u SDK. When you build on OS X 10.5 or later
 you can use the system headers instead of an SDK::
 
   $ ./configure --enable-universalsdk=/
@@ -88,34 +101,51 @@
 Python Developer's Guide (http://docs.python.org/devguide/setup.html)
 for more information.
 
-2.1 Flavours of universal binaries
-..................................
+2.1 Flavors of universal binaries
+.................................
 
-It is possible to build a number of flavours of the universal binary build,
-the default is a 32-bit only binary (i386 and ppc). The flavour can be
+It is possible to build a number of flavors of the universal binary build,
+the default is a 32-bit only binary (i386 and ppc). Note that starting with
+Xcode 4, the build tools no longer support ppc. The flavor can be
 specified using the option ``--with-universal-archs=VALUE``. The following
 values are available:
 
+  * ``intel``:	  ``i386``, ``x86_64``
+
   * ``32-bit``:   ``ppc``, ``i386``
 
+  * ``3-way``:	  ``i386``, ``x86_64``, ``ppc``
+
   * ``64-bit``:   ``ppc64``, ``x86_64``
 
   * ``all``:      ``ppc``, ``ppc64``, ``i386``, ``x86_64``
 
-  * ``3-way``:	  ``ppc``, ``i386`` and ``x86_64``
+To build a universal binary that includes a 64-bit architecture, you must build
+on a system running OS X 10.5 or later.  The ``all`` and ``64-bit`` flavors can
+only be built with an 10.5 SDK because ``ppc64`` support was only included with
+OS X 10.5.  Although legacy ``ppc`` support was included with Xcode 3 on OS X
+10.6, it was removed in Xcode 4, versions of which were released on OS X 10.6
+and which is the standard for OS X 10.7.  To summarize, the
+following combinations of SDKs and universal-archs flavors are available:
 
-  * ``intel``:	  ``i386``, ``x86_64``
+  * 10.4u SDK with Xcode 2 supports ``32-bit`` only
 
-To build a universal binary that includes a 64-bit architecture, you must build
-on a system running OSX 10.5 or later. The ``all`` flavour can only be built on
-OSX 10.5.
+  * 10.5 SDK with Xcode 3.1.x supports all flavors
 
-The makefile for a framework build will install ``python32`` and ``pythonw32`` 
-binaries when the universal architecures includes at least one 32-bit architecture
-(that is, for all flavours but ``64-bit``).
+  * 10.6 SDK with Xcode 3.2.x supports ``intel``, ``3-way``, and ``32-bit``
 
-Running a specific archicture
-.............................
+  * 10.6 SDK with Xcode 4 supports ``intel`` only
+
+  * 10.7 and 10.8 SDKs with Xcode 4 support ``intel`` only
+
+  * 10.8 and 10.9 SDKs with Xcode 5 support ``intel`` only
+
+The makefile for a framework build will also install ``python2.7-32``
+binaries when the universal architecture includes at least one 32-bit
+architecture (that is, for all flavors but ``64-bit``).
+
+Running a specific architecture
+...............................
 
 You can run code using a specific architecture using the ``arch`` command::
 
@@ -130,6 +160,13 @@
 wrapper tools that execute the real interpreter without ensuring that the
 real interpreter runs with the same architecture.
 
+Using ``arch`` is not a perfect solution as the selected architecture will
+not automatically carry through to subprocesses launched by programs and tests
+under that Python.  If you want to ensure that Python interpreters launched in
+subprocesses also run in 32-bit-mode if the main interpreter does, use
+a ``python2.7-32`` binary and use the value of ``sys.executable`` as the
+``subprocess`` ``Popen`` executable value.
+
 Building and using a framework-based Python on Mac OS X.
 ========================================================
 
@@ -139,16 +176,17 @@
 
 The main reason is because you want to create GUI programs in Python. With the
 exception of X11/XDarwin-based GUI toolkits all GUI programs need to be run 
-from a fullblown MacOSX application (a ".app" bundle).
+from a Mac OS X application bundle (".app").
 
 While it is technically possible to create a .app without using frameworks you
 will have to do the work yourself if you really want this.
 
 A second reason for using frameworks is that they put Python-related items in
 only two places: "/Library/Framework/Python.framework" and 
-"/Applications/MacPython 2.6".  This simplifies matters for users installing 
+"/Applications/Python <VERSION>" where ``<VERSION>`` can be e.g. "3.4",
+"2.7", etc.  This simplifies matters for users installing
 Python from a binary distribution if they want to get rid of it again. Moreover,
-due to the way frameworks work a user without admin privileges can install a 
+due to the way frameworks work, a user without admin privileges can install a
 binary distribution in his or her home directory without recompilation.
 
 2. How does a framework Python differ from a normal static Python?
@@ -163,43 +201,55 @@
 3. Do I need extra packages?
 ----------------------------
 
-Yes, probably.  If you want Tkinter support you need to get the OSX AquaTk 
-distribution, this is installed by default on Mac OS X 10.4 or later. If
-you want wxPython you need to get that. If you want Cocoa you need to get
-PyObjC. 
+Yes, probably.  If you want Tkinter support you need to get the OS X AquaTk
+distribution, this is installed by default on Mac OS X 10.4 or later.  Be
+aware, though, that the Cocoa-based AquaTk's supplied starting with OS X
+10.6 have proven to be unstable.  If possible, you should consider
+installing a newer version before building on OS X 10.6 or later, such as
+the ActiveTcl 8.5.  See http://www.python.org/download/mac/tcltk/.  If you
+are building with an SDK, ensure that the newer Tcl and Tk frameworks are
+seen in the SDK's ``Library/Frameworks`` directory; you may need to
+manually create symlinks to their installed location, ``/Library/Frameworks``.
+If you want wxPython you need to get that.
+If you want Cocoa you need to get PyObjC.
 
 4. How do I build a framework Python?
 -------------------------------------
 
 This directory contains a Makefile that will create a couple of python-related
-applications (fullblown OSX .app applications, that is) in
-"/Applications/MacPython 2.6", and a hidden helper application Python.app 
-inside the Python.framework, and unix tools "python" and "pythonw" into 
-/usr/local/bin.  In addition it has a target "installmacsubtree" that installs 
+applications (full-blown OS X .app applications, that is) in
+"/Applications/Python <VERSION>", and a hidden helper application Python.app
+inside the Python.framework, and unix tools "python" and "pythonw" into
+/usr/local/bin.  In addition it has a target "installmacsubtree" that installs
 the relevant portions of the Mac subtree into the Python.framework.
 
 It is normally invoked indirectly through the main Makefile, as the last step
-in the sequence::
+in the sequence
 
- $ ./configure --enable-framework
- $ make
- $ make install
+ 1. ./configure --enable-framework
 
-This sequence will put the framework in /Library/Framework/Python.framework,
-the applications in "/Applications/MacPython 2.6" and the unix tools in 
-/usr/local/bin.
+ 2. make
+ 
+ 3. make install
 
-It is possible to select a different name for the framework using the configure 
-option ``--with-framework-name=NAME``. This makes it possible to have several
-parallel installs of a Python framework.
+This sequence will put the framework in ``/Library/Framework/Python.framework``,
+the applications in ``/Applications/Python <VERSION>`` and the unix tools in 
+``/usr/local/bin``.
 
-Installing in another place, for instance $HOME/Library/Frameworks if you have
-no admin privileges on your machine, has only been tested very lightly. This
-can be done by configuring with --enable-framework=$HOME/Library/Frameworks.
-The other two directories, "/Applications/MacPython-2.6" and /usr/local/bin, 
-will then also be deposited in $HOME. This is sub-optimal for the unix tools, 
-which you would want in $HOME/bin, but there is no easy way to fix this right 
-now.
+Installing in another place, for instance ``$HOME/Library/Frameworks`` if you
+have no admin privileges on your machine, is possible. This can be accomplished
+by configuring with ``--enable-framework=$HOME/Library/Frameworks``.
+The other two directories will then also be installed in your home directory,
+at ``$HOME/Applications/Python-<VERSION>`` and ``$HOME/bin``.
+
+If you want to install some part, but not all, read the main Makefile. The
+frameworkinstall is composed of a couple of sub-targets that install the
+framework itself, the Mac subtree, the applications and the unix tools.
+
+There is an extra target frameworkinstallextras that is not part of the
+normal frameworkinstall which installs the Tools directory into
+"/Applications/Python <VERSION>", this is useful for binary
+distributions.
 
 What do all these programs do?
 ===============================
@@ -207,35 +257,54 @@
 "IDLE.app" is an integrated development environment for Python: editor,
 debugger, etc.
 
-"PythonLauncher.app" is a helper application that will handle things when you
+"Python Launcher.app" is a helper application that will handle things when you
 double-click a .py, .pyc or .pyw file. For the first two it creates a Terminal
 window and runs the scripts with the normal command-line Python. For the
 latter it runs the script in the Python.app interpreter so the script can do
-GUI-things. Keep the "alt" key depressed while dragging or double-clicking a
-script to set runtime options. These options can be set once and for all
-through PythonLauncher's preferences dialog.
+GUI-things. Keep the ``Option`` key depressed while dragging or double-clicking
+a script to set runtime options. These options can be set persistently
+through Python Launcher's preferences dialog.
 
-"BuildApplet.app" creates an applet from a Python script. Drop the script on it
-and out comes a full-featured MacOS application.  BuildApplet.app is now
+"Build Applet.app" creates an applet from a Python script. Drop the script on it
+and out comes a full-featured Mac OS X application.  "Build Applet.app" is now
 deprecated and has been removed in Python 3.  As of OS X 10.8, Xcode 4 no
 longer supplies the headers for the deprecated QuickDraw APIs used by
 the EasyDialogs module making BuildApplet unusable as an app.  It will
 not be built by the Mac/Makefile in this case.
 
-The commandline scripts /usr/local/bin/python and pythonw can be used to run
-non-GUI and GUI python scripts from the command line, respectively.
+The program ``pythonx.x`` runs python scripts from the command line.  Various
+compatibility aliases are also installed, including ``pythonwx.x`` which
+in early releases of Python on OS X was required to run GUI programs.  In
+current releases, the ``pythonx.x`` and ``pythonwx.x`` commands are identical
+and the use of ``pythonwx.x`` should be avoided as it has been removed in
+current versions of Python 3.
 
 How do I create a binary distribution?
 ======================================
 
-Go to the directory "Mac/OSX/BuildScript". There you'll find a script 
-"build-installer.py" that does all the work. This will download and build
+Download and unpack the source release from http://www.python.org/download/.
+Go to the directory ``Mac/BuildScript``. There you will find a script
+``build-installer.py`` that does all the work. This will download and build
 a number of 3rd-party libaries, configures and builds a framework Python,
 installs it, creates the installer package files and then packs this in a
-DMG image.
+DMG image.  The script also builds an HTML copy of the current Python
+documentation set for this release for inclusion in the framework.  The
+installer package will create links to the documentation for use by IDLE,
+pydoc, shell users, and Finder user.
 
-The script will build a universal binary, you'll therefore have to run this
+The script will build a universal binary so you'll therefore have to run this
 script on Mac OS X 10.4 or later and with Xcode 2.1 or later installed.
+However, the Python build process itself has several build dependencies not
+available out of the box with OS X 10.4 so you may have to install
+additional software beyond what is provided with Xcode 2.  OS X 10.5
+provides a recent enough system Python (in ``/usr/bin``) to build
+the Python documentation set.  It should be possible to use SDKs and/or older
+versions of Xcode to build installers that are compatible with older systems
+on a newer system but this may not be completely foolproof so the resulting
+executables, shared libraries, and ``.so`` bundles should be carefully
+examined and tested on all supported systems for proper dynamic linking
+dependencies.  It is safest to build the distribution on a system running the
+minimum OS X version supported.
 
 All of this is normally done completely isolated in /tmp/_py, so it does not
 use your normal build directory nor does it install into /.
@@ -260,7 +329,7 @@
    configure: WARNING:     ## -------------------------------------- ##
 
 This almost always means you are trying to build a universal binary for
-Python and have libaries in ``/usr/local`` that don't contain the required
+Python and have libraries in ``/usr/local`` that don't contain the required
 architectures. Temporarily move ``/usr/local`` aside to finish the build.
 
 
@@ -269,7 +338,7 @@
 
 Uninstalling a framework can be done by manually removing all bits that got installed.
 That's true for both installations from source and installations using the binary installer.
-Sadly enough OSX does not have a central uninstaller.
+OS X does not provide a central uninstaller.
 
 The main bit of a framework install is the framework itself, installed in
 ``/Library/Frameworks/Python.framework``. This can contain multiple versions
@@ -283,14 +352,12 @@
 And lastly a framework installation installs files in ``/usr/local/bin``, all of
 them symbolic links to files in ``/Library/Frameworks/Python.framework/Versions/X.Y/bin``.
 
-Odds and ends
-=============
 
-Something to take note of is that the ".rsrc" files in the distribution are
-not actually resource files, they're AppleSingle encoded resource files. The
-macresource module and the Mac/OSX/Makefile cater for this, and create
-".rsrc.df.rsrc" files on the fly that are normal datafork-based resource
-files.
+Resources
+=========
 
-	Jack Jansen, Jack.Jansen at cwi.nl, 15-Jul-2004.
-	Ronald Oussoren, RonaldOussoren at mac.com, 30-April-2010
+  *  http://www.python.org/download/mac/
+
+  *  http://www.python.org/community/sigs/current/pythonmac-sig/
+
+  *  http://docs.python.org/devguide/
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -436,6 +436,20 @@
 - Issue #21303, #20565: Updated the version of Tcl/Tk included in the
   installer from 8.5.2 to 8.5.15.
 
+Mac OS X
+--------
+
+- As of 2.7.8, the 32-bit-only installer will support OS X 10.5
+  and later systems as is currently done for Python 3.x installers.
+  For 2.7.7 only, we will provide three installers:
+  the legacy deprecated 10.3+ 32-bit-only format;
+  the newer 10.5+ 32-bit-only format;
+  and the unchanged 10.6+ 64-/32-bit format.
+  Although binary installers will no longer be available from
+  python.org as of 2.7.8, it will still be possible to build from
+  source on 10.3.9 and 10.4 systems if necessary.
+  See Mac/BuildScript/README.txt for more information.
+
 
 Whats' New in Python 2.7.6?
 ===========================

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat May 17 23:32:09 2014
From: python-checkins at python.org (ned.deily)
Date: Sat, 17 May 2014 23:32:09 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Miscellaneous_?=
 =?utf-8?q?corrections_and_updates_to_the_OS_X_README_file=2E?=
Message-ID: <3gWKSn0mwfz7Ljg@mail.python.org>

http://hg.python.org/cpython/rev/6f1db02a8428
changeset:   90733:6f1db02a8428
branch:      3.4
parent:      90723:6d2982ff441f
user:        Ned Deily <nad at acm.org>
date:        Sat May 17 14:30:09 2014 -0700
summary:
  Miscellaneous corrections and updates to the OS X README file.

files:
  Mac/README |  30 +++++++++++++++++-------------
  1 files changed, 17 insertions(+), 13 deletions(-)


diff --git a/Mac/README b/Mac/README
--- a/Mac/README
+++ b/Mac/README
@@ -12,6 +12,9 @@
 This document provides a quick overview of some Mac OS X specific features in
 the Python distribution.
 
+OS X specific arguments to configure
+====================================
+
 * ``--enable-framework[=DIR]``
 
   If this argument is specified the build will create a Python.framework rather
@@ -121,7 +124,7 @@
 only be built with an 10.5 SDK because ``ppc64`` support was only included with
 OS X 10.5.  Although legacy ``ppc`` support was included with Xcode 3 on OS X
 10.6, it was removed in Xcode 4, versions of which were released on OS X 10.6
-and which is the current standard for OS X 10.7 and 10.8.  To summarize, the
+and which is the standard for OS X 10.7.  To summarize, the
 following combinations of SDKs and universal-archs flavors are available:
 
   * 10.4u SDK with Xcode 2 supports ``32-bit`` only
@@ -134,6 +137,8 @@
 
   * 10.7 and 10.8 SDKs with Xcode 4 support ``intel`` only
 
+  * 10.8 and 10.9 SDKs with Xcode 5 support ``intel`` only
+
 The makefile for a framework build will also install ``python3.4-32``
 binaries when the universal architecture includes at least one 32-bit
 architecture (that is, for all flavors but ``64-bit``).
@@ -161,7 +166,6 @@
 a ``python3.4-32`` binary and use the value of ``sys.executable`` as the
 ``subprocess`` ``Popen`` executable value.
 
-
 Building and using a framework-based Python on Mac OS X.
 ========================================================
 
@@ -171,7 +175,7 @@
 
 The main reason is because you want to create GUI programs in Python. With the
 exception of X11/XDarwin-based GUI toolkits all GUI programs need to be run 
-from a Mac OSX application bundle (".app").
+from a Mac OS X application bundle (".app").
 
 While it is technically possible to create a .app without using frameworks you
 will have to do the work yourself if you really want this.
@@ -196,7 +200,7 @@
 3. Do I need extra packages?
 ----------------------------
 
-Yes, probably.  If you want Tkinter support you need to get the OSX AquaTk 
+Yes, probably.  If you want Tkinter support you need to get the OS X AquaTk
 distribution, this is installed by default on Mac OS X 10.4 or later.  Be
 aware, though, that the Cocoa-based AquaTk's supplied starting with OS X
 10.6 have proven to be unstable.  If possible, you should consider
@@ -212,9 +216,9 @@
 -------------------------------------
 
 This directory contains a Makefile that will create a couple of python-related
-applications (full-blown OSX .app applications, that is) in
+applications (full-blown OS X .app applications, that is) in
 "/Applications/Python <VERSION>", and a hidden helper application Python.app
-inside the Python.framework, and unix tools "python" and "pythonw" into
+inside the Python.framework, and unix tools including "python" into
 /usr/local/bin.  In addition it has a target "installmacsubtree" that installs
 the relevant portions of the Mac subtree into the Python.framework.
 
@@ -252,18 +256,18 @@
 "IDLE.app" is an integrated development environment for Python: editor,
 debugger, etc.
 
-"PythonLauncher.app" is a helper application that will handle things when you
+"Python Launcher.app" is a helper application that will handle things when you
 double-click a .py, .pyc or .pyw file. For the first two it creates a Terminal
 window and runs the scripts with the normal command-line Python. For the
 latter it runs the script in the Python.app interpreter so the script can do
 GUI-things. Keep the ``Option`` key depressed while dragging or double-clicking
 a script to set runtime options. These options can be set persistently
-through PythonLauncher's preferences dialog.
+through Python Launcher's preferences dialog.
 
-The program ``pythonx.x`` runs python scripts from the command line.  Various
-compatibility aliases are also installed, including ``pythonwx.x`` which
-in early releases of Python on OS X was required to run GUI programs.  In
-current releases, the ``pythonx.x`` and ``pythonwx.x`` commands are identical.
+The program ``pythonx.x`` runs python scripts from the command line.
+Previously, various compatibility aliases were also installed, including
+``pythonwx.x`` which in early releases of Python on OS X was required to run
+GUI programs.  As of 3.4.0, the ``pythonwx.x`` aliases are no longer installed.
 
 How do I create a binary distribution?
 ======================================
@@ -308,7 +312,7 @@
    configure: WARNING: libintl.h:     check for missing prerequisite headers?
    configure: WARNING: libintl.h: see the Autoconf documentation
    configure: WARNING: libintl.h:     section "Present But Cannot Be Compiled"
-   configure: WARNING: libintl.h: proceeding with the preprocessor's result   
+   configure: WARNING: libintl.h: proceeding with the preprocessor's result
    configure: WARNING: libintl.h: in the future, the compiler will take precedence
    configure: WARNING:     ## -------------------------------------- ##
    configure: WARNING:     ## Report this to http://bugs.python.org/ ##

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat May 17 23:32:10 2014
From: python-checkins at python.org (ned.deily)
Date: Sat, 17 May 2014 23:32:10 +0200 (CEST)
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Miscellaneous_corrections_and_updates_to_the_OS_X_README?=
 =?utf-8?q?_file=2E?=
Message-ID: <3gWKSp3lWnz7Ljq@mail.python.org>

http://hg.python.org/cpython/rev/b2e33f895bdd
changeset:   90734:b2e33f895bdd
parent:      90731:3024ad49f00e
parent:      90733:6f1db02a8428
user:        Ned Deily <nad at acm.org>
date:        Sat May 17 14:31:34 2014 -0700
summary:
  Miscellaneous corrections and updates to the OS X README file.

files:
  Mac/README |  30 +++++++++++++++++-------------
  1 files changed, 17 insertions(+), 13 deletions(-)


diff --git a/Mac/README b/Mac/README
--- a/Mac/README
+++ b/Mac/README
@@ -12,6 +12,9 @@
 This document provides a quick overview of some Mac OS X specific features in
 the Python distribution.
 
+OS X specific arguments to configure
+====================================
+
 * ``--enable-framework[=DIR]``
 
   If this argument is specified the build will create a Python.framework rather
@@ -121,7 +124,7 @@
 only be built with an 10.5 SDK because ``ppc64`` support was only included with
 OS X 10.5.  Although legacy ``ppc`` support was included with Xcode 3 on OS X
 10.6, it was removed in Xcode 4, versions of which were released on OS X 10.6
-and which is the current standard for OS X 10.7 and 10.8.  To summarize, the
+and which is the standard for OS X 10.7.  To summarize, the
 following combinations of SDKs and universal-archs flavors are available:
 
   * 10.4u SDK with Xcode 2 supports ``32-bit`` only
@@ -134,6 +137,8 @@
 
   * 10.7 and 10.8 SDKs with Xcode 4 support ``intel`` only
 
+  * 10.8 and 10.9 SDKs with Xcode 5 support ``intel`` only
+
 The makefile for a framework build will also install ``python3.4-32``
 binaries when the universal architecture includes at least one 32-bit
 architecture (that is, for all flavors but ``64-bit``).
@@ -161,7 +166,6 @@
 a ``python3.4-32`` binary and use the value of ``sys.executable`` as the
 ``subprocess`` ``Popen`` executable value.
 
-
 Building and using a framework-based Python on Mac OS X.
 ========================================================
 
@@ -171,7 +175,7 @@
 
 The main reason is because you want to create GUI programs in Python. With the
 exception of X11/XDarwin-based GUI toolkits all GUI programs need to be run 
-from a Mac OSX application bundle (".app").
+from a Mac OS X application bundle (".app").
 
 While it is technically possible to create a .app without using frameworks you
 will have to do the work yourself if you really want this.
@@ -196,7 +200,7 @@
 3. Do I need extra packages?
 ----------------------------
 
-Yes, probably.  If you want Tkinter support you need to get the OSX AquaTk 
+Yes, probably.  If you want Tkinter support you need to get the OS X AquaTk
 distribution, this is installed by default on Mac OS X 10.4 or later.  Be
 aware, though, that the Cocoa-based AquaTk's supplied starting with OS X
 10.6 have proven to be unstable.  If possible, you should consider
@@ -212,9 +216,9 @@
 -------------------------------------
 
 This directory contains a Makefile that will create a couple of python-related
-applications (full-blown OSX .app applications, that is) in
+applications (full-blown OS X .app applications, that is) in
 "/Applications/Python <VERSION>", and a hidden helper application Python.app
-inside the Python.framework, and unix tools "python" and "pythonw" into
+inside the Python.framework, and unix tools including "python" into
 /usr/local/bin.  In addition it has a target "installmacsubtree" that installs
 the relevant portions of the Mac subtree into the Python.framework.
 
@@ -252,18 +256,18 @@
 "IDLE.app" is an integrated development environment for Python