Mailman 3 August 2012 - Python-checkins

cpython: Fix findnocoding.p and pysource.py scripts
by victor.stinner Aug. 1, 2012

Aug. 1, 2012

http://hg.python.org/cpython/rev/67d36e8ddcfc changeset: 78375:67d36e8ddcfc user: Victor Stinner <victor.stinner(a)gmail.com> date: Wed Aug 01 20:12:51 2012 +0200 summary: Fix findnocoding.p and pysource.py scripts I suppose that these scripts didn't work since Python 3.0. files: Tools/scripts/findnocoding.py | 22 +++++++++++----------- Tools/scripts/pysource.py | 6 +++--- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/Tools/scripts/findnocoding.py b/Tools/scripts/findnocoding.py --- a/Tools/scripts/findnocoding.py +++ b/Tools/scripts/findnocoding.py @@ -32,7 +32,7 @@ "no sophisticated Python source file search will be done.", file=sys.stderr) -decl_re = re.compile(r"coding[=:]\s*([-\w.]+)") +decl_re = re.compile(rb"coding[=:]\s*([-\w.]+)") def get_declaration(line): match = decl_re.search(line) @@ -50,21 +50,21 @@ def needs_declaration(fullpath): try: - infile = open(fullpath) + infile = open(fullpath, 'rb') except IOError: # Oops, the file was removed - ignore it return None - line1 = infile.readline() - line2 = infile.readline() + with infile: + line1 = infile.readline() + line2 = infile.readline() - if get_declaration(line1) or get_declaration(line2): - # the file does have an encoding declaration, so trust it - infile.close() - return False + if get_declaration(line1) or get_declaration(line2): + # the file does have an encoding declaration, so trust it + infile.close() + return False - # check the whole file for non utf-8 characters - rest = infile.read() - infile.close() + # check the whole file for non utf-8 characters + rest = infile.read() if has_correct_encoding(line1+line2+rest, "utf-8"): return False diff --git a/Tools/scripts/pysource.py b/Tools/scripts/pysource.py --- a/Tools/scripts/pysource.py +++ b/Tools/scripts/pysource.py @@ -22,7 +22,7 @@ import os, re -binary_re = re.compile('[\x00-\x08\x0E-\x1F\x7F]') +binary_re = re.compile(br'[\x00-\x08\x0E-\x1F\x7F]') debug = False @@ -42,7 +42,7 @@ return None try: - return open(fullpath) + return open(fullpath, "rb") except IOError as err: # Access denied, or a special file - ignore it print_debug("%s: access denied: %s" % (fullpath, err)) return None @@ -65,7 +65,7 @@ if fullpath.endswith(".py") or fullpath.endswith(".pyw"): return True - elif "python" in line: + elif b"python" in line: # disguised Python script (e.g. CGI) return True -- Repository URL: http://hg.python.org/cpython

3 2

cpython: Cleanup findnocoding.py and pysource.py scripts (with infile/infile.close)
by victor.stinner Aug. 1, 2012

Aug. 1, 2012

http://hg.python.org/cpython/rev/8ace059cdffd changeset: 78376:8ace059cdffd user: Victor Stinner <victor.stinner(a)gmail.com> date: Thu Aug 02 00:05:41 2012 +0200 summary: Cleanup findnocoding.py and pysource.py scripts (with infile/infile.close) files: Tools/scripts/findnocoding.py | 1 - Tools/scripts/pysource.py | 8 ++++---- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/Tools/scripts/findnocoding.py b/Tools/scripts/findnocoding.py --- a/Tools/scripts/findnocoding.py +++ b/Tools/scripts/findnocoding.py @@ -60,7 +60,6 @@ if get_declaration(line1) or get_declaration(line2): # the file does have an encoding declaration, so trust it - infile.close() return False # check the whole file for non utf-8 characters diff --git a/Tools/scripts/pysource.py b/Tools/scripts/pysource.py --- a/Tools/scripts/pysource.py +++ b/Tools/scripts/pysource.py @@ -55,8 +55,8 @@ if infile is None: return False - line = infile.readline() - infile.close() + with infile: + line = infile.readline() if binary_re.search(line): # file appears to be binary @@ -76,8 +76,8 @@ if infile is None: return False - code = infile.read() - infile.close() + with infile: + code = infile.read() try: compile(code, fullpath, "exec") -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #15441: Skip test_nonascii_abspath() of test_genericpath on Windows
by victor.stinner Aug. 1, 2012

Aug. 1, 2012

http://hg.python.org/cpython/rev/3edc71ed19e7 changeset: 78374:3edc71ed19e7 user: Victor Stinner <victor.stinner(a)gmail.com> date: Wed Aug 01 20:03:49 2012 +0200 summary: Issue #15441: Skip test_nonascii_abspath() of test_genericpath on Windows if the bytes filenames cannot be encoded from the file system (ANSI) code page files: Lib/test/test_genericpath.py | 14 +++++++++++--- 1 files changed, 11 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_genericpath.py b/Lib/test/test_genericpath.py --- a/Lib/test/test_genericpath.py +++ b/Lib/test/test_genericpath.py @@ -299,8 +299,7 @@ unicwd = '\xe7w\xf0' try: - fsencoding = support.TESTFN_ENCODING or "ascii" - unicwd.encode(fsencoding) + os.fsencode(unicwd) except (AttributeError, UnicodeEncodeError): # FS encoding is probably ASCII pass @@ -312,10 +311,19 @@ @unittest.skipIf(sys.platform == 'darwin', "Mac OS X denies the creation of a directory with an invalid utf8 name") def test_nonascii_abspath(self): + name = b'\xe7w\xf0' + if sys.platform == 'win32': + try: + os.fsdecode(name) + except UnicodeDecodeError: + self.skipTest("the filename %a is not decodable " + "from the ANSI code page %s" + % (name, sys.getfilesystemencoding())) + # Test non-ASCII, non-UTF8 bytes in the path. with warnings.catch_warnings(): warnings.simplefilter("ignore", DeprecationWarning) - with support.temp_cwd(b'\xe7w\xf0'): + with support.temp_cwd(name): self.test_abspath() -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #15463: Write a test for faulthandler truncating the name of functions
by victor.stinner Aug. 1, 2012

Aug. 1, 2012

http://hg.python.org/cpython/rev/6e03f9b72c61 changeset: 78373:6e03f9b72c61 user: Victor Stinner <victor.stinner(a)gmail.com> date: Wed Aug 01 19:45:34 2012 +0200 summary: Issue #15463: Write a test for faulthandler truncating the name of functions to 500 characters. files: Lib/test/test_faulthandler.py | 24 +++++++++++++++++++++++ 1 files changed, 24 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_faulthandler.py b/Lib/test/test_faulthandler.py --- a/Lib/test/test_faulthandler.py +++ b/Lib/test/test_faulthandler.py @@ -316,6 +316,30 @@ with temporary_filename() as filename: self.check_dump_traceback(filename) + def test_truncate(self): + maxlen = 500 + func_name = 'x' * (maxlen + 50) + truncated = 'x' * maxlen + '...' + code = """ +import faulthandler + +def {func_name}(): + faulthandler.dump_traceback(all_threads=False) + +{func_name}() +""".strip() + code = code.format( + func_name=func_name, + ) + expected = [ + 'Traceback (most recent call first):', + ' File "<string>", line 4 in %s' % truncated, + ' File "<string>", line 6 in <module>' + ] + trace, exitcode = self.get_output(code) + self.assertEqual(trace, expected) + self.assertEqual(exitcode, 0) + @unittest.skipIf(not HAVE_THREADS, 'need threads') def check_dump_traceback_threads(self, filename): """ -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Fix the user signal handler of faulthandler
by victor.stinner Aug. 1, 2012

Aug. 1, 2012

http://hg.python.org/cpython/rev/2dd1aaf41319 changeset: 78372:2dd1aaf41319 user: Victor Stinner <victor.stinner(a)gmail.com> date: Wed Aug 01 19:36:36 2012 +0200 summary: Fix the user signal handler of faulthandler Don't exit the tstate is NULL to restore the errno and chain the signal handler if needed. files: Modules/faulthandler.c | 5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-) diff --git a/Modules/faulthandler.c b/Modules/faulthandler.c --- a/Modules/faulthandler.c +++ b/Modules/faulthandler.c @@ -653,9 +653,8 @@ if (user->all_threads) _Py_DumpTracebackThreads(user->fd, user->interp, tstate); else { - if (tstate == NULL) - return; - _Py_DumpTraceback(user->fd, tstate); + if (tstate != NULL) + _Py_DumpTraceback(user->fd, tstate); } #ifdef HAVE_SIGACTION if (user->chain) { -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #15525: Increase timeout when TerminateProcess() fails
by richard.oudkerk Aug. 1, 2012

Aug. 1, 2012

http://hg.python.org/cpython/rev/b99a82bc2cde changeset: 78371:b99a82bc2cde user: Richard Oudkerk <shibturn(a)gmail.com> date: Wed Aug 01 17:44:18 2012 +0100 summary: Issue #15525: Increase timeout when TerminateProcess() fails files: Lib/multiprocessing/forking.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/multiprocessing/forking.py b/Lib/multiprocessing/forking.py --- a/Lib/multiprocessing/forking.py +++ b/Lib/multiprocessing/forking.py @@ -273,8 +273,8 @@ if self.returncode is None: try: _winapi.TerminateProcess(int(self._handle), TERMINATE) - except WindowsError: - if self.wait(timeout=0.1) is None: + except OSError: + if self.wait(timeout=1.0) is None: raise # -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Improve wording for __hash__
by antoine.pitrou Aug. 1, 2012

Aug. 1, 2012

http://hg.python.org/cpython/rev/2dd1b056d663 changeset: 78370:2dd1b056d663 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Wed Aug 01 14:53:22 2012 +0200 summary: Improve wording for __hash__ files: Doc/reference/datamodel.rst | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1291,7 +1291,7 @@ .. note:: - Note by default the :meth:`__hash__` values of str, bytes and datetime + By default, the :meth:`__hash__` values of str, bytes and datetime objects are "salted" with an unpredictable random value. Although they remain constant within an individual Python process, they are not predictable between repeated invocations of Python. @@ -1301,9 +1301,9 @@ dict insertion, O(n^2) complexity. See http://www.ocert.org/advisories/ocert-2011-003.html for details. - Changing hash values affects the order in which keys are retrieved from a - dict. Note Python has never made guarantees about this ordering (and it - typically varies between 32-bit and 64-bit builds). + Changing hash values affects the iteration order of dicts, sets and + other mappings. Python has never made guarantees about this ordering + (and it typically varies between 32-bit and 64-bit builds). See also :envvar:`PYTHONHASHSEED`. -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Clarify that hash randomization is on by default
by antoine.pitrou Aug. 1, 2012

Aug. 1, 2012

http://hg.python.org/cpython/rev/7e438f088129 changeset: 78369:7e438f088129 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Wed Aug 01 14:53:16 2012 +0200 summary: Clarify that hash randomization is on by default files: Doc/using/cmdline.rst | 31 +++++++++++++++---------------- 1 files changed, 15 insertions(+), 16 deletions(-) diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -229,23 +229,22 @@ .. cmdoption:: -R - Turn on hash randomization, so that the :meth:`__hash__` values of str, bytes - and datetime objects are "salted" with an unpredictable random value. - Although they remain constant within an individual Python process, they are - not predictable between repeated invocations of Python. + Kept for compatibility. On Python 3.3 and greater, hash randomization is + turned on by default. - This is intended to provide protection against a denial-of-service caused by - carefully-chosen inputs that exploit the worst case performance of a dict - construction, O(n^2) complexity. See + On previous versions of Python, this option turns on hash randomization, + so that the :meth:`__hash__` values of str, bytes and datetime + are "salted" with an unpredictable random value. Although they remain + constant within an individual Python process, they are not predictable + between repeated invocations of Python. + + Hash randomization is intended to provide protection against a + denial-of-service caused by carefully-chosen inputs that exploit the worst + case performance of a dict construction, O(n^2) complexity. See http://www.ocert.org/advisories/ocert-2011-003.html for details. - Changing hash values affects the order in which keys are retrieved from a - dict. Although Python has never made guarantees about this ordering (and it - typically varies between 32-bit and 64-bit builds), enough real-world code - implicitly relies on this non-guaranteed behavior that the randomization is - disabled by default. - - See also :envvar:`PYTHONHASHSEED`. + :envvar:`PYTHONHASHSEED` allows you to set a fixed value for the hash + seed secret. .. versionadded:: 3.2.3 @@ -486,8 +485,8 @@ .. envvar:: PYTHONHASHSEED - If this variable is set to ``random``, a random value is used to seed the - hashes of str, bytes and datetime objects. + If this variable is not set or set to ``random``, a random value is used + to seed the hashes of str, bytes and datetime objects. If :envvar:`PYTHONHASHSEED` is set to an integer value, it is used as a fixed seed for generating the hash() of the types covered by the hash -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Issue #8847: Merge with 3.2
by martin.v.loewis Aug. 1, 2012

Aug. 1, 2012

http://hg.python.org/cpython/rev/d3afe5d8a4da changeset: 78368:d3afe5d8a4da parent: 78367:029cde4e58c5 parent: 78366:2638ce032151 user: Martin v. Löwis <martin(a)v.loewis.de> date: Wed Aug 01 11:09:55 2012 +0200 summary: Issue #8847: Merge with 3.2 files: Lib/test/test_list.py | 8 ++++++++ Lib/test/test_tuple.py | 8 ++++++++ Misc/NEWS | 2 ++ PC/VS9.0/pginstrument.vsprops | 2 +- 4 files changed, 19 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_list.py b/Lib/test/test_list.py --- a/Lib/test/test_list.py +++ b/Lib/test/test_list.py @@ -98,6 +98,14 @@ d = pickle.dumps(it) self.assertEqual(self.type2test(it), self.type2test(reversed(data))[1:]) + def test_no_comdat_folding(self): + # Issue 8847: In the PGO build, the MSVC linker's COMDAT folding + # optimization causes failures in code that relies on distinct + # function addresses. + class L(list): pass + with self.assertRaises(TypeError): + (3,) + L([1,2]) + def test_main(verbose=None): support.run_unittest(ListTest) diff --git a/Lib/test/test_tuple.py b/Lib/test/test_tuple.py --- a/Lib/test/test_tuple.py +++ b/Lib/test/test_tuple.py @@ -193,6 +193,14 @@ d = pickle.dumps(it) self.assertEqual(self.type2test(it), self.type2test(reversed(data))[1:]) + def test_no_comdat_folding(self): + # Issue 8847: In the PGO build, the MSVC linker's COMDAT folding + # optimization causes failures in code that relies on distinct + # function addresses. + class T(tuple): pass + with self.assertRaises(TypeError): + [3,] + T((1,2)) + def test_main(): support.run_unittest(TupleTest) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -276,6 +276,8 @@ Build ----- +- Issue #8847: Disable COMDAT folding in Windows PGO builds. + - Issue #14018: Fix OS X Tcl/Tk framework checking when using OS X SDKs. - Issue #15431: Add _freeze_importlib project to regenerate importlib.h diff --git a/PC/VS9.0/pginstrument.vsprops b/PC/VS9.0/pginstrument.vsprops --- a/PC/VS9.0/pginstrument.vsprops +++ b/PC/VS9.0/pginstrument.vsprops @@ -22,7 +22,7 @@ <Tool Name="VCLinkerTool" OptimizeReferences="2" - EnableCOMDATFolding="2" + EnableCOMDATFolding="1" LinkTimeCodeGeneration="2" ProfileGuidedDatabase="$(SolutionDir)$(PlatformName)-pgi\$(TargetName).pgd" ImportLibrary="$(OutDirPGI)\$(TargetName).lib" -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #8847: Disable COMDAT folding in Windows PGO builds.
by martin.v.loewis Aug. 1, 2012

Aug. 1, 2012

http://hg.python.org/cpython/rev/029cde4e58c5 changeset: 78367:029cde4e58c5 parent: 78364:c58f65e22fe9 user: Martin v. Löwis <martin(a)v.loewis.de> date: Wed Aug 01 11:06:53 2012 +0200 summary: Issue #8847: Disable COMDAT folding in Windows PGO builds. Patch by Stefan Krah. files: PCbuild/pginstrument.props | 4 ++-- PCbuild/pylauncher.vcxproj | 14 +++++++------- PCbuild/pywlauncher.vcxproj | 6 +++--- 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/PCbuild/pginstrument.props b/PCbuild/pginstrument.props --- a/PCbuild/pginstrument.props +++ b/PCbuild/pginstrument.props @@ -24,7 +24,7 @@ </ClCompile> <Link> <OptimizeReferences>true</OptimizeReferences> - <EnableCOMDATFolding>true</EnableCOMDATFolding> + <EnableCOMDATFolding>false</EnableCOMDATFolding> <LinkTimeCodeGeneration>PGInstrument</LinkTimeCodeGeneration> <ProfileGuidedDatabase>$(SolutionDir)$(Platform)-pgi\$(TargetName).pgd</ProfileGuidedDatabase> <ImportLibrary>$(OutDirPGI)\$(TargetName).lib</ImportLibrary> @@ -35,4 +35,4 @@ <Value>$(OutDirPGI)</Value> </BuildMacro> </ItemGroup> -</Project> \ No newline at end of file +</Project> diff --git a/PCbuild/pylauncher.vcxproj b/PCbuild/pylauncher.vcxproj --- a/PCbuild/pylauncher.vcxproj +++ b/PCbuild/pylauncher.vcxproj @@ -199,7 +199,7 @@ </ClCompile> <Link> <GenerateDebugInformation>true</GenerateDebugInformation> - <EnableCOMDATFolding>true</EnableCOMDATFolding> + <EnableCOMDATFolding>false</EnableCOMDATFolding> <OptimizeReferences>true</OptimizeReferences> <IgnoreAllDefaultLibraries>false</IgnoreAllDefaultLibraries> <AdditionalDependencies>version.lib;%(AdditionalDependencies)</AdditionalDependencies> @@ -216,7 +216,7 @@ </ClCompile> <Link> <GenerateDebugInformation>true</GenerateDebugInformation> - <EnableCOMDATFolding>true</EnableCOMDATFolding> + <EnableCOMDATFolding>false</EnableCOMDATFolding> <OptimizeReferences>true</OptimizeReferences> <IgnoreAllDefaultLibraries>false</IgnoreAllDefaultLibraries> <AdditionalDependencies>version.lib;%(AdditionalDependencies)</AdditionalDependencies> @@ -233,7 +233,7 @@ </ClCompile> <Link> <GenerateDebugInformation>true</GenerateDebugInformation> - <EnableCOMDATFolding>true</EnableCOMDATFolding> + <EnableCOMDATFolding>false</EnableCOMDATFolding> <OptimizeReferences>true</OptimizeReferences> <IgnoreAllDefaultLibraries>false</IgnoreAllDefaultLibraries> <AdditionalDependencies>version.lib;%(AdditionalDependencies)</AdditionalDependencies> @@ -250,7 +250,7 @@ </ClCompile> <Link> <GenerateDebugInformation>true</GenerateDebugInformation> - <EnableCOMDATFolding>true</EnableCOMDATFolding> + <EnableCOMDATFolding>false</EnableCOMDATFolding> <OptimizeReferences>true</OptimizeReferences> <IgnoreAllDefaultLibraries>false</IgnoreAllDefaultLibraries> <AdditionalDependencies>version.lib;%(AdditionalDependencies)</AdditionalDependencies> @@ -267,7 +267,7 @@ </ClCompile> <Link> <GenerateDebugInformation>true</GenerateDebugInformation> - <EnableCOMDATFolding>true</EnableCOMDATFolding> + <EnableCOMDATFolding>false</EnableCOMDATFolding> <OptimizeReferences>true</OptimizeReferences> <IgnoreAllDefaultLibraries>false</IgnoreAllDefaultLibraries> <AdditionalDependencies>version.lib;%(AdditionalDependencies)</AdditionalDependencies> @@ -284,7 +284,7 @@ </ClCompile> <Link> <GenerateDebugInformation>true</GenerateDebugInformation> - <EnableCOMDATFolding>true</EnableCOMDATFolding> + <EnableCOMDATFolding>false</EnableCOMDATFolding> <OptimizeReferences>true</OptimizeReferences> <IgnoreAllDefaultLibraries>false</IgnoreAllDefaultLibraries> <AdditionalDependencies>version.lib;%(AdditionalDependencies)</AdditionalDependencies> @@ -308,4 +308,4 @@ <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" /> <ImportGroup Label="ExtensionTargets"> </ImportGroup> -</Project> \ No newline at end of file +</Project> diff --git a/PCbuild/pywlauncher.vcxproj b/PCbuild/pywlauncher.vcxproj --- a/PCbuild/pywlauncher.vcxproj +++ b/PCbuild/pywlauncher.vcxproj @@ -143,7 +143,7 @@ </ClCompile> <Link> <GenerateDebugInformation>true</GenerateDebugInformation> - <EnableCOMDATFolding>true</EnableCOMDATFolding> + <EnableCOMDATFolding>false</EnableCOMDATFolding> <OptimizeReferences>true</OptimizeReferences> <IgnoreAllDefaultLibraries>false</IgnoreAllDefaultLibraries> <AdditionalDependencies>version.lib;%(AdditionalDependencies)</AdditionalDependencies> @@ -160,7 +160,7 @@ </ClCompile> <Link> <GenerateDebugInformation>true</GenerateDebugInformation> - <EnableCOMDATFolding>true</EnableCOMDATFolding> + <EnableCOMDATFolding>false</EnableCOMDATFolding> <OptimizeReferences>true</OptimizeReferences> <IgnoreAllDefaultLibraries>false</IgnoreAllDefaultLibraries> <AdditionalDependencies>version.lib;%(AdditionalDependencies)</AdditionalDependencies> @@ -199,4 +199,4 @@ <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" /> <ImportGroup Label="ExtensionTargets"> </ImportGroup> -</Project> \ No newline at end of file +</Project> -- Repository URL: http://hg.python.org/cpython

1 0