Mailman 3 December 2012 - Python-checkins

cpython (merge 3.3 -> default): Make indentation consistent and remove dead commented-out code.
by eli.bendersky 31 Dec '12

31 Dec '12

http://hg.python.org/cpython/rev/0a095821c74e changeset: 81179:0a095821c74e parent: 81176:3738d270c54a parent: 81178:3ca72d4e9762 user: Eli Bendersky <eliben(a)gmail.com> date: Mon Dec 31 15:34:43 2012 -0800 summary: Make indentation consistent and remove dead commented-out code. files: Modules/arraymodule.c | 12 +++++------- 1 files changed, 5 insertions(+), 7 deletions(-) diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c --- a/Modules/arraymodule.c +++ b/Modules/arraymodule.c @@ -2111,7 +2111,7 @@ static PyMethodDef array_methods[] = { {"append", (PyCFunction)array_append, METH_O, append_doc}, - {"buffer_info", (PyCFunction)array_buffer_info, METH_NOARGS, + {"buffer_info", (PyCFunction)array_buffer_info, METH_NOARGS, buffer_info_doc}, {"byteswap", (PyCFunction)array_byteswap, METH_NOARGS, byteswap_doc}, @@ -2119,9 +2119,9 @@ copy_doc}, {"count", (PyCFunction)array_count, METH_O, count_doc}, - {"__deepcopy__",(PyCFunction)array_copy, METH_O, + {"__deepcopy__", (PyCFunction)array_copy, METH_O, copy_doc}, - {"extend", (PyCFunction)array_extend, METH_O, + {"extend", (PyCFunction)array_extend, METH_O, extend_doc}, {"fromfile", (PyCFunction)array_fromfile, METH_VARARGS, fromfile_doc}, @@ -2139,14 +2139,12 @@ insert_doc}, {"pop", (PyCFunction)array_pop, METH_VARARGS, pop_doc}, - {"__reduce_ex__", (PyCFunction)array_reduce_ex, METH_O, + {"__reduce_ex__", (PyCFunction)array_reduce_ex, METH_O, reduce_doc}, {"remove", (PyCFunction)array_remove, METH_O, remove_doc}, {"reverse", (PyCFunction)array_reverse, METH_NOARGS, reverse_doc}, -/* {"sort", (PyCFunction)array_sort, METH_VARARGS, - sort_doc},*/ {"tofile", (PyCFunction)array_tofile, METH_O, tofile_doc}, {"tolist", (PyCFunction)array_tolist, METH_NOARGS, @@ -2155,7 +2153,7 @@ tostring_doc}, {"tobytes", (PyCFunction)array_tobytes, METH_NOARGS, tobytes_doc}, - {"tounicode", (PyCFunction)array_tounicode, METH_NOARGS, + {"tounicode", (PyCFunction)array_tounicode, METH_NOARGS, tounicode_doc}, {"__sizeof__", (PyCFunction)array_sizeof, METH_NOARGS, sizeof_doc}, -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.3): Make indentation consistent and remove dead commented-out code.
by eli.bendersky 31 Dec '12

31 Dec '12

http://hg.python.org/cpython/rev/3ca72d4e9762 changeset: 81178:3ca72d4e9762 branch: 3.3 parent: 81173:d89891f3f769 user: Eli Bendersky <eliben(a)gmail.com> date: Mon Dec 31 15:34:20 2012 -0800 summary: Make indentation consistent and remove dead commented-out code. files: Modules/arraymodule.c | 12 +++++------- 1 files changed, 5 insertions(+), 7 deletions(-) diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c --- a/Modules/arraymodule.c +++ b/Modules/arraymodule.c @@ -2114,7 +2114,7 @@ static PyMethodDef array_methods[] = { {"append", (PyCFunction)array_append, METH_O, append_doc}, - {"buffer_info", (PyCFunction)array_buffer_info, METH_NOARGS, + {"buffer_info", (PyCFunction)array_buffer_info, METH_NOARGS, buffer_info_doc}, {"byteswap", (PyCFunction)array_byteswap, METH_NOARGS, byteswap_doc}, @@ -2122,9 +2122,9 @@ copy_doc}, {"count", (PyCFunction)array_count, METH_O, count_doc}, - {"__deepcopy__",(PyCFunction)array_copy, METH_O, + {"__deepcopy__", (PyCFunction)array_copy, METH_O, copy_doc}, - {"extend", (PyCFunction)array_extend, METH_O, + {"extend", (PyCFunction)array_extend, METH_O, extend_doc}, {"fromfile", (PyCFunction)array_fromfile, METH_VARARGS, fromfile_doc}, @@ -2142,14 +2142,12 @@ insert_doc}, {"pop", (PyCFunction)array_pop, METH_VARARGS, pop_doc}, - {"__reduce_ex__", (PyCFunction)array_reduce_ex, METH_O, + {"__reduce_ex__", (PyCFunction)array_reduce_ex, METH_O, reduce_doc}, {"remove", (PyCFunction)array_remove, METH_O, remove_doc}, {"reverse", (PyCFunction)array_reverse, METH_NOARGS, reverse_doc}, -/* {"sort", (PyCFunction)array_sort, METH_VARARGS, - sort_doc},*/ {"tofile", (PyCFunction)array_tofile, METH_O, tofile_doc}, {"tolist", (PyCFunction)array_tolist, METH_NOARGS, @@ -2158,7 +2156,7 @@ tostring_doc}, {"tobytes", (PyCFunction)array_tobytes, METH_NOARGS, tobytes_doc}, - {"tounicode", (PyCFunction)array_tounicode, METH_NOARGS, + {"tounicode", (PyCFunction)array_tounicode, METH_NOARGS, tounicode_doc}, {"__sizeof__", (PyCFunction)array_sizeof, METH_NOARGS, sizeof_doc}, -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #14958: backport to 2.7.x from 3.3 (patch by Roger Serwy)
by ned.deily 31 Dec '12

31 Dec '12

http://hg.python.org/cpython/rev/0dbdb85fd141 changeset: 81177:0dbdb85fd141 branch: 2.7 parent: 81175:c5c27b84d7af user: Ned Deily <nad(a)acm.org> date: Mon Dec 31 15:06:38 2012 -0800 summary: Issue #14958: backport to 2.7.x from 3.3 (patch by Roger Serwy) files: Lib/idlelib/ColorDelegator.py | 9 +++++---- Misc/NEWS | 3 +++ 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/Lib/idlelib/ColorDelegator.py b/Lib/idlelib/ColorDelegator.py --- a/Lib/idlelib/ColorDelegator.py +++ b/Lib/idlelib/ColorDelegator.py @@ -20,10 +20,11 @@ # 1st 'file' colorized normal, 2nd as builtin, 3rd as string builtin = r"([^.'\"\\#]\b|^)" + any("BUILTIN", builtinlist) + r"\b" comment = any("COMMENT", [r"#[^\n]*"]) - sqstring = r"(\b[rRuU])?'[^'\\\n]*(\\.[^'\\\n]*)*'?" - dqstring = r'(\b[rRuU])?"[^"\\\n]*(\\.[^"\\\n]*)*"?' - sq3string = r"(\b[rRuU])?'''[^'\\]*((\\.|'(?!''))[^'\\]*)*(''')?" - dq3string = r'(\b[rRuU])?"""[^"\\]*((\\.|"(?!""))[^"\\]*)*(""")?' + stringprefix = r"(\br|u|ur|R|U|UR|Ur|uR|b|B|br|Br|bR|BR)?" + sqstring = stringprefix + r"'[^'\\\n]*(\\.[^'\\\n]*)*'?" + dqstring = stringprefix + r'"[^"\\\n]*(\\.[^"\\\n]*)*"?' + sq3string = stringprefix + r"'''[^'\\]*((\\.|'(?!''))[^'\\]*)*(''')?" + dq3string = stringprefix + r'"""[^"\\]*((\\.|"(?!""))[^"\\]*)*(""")?' string = any("STRING", [sq3string, dq3string, sqstring, dqstring]) return kw + "|" + builtin + "|" + comment + "|" + string +\ "|" + any("SYNC", [r"\n"]) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -430,6 +430,9 @@ - Issue #12157: Make pool.map() empty iterables correctly. Initial patch by mouad. +- Issue #14958: Change IDLE systax highlighting to recognize all string and byte + literals currently supported in Python 2.7. + - Issue #14962: Update text coloring in IDLE shell window after changing options. Patch by Roger Serwy. -- Repository URL: http://hg.python.org/cpython

1 0

cpython: st_dev/st_rdev should be unsigned long as dwVolumeSerialNumber, which it is set
by brian.curtin 31 Dec '12

31 Dec '12

http://hg.python.org/cpython/rev/3738d270c54a changeset: 81176:3738d270c54a parent: 81174:e971a70984b8 user: Brian Curtin <brian(a)python.org> date: Mon Dec 31 11:59:48 2012 -0600 summary: st_dev/st_rdev should be unsigned long as dwVolumeSerialNumber, which it is set to, is a DWORD. This was fixed in #11939 and the overflow was mentioned in #10657 and seen by me on some machines. files: Modules/posixmodule.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -1184,13 +1184,13 @@ #define HAVE_STAT_NSEC 1 struct win32_stat{ - int st_dev; + unsigned long st_dev; __int64 st_ino; unsigned short st_mode; int st_nlink; int st_uid; int st_gid; - int st_rdev; + unsigned long st_rdev; __int64 st_size; time_t st_atime; int st_atime_nsec; -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Fix issue 10527: make multiprocessing use poll() instead of select() if
by giampaolo.rodola 31 Dec '12

31 Dec '12

http://hg.python.org/cpython/rev/c5c27b84d7af changeset: 81175:c5c27b84d7af branch: 2.7 parent: 81155:503b889668fc user: Giampaolo Rodola' <g.rodola(a)gmail.com> date: Mon Dec 31 17:23:09 2012 +0100 summary: Fix issue 10527: make multiprocessing use poll() instead of select() if available. files: Lib/multiprocessing/connection.py | 21 +++++++++++++++++++ Lib/test/test_multiprocessing.py | 1 + Misc/NEWS | 2 + 3 files changed, 24 insertions(+), 0 deletions(-) diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -200,6 +200,27 @@ return c1, c2 else: + if hasattr(select, 'poll'): + def _poll(fds, timeout): + if timeout is not None: + timeout = int(timeout) * 1000 # timeout is in milliseconds + fd_map = {} + pollster = select.poll() + for fd in fds: + pollster.register(fd, select.POLLIN) + if hasattr(fd, 'fileno'): + fd_map[fd.fileno()] = fd + else: + fd_map[fd] = fd + ls = [] + for fd, event in pollster.poll(timeout): + if event & select.POLLNVAL: + raise ValueError('invalid file descriptor %i' % fd) + ls.append(fd_map[fd]) + return ls + else: + def _poll(fds, timeout): + return select.select(fds, [], [], timeout)[0] from _multiprocessing import win32 diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -1513,6 +1513,7 @@ self.assertTimingAlmostEqual(poll.elapsed, TIMEOUT1) conn.send(None) + time.sleep(.1) self.assertEqual(poll(TIMEOUT1), True) self.assertTimingAlmostEqual(poll.elapsed, 0) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -175,6 +175,8 @@ Library ------- +- Issue 10527: make multiprocessing use poll() instead of select() if available. + - Issue #16485: Fix file descriptor not being closed if file header patching fails on closing of aifc file. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.3 -> default): Fix issue 10527: make multiprocessing use poll() instead of select() if
by giampaolo.rodola 31 Dec '12

31 Dec '12

http://hg.python.org/cpython/rev/e971a70984b8 changeset: 81174:e971a70984b8 parent: 81171:f6fb5a5748f0 parent: 81173:d89891f3f769 user: Giampaolo Rodola' <g.rodola(a)gmail.com> date: Mon Dec 31 17:38:46 2012 +0100 summary: Fix issue 10527: make multiprocessing use poll() instead of select() if available. files: Lib/multiprocessing/connection.py | 21 +++++++++++++++++++ Lib/test/test_multiprocessing.py | 6 +++- Misc/NEWS | 2 + 3 files changed, 27 insertions(+), 2 deletions(-) diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -509,6 +509,27 @@ return c1, c2 else: + if hasattr(select, 'poll'): + def _poll(fds, timeout): + if timeout is not None: + timeout = int(timeout) * 1000 # timeout is in milliseconds + fd_map = {} + pollster = select.poll() + for fd in fds: + pollster.register(fd, select.POLLIN) + if hasattr(fd, 'fileno'): + fd_map[fd.fileno()] = fd + else: + fd_map[fd] = fd + ls = [] + for fd, event in pollster.poll(timeout): + if event & select.POLLNVAL: + raise ValueError('invalid file descriptor %i' % fd) + ls.append(fd_map[fd]) + return ls + else: + def _poll(fds, timeout): + return select.select(fds, [], [], timeout)[0] def Pipe(duplex=True): ''' diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -2102,6 +2102,7 @@ self.assertTimingAlmostEqual(poll.elapsed, TIMEOUT1) conn.send(None) + time.sleep(.1) self.assertEqual(poll(TIMEOUT1), True) self.assertTimingAlmostEqual(poll.elapsed, 0) @@ -3251,6 +3252,7 @@ from multiprocessing.connection import wait expected = 3 + sorted_ = lambda l: sorted(l, key=lambda x: isinstance(x, int)) sem = multiprocessing.Semaphore(0) a, b = multiprocessing.Pipe() p = multiprocessing.Process(target=self.signal_and_sleep, @@ -3274,7 +3276,7 @@ res = wait([a, p.sentinel, b], 20) delta = time.time() - start - self.assertEqual(res, [p.sentinel, b]) + self.assertEqual(sorted_(res), sorted_([p.sentinel, b])) self.assertLess(delta, 0.4) b.send(None) @@ -3283,7 +3285,7 @@ res = wait([a, p.sentinel, b], 20) delta = time.time() - start - self.assertEqual(res, [a, p.sentinel, b]) + self.assertEqual(sorted_(res), sorted_([a, p.sentinel, b])) self.assertLess(delta, 0.4) p.terminate() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -200,6 +200,8 @@ Library ------- +- Issue 10527: make multiprocessing use poll() instead of select() if available. + - Issue #16688: Fix backreferences did make case-insensitive regex fail on non-ASCII strings. Patch by Matthew Barnett. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> 3.3): Fix issue 10527: make multiprocessing use poll() instead of select() if
by giampaolo.rodola 31 Dec '12

31 Dec '12

http://hg.python.org/cpython/rev/d89891f3f769 changeset: 81173:d89891f3f769 branch: 3.3 parent: 81170:459a23083b66 parent: 81172:5530251d9cac user: Giampaolo Rodola' <g.rodola(a)gmail.com> date: Mon Dec 31 17:38:17 2012 +0100 summary: Fix issue 10527: make multiprocessing use poll() instead of select() if available. files: Lib/multiprocessing/connection.py | 21 +++++++++++++++++++ Lib/test/test_multiprocessing.py | 6 +++- Misc/NEWS | 2 + 3 files changed, 27 insertions(+), 2 deletions(-) diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -509,6 +509,27 @@ return c1, c2 else: + if hasattr(select, 'poll'): + def _poll(fds, timeout): + if timeout is not None: + timeout = int(timeout) * 1000 # timeout is in milliseconds + fd_map = {} + pollster = select.poll() + for fd in fds: + pollster.register(fd, select.POLLIN) + if hasattr(fd, 'fileno'): + fd_map[fd.fileno()] = fd + else: + fd_map[fd] = fd + ls = [] + for fd, event in pollster.poll(timeout): + if event & select.POLLNVAL: + raise ValueError('invalid file descriptor %i' % fd) + ls.append(fd_map[fd]) + return ls + else: + def _poll(fds, timeout): + return select.select(fds, [], [], timeout)[0] def Pipe(duplex=True): ''' diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -2127,6 +2127,7 @@ self.assertTimingAlmostEqual(poll.elapsed, TIMEOUT1) conn.send(None) + time.sleep(.1) self.assertEqual(poll(TIMEOUT1), True) self.assertTimingAlmostEqual(poll.elapsed, 0) @@ -3232,6 +3233,7 @@ from multiprocessing.connection import wait expected = 3 + sorted_ = lambda l: sorted(l, key=lambda x: isinstance(x, int)) sem = multiprocessing.Semaphore(0) a, b = multiprocessing.Pipe() p = multiprocessing.Process(target=self.signal_and_sleep, @@ -3255,7 +3257,7 @@ res = wait([a, p.sentinel, b], 20) delta = time.time() - start - self.assertEqual(res, [p.sentinel, b]) + self.assertEqual(sorted_(res), sorted_([p.sentinel, b])) self.assertLess(delta, 0.4) b.send(None) @@ -3264,7 +3266,7 @@ res = wait([a, p.sentinel, b], 20) delta = time.time() - start - self.assertEqual(res, [a, p.sentinel, b]) + self.assertEqual(sorted_(res), sorted_([a, p.sentinel, b])) self.assertLess(delta, 0.4) p.terminate() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -124,6 +124,8 @@ Library ------- +- Issue 10527: make multiprocessing use poll() instead of select() if available. + - Issue #16688: Fix backreferences did make case-insensitive regex fail on non-ASCII strings. Patch by Matthew Barnett. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Fix issue 10527: make multiprocessing use poll() instead of select() if
by giampaolo.rodola 31 Dec '12

31 Dec '12

http://hg.python.org/cpython/rev/5530251d9cac changeset: 81172:5530251d9cac branch: 3.2 parent: 81165:0786dfc3b2b4 user: Giampaolo Rodola' <g.rodola(a)gmail.com> date: Mon Dec 31 17:23:09 2012 +0100 summary: Fix issue 10527: make multiprocessing use poll() instead of select() if available. files: Lib/multiprocessing/connection.py | 21 +++++++++++++++++++ Lib/test/test_multiprocessing.py | 1 + Misc/NEWS | 2 + 3 files changed, 24 insertions(+), 0 deletions(-) diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -213,6 +213,27 @@ return c1, c2 else: + if hasattr(select, 'poll'): + def _poll(fds, timeout): + if timeout is not None: + timeout = int(timeout) * 1000 # timeout is in milliseconds + fd_map = {} + pollster = select.poll() + for fd in fds: + pollster.register(fd, select.POLLIN) + if hasattr(fd, 'fileno'): + fd_map[fd.fileno()] = fd + else: + fd_map[fd] = fd + ls = [] + for fd, event in pollster.poll(timeout): + if event & select.POLLNVAL: + raise ValueError('invalid file descriptor %i' % fd) + ls.append(fd_map[fd]) + return ls + else: + def _poll(fds, timeout): + return select.select(fds, [], [], timeout)[0] from _multiprocessing import win32 diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -1574,6 +1574,7 @@ self.assertTimingAlmostEqual(poll.elapsed, TIMEOUT1) conn.send(None) + time.sleep(.1) self.assertEqual(poll(TIMEOUT1), True) self.assertTimingAlmostEqual(poll.elapsed, 0) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -189,6 +189,8 @@ Library ------- +- Issue 10527: make multiprocessing use poll() instead of select() if available. + - Issue #16485: Fix file descriptor not being closed if file header patching fails on closing of aifc file. -- Repository URL: http://hg.python.org/cpython

1 0

peps: Claim the packaging PEPs (and strip trailing whitespace)
by nick.coghlan 31 Dec '12

31 Dec '12

http://hg.python.org/peps/rev/7f88cb558d79 changeset: 4645:7f88cb558d79 user: Nick Coghlan <ncoghlan(a)gmail.com> date: Tue Jan 01 01:45:03 2013 +1000 summary: Claim the packaging PEPs (and strip trailing whitespace) files: pep-0425.txt | 7 ++++--- pep-0426.txt | 23 ++++++++++++----------- pep-0427.txt | 5 +++-- 3 files changed, 19 insertions(+), 16 deletions(-) diff --git a/pep-0425.txt b/pep-0425.txt --- a/pep-0425.txt +++ b/pep-0425.txt @@ -1,8 +1,9 @@ PEP: 425 -Title: Compatibility Tags for Built Distributions +Title: Compatibility Tags for Built Distributions Version: $Revision$ Last-Modified: 07-Aug-2012 Author: Daniel Holth <dholth(a)fastmail.fm> +BDFL-Delegate: Nick Coghlan <ncoghlan(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst @@ -216,7 +217,7 @@ Who will maintain the registry of abbreviated implementations? New two-letter abbreviations can be requested on the python-dev mailing list. As a rule of thumb, abbreviations are reserved for - the current 4 most prominent implementations. + the current 4 most prominent implementations. Does the compatibility tag go into METADATA or PKG-INFO? No. The compatibility tag is part of the built distribution's @@ -253,7 +254,7 @@ This document has been placed in the public domain. - + .. Local Variables: mode: indented-text diff --git a/pep-0426.txt b/pep-0426.txt --- a/pep-0426.txt +++ b/pep-0426.txt @@ -3,6 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth <dholth(a)fastmail.fm> +BDFL-Delegate: Nick Coghlan <ncoghlan(a)gmail.com> Discussions-To: Distutils SIG Status: Draft Type: Standards Track @@ -618,7 +619,7 @@ - Extension - Provides-Extra - Setup-Requires-Dist - + References ========== @@ -643,15 +644,15 @@ # Metadata 1.3 demo from email.generator import Generator - from email import header - from email.parser import Parser - from email.policy import Compat32 + from email import header + from email.parser import Parser + from email.policy import Compat32 from email.utils import _has_surrogates class MetadataPolicy(Compat32): max_line_length = 0 continuation_whitespace = '\t' - + def _sanitize_header(self, name, value): if not isinstance(value, str): return value @@ -676,8 +677,8 @@ Summary: A package. Description: Description =========== - - + + A description of the package. """ @@ -687,9 +688,9 @@ m['License'] = 'GPL' description = m['Description'] description_lines = description.splitlines() - m.set_payload(description_lines[0] - + '\n' - + textwrap.dedent('\n'.join(description_lines[1:])) + m.set_payload(description_lines[0] + + '\n' + + textwrap.dedent('\n'.join(description_lines[1:])) + '\n') del m['Description'] @@ -701,7 +702,7 @@ This document has been placed in the public domain. - + .. Local Variables: mode: indented-text diff --git a/pep-0427.txt b/pep-0427.txt --- a/pep-0427.txt +++ b/pep-0427.txt @@ -3,6 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth <dholth(a)fastmail.fm> +BDFL-Delegate: Nick Coghlan <ncoghlan(a)gmail.com> Discussions-To: <distutils-sig(a)python.org> Status: Draft Type: Standards Track @@ -319,7 +320,7 @@ ======== Example urlsafe-base64-nopad implementation:: - + # urlsafe-base64-nopad for Python 3 import base64 @@ -337,7 +338,7 @@ This document has been placed into the public domain. - + .. Local Variables: mode: indented-text -- Repository URL: http://hg.python.org/peps

1 0

peps: Further PEP 432 updates
by nick.coghlan 31 Dec '12

31 Dec '12

http://hg.python.org/peps/rev/bd623fe7ad27 changeset: 4641:bd623fe7ad27 user: Nick Coghlan <ncoghlan(a)gmail.com> date: Sun Dec 30 23:39:20 2012 +1000 summary: Further PEP 432 updates - describe all 4 proposed initialisation phases - more detailed interface for hash seed handling - move ignore environment flag into core config - consistently use American spelling of initialize - preliminary concept for main execution API - misc notes on status quo files: pep-0432.txt | 246 ++++++++++++++++++++++++++++---------- 1 files changed, 181 insertions(+), 65 deletions(-) diff --git a/pep-0432.txt b/pep-0432.txt --- a/pep-0432.txt +++ b/pep-0432.txt @@ -15,7 +15,7 @@ ======== This PEP proposes a mechanism for simplifying the startup sequence for -CPython, making it easier to modify the initialisation behaviour of the +CPython, making it easier to modify the initialization behaviour of the reference interpreter executable, as well as making it easier to control CPython's startup behaviour when creating an alternate executable or embedding it as a Python execution engine inside a larger application. @@ -25,15 +25,24 @@ implementation is developed. -Proposal Summary -================ +Proposal +======== -This PEP proposes that CPython move to an explicit 2-phase initialisation +This PEP proposes that CPython move to an explicit multi-phase initialization process, where a preliminary interpreter is put in place with limited OS interaction capabilities early in the startup sequence. This essential core remains in place while all of the configuration settings are determined, until a final configuration call takes those settings and finishes -bootstrapping the interpreter immediately before executing the main module. +bootstrapping the interpreter immediately before locating and executing +the main module. + +In the new design, the interpreter will move through the following +well-defined phases during the startup sequence: + +* Pre-Initialization - no interpreter available +* Initialization - limited interpreter available +* Pre-Main - full interpreter available, __main__ related metadata incomplete +* Main Execution - normal interpreter operation As a concrete use case to help guide any design changes, and to solve a known problem where the appropriate defaults for system utilities differ from those @@ -46,20 +55,21 @@ To keep the implementation complexity under control, this PEP does *not* propose wholesale changes to the way the interpreter state is accessed at runtime, nor does it propose changes to the way subinterpreters are -created after the main interpreter has already been initialised. Changing -the order in which the existing initialisation steps occur to make the -startup sequence easier to maintain is already a substantial change, and +created after the main interpreter has already been initialized. Changing +the order in which the existing initialization steps occur in order to make +the startup sequence easier to maintain is already a substantial change, and attempting to make those other changes at the same time will make the change significantly more invasive and much harder to review. However, such proposals may be suitable topics for follow-on PEPs or patches - one key benefit of this PEP is decreasing the coupling between the internal storage -model and the configuration interface. +model and the configuration interface, so such changes should be easier +once this PEP has been implemented. Background ========== -Over time, CPython's initialisation sequence has become progressively more +Over time, CPython's initialization sequence has become progressively more complicated, offering more options, as well as performing more complex tasks (such as configuring the Unicode settings for OS interfaces in Python 3 as well as bootstrapping a pure Python implementation of the import system). @@ -72,7 +82,7 @@ safely. A number of proposals are on the table for even *more* sophisticated -startup behaviour, such as better control over ``sys.path`` initialisation +startup behaviour, such as better control over ``sys.path`` initialization (easily adding additional directories on the command line in a cross-platform fashion, as well as controlling the configuration of ``sys.path[0]``), easier configuration of utilities like coverage tracing when launching Python @@ -96,14 +106,14 @@ The current CPython startup sequence is difficult to understand, and even more difficult to modify. It is not clear what state the interpreter is in -while much of the initialisation code executes, leading to behaviour such +while much of the initialization code executes, leading to behaviour such as lists, dictionaries and Unicode values being created prior to the call to ``Py_Initialize`` when the ``-X`` or ``-W`` options are used [1_]. -By moving to a 2-phase startup sequence, developers should only need to -understand which features are not available in the core bootstrapping state, -as the vast majority of the configuration process will now take place in -that state. +By moving to an explicitly multi-phase startup sequence, developers should +only need to understand which features are not available in the core +bootstrapping state, as the vast majority of the configuration process +will now take place in that state. By basing the new design on a combination of C structures and Python dictionaries, it should also be easier to modify the system in the @@ -114,7 +124,7 @@ ----------- CPython is used heavily to run short scripts where the runtime is dominated -by the interpreter initialisation time. Any changes to the startup sequence +by the interpreter initialization time. Any changes to the startup sequence should minimise their impact on the startup overhead. Experience with the importlib migration suggests that the startup time is @@ -141,15 +151,15 @@ Improvements in the import system and the Unicode support already resulted in a more than 30% improvement in startup time in Python 3.3 relative to 3.2. Python 3.3 is still slightly slower to start than Python 2.7 due to the -additional infrastructure that needs to be put in place to support the Unicode -based text model. +additional infrastructure that needs to be put in place to support the +Unicode based text model. This PEP is not expected to have any significant effect on the startup time, -as it is aimed primarily at *reordering* the existing initialisation +as it is aimed primarily at *reordering* the existing initialization sequence, without making substantial changes to the individual steps. However, if this simple check suggests that the proposed changes to the -initialisation sequence may pose a performance problem, then a more +initialization sequence may pose a performance problem, then a more sophisticated microbenchmark will be developed to assist in investigation. @@ -198,7 +208,7 @@ * ``no_site`` (don't implicitly import site during startup) * ``ignore_environment`` (whether environment vars are used during config) * ``verbose`` (enable all sorts of random output) - * ``bytes_warning`` + * ``bytes_warning`` (warnings/errors for implicit str/bytes interaction) * ``quiet`` (disable banner output even if verbose is also enabled or stdin is a tty and the interpreter is launched in interactive mode) @@ -219,7 +229,7 @@ Note that this just covers settings that are currently configurable in some manner when using the main CPython executable. While this PEP aims to make adding additional configuration settings easier in the future, it -deliberately avoids any new settings of its own. +deliberately avoids adding any new settings of its own. The Status Quo @@ -238,13 +248,14 @@ ------------------------------ The ``-E`` command line option allows all environment variables to be -ignored when initialising the Python interpreter. An embedding application +ignored when initializing the Python interpreter. An embedding application can enable this behaviour by setting ``Py_IgnoreEnvironmentFlag`` before calling ``Py_Initialize()``. In the CPython source code, the ``Py_GETENV`` macro implicitly checks this flag, and always produces ``NULL`` if it is set. +<TBD: I believe PYTHONCASEOK is checked regardless of this setting > <TBD: Does -E also ignore Windows registry keys? > @@ -266,7 +277,8 @@ The new configuration API should make it straightforward for an embedding application to reuse the ``PYTHONHASHSEED`` processing with -a text based configuration setting provided by other means. +a text based configuration setting provided by other means (e.g. a +config file or separate environment variable). Locating Python and the standard library @@ -301,7 +313,7 @@ An embedding application may call ``Py_SetPath()`` prior to ``Py_Initialize()`` to completely override the calculation of ``sys.path``. It is not straightforward to only allow *some* of the -calculations, as modifying ``sys.path`` after initialisation is +calculations, as modifying ``sys.path`` after initialization is already complete means those modifications will not be in effect when standard library modules are imported during the startup sequence. @@ -332,10 +344,10 @@ (Note: you can see similar information using ``-m site`` instead of ``-c``, but this is slightly misleading as it calls ``os.abspath`` on all of the -path entries (making relative path entries look absolute), and also causes -problems in the last case, as on Python versions prior to 3.3, explicitly -importing site will carry out the path modifications ``-S`` avoids, while on -3.3+ combining ``-m site`` with ``-S`` currently fails) +path entries, making relative path entries look absolute. Using the ``site`` +module also causes problems in the last case, as on Python versions prior to +3.3, explicitly importing site will carry out the path modifications ``-S`` +avoids, while on 3.3+ combining ``-m site`` with ``-S`` currently fails) The calculation of ``sys.path[0]`` is comparatively straightforward: @@ -386,7 +398,7 @@ Other configuration settings ---------------------------- -TBD: Cover the initialisation of the following in more detail: +TBD: Cover the initialization of the following in more detail: * The initial warning system state: * ``sys.warnoptions`` @@ -419,7 +431,7 @@ * ``no_site`` (don't implicitly import site during startup) * ``ignore_environment`` (whether environment vars are used during config) * ``verbose`` (enable all sorts of random output) - * ``bytes_warning`` (This may be obsolete in Py3k...) + * ``bytes_warning`` (warnings/errors for implicit str/bytes interaction) * ``quiet`` (disable banner output even if verbose is also enabled or stdin is a tty and the interpreter is launched in interactive mode) @@ -428,15 +440,15 @@ Much of the configuration of CPython is currently handled through C level global variables:: - Py_BytesWarningFlag + Py_BytesWarningFlag (-b) Py_DebugFlag (-d option) Py_InspectFlag (-i option, PYTHONINSPECT) - Py_InteractiveFlag + Py_InteractiveFlag (property of stdin, cannot be overridden) Py_OptimizeFlag (-O option, PYTHONOPTIMIZE) Py_DontWriteBytecodeFlag (-B option, PYTHONDONTWRITEBYTECODE) Py_NoUserSiteDirectory (-s option, PYTHONNOUSERSITE) Py_NoSiteFlag (-S option) - Py_UnbufferedStdioFlag + Py_UnbufferedStdioFlag (-u, PYTHONUNBUFFEREDIO) Py_VerboseFlag (-v option, PYTHONVERBOSE) For the above variables, the conversion of command line options and @@ -463,34 +475,63 @@ Also see detailed sequence of operations notes at [1_] -Proposal -======== +Design Details +============== (Note: details here are still very much in flux, but preliminary feedback is appreciated anyway) The main theme of this proposal is to create the interpreter state for the main interpreter *much* earlier in the startup process. This will allow -most of the CPython API to be used during the remainder of the initialisation +most of the CPython API to be used during the remainder of the initialization process, potentially simplifying a number of operations that currently need to rely on basic C functionality rather than being able to use the richer data structures provided by the CPython C API. +In the following, the term "embedding application" also covers the standard +CPython command line application. -Core Interpreter Initialisation -------------------------------- -The only configuration that currently absolutely needs to be in place -before even the interpreter core can be initialised is a flag indicating -whether or not to use a specific seed value for the randomised hashes, and -if so, the specific value for the seed (a seed value of zero disables -randomised hashing). +Startup Phases +-------------- + +Four distinct phases are proposed: + +* Pre-Initialization: no interpreter is available. Embedding application + determines the settings required to create the core interpreter and + moves to the next phase by calling ``Py_BeginInitialization``. +* Initialization - a limited interpreter is available. Embedding application + determines and applies the settings required to complete the initialization + process by calling ``Py_ReadConfiguration`` and ``Py_EndInitialization``. +* Pre-Main - the full interpreter is available, but ``__main__`` related + metadata is incomplete. +* Main Execution - normal interpreter operation + +All 4 phases will be used by the standard CPython interpreter and the +proposed System Python interpreter. Other embedding applications may +choose to skip the step of executing code in the ``__main__`` module. + +Pre-Initialization Phase +------------------------ + +The pre-initialization phase is where an embedding application determines +the settings which are absolutely required before the interpreter can be +initialized at all. Currently, the only configuration settings in this +category are those related to the randomised hash algorithm - the hash +algorithms must be consistent for the lifetime of the process, and so they +must be in place before the core interpreter is created. + +The specific settings needed are a flag indicating whether or not to use a +specific seed value for the randomised hashes, and if so, the specific value +for the seed (a seed value of zero disables randomised hashing). In addition, +the question of whether or not to consider environment variables must be +addressed early. The proposed API for this step in the startup sequence is:: void Py_BeginInitialization(Py_CoreConfig *config); -Like Py_Initialize, this part of the new API treats initialisation failures +Like Py_Initialize, this part of the new API treats initialization failures as fatal errors. While that's still not particularly embedding friendly, the operations in this step *really* shouldn't be failing, and changing them to return error codes instead of aborting would be an even larger task than @@ -500,16 +541,50 @@ configuration:: typedef struct { + int ignore_environment; int use_hash_seed; unsigned long hash_seed; } Py_CoreConfig; -To disable hash randomisation, set "use_hash_seed" and pass a hash seed of -zero. (This is the same approach already used when interpreting the -``PYTHONHASHSEED`` environment variable) +The core configuration settings pointer may be ``NULL``, in which case the +default values are ``ignore_environment = 0`` and ``use_hash_seed = -1``. -The core configuration settings pointer may be NULL, in which case the -default behaviour of randomised hashes with a random seed will be used. +``ignore_environment`` controls the processing of all Python related +environment variables. If the flag is zero, then environment variables are +processed normally. Otherwise, all Python-specific environment variables +are considered undefined (exceptions may be made for some OS specific +environment variables, such as those used on Mac OS X to communicate +between the App bundle and the main Python binary). + +``use_hash_seed`` controls the configuration of the randomised hash +algorithm. If it is zero, then randomised hashes with a random seed will +be used. It it is positive, then the value in ``hash_seed`` will be used +to seed the random number generator. If the ``hash_seed`` is zero in this +case, then the randomised hashing is disabled completely. + +If ``use_hash_seed`` is negative (and ``ignore_environment`` is zero), +then CPython will inspect the ``PYTHONHASHSEED`` environment variable. If it +is not set, is set to the empty string, or to the value ``"random"``, then +randomised hashes with a random seed will be used. If it is set to the string +``"0"`` the randomised hashing will be disabled. Otherwise, the hash seed is +expected to be a string representation of an integer in the range +``[0; 4294967295]``. + +To make it easier for embedding applications to use the ``PYTHONHASHSEED`` +processing with a different data source, the following helper function +will be added to the C API:: + + int Py_ReadHashSeed(char *seed_text, + int *use_hash_seed, + unsigned long *hash_seed); + +This function accepts a seed string in ``seed_text`` and converts it to +the appropriate flag and seed values. If ``seed_text`` is ``NULL``, +the empty string or the value ``"random"``, both ``use_hash_seed`` and +``hash_seed`` will be set to zero. Otherwise, ``use_hash_seed`` will be set to +``1`` and the seed text will be interpreted as an integer and reported as +``hash_seed``. On success the function will return zero. A non-zero return +value indicates an error (most likely in the conversion to an integer). The aim is to keep this initial level of configuration as small as possible in order to keep the bootstrapping environment consistent across @@ -519,14 +594,14 @@ A new query API will allow code to determine if the interpreter is in the bootstrapping state between the creation of the interpreter state and the -completion of the bulk of the initialisation process:: +completion of the bulk of the initialization process:: int Py_IsInitializing(); Attempting to call ``Py_BeginInitialization()`` again when ``Py_IsInitializing()`` or ``Py_IsInitialized()`` is true is a fatal error. -While in the initialising state, the interpreter should be fully functional +While in the initializing state, the interpreter should be fully functional except that: * compilation is not allowed (as the parser and compiler are not yet @@ -551,7 +626,7 @@ * only builtin and frozen modules may be imported (due to above limitations) * ``sys.stderr`` is set to a temporary IO object using unbuffered binary mode -* The ``warnings`` module is not yet initialised +* The ``warnings`` module is not yet initialized * The ``__main__`` module does not yet exist <TBD: identify any other notable missing functionality> @@ -573,7 +648,7 @@ Determining the remaining configuration settings ------------------------------------------------ -The next step in the initialisation sequence is to determine the full +The next step in the initialization sequence is to determine the full settings needed to complete the process. No changes are made to the interpreter state at this point. The core API for this step is:: @@ -630,11 +705,12 @@ <TBD: at least more from sys.flags need to go here> -Completing the interpreter initialisation +Completing the interpreter initialization ----------------------------------------- -The final step in the process is to actually put the configuration settings -into effect and finish bootstrapping the interpreter up to full operation:: +The final step in the initialization process is to actually put the +configuration settings into effect and finish bootstrapping the interpreter +up to full operation:: int Py_EndInitialization(PyObject *config); @@ -648,7 +724,48 @@ After a successful call, Py_IsInitializing() will be false, while Py_IsInitialized() will become true. The caveats described above for the -interpreter during the initialisation phase will no longer hold. +interpreter during the initialization phase will no longer hold. + +However, some metadata related to the ``__main__`` module may still be +incomplete: + +* ``sys.argv[0]`` may not yet have its final value + * it will be ``-m`` when executing a module or package with CPython + * it will be the same as ``sys.path[0]`` rather than the location of + the ``__main__`` module when executing a valid ``sys.path`` entry + (typically a zipfile or directory) +* the metadata in the ``__main__`` module will still indicate it is a + builtin module + + +Executing the main module +------------------------- + +<TBD> + +Initial thought is that hiding the various options behind a single API +would make that API too complicated, so 3 separate APIs is more likely:: + + Py_RunPathAsMain + Py_RunModuleAsMain + Py_RunStreamAsMain + + +Internal Storage of Configuration Data +-------------------------------------- + +The interpreter state will be updated to include details of the configuration +settings supplied during initialization by extending the interpreter state +object with an embedded copy of the ``Py_CoreConfig`` struct and an +additional ``PyObject`` pointer to hold a reference to a copy of the +supplied configuration dictionary. + +For debugging purposes, the copied configuration dictionary will be +exposed as ``sys._configuration``. It will include additional keys for +the fields in the ``Py_CoreConfig`` struct. + +These are *snapshots* of the initial configuration settings. They are not +consulted by the interpreter during runtime. Stable ABI @@ -670,7 +787,7 @@ One acknowledged incompatiblity is that some environment variables which are currently read lazily may instead be read once during interpreter -initialisation. As the PEP matures, these will be discussed in more detail +initialization. As the PEP matures, these will be discussed in more detail on a case by case basis. The environment variables which are currently known to be looked up dynamically are: @@ -680,7 +797,7 @@ * ``PYTHONINSPECT``: ``os.environ['PYTHONINSPECT']`` will still be checked after execution of the ``__main__`` module terminates -The ``Py_Initialize()`` style of initialisation will continue to be +The ``Py_Initialize()`` style of initialization will continue to be supported. It will use (at least some elements of) the new API internally, but will continue to exhibit the same behaviour as it does today, ensuring that ``sys.argv`` is not populated until a subsequent @@ -691,7 +808,7 @@ To minimise unnecessary code churn, and to ensure the backwards compatibility is well tested, the main CPython executable may continue to use some elements -of the old style initialisation API. (very much TBC) +of the old style initialization API. (very much TBC) A System Python Executable @@ -712,8 +829,8 @@ change in the PEP is designed to help avoid acceptance of a design that sounds good in theory but proves to be problematic in practice. -One final aspect not addressed by the general embedding changes above is -the current inaccessibility of the core logic for deciding between the +Better supporting this kind of "alternate CLI" is the main reason for the +proposed changes to better expose the core logic for deciding between the different execution modes supported by CPython: * script execution @@ -723,7 +840,6 @@ * execution from stdin (non-interactive) * interactive stdin -<TBD: concrete proposal for better exposing the __main__ execution step> Implementation ============== -- Repository URL: http://hg.python.org/peps

3 2