From python-checkins at python.org Sun Dec 1 00:03:25 2013 From: python-checkins at python.org (zach.ware) Date: Sun, 1 Dec 2013 00:03:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5ODQ1?= =?utf-8?q?=3A_Updated_the_Compiling_Python_on_Windows_docs=2E?= Message-ID: <3dX7Rd6xp6z7Lk2@mail.python.org> http://hg.python.org/cpython/rev/a239897084bf changeset: 87662:a239897084bf branch: 3.3 parent: 87660:5c8130b85c17 user: Zachary Ware date: Sat Nov 30 16:59:33 2013 -0600 summary: Issue #19845: Updated the Compiling Python on Windows docs. files: Doc/using/windows.rst | 18 ++++++++++-------- Misc/NEWS | 2 ++ 2 files changed, 12 insertions(+), 8 deletions(-) diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -556,9 +556,9 @@ latest release's source or just grab a fresh `checkout `_. -For Microsoft Visual C++, which is the compiler with which official Python -releases are built, the source tree contains solutions/project files. View the -:file:`readme.txt` in their respective directories: +The source tree contains a build solution and project files for Microsoft +Visual C++, which is the compiler used to build the official Python releases. +View the :file:`readme.txt` in their respective directories: +--------------------+--------------+-----------------------+ | Directory | MSVC version | Visual Studio version | @@ -569,14 +569,16 @@ +--------------------+--------------+-----------------------+ | :file:`PC/VS8.0/` | 8.0 | 2005 | +--------------------+--------------+-----------------------+ -| :file:`PCbuild/` | 9.0 | 2008 | +| :file:`PC/VS9.0/` | 9.0 | 2008 | ++--------------------+--------------+-----------------------+ +| :file:`PCbuild/` | 10.0 | 2010 | +--------------------+--------------+-----------------------+ -Note that not all of these build directories are fully supported. Read the -release notes to see which compiler version the official releases for your -version are built with. +Note that any build directories within the :file:`PC` directory are not +necessarily fully supported. The :file:`PCbuild` directory contains the files +for the compiler used to build the official release. -Check :file:`PC/readme.txt` for general information on the build process. +Check :file:`PCbuild/readme.txt` for general information on the build process. For extension modules, consult :ref:`building-on-windows`. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -95,6 +95,8 @@ Documentation ------------- +- Issue #19845: Updated the Compiling Python on Windows section. + - Issue #19795: Improved markup of True/False constants. - Issue #18326: Clarify that list.sort's arguments are keyword-only. Also, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 00:03:27 2013 From: python-checkins at python.org (zach.ware) Date: Sun, 1 Dec 2013 00:03:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319845=3A_Updated_the_Compiling_Python_on_Window?= =?utf-8?q?s_docs=2E?= Message-ID: <3dX7Rg1snGz7Lky@mail.python.org> http://hg.python.org/cpython/rev/f4eedd29706b changeset: 87663:f4eedd29706b parent: 87661:eae4b83108fb parent: 87662:a239897084bf user: Zachary Ware date: Sat Nov 30 17:03:03 2013 -0600 summary: Issue #19845: Updated the Compiling Python on Windows docs. files: Doc/using/windows.rst | 22 +++++++++------------- Misc/NEWS | 2 ++ 2 files changed, 11 insertions(+), 13 deletions(-) diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -556,27 +556,23 @@ latest release's source or just grab a fresh `checkout `_. -For Microsoft Visual C++, which is the compiler with which official Python -releases are built, the source tree contains solutions/project files. View the -:file:`readme.txt` in their respective directories: +The source tree contains a build solution and project files for Microsoft +Visual C++, which is the compiler used to build the official Python releases. +View the :file:`readme.txt` in their respective directories: +--------------------+--------------+-----------------------+ | Directory | MSVC version | Visual Studio version | +====================+==============+=======================+ -| :file:`PC/VC6/` | 6.0 | 97 | +| :file:`PC/VS9.0/` | 9.0 | 2008 | +--------------------+--------------+-----------------------+ -| :file:`PC/VS7.1/` | 7.1 | 2003 | -+--------------------+--------------+-----------------------+ -| :file:`PC/VS8.0/` | 8.0 | 2005 | -+--------------------+--------------+-----------------------+ -| :file:`PCbuild/` | 9.0 | 2008 | +| :file:`PCbuild/` | 10.0 | 2010 | +--------------------+--------------+-----------------------+ -Note that not all of these build directories are fully supported. Read the -release notes to see which compiler version the official releases for your -version are built with. +Note that any build directories within the :file:`PC` directory are not +necessarily fully supported. The :file:`PCbuild` directory contains the files +for the compiler used to build the official release. -Check :file:`PC/readme.txt` for general information on the build process. +Check :file:`PCbuild/readme.txt` for general information on the build process. For extension modules, consult :ref:`building-on-windows`. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -63,6 +63,8 @@ Documentation ------------- +- Issue #19845: Updated the Compiling Python on Windows section. + - Issue #19795: Improved markup of True/False constants. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 00:35:53 2013 From: python-checkins at python.org (guido.van.rossum) Date: Sun, 1 Dec 2013 00:35:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_Use_Interface_i?= =?utf-8?q?nstead_of_ABC=2E__Fixes_issue_19726=2E?= Message-ID: <3dX8950mryz7Ljl@mail.python.org> http://hg.python.org/cpython/rev/5469e1a68dbd changeset: 87664:5469e1a68dbd user: Guido van Rossum date: Sat Nov 30 15:35:42 2013 -0800 summary: asyncio: Use Interface instead of ABC. Fixes issue 19726. files: Lib/asyncio/events.py | 4 ++-- Lib/asyncio/protocols.py | 8 ++++---- Lib/asyncio/transports.py | 10 +++++----- 3 files changed, 11 insertions(+), 11 deletions(-) diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py --- a/Lib/asyncio/events.py +++ b/Lib/asyncio/events.py @@ -234,7 +234,7 @@ protocol_factory should instantiate object with Protocol interface. pipe is file-like object already switched to nonblocking. Return pair (transport, protocol), where transport support - ReadTransport ABC""" + ReadTransport interface.""" # The reason to accept file-like object instead of just file descriptor # is: we need to own pipe and close it at transport finishing # Can got complicated errors if pass f.fileno(), @@ -247,7 +247,7 @@ protocol_factory should instantiate object with BaseProtocol interface. Pipe is file-like object already switched to nonblocking. Return pair (transport, protocol), where transport support - WriteTransport ABC""" + WriteTransport interface.""" # The reason to accept file-like object instead of just file descriptor # is: we need to own pipe and close it at transport finishing # Can got complicated errors if pass f.fileno(), diff --git a/Lib/asyncio/protocols.py b/Lib/asyncio/protocols.py --- a/Lib/asyncio/protocols.py +++ b/Lib/asyncio/protocols.py @@ -4,7 +4,7 @@ class BaseProtocol: - """ABC for base protocol class. + """Common base class for protocol interfaces. Usually user implements protocols that derived from BaseProtocol like Protocol or ProcessProtocol. @@ -59,7 +59,7 @@ class Protocol(BaseProtocol): - """ABC representing a protocol. + """Interface for stream protocol. The user should implement this interface. They can inherit from this class but don't need to. The implementations here do @@ -95,7 +95,7 @@ class DatagramProtocol(BaseProtocol): - """ABC representing a datagram protocol.""" + """Interface for datagram protocol.""" def datagram_received(self, data, addr): """Called when some datagram is received.""" @@ -108,7 +108,7 @@ class SubprocessProtocol(BaseProtocol): - """ABC representing a protocol for subprocess calls.""" + """Interface for protocol for subprocess calls.""" def pipe_data_received(self, fd, data): """Called when the subprocess writes data into stdout/stderr pipe. diff --git a/Lib/asyncio/transports.py b/Lib/asyncio/transports.py --- a/Lib/asyncio/transports.py +++ b/Lib/asyncio/transports.py @@ -4,7 +4,7 @@ class BaseTransport: - """Base ABC for transports.""" + """Base class for transports.""" def __init__(self, extra=None): if extra is None: @@ -27,7 +27,7 @@ class ReadTransport(BaseTransport): - """ABC for read-only transports.""" + """Interface for read-only transports.""" def pause_reading(self): """Pause the receiving end. @@ -47,7 +47,7 @@ class WriteTransport(BaseTransport): - """ABC for write-only transports.""" + """Interface for write-only transports.""" def set_write_buffer_limits(self, high=None, low=None): """Set the high- and low-water limits for write flow control. @@ -115,7 +115,7 @@ class Transport(ReadTransport, WriteTransport): - """ABC representing a bidirectional transport. + """Interface representing a bidirectional transport. There may be several implementations, but typically, the user does not implement new transports; rather, the platform provides some @@ -137,7 +137,7 @@ class DatagramTransport(BaseTransport): - """ABC for datagram (UDP) transports.""" + """Interface for datagram (UDP) transports.""" def sendto(self, data, addr=None): """Send data to the transport. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 01:21:43 2013 From: python-checkins at python.org (alexandre.vassalotti) Date: Sun, 1 Dec 2013 01:21:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzY0Nzc6?= =?utf-8?q?_Added_support_for_pickling_the_types_of_built-in_singletons=2E?= Message-ID: <3dX99z06sGz7Ljm@mail.python.org> http://hg.python.org/cpython/rev/16eba94d3cfe changeset: 87665:16eba94d3cfe branch: 3.3 parent: 87662:a239897084bf user: Alexandre Vassalotti date: Sat Nov 30 16:06:39 2013 -0800 summary: Issue #6477: Added support for pickling the types of built-in singletons. files: Include/object.h | 3 ++ Lib/pickle.py | 11 ++++++++- Lib/test/pickletester.py | 9 +++++++ Misc/NEWS | 3 ++ Modules/_pickle.c | 32 +++++++++++++++++++++++++++- Objects/object.c | 4 +- 6 files changed, 58 insertions(+), 4 deletions(-) diff --git a/Include/object.h b/Include/object.h --- a/Include/object.h +++ b/Include/object.h @@ -831,6 +831,9 @@ PyAPI_FUNC(void) Py_IncRef(PyObject *); PyAPI_FUNC(void) Py_DecRef(PyObject *); +PyAPI_DATA(PyTypeObject) PyNone_Type; +PyAPI_DATA(PyTypeObject) PyNotImplemented_Type; + /* _Py_NoneStruct is an object of undefined type which can be used in contexts where NULL (nil) is not suitable (since NULL often means 'error'). diff --git a/Lib/pickle.py b/Lib/pickle.py --- a/Lib/pickle.py +++ b/Lib/pickle.py @@ -728,9 +728,18 @@ self.memoize(obj) + def save_type(self, obj): + if obj is type(None): + return self.save_reduce(type, (None,), obj=obj) + elif obj is type(NotImplemented): + return self.save_reduce(type, (NotImplemented,), obj=obj) + elif obj is type(...): + return self.save_reduce(type, (...,), obj=obj) + return self.save_global(obj) + dispatch[FunctionType] = save_global dispatch[BuiltinFunctionType] = save_global - dispatch[type] = save_global + dispatch[type] = save_type # Pickling helpers diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -768,6 +768,15 @@ u = self.loads(s) self.assertEqual(NotImplemented, u) + def test_singleton_types(self): + # Issue #6477: Test that types of built-in singletons can be pickled. + singletons = [None, ..., NotImplemented] + for singleton in singletons: + for proto in protocols: + s = self.dumps(type(singleton), proto) + u = self.loads(s) + self.assertIs(type(singleton), u) + # Tests for protocol 2 def test_proto(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -24,6 +24,9 @@ - Fixed _pickle.Unpickler to not fail when loading empty strings as persistent IDs. +- Issue #6477: Added support for pickling the types of built-in singletons + (i.e., Ellipsis, NotImplemented, None). + - Issue #11508: Fixed uuid.getnode() and uuid.uuid1() on environment with virtual interface. Original patch by Kent Frazier. diff --git a/Modules/_pickle.c b/Modules/_pickle.c --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -2836,6 +2836,36 @@ } static int +save_singleton_type(PicklerObject *self, PyObject *obj, PyObject *singleton) +{ + PyObject *reduce_value; + int status; + + reduce_value = Py_BuildValue("O(O)", &PyType_Type, singleton); + if (reduce_value == NULL) { + return -1; + } + status = save_reduce(self, reduce_value, obj); + Py_DECREF(reduce_value); + return status; +} + +static int +save_type(PicklerObject *self, PyObject *obj) +{ + if (obj == (PyObject *)&PyNone_Type) { + return save_singleton_type(self, obj, Py_None); + } + else if (obj == (PyObject *)&PyEllipsis_Type) { + return save_singleton_type(self, obj, Py_Ellipsis); + } + else if (obj == (PyObject *)&PyNotImplemented_Type) { + return save_singleton_type(self, obj, Py_NotImplemented); + } + return save_global(self, obj, NULL); +} + +static int save_pers(PicklerObject *self, PyObject *obj, PyObject *func) { PyObject *pid = NULL; @@ -3189,7 +3219,7 @@ goto done; } else if (type == &PyType_Type) { - status = save_global(self, obj, NULL); + status = save_type(self, obj); goto done; } else if (type == &PyFunction_Type) { diff --git a/Objects/object.c b/Objects/object.c --- a/Objects/object.c +++ b/Objects/object.c @@ -1459,7 +1459,7 @@ 0, /* nb_index */ }; -static PyTypeObject PyNone_Type = { +PyTypeObject PyNone_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "NoneType", 0, @@ -1524,7 +1524,7 @@ Py_RETURN_NOTIMPLEMENTED; } -static PyTypeObject PyNotImplemented_Type = { +PyTypeObject PyNotImplemented_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "NotImplementedType", 0, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 01:21:44 2013 From: python-checkins at python.org (alexandre.vassalotti) Date: Sun, 1 Dec 2013 01:21:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=236477=3A_Merge_with_3=2E3=2E?= Message-ID: <3dX9B03Jxnz7Lky@mail.python.org> http://hg.python.org/cpython/rev/ff56f48b3277 changeset: 87666:ff56f48b3277 parent: 87664:5469e1a68dbd parent: 87665:16eba94d3cfe user: Alexandre Vassalotti date: Sat Nov 30 16:21:20 2013 -0800 summary: Issue #6477: Merge with 3.3. files: Include/object.h | 3 ++ Lib/pickle.py | 11 ++++++++- Lib/test/pickletester.py | 9 +++++++ Misc/NEWS | 3 ++ Modules/_pickle.c | 32 +++++++++++++++++++++++++++- Objects/object.c | 4 +- 6 files changed, 58 insertions(+), 4 deletions(-) diff --git a/Include/object.h b/Include/object.h --- a/Include/object.h +++ b/Include/object.h @@ -829,6 +829,9 @@ PyAPI_FUNC(void) Py_IncRef(PyObject *); PyAPI_FUNC(void) Py_DecRef(PyObject *); +PyAPI_DATA(PyTypeObject) PyNone_Type; +PyAPI_DATA(PyTypeObject) PyNotImplemented_Type; + /* _Py_NoneStruct is an object of undefined type which can be used in contexts where NULL (nil) is not suitable (since NULL often means 'error'). diff --git a/Lib/pickle.py b/Lib/pickle.py --- a/Lib/pickle.py +++ b/Lib/pickle.py @@ -954,8 +954,17 @@ self.memoize(obj) + def save_type(self, obj): + if obj is type(None): + return self.save_reduce(type, (None,), obj=obj) + elif obj is type(NotImplemented): + return self.save_reduce(type, (NotImplemented,), obj=obj) + elif obj is type(...): + return self.save_reduce(type, (...,), obj=obj) + return self.save_global(obj) + dispatch[FunctionType] = save_global - dispatch[type] = save_global + dispatch[type] = save_type # Unpickling machinery diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -804,6 +804,15 @@ u = self.loads(s) self.assertIs(NotImplemented, u) + def test_singleton_types(self): + # Issue #6477: Test that types of built-in singletons can be pickled. + singletons = [None, ..., NotImplemented] + for singleton in singletons: + for proto in protocols: + s = self.dumps(type(singleton), proto) + u = self.loads(s) + self.assertIs(type(singleton), u) + # Tests for protocol 2 def test_proto(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -33,6 +33,9 @@ - Fixed _pickle.Unpickler to not fail when loading empty strings as persistent IDs. +- Issue #6477: Added support for pickling the types of built-in singletons + (i.e., Ellipsis, NotImplemented, None). + - ssl.create_default_context() sets OP_NO_COMPRESSION to prevent CRIME. - Issue #19802: Add socket.SO_PRIORITY. diff --git a/Modules/_pickle.c b/Modules/_pickle.c --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -3288,6 +3288,36 @@ } static int +save_singleton_type(PicklerObject *self, PyObject *obj, PyObject *singleton) +{ + PyObject *reduce_value; + int status; + + reduce_value = Py_BuildValue("O(O)", &PyType_Type, singleton); + if (reduce_value == NULL) { + return -1; + } + status = save_reduce(self, reduce_value, obj); + Py_DECREF(reduce_value); + return status; +} + +static int +save_type(PicklerObject *self, PyObject *obj) +{ + if (obj == (PyObject *)&PyNone_Type) { + return save_singleton_type(self, obj, Py_None); + } + else if (obj == (PyObject *)&PyEllipsis_Type) { + return save_singleton_type(self, obj, Py_Ellipsis); + } + else if (obj == (PyObject *)&PyNotImplemented_Type) { + return save_singleton_type(self, obj, Py_NotImplemented); + } + return save_global(self, obj, NULL); +} + +static int save_pers(PicklerObject *self, PyObject *obj, PyObject *func) { PyObject *pid = NULL; @@ -3696,7 +3726,7 @@ goto done; } else if (type == &PyType_Type) { - status = save_global(self, obj, NULL); + status = save_type(self, obj); goto done; } else if (type == &PyFunction_Type) { diff --git a/Objects/object.c b/Objects/object.c --- a/Objects/object.c +++ b/Objects/object.c @@ -1409,7 +1409,7 @@ 0, /* nb_index */ }; -static PyTypeObject PyNone_Type = { +PyTypeObject PyNone_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "NoneType", 0, @@ -1494,7 +1494,7 @@ Py_FatalError("deallocating NotImplemented"); } -static PyTypeObject PyNotImplemented_Type = { +PyTypeObject PyNotImplemented_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "NotImplementedType", 0, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 01:57:22 2013 From: python-checkins at python.org (alexandre.vassalotti) Date: Sun, 1 Dec 2013 01:57:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzY0Nzc6?= =?utf-8?q?_Added_pickling_support_for_singletons_and_their_types=2E?= Message-ID: <3dX9z61BQczQLG@mail.python.org> http://hg.python.org/cpython/rev/fbb97f6eb3b3 changeset: 87667:fbb97f6eb3b3 branch: 2.7 parent: 87658:ce55df8ee815 user: Alexandre Vassalotti date: Sat Nov 30 16:52:03 2013 -0800 summary: Issue #6477: Added pickling support for singletons and their types. files: Include/object.h | 3 + Lib/pickle.py | 21 ++++++++- Lib/test/pickletester.py | 9 +++ Misc/NEWS | 3 + Modules/cPickle.c | 64 +++++++++++++++++++++++++++- Objects/object.c | 4 +- 6 files changed, 100 insertions(+), 4 deletions(-) diff --git a/Include/object.h b/Include/object.h --- a/Include/object.h +++ b/Include/object.h @@ -829,6 +829,9 @@ PyAPI_FUNC(void) Py_IncRef(PyObject *); PyAPI_FUNC(void) Py_DecRef(PyObject *); +PyAPI_DATA(PyTypeObject) PyNone_Type; +PyAPI_DATA(PyTypeObject) PyNotImplemented_Type; + /* _Py_NoneStruct is an object of undefined type which can be used in contexts where NULL (nil) is not suitable (since NULL often means 'error'). diff --git a/Lib/pickle.py b/Lib/pickle.py --- a/Lib/pickle.py +++ b/Lib/pickle.py @@ -427,6 +427,14 @@ self.write(NONE) dispatch[NoneType] = save_none + def save_ellipsis(self, obj): + self.save_global(Ellipsis, 'Ellipsis') + dispatch[type(Ellipsis)] = save_ellipsis + + def save_notimplemented(self, obj): + self.save_global(NotImplemented, 'NotImplemented') + dispatch[type(NotImplemented)] = save_notimplemented + def save_bool(self, obj): if self.proto >= 2: self.write(obj and NEWTRUE or NEWFALSE) @@ -770,7 +778,18 @@ dispatch[ClassType] = save_global dispatch[FunctionType] = save_global dispatch[BuiltinFunctionType] = save_global - dispatch[TypeType] = save_global + + def save_type(self, obj): + if obj is type(None): + return self.save_reduce(type, (None,), obj=obj) + elif obj is type(NotImplemented): + return self.save_reduce(type, (NotImplemented,), obj=obj) + elif obj is type(Ellipsis): + return self.save_reduce(type, (Ellipsis,), obj=obj) + return self.save_global(obj) + + dispatch[TypeType] = save_type + # Pickling helpers diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -661,6 +661,15 @@ u = self.loads(s) self.assertEqual(t, u) + def test_singleton_types(self): + # Issue #6477: Test that types of built-in singletons can be pickled. + singletons = [None, Ellipsis, NotImplemented] + for singleton in singletons: + for proto in protocols: + s = self.dumps(type(singleton), proto) + u = self.loads(s) + self.assertIs(type(singleton), u) + # Tests for protocol 2 def test_proto(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,9 @@ Library ------- +- Issue #6477: Added pickling support for built-in singletons (i.e., None, + Ellipsis and NotImplemented) and their respective types. + - Issue #16231: Fixed pickle.Pickler to only fallback to its default pickling behaviour when Pickler.persistent_id returns None, but not for any other false values. This allows false values other than None to be used as diff --git a/Modules/cPickle.c b/Modules/cPickle.c --- a/Modules/cPickle.c +++ b/Modules/cPickle.c @@ -2559,6 +2559,60 @@ } static int +save_ellipsis(Picklerobject *self, PyObject *obj) +{ + PyObject *str = PyString_FromString("Ellipsis"); + int res; + if (str == NULL) + return -1; + res = save_global(self, Py_Ellipsis, str); + Py_DECREF(str); + return res; +} + +static int +save_notimplemented(Picklerobject *self, PyObject *obj) +{ + PyObject *str = PyString_FromString("NotImplemented"); + int res; + if (str == NULL) + return -1; + res = save_global(self, Py_NotImplemented, str); + Py_DECREF(str); + return res; +} + +static int +save_singleton_type(Picklerobject *self, PyObject *obj, PyObject *singleton) +{ + PyObject *reduce_value; + int status; + + reduce_value = Py_BuildValue("O(O)", &PyType_Type, singleton); + if (reduce_value == NULL) { + return -1; + } + status = save_reduce(self, reduce_value, (PyObject *)self, obj); + Py_DECREF(reduce_value); + return status; +} + +static int +save_type(Picklerobject *self, PyObject *obj) +{ + if (obj == (PyObject *)&PyNone_Type) { + return save_singleton_type(self, obj, Py_None); + } + else if (obj == (PyObject *)&PyEllipsis_Type) { + return save_singleton_type(self, obj, Py_Ellipsis); + } + else if (obj == (PyObject *)&PyNotImplemented_Type) { + return save_singleton_type(self, obj, Py_NotImplemented); + } + return save_global(self, obj, NULL); +} + +static int save(Picklerobject *self, PyObject *args, int pers_save) { PyTypeObject *type; @@ -2580,6 +2634,14 @@ res = save_none(self, args); goto finally; } + else if (args == Py_Ellipsis) { + res = save_ellipsis(self, args); + goto finally; + } + else if (args == Py_NotImplemented) { + res = save_notimplemented(self, args); + goto finally; + } type = Py_TYPE(args); @@ -2671,7 +2733,7 @@ goto finally; } if (type == &PyType_Type) { - res = save_global(self, args, NULL); + res = save_type(self, args); goto finally; } break; diff --git a/Objects/object.c b/Objects/object.c --- a/Objects/object.c +++ b/Objects/object.c @@ -2012,7 +2012,7 @@ } -static PyTypeObject PyNone_Type = { +PyTypeObject PyNone_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "NoneType", 0, @@ -2043,7 +2043,7 @@ return PyString_FromString("NotImplemented"); } -static PyTypeObject PyNotImplemented_Type = { +PyTypeObject PyNotImplemented_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "NotImplementedType", 0, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 01:57:23 2013 From: python-checkins at python.org (alexandre.vassalotti) Date: Sun, 1 Dec 2013 01:57:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf-8?q?_Merge_heads=2E?= Message-ID: <3dX9z731sLz7LlN@mail.python.org> http://hg.python.org/cpython/rev/9ca47b3babe5 changeset: 87668:9ca47b3babe5 branch: 2.7 parent: 87667:fbb97f6eb3b3 parent: 87659:b6377ca8087a user: Alexandre Vassalotti date: Sat Nov 30 16:56:36 2013 -0800 summary: Merge heads. files: Doc/library/logging.rst | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -837,8 +837,10 @@ effect is to disable all logging calls of severity *lvl* and below, so that if you call it with a value of INFO, then all INFO and DEBUG events would be discarded, whereas those of severity WARNING and above would be processed - according to the logger's effective level. To undo the effect of a call to - ``logging.disable(lvl)``, call ``logging.disable(logging.NOTSET)``. + according to the logger's effective level. If + ``logging.disable(logging.NOTSET)`` is called, it effectively removes this + overriding level, so that logging output again depends on the effective + levels of individual loggers. .. function:: addLevelName(lvl, levelName) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 02:44:24 2013 From: python-checkins at python.org (alexandre.vassalotti) Date: Sun, 1 Dec 2013 02:44:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzY0Nzc6?= =?utf-8?q?_Revert_fbb97f6eb3b3_as_it_broke_test=5Fxpickle=2E?= Message-ID: <3dXC1N6drHz7Lk2@mail.python.org> http://hg.python.org/cpython/rev/d964d7023aa4 changeset: 87669:d964d7023aa4 branch: 2.7 user: Alexandre Vassalotti date: Sat Nov 30 17:43:42 2013 -0800 summary: Issue #6477: Revert fbb97f6eb3b3 as it broke test_xpickle. files: Include/object.h | 3 - Lib/pickle.py | 21 +-------- Lib/test/pickletester.py | 9 --- Misc/NEWS | 3 - Modules/cPickle.c | 64 +--------------------------- Objects/object.c | 4 +- 6 files changed, 4 insertions(+), 100 deletions(-) diff --git a/Include/object.h b/Include/object.h --- a/Include/object.h +++ b/Include/object.h @@ -829,9 +829,6 @@ PyAPI_FUNC(void) Py_IncRef(PyObject *); PyAPI_FUNC(void) Py_DecRef(PyObject *); -PyAPI_DATA(PyTypeObject) PyNone_Type; -PyAPI_DATA(PyTypeObject) PyNotImplemented_Type; - /* _Py_NoneStruct is an object of undefined type which can be used in contexts where NULL (nil) is not suitable (since NULL often means 'error'). diff --git a/Lib/pickle.py b/Lib/pickle.py --- a/Lib/pickle.py +++ b/Lib/pickle.py @@ -427,14 +427,6 @@ self.write(NONE) dispatch[NoneType] = save_none - def save_ellipsis(self, obj): - self.save_global(Ellipsis, 'Ellipsis') - dispatch[type(Ellipsis)] = save_ellipsis - - def save_notimplemented(self, obj): - self.save_global(NotImplemented, 'NotImplemented') - dispatch[type(NotImplemented)] = save_notimplemented - def save_bool(self, obj): if self.proto >= 2: self.write(obj and NEWTRUE or NEWFALSE) @@ -778,18 +770,7 @@ dispatch[ClassType] = save_global dispatch[FunctionType] = save_global dispatch[BuiltinFunctionType] = save_global - - def save_type(self, obj): - if obj is type(None): - return self.save_reduce(type, (None,), obj=obj) - elif obj is type(NotImplemented): - return self.save_reduce(type, (NotImplemented,), obj=obj) - elif obj is type(Ellipsis): - return self.save_reduce(type, (Ellipsis,), obj=obj) - return self.save_global(obj) - - dispatch[TypeType] = save_type - + dispatch[TypeType] = save_global # Pickling helpers diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -661,15 +661,6 @@ u = self.loads(s) self.assertEqual(t, u) - def test_singleton_types(self): - # Issue #6477: Test that types of built-in singletons can be pickled. - singletons = [None, Ellipsis, NotImplemented] - for singleton in singletons: - for proto in protocols: - s = self.dumps(type(singleton), proto) - u = self.loads(s) - self.assertIs(type(singleton), u) - # Tests for protocol 2 def test_proto(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,9 +15,6 @@ Library ------- -- Issue #6477: Added pickling support for built-in singletons (i.e., None, - Ellipsis and NotImplemented) and their respective types. - - Issue #16231: Fixed pickle.Pickler to only fallback to its default pickling behaviour when Pickler.persistent_id returns None, but not for any other false values. This allows false values other than None to be used as diff --git a/Modules/cPickle.c b/Modules/cPickle.c --- a/Modules/cPickle.c +++ b/Modules/cPickle.c @@ -2559,60 +2559,6 @@ } static int -save_ellipsis(Picklerobject *self, PyObject *obj) -{ - PyObject *str = PyString_FromString("Ellipsis"); - int res; - if (str == NULL) - return -1; - res = save_global(self, Py_Ellipsis, str); - Py_DECREF(str); - return res; -} - -static int -save_notimplemented(Picklerobject *self, PyObject *obj) -{ - PyObject *str = PyString_FromString("NotImplemented"); - int res; - if (str == NULL) - return -1; - res = save_global(self, Py_NotImplemented, str); - Py_DECREF(str); - return res; -} - -static int -save_singleton_type(Picklerobject *self, PyObject *obj, PyObject *singleton) -{ - PyObject *reduce_value; - int status; - - reduce_value = Py_BuildValue("O(O)", &PyType_Type, singleton); - if (reduce_value == NULL) { - return -1; - } - status = save_reduce(self, reduce_value, (PyObject *)self, obj); - Py_DECREF(reduce_value); - return status; -} - -static int -save_type(Picklerobject *self, PyObject *obj) -{ - if (obj == (PyObject *)&PyNone_Type) { - return save_singleton_type(self, obj, Py_None); - } - else if (obj == (PyObject *)&PyEllipsis_Type) { - return save_singleton_type(self, obj, Py_Ellipsis); - } - else if (obj == (PyObject *)&PyNotImplemented_Type) { - return save_singleton_type(self, obj, Py_NotImplemented); - } - return save_global(self, obj, NULL); -} - -static int save(Picklerobject *self, PyObject *args, int pers_save) { PyTypeObject *type; @@ -2634,14 +2580,6 @@ res = save_none(self, args); goto finally; } - else if (args == Py_Ellipsis) { - res = save_ellipsis(self, args); - goto finally; - } - else if (args == Py_NotImplemented) { - res = save_notimplemented(self, args); - goto finally; - } type = Py_TYPE(args); @@ -2733,7 +2671,7 @@ goto finally; } if (type == &PyType_Type) { - res = save_type(self, args); + res = save_global(self, args, NULL); goto finally; } break; diff --git a/Objects/object.c b/Objects/object.c --- a/Objects/object.c +++ b/Objects/object.c @@ -2012,7 +2012,7 @@ } -PyTypeObject PyNone_Type = { +static PyTypeObject PyNone_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "NoneType", 0, @@ -2043,7 +2043,7 @@ return PyString_FromString("NotImplemented"); } -PyTypeObject PyNotImplemented_Type = { +static PyTypeObject PyNotImplemented_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "NotImplementedType", 0, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 03:04:48 2013 From: python-checkins at python.org (alexandre.vassalotti) Date: Sun, 1 Dec 2013 03:04:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzY0Nzc6?= =?utf-8?q?_Keep_PyNotImplemented=5FType_and_PyNone=5FType_private=2E?= Message-ID: <3dXCSw50RGz7Llv@mail.python.org> http://hg.python.org/cpython/rev/9fcba15d7685 changeset: 87670:9fcba15d7685 branch: 3.3 parent: 87665:16eba94d3cfe user: Alexandre Vassalotti date: Sat Nov 30 17:55:48 2013 -0800 summary: Issue #6477: Keep PyNotImplemented_Type and PyNone_Type private. files: Include/object.h | 4 ++-- Modules/_pickle.c | 4 ++-- Objects/object.c | 12 ++++++------ 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Include/object.h b/Include/object.h --- a/Include/object.h +++ b/Include/object.h @@ -831,8 +831,8 @@ PyAPI_FUNC(void) Py_IncRef(PyObject *); PyAPI_FUNC(void) Py_DecRef(PyObject *); -PyAPI_DATA(PyTypeObject) PyNone_Type; -PyAPI_DATA(PyTypeObject) PyNotImplemented_Type; +PyAPI_DATA(PyTypeObject) _PyNone_Type; +PyAPI_DATA(PyTypeObject) _PyNotImplemented_Type; /* _Py_NoneStruct is an object of undefined type which can be used in contexts diff --git a/Modules/_pickle.c b/Modules/_pickle.c --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -2853,13 +2853,13 @@ static int save_type(PicklerObject *self, PyObject *obj) { - if (obj == (PyObject *)&PyNone_Type) { + if (obj == (PyObject *)&_PyNone_Type) { return save_singleton_type(self, obj, Py_None); } else if (obj == (PyObject *)&PyEllipsis_Type) { return save_singleton_type(self, obj, Py_Ellipsis); } - else if (obj == (PyObject *)&PyNotImplemented_Type) { + else if (obj == (PyObject *)&_PyNotImplemented_Type) { return save_singleton_type(self, obj, Py_NotImplemented); } return save_global(self, obj, NULL); diff --git a/Objects/object.c b/Objects/object.c --- a/Objects/object.c +++ b/Objects/object.c @@ -1459,7 +1459,7 @@ 0, /* nb_index */ }; -PyTypeObject PyNone_Type = { +PyTypeObject _PyNone_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "NoneType", 0, @@ -1502,7 +1502,7 @@ PyObject _Py_NoneStruct = { _PyObject_EXTRA_INIT - 1, &PyNone_Type + 1, &_PyNone_Type }; /* NotImplemented is an object that can be used to signal that an @@ -1524,7 +1524,7 @@ Py_RETURN_NOTIMPLEMENTED; } -PyTypeObject PyNotImplemented_Type = { +PyTypeObject _PyNotImplemented_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "NotImplementedType", 0, @@ -1567,7 +1567,7 @@ PyObject _Py_NotImplementedStruct = { _PyObject_EXTRA_INIT - 1, &PyNotImplemented_Type + 1, &_PyNotImplemented_Type }; void @@ -1597,10 +1597,10 @@ if (PyType_Ready(&PyList_Type) < 0) Py_FatalError("Can't initialize list type"); - if (PyType_Ready(&PyNone_Type) < 0) + if (PyType_Ready(&_PyNone_Type) < 0) Py_FatalError("Can't initialize None type"); - if (PyType_Ready(&PyNotImplemented_Type) < 0) + if (PyType_Ready(&_PyNotImplemented_Type) < 0) Py_FatalError("Can't initialize NotImplemented type"); if (PyType_Ready(&PyTraceBack_Type) < 0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 03:04:49 2013 From: python-checkins at python.org (alexandre.vassalotti) Date: Sun, 1 Dec 2013 03:04:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=236477=3A_Merge_with_3=2E3=2E?= Message-ID: <3dXCSx72vmz7Lm5@mail.python.org> http://hg.python.org/cpython/rev/7d6c27fa7f32 changeset: 87671:7d6c27fa7f32 parent: 87666:ff56f48b3277 parent: 87670:9fcba15d7685 user: Alexandre Vassalotti date: Sat Nov 30 17:58:53 2013 -0800 summary: Issue #6477: Merge with 3.3. files: Include/object.h | 4 ++-- Modules/_pickle.c | 4 ++-- Objects/object.c | 12 ++++++------ 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Include/object.h b/Include/object.h --- a/Include/object.h +++ b/Include/object.h @@ -829,8 +829,8 @@ PyAPI_FUNC(void) Py_IncRef(PyObject *); PyAPI_FUNC(void) Py_DecRef(PyObject *); -PyAPI_DATA(PyTypeObject) PyNone_Type; -PyAPI_DATA(PyTypeObject) PyNotImplemented_Type; +PyAPI_DATA(PyTypeObject) _PyNone_Type; +PyAPI_DATA(PyTypeObject) _PyNotImplemented_Type; /* _Py_NoneStruct is an object of undefined type which can be used in contexts diff --git a/Modules/_pickle.c b/Modules/_pickle.c --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -3305,13 +3305,13 @@ static int save_type(PicklerObject *self, PyObject *obj) { - if (obj == (PyObject *)&PyNone_Type) { + if (obj == (PyObject *)&_PyNone_Type) { return save_singleton_type(self, obj, Py_None); } else if (obj == (PyObject *)&PyEllipsis_Type) { return save_singleton_type(self, obj, Py_Ellipsis); } - else if (obj == (PyObject *)&PyNotImplemented_Type) { + else if (obj == (PyObject *)&_PyNotImplemented_Type) { return save_singleton_type(self, obj, Py_NotImplemented); } return save_global(self, obj, NULL); diff --git a/Objects/object.c b/Objects/object.c --- a/Objects/object.c +++ b/Objects/object.c @@ -1409,7 +1409,7 @@ 0, /* nb_index */ }; -PyTypeObject PyNone_Type = { +PyTypeObject _PyNone_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "NoneType", 0, @@ -1452,7 +1452,7 @@ PyObject _Py_NoneStruct = { _PyObject_EXTRA_INIT - 1, &PyNone_Type + 1, &_PyNone_Type }; /* NotImplemented is an object that can be used to signal that an @@ -1494,7 +1494,7 @@ Py_FatalError("deallocating NotImplemented"); } -PyTypeObject PyNotImplemented_Type = { +PyTypeObject _PyNotImplemented_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) "NotImplementedType", 0, @@ -1537,7 +1537,7 @@ PyObject _Py_NotImplementedStruct = { _PyObject_EXTRA_INIT - 1, &PyNotImplemented_Type + 1, &_PyNotImplemented_Type }; void @@ -1567,10 +1567,10 @@ if (PyType_Ready(&PyList_Type) < 0) Py_FatalError("Can't initialize list type"); - if (PyType_Ready(&PyNone_Type) < 0) + if (PyType_Ready(&_PyNone_Type) < 0) Py_FatalError("Can't initialize None type"); - if (PyType_Ready(&PyNotImplemented_Type) < 0) + if (PyType_Ready(&_PyNotImplemented_Type) < 0) Py_FatalError("Can't initialize NotImplemented type"); if (PyType_Ready(&PyTraceBack_Type) < 0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 04:04:14 2013 From: python-checkins at python.org (gregory.p.smith) Date: Sun, 1 Dec 2013 04:04:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Fixes_Issue_?= =?utf-8?q?=2315798_-_subprocess=2EPopen=28=29_no_longer_fails_if_file?= Message-ID: <3dXDnV3MRyz7Lk7@mail.python.org> http://hg.python.org/cpython/rev/c4cd891cf167 changeset: 87672:c4cd891cf167 branch: 3.3 parent: 87670:9fcba15d7685 user: Gregory P. Smith date: Sat Nov 30 19:02:57 2013 -0800 summary: Fixes Issue #15798 - subprocess.Popen() no longer fails if file descriptor 0, 1 or 2 is closed. files: Lib/subprocess.py | 5 ++++- Lib/test/test_subprocess.py | 21 +++++++++++++++++++++ Misc/NEWS | 3 +++ Modules/_posixsubprocess.c | 6 +----- 4 files changed, 29 insertions(+), 6 deletions(-) diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -1368,7 +1368,10 @@ executable_list = tuple( os.path.join(os.fsencode(dir), executable) for dir in os.get_exec_path(env)) - fds_to_keep = set(pass_fds) + # Never close stdin, stdout and stderr for the child. + fds_to_keep = {0,1,2} + fds_to_keep.update(pass_fds) + # Our child uses this one to signal error before exec(). fds_to_keep.add(errpipe_write) self.pid = _posixsubprocess.fork_exec( args, executable_list, diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -1559,6 +1559,27 @@ # all standard fds closed. self.check_close_std_fds([0, 1, 2]) + def test_small_errpipe_write_fd(self): + """Issue #15798: Popen should work when stdio fds are available.""" + new_stdin = os.dup(0) + new_stdout = os.dup(1) + try: + os.close(0) + os.close(1) + + # Side test: if errpipe_write fails to have its CLOEXEC + # flag set this should cause the parent to think the exec + # failed. Extremely unlikely: everyone supports CLOEXEC. + subprocess.Popen([ + sys.executable, "-c", + "print('AssertionError:0:CLOEXEC failure.')"]).wait() + finally: + # Restore original stdin and stdout + os.dup2(new_stdin, 0) + os.dup2(new_stdout, 1) + os.close(new_stdin) + os.close(new_stdout) + def test_remapping_std_fds(self): # open up some temporary files temps = [mkstemp() for i in range(3)] diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,9 @@ Library ------- +- Issue #15798: Fixed subprocess.Popen() to no longer fail if file + descriptor 0, 1 or 2 is closed. + - Issue #19088: Fixed incorrect caching of the copyreg module in object.__reduce__() and object.__reduce_ex__(). diff --git a/Modules/_posixsubprocess.c b/Modules/_posixsubprocess.c --- a/Modules/_posixsubprocess.c +++ b/Modules/_posixsubprocess.c @@ -449,7 +449,7 @@ local_max_fd = max_fd; #endif /* TODO HP-UX could use pstat_getproc() if anyone cares about it. */ - _close_open_fd_range(3, local_max_fd, py_fds_to_keep); + _close_open_fd_range(0, local_max_fd, py_fds_to_keep); } /* This loop matches the Lib/os.py _execvpe()'s PATH search when */ @@ -526,10 +526,6 @@ &restore_signals, &call_setsid, &preexec_fn)) return NULL; - if (close_fds && errpipe_write < 3) { /* precondition */ - PyErr_SetString(PyExc_ValueError, "errpipe_write must be >= 3"); - return NULL; - } if (PySequence_Length(py_fds_to_keep) < 0) { PyErr_SetString(PyExc_ValueError, "cannot get length of fds_to_keep"); return NULL; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 04:04:16 2013 From: python-checkins at python.org (gregory.p.smith) Date: Sun, 1 Dec 2013 04:04:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Fixes_Issue_=2315798_-_subprocess=2EPopen=28=29_no_longe?= =?utf-8?q?r_fails_if_file?= Message-ID: <3dXDnX0NCTz7Llq@mail.python.org> http://hg.python.org/cpython/rev/0387054b2038 changeset: 87673:0387054b2038 parent: 87671:7d6c27fa7f32 parent: 87672:c4cd891cf167 user: Gregory P. Smith date: Sat Nov 30 19:04:00 2013 -0800 summary: Fixes Issue #15798 - subprocess.Popen() no longer fails if file descriptor 0, 1 or 2 is closed. files: Lib/subprocess.py | 5 ++++- Lib/test/test_subprocess.py | 21 +++++++++++++++++++++ Misc/NEWS | 3 +++ Modules/_posixsubprocess.c | 6 +----- 4 files changed, 29 insertions(+), 6 deletions(-) diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -1361,7 +1361,10 @@ executable_list = tuple( os.path.join(os.fsencode(dir), executable) for dir in os.get_exec_path(env)) - fds_to_keep = set(pass_fds) + # Never close stdin, stdout and stderr for the child. + fds_to_keep = {0,1,2} + fds_to_keep.update(pass_fds) + # Our child uses this one to signal error before exec(). fds_to_keep.add(errpipe_write) self.pid = _posixsubprocess.fork_exec( args, executable_list, diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -1559,6 +1559,27 @@ # all standard fds closed. self.check_close_std_fds([0, 1, 2]) + def test_small_errpipe_write_fd(self): + """Issue #15798: Popen should work when stdio fds are available.""" + new_stdin = os.dup(0) + new_stdout = os.dup(1) + try: + os.close(0) + os.close(1) + + # Side test: if errpipe_write fails to have its CLOEXEC + # flag set this should cause the parent to think the exec + # failed. Extremely unlikely: everyone supports CLOEXEC. + subprocess.Popen([ + sys.executable, "-c", + "print('AssertionError:0:CLOEXEC failure.')"]).wait() + finally: + # Restore original stdin and stdout + os.dup2(new_stdin, 0) + os.dup2(new_stdout, 1) + os.close(new_stdin) + os.close(new_stdout) + def test_remapping_std_fds(self): # open up some temporary files temps = [mkstemp() for i in range(3)] diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,9 @@ Library ------- +- Issue #15798: Fixed subprocess.Popen() to no longer fail if file + descriptor 0, 1 or 2 is closed. + - Issue #17897: Optimized unpickle prefetching. - Issue #3693: Make the error message more helpful when the array.array() diff --git a/Modules/_posixsubprocess.c b/Modules/_posixsubprocess.c --- a/Modules/_posixsubprocess.c +++ b/Modules/_posixsubprocess.c @@ -458,7 +458,7 @@ local_max_fd = max_fd; #endif /* TODO HP-UX could use pstat_getproc() if anyone cares about it. */ - _close_open_fd_range(3, local_max_fd, py_fds_to_keep); + _close_open_fd_range(0, local_max_fd, py_fds_to_keep); } /* This loop matches the Lib/os.py _execvpe()'s PATH search when */ @@ -535,10 +535,6 @@ &restore_signals, &call_setsid, &preexec_fn)) return NULL; - if (close_fds && errpipe_write < 3) { /* precondition */ - PyErr_SetString(PyExc_ValueError, "errpipe_write must be >= 3"); - return NULL; - } if (PySequence_Length(py_fds_to_keep) < 0) { PyErr_SetString(PyExc_ValueError, "cannot get length of fds_to_keep"); return NULL; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 09:13:46 2013 From: python-checkins at python.org (gregory.p.smith) Date: Sun, 1 Dec 2013 09:13:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Undo_supposed_?= =?utf-8?q?fix_for_Issue_=2315798_until_I_understand_why_this_is?= Message-ID: <3dXMff2nyzz7Lk2@mail.python.org> http://hg.python.org/cpython/rev/efcdf2a70f2a changeset: 87674:efcdf2a70f2a branch: 3.3 parent: 87672:c4cd891cf167 user: Gregory P. Smith date: Sun Dec 01 00:12:24 2013 -0800 summary: Undo supposed fix for Issue #15798 until I understand why this is causing test_multiprocessing_forkserver and test_multiprocessing_spawn failures on head (3.4). files: Lib/subprocess.py | 5 +---- Lib/test/test_subprocess.py | 21 --------------------- Misc/NEWS | 3 --- Modules/_posixsubprocess.c | 6 +++++- 4 files changed, 6 insertions(+), 29 deletions(-) diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -1368,10 +1368,7 @@ executable_list = tuple( os.path.join(os.fsencode(dir), executable) for dir in os.get_exec_path(env)) - # Never close stdin, stdout and stderr for the child. - fds_to_keep = {0,1,2} - fds_to_keep.update(pass_fds) - # Our child uses this one to signal error before exec(). + fds_to_keep = set(pass_fds) fds_to_keep.add(errpipe_write) self.pid = _posixsubprocess.fork_exec( args, executable_list, diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -1559,27 +1559,6 @@ # all standard fds closed. self.check_close_std_fds([0, 1, 2]) - def test_small_errpipe_write_fd(self): - """Issue #15798: Popen should work when stdio fds are available.""" - new_stdin = os.dup(0) - new_stdout = os.dup(1) - try: - os.close(0) - os.close(1) - - # Side test: if errpipe_write fails to have its CLOEXEC - # flag set this should cause the parent to think the exec - # failed. Extremely unlikely: everyone supports CLOEXEC. - subprocess.Popen([ - sys.executable, "-c", - "print('AssertionError:0:CLOEXEC failure.')"]).wait() - finally: - # Restore original stdin and stdout - os.dup2(new_stdin, 0) - os.dup2(new_stdout, 1) - os.close(new_stdin) - os.close(new_stdout) - def test_remapping_std_fds(self): # open up some temporary files temps = [mkstemp() for i in range(3)] diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,9 +18,6 @@ Library ------- -- Issue #15798: Fixed subprocess.Popen() to no longer fail if file - descriptor 0, 1 or 2 is closed. - - Issue #19088: Fixed incorrect caching of the copyreg module in object.__reduce__() and object.__reduce_ex__(). diff --git a/Modules/_posixsubprocess.c b/Modules/_posixsubprocess.c --- a/Modules/_posixsubprocess.c +++ b/Modules/_posixsubprocess.c @@ -449,7 +449,7 @@ local_max_fd = max_fd; #endif /* TODO HP-UX could use pstat_getproc() if anyone cares about it. */ - _close_open_fd_range(0, local_max_fd, py_fds_to_keep); + _close_open_fd_range(3, local_max_fd, py_fds_to_keep); } /* This loop matches the Lib/os.py _execvpe()'s PATH search when */ @@ -526,6 +526,10 @@ &restore_signals, &call_setsid, &preexec_fn)) return NULL; + if (close_fds && errpipe_write < 3) { /* precondition */ + PyErr_SetString(PyExc_ValueError, "errpipe_write must be >= 3"); + return NULL; + } if (PySequence_Length(py_fds_to_keep) < 0) { PyErr_SetString(PyExc_ValueError, "cannot get length of fds_to_keep"); return NULL; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 09:13:47 2013 From: python-checkins at python.org (gregory.p.smith) Date: Sun, 1 Dec 2013 09:13:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Undo_supposed_fix_for_Issue_=2315798_until_I_understand_?= =?utf-8?q?why_this_is?= Message-ID: <3dXMfg5tBkz7Llv@mail.python.org> http://hg.python.org/cpython/rev/ddbf9632795b changeset: 87675:ddbf9632795b parent: 87673:0387054b2038 parent: 87674:efcdf2a70f2a user: Gregory P. Smith date: Sun Dec 01 00:13:35 2013 -0800 summary: Undo supposed fix for Issue #15798 until I understand why this is causing test_multiprocessing_forkserver and test_multiprocessing_spawn failures on head (3.4). files: Lib/subprocess.py | 5 +---- Lib/test/test_subprocess.py | 21 --------------------- Misc/NEWS | 3 --- Modules/_posixsubprocess.c | 6 +++++- 4 files changed, 6 insertions(+), 29 deletions(-) diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -1361,10 +1361,7 @@ executable_list = tuple( os.path.join(os.fsencode(dir), executable) for dir in os.get_exec_path(env)) - # Never close stdin, stdout and stderr for the child. - fds_to_keep = {0,1,2} - fds_to_keep.update(pass_fds) - # Our child uses this one to signal error before exec(). + fds_to_keep = set(pass_fds) fds_to_keep.add(errpipe_write) self.pid = _posixsubprocess.fork_exec( args, executable_list, diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -1559,27 +1559,6 @@ # all standard fds closed. self.check_close_std_fds([0, 1, 2]) - def test_small_errpipe_write_fd(self): - """Issue #15798: Popen should work when stdio fds are available.""" - new_stdin = os.dup(0) - new_stdout = os.dup(1) - try: - os.close(0) - os.close(1) - - # Side test: if errpipe_write fails to have its CLOEXEC - # flag set this should cause the parent to think the exec - # failed. Extremely unlikely: everyone supports CLOEXEC. - subprocess.Popen([ - sys.executable, "-c", - "print('AssertionError:0:CLOEXEC failure.')"]).wait() - finally: - # Restore original stdin and stdout - os.dup2(new_stdin, 0) - os.dup2(new_stdout, 1) - os.close(new_stdin) - os.close(new_stdout) - def test_remapping_std_fds(self): # open up some temporary files temps = [mkstemp() for i in range(3)] diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,9 +18,6 @@ Library ------- -- Issue #15798: Fixed subprocess.Popen() to no longer fail if file - descriptor 0, 1 or 2 is closed. - - Issue #17897: Optimized unpickle prefetching. - Issue #3693: Make the error message more helpful when the array.array() diff --git a/Modules/_posixsubprocess.c b/Modules/_posixsubprocess.c --- a/Modules/_posixsubprocess.c +++ b/Modules/_posixsubprocess.c @@ -458,7 +458,7 @@ local_max_fd = max_fd; #endif /* TODO HP-UX could use pstat_getproc() if anyone cares about it. */ - _close_open_fd_range(0, local_max_fd, py_fds_to_keep); + _close_open_fd_range(3, local_max_fd, py_fds_to_keep); } /* This loop matches the Lib/os.py _execvpe()'s PATH search when */ @@ -535,6 +535,10 @@ &restore_signals, &call_setsid, &preexec_fn)) return NULL; + if (close_fds && errpipe_write < 3) { /* precondition */ + PyErr_SetString(PyExc_ValueError, "errpipe_write must be >= 3"); + return NULL; + } if (PySequence_Length(py_fds_to_keep) < 0) { PyErr_SetString(PyExc_ValueError, "cannot get length of fds_to_keep"); return NULL; -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sun Dec 1 09:39:08 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 01 Dec 2013 09:39:08 +0100 Subject: [Python-checkins] Daily reference leaks (7d6c27fa7f32): sum=0 Message-ID: results for 7d6c27fa7f32 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogIGGZQS', '-x'] From python-checkins at python.org Sun Dec 1 10:05:58 2013 From: python-checkins at python.org (victor.stinner) Date: Sun, 1 Dec 2013 10:05:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Closes_=2319831=3A_Stop_tr?= =?utf-8?q?acemalloc_later_at_Python_shutdown_to_be_able_to_use?= Message-ID: <3dXNpt42znz7Lk2@mail.python.org> http://hg.python.org/cpython/rev/cc8953ea3c7e changeset: 87676:cc8953ea3c7e user: Victor Stinner date: Sun Dec 01 10:03:26 2013 +0100 summary: Closes #19831: Stop tracemalloc later at Python shutdown to be able to use tracemalloc in objects destructor Replace atexit handler with an harcoded C function _PyTraceMalloc_Fini(). files: Modules/_tracemalloc.c | 77 +++-------------------------- Python/pythonrun.c | 5 + 2 files changed, 15 insertions(+), 67 deletions(-) diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -9,7 +9,6 @@ /* Forward declaration */ static void tracemalloc_stop(void); -static int tracemalloc_atexit_register(void); static void* raw_malloc(size_t size); static void raw_free(void *ptr); @@ -36,9 +35,6 @@ TRACEMALLOC_FINALIZED } initialized; - /* atexit handler registered? */ - int atexit_registered; - /* Is tracemalloc tracing memory allocations? Variable protected by the GIL */ int tracing; @@ -46,7 +42,7 @@ /* limit of the number of frames in a traceback, 1 by default. Variable protected by the GIL. */ int max_nframe; -} tracemalloc_config = {TRACEMALLOC_NOT_INITIALIZED, 0, 0, 1}; +} tracemalloc_config = {TRACEMALLOC_NOT_INITIALIZED, 0, 1}; #if defined(TRACE_RAW_MALLOC) && defined(WITH_THREAD) /* This lock is needed because tracemalloc_free() is called without @@ -802,9 +798,6 @@ return 0; } - if (tracemalloc_atexit_register() < 0) - return -1; - assert(1 <= max_nframe && max_nframe <= MAX_NFRAME); tracemalloc_config.max_nframe = max_nframe; @@ -1140,65 +1133,6 @@ return traceback_to_pyobject(trace.traceback, NULL); } -static PyObject* -tracemalloc_atexit(PyObject *self) -{ -#ifdef WITH_THREAD - assert(PyGILState_Check()); -#endif - tracemalloc_deinit(); - Py_RETURN_NONE; -} - -static PyMethodDef atexit_method = { - "_atexit", (PyCFunction)tracemalloc_atexit, METH_NOARGS, NULL}; - -static int -tracemalloc_atexit_register(void) -{ - PyObject *method = NULL, *atexit = NULL, *func = NULL; - PyObject *result; - int ret = -1; - - if (tracemalloc_config.atexit_registered) - return 0; - tracemalloc_config.atexit_registered = 1; - - /* private functions */ - method = PyCFunction_New(&atexit_method, NULL); - if (method == NULL) - goto done; - - atexit = PyImport_ImportModule("atexit"); - if (atexit == NULL) { - if (!PyErr_Warn(PyExc_ImportWarning, - "atexit module is missing: " - "cannot automatically disable tracemalloc at exit")) - { - PyErr_Clear(); - return 0; - } - goto done; - } - - func = PyObject_GetAttrString(atexit, "register"); - if (func == NULL) - goto done; - - result = PyObject_CallFunction(func, "O", method); - if (result == NULL) - goto done; - Py_DECREF(result); - - ret = 0; - -done: - Py_XDECREF(method); - Py_XDECREF(func); - Py_XDECREF(atexit); - return ret; -} - PyDoc_STRVAR(tracemalloc_start_doc, "start(nframe: int=1)\n" "\n" @@ -1439,3 +1373,12 @@ return tracemalloc_start(nframe); } +void +_PyTraceMalloc_Fini(void) +{ +#ifdef WITH_THREAD + assert(PyGILState_Check()); +#endif + tracemalloc_deinit(); +} + diff --git a/Python/pythonrun.c b/Python/pythonrun.c --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -106,6 +106,7 @@ extern void _PyFaulthandler_Fini(void); extern void _PyHash_Fini(void); extern int _PyTraceMalloc_Init(void); +extern int _PyTraceMalloc_Fini(void); #ifdef WITH_THREAD extern void _PyGILState_Init(PyInterpreterState *, PyThreadState *); @@ -642,6 +643,10 @@ PyGC_Collect(); #endif + /* Disable tracemalloc after all Python objects have been destroyed, + so it is possible to use tracemalloc in objects destructor. */ + _PyTraceMalloc_Fini(); + /* Destroy the database used by _PyImport_{Fixup,Find}Extension */ _PyImport_Fini(); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 11:05:00 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sun, 1 Dec 2013 11:05:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319842=3A_Refactor?= =?utf-8?q?_BaseSelector_to_make_it_an_actual_usable_ABC=2E?= Message-ID: <3dXQ706cbFz7LjZ@mail.python.org> http://hg.python.org/cpython/rev/f48f302f54aa changeset: 87677:f48f302f54aa user: Charles-Fran?ois Natali date: Sun Dec 01 11:04:17 2013 +0100 summary: Issue #19842: Refactor BaseSelector to make it an actual usable ABC. files: Lib/asyncio/test_utils.py | 14 +++ Lib/selectors.py | 112 +++++++++++++++--------- Lib/test/test_telnetlib.py | 7 +- 3 files changed, 86 insertions(+), 47 deletions(-) diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py --- a/Lib/asyncio/test_utils.py +++ b/Lib/asyncio/test_utils.py @@ -142,9 +142,23 @@ class TestSelector(selectors.BaseSelector): + def __init__(self): + self.keys = {} + + def register(self, fileobj, events, data=None): + key = selectors.SelectorKey(fileobj, 0, events, data) + self.keys[fileobj] = key + return key + + def unregister(self, fileobj): + return self.keys.pop(fileobj) + def select(self, timeout): return [] + def get_map(self): + return self.keys + class TestLoop(base_events.BaseEventLoop): """Loop for unittests. diff --git a/Lib/selectors.py b/Lib/selectors.py --- a/Lib/selectors.py +++ b/Lib/selectors.py @@ -64,7 +64,7 @@ class BaseSelector(metaclass=ABCMeta): - """Base selector class. + """Selector abstract base class. A selector supports registering file objects to be monitored for specific I/O events. @@ -78,12 +78,7 @@ performant implementation on the current platform. """ - def __init__(self): - # this maps file descriptors to keys - self._fd_to_key = {} - # read-only mapping returned by get_map() - self._map = _SelectorMapping(self) - + @abstractmethod def register(self, fileobj, events, data=None): """Register a file object. @@ -95,18 +90,9 @@ Returns: SelectorKey instance """ - if (not events) or (events & ~(EVENT_READ | EVENT_WRITE)): - raise ValueError("Invalid events: {!r}".format(events)) + raise NotImplementedError - key = SelectorKey(fileobj, _fileobj_to_fd(fileobj), events, data) - - if key.fd in self._fd_to_key: - raise KeyError("{!r} (FD {}) is already " - "registered".format(fileobj, key.fd)) - - self._fd_to_key[key.fd] = key - return key - + @abstractmethod def unregister(self, fileobj): """Unregister a file object. @@ -116,11 +102,7 @@ Returns: SelectorKey instance """ - try: - key = self._fd_to_key.pop(_fileobj_to_fd(fileobj)) - except KeyError: - raise KeyError("{!r} is not registered".format(fileobj)) from None - return key + raise NotImplementedError def modify(self, fileobj, events, data=None): """Change a registered file object monitored events or attached data. @@ -133,19 +115,8 @@ Returns: SelectorKey instance """ - # TODO: Subclasses can probably optimize this even further. - try: - key = self._fd_to_key[_fileobj_to_fd(fileobj)] - except KeyError: - raise KeyError("{!r} is not registered".format(fileobj)) from None - if events != key.events: - self.unregister(fileobj) - key = self.register(fileobj, events, data) - elif data != key.data: - # Use a shortcut to update the data. - key = key._replace(data=data) - self._fd_to_key[key.fd] = key - return key + self.unregister(fileobj) + return self.register(fileobj, events, data) @abstractmethod def select(self, timeout=None): @@ -164,14 +135,14 @@ list of (key, events) for ready file objects `events` is a bitwise mask of EVENT_READ|EVENT_WRITE """ - raise NotImplementedError() + raise NotImplementedError def close(self): """Close the selector. This must be called to make sure that any underlying resource is freed. """ - self._fd_to_key.clear() + pass def get_key(self, fileobj): """Return the key associated to a registered file object. @@ -179,14 +150,16 @@ Returns: SelectorKey for this file object """ + mapping = self.get_map() try: - return self._fd_to_key[_fileobj_to_fd(fileobj)] + return mapping[fileobj] except KeyError: raise KeyError("{!r} is not registered".format(fileobj)) from None + @abstractmethod def get_map(self): """Return a mapping of file objects to selector keys.""" - return self._map + raise NotImplementedError def __enter__(self): return self @@ -194,6 +167,57 @@ def __exit__(self, *args): self.close() + +class _BaseSelectorImpl(BaseSelector): + """Base selector implementation.""" + + def __init__(self): + # this maps file descriptors to keys + self._fd_to_key = {} + # read-only mapping returned by get_map() + self._map = _SelectorMapping(self) + + def register(self, fileobj, events, data=None): + if (not events) or (events & ~(EVENT_READ | EVENT_WRITE)): + raise ValueError("Invalid events: {!r}".format(events)) + + key = SelectorKey(fileobj, _fileobj_to_fd(fileobj), events, data) + + if key.fd in self._fd_to_key: + raise KeyError("{!r} (FD {}) is already " + "registered".format(fileobj, key.fd)) + + self._fd_to_key[key.fd] = key + return key + + def unregister(self, fileobj): + try: + key = self._fd_to_key.pop(_fileobj_to_fd(fileobj)) + except KeyError: + raise KeyError("{!r} is not registered".format(fileobj)) from None + return key + + def modify(self, fileobj, events, data=None): + # TODO: Subclasses can probably optimize this even further. + try: + key = self._fd_to_key[_fileobj_to_fd(fileobj)] + except KeyError: + raise KeyError("{!r} is not registered".format(fileobj)) from None + if events != key.events: + self.unregister(fileobj) + key = self.register(fileobj, events, data) + elif data != key.data: + # Use a shortcut to update the data. + key = key._replace(data=data) + self._fd_to_key[key.fd] = key + return key + + def close(self): + self._fd_to_key.clear() + + def get_map(self): + return self._map + def _key_from_fd(self, fd): """Return the key associated to a given file descriptor. @@ -209,7 +233,7 @@ return None -class SelectSelector(BaseSelector): +class SelectSelector(_BaseSelectorImpl): """Select-based selector.""" def __init__(self): @@ -262,7 +286,7 @@ if hasattr(select, 'poll'): - class PollSelector(BaseSelector): + class PollSelector(_BaseSelectorImpl): """Poll-based selector.""" def __init__(self): @@ -306,7 +330,7 @@ if hasattr(select, 'epoll'): - class EpollSelector(BaseSelector): + class EpollSelector(_BaseSelectorImpl): """Epoll-based selector.""" def __init__(self): @@ -358,7 +382,7 @@ if hasattr(select, 'kqueue'): - class KqueueSelector(BaseSelector): + class KqueueSelector(_BaseSelectorImpl): """Kqueue-based selector.""" def __init__(self): diff --git a/Lib/test/test_telnetlib.py b/Lib/test/test_telnetlib.py --- a/Lib/test/test_telnetlib.py +++ b/Lib/test/test_telnetlib.py @@ -114,7 +114,6 @@ class MockSelector(selectors.BaseSelector): def __init__(self): - super().__init__() self.keys = {} def register(self, fileobj, events, data=None): @@ -123,8 +122,7 @@ return key def unregister(self, fileobj): - key = self.keys.pop(fileobj) - return key + return self.keys.pop(fileobj) def select(self, timeout=None): block = False @@ -137,6 +135,9 @@ else: return [(key, key.events) for key in self.keys.values()] + def get_map(self): + return self.keys + @contextlib.contextmanager def test_socket(reads): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 13:31:31 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sun, 1 Dec 2013 13:31:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319849=3A_selector?= =?utf-8?q?s=3A_Document_the_possibility_of_early_select=28=29_wakeup_upon?= Message-ID: <3dXTN302Wqz7Lk7@mail.python.org> http://hg.python.org/cpython/rev/b0c4c7f04f05 changeset: 87678:b0c4c7f04f05 user: Charles-Fran?ois Natali date: Sun Dec 01 13:23:48 2013 +0100 summary: Issue #19849: selectors: Document the possibility of early select() wakeup upon EINTR. files: Doc/library/selectors.rst | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/Doc/library/selectors.rst b/Doc/library/selectors.rst --- a/Doc/library/selectors.rst +++ b/Doc/library/selectors.rst @@ -150,6 +150,11 @@ object. *events* is a bitmask of events ready on this file object. + .. note:: + This method can return before any file object becomes ready or the + timeout has elapsed if the current process receives a signal: in this + case, an empty list will be returned. + .. method:: close() Close the selector. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 15:53:51 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sun, 1 Dec 2013 15:53:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2318994=3A_Add_a_mi?= =?utf-8?q?ssing_check_for_a_return_value_in_fcntmodule=2E_Patch_by?= Message-ID: <3dXXXH0XwMz7LkK@mail.python.org> http://hg.python.org/cpython/rev/84148898a606 changeset: 87679:84148898a606 user: Charles-Fran?ois Natali date: Sun Dec 01 14:30:47 2013 +0100 summary: Issue #18994: Add a missing check for a return value in fcntmodule. Patch by Vajrasky Kok. files: Modules/fcntlmodule.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Modules/fcntlmodule.c b/Modules/fcntlmodule.c --- a/Modules/fcntlmodule.c +++ b/Modules/fcntlmodule.c @@ -628,6 +628,8 @@ return NULL; /* Add some symbolic constants to the module */ - all_ins(m); + if (all_ins(m) < 0) + return NULL; + return m; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 16:31:25 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 1 Dec 2013 16:31:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Fixed_JSON_tes?= =?utf-8?q?ts_on_wide_build_when_ran_from_*=2Epyc_files_=28issue_=2311489?= =?utf-8?b?KS4=?= Message-ID: <3dXYMd41JSz7Ll1@mail.python.org> http://hg.python.org/cpython/rev/02d186e3af09 changeset: 87680:02d186e3af09 branch: 2.7 parent: 87669:d964d7023aa4 user: Serhiy Storchaka date: Sun Dec 01 17:30:55 2013 +0200 summary: Fixed JSON tests on wide build when ran from *.pyc files (issue #11489). files: Lib/json/tests/test_scanstring.py | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Lib/json/tests/test_scanstring.py b/Lib/json/tests/test_scanstring.py --- a/Lib/json/tests/test_scanstring.py +++ b/Lib/json/tests/test_scanstring.py @@ -100,14 +100,15 @@ self.assertEqual(scanstring(given, 1, None, True), (expect, len(given))) + surrogates = unichr(0xd834) + unichr(0xdd20) assertScan('"z\\ud834\\u0079x"', u'z\ud834yx') assertScan('"z\\ud834\\udd20x"', u'z\U0001d120x') assertScan('"z\\ud834\\ud834\\udd20x"', u'z\ud834\U0001d120x') assertScan('"z\\ud834x"', u'z\ud834x') - assertScan(u'"z\\ud834\udd20x12345"', u'z\ud834\udd20x12345') + assertScan(u'"z\\ud834\udd20x12345"', u'z%sx12345' % surrogates) assertScan('"z\\udd20x"', u'z\udd20x') assertScan(u'"z\ud834\udd20x"', u'z\ud834\udd20x') - assertScan(u'"z\ud834\\udd20x"', u'z\ud834\udd20x') + assertScan(u'"z\ud834\\udd20x"', u'z%sx' % surrogates) assertScan(u'"z\ud834x"', u'z\ud834x') def test_bad_escapes(self): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 22:26:59 2013 From: python-checkins at python.org (alexandre.vassalotti) Date: Sun, 1 Dec 2013 22:26:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzExNDgw?= =?utf-8?q?=3A_Fixed_copy=2Ecopy_to_work_with_classes_with_custom_metaclas?= =?utf-8?q?ses=2E?= Message-ID: <3dXjFv1bZRz7LmB@mail.python.org> http://hg.python.org/cpython/rev/c6bb6f304f75 changeset: 87681:c6bb6f304f75 branch: 3.3 parent: 87674:efcdf2a70f2a user: Alexandre Vassalotti date: Sun Dec 01 13:25:26 2013 -0800 summary: Issue #11480: Fixed copy.copy to work with classes with custom metaclasses. Patch by Daniel Urban. files: Lib/copy.py | 8 ++++++++ Lib/test/test_copy.py | 5 ++++- Misc/NEWS | 3 +++ 3 files changed, 15 insertions(+), 1 deletions(-) diff --git a/Lib/copy.py b/Lib/copy.py --- a/Lib/copy.py +++ b/Lib/copy.py @@ -76,6 +76,14 @@ if copier: return copier(x) + try: + issc = issubclass(cls, type) + except TypeError: # cls is not a class + issc = False + if issc: + # treat it as a regular class: + return _copy_immutable(x) + copier = getattr(cls, "__copy__", None) if copier: return copier(x) diff --git a/Lib/test/test_copy.py b/Lib/test/test_copy.py --- a/Lib/test/test_copy.py +++ b/Lib/test/test_copy.py @@ -3,6 +3,7 @@ import copy import copyreg import weakref +import abc from operator import le, lt, ge, gt, eq, ne import unittest @@ -93,9 +94,11 @@ pass def f(): pass + class WithMetaclass(metaclass=abc.ABCMeta): + pass tests = [None, 42, 2**100, 3.14, True, False, 1j, "hello", "hello\u1234", f.__code__, - NewStyle, range(10), Classic, max] + NewStyle, range(10), Classic, max, WithMetaclass] for x in tests: self.assertIs(copy.copy(x), x) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -24,6 +24,9 @@ - Fixed _pickle.Unpickler to not fail when loading empty strings as persistent IDs. +- Issue #11480: Fixed copy.copy to work with classes with custom metaclasses. + Patch by Daniel Urban. + - Issue #6477: Added support for pickling the types of built-in singletons (i.e., Ellipsis, NotImplemented, None). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 1 22:27:00 2013 From: python-checkins at python.org (alexandre.vassalotti) Date: Sun, 1 Dec 2013 22:27:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2311480=3A_Merge_with_3=2E3=2E?= Message-ID: <3dXjFw3rvTz7LmZ@mail.python.org> http://hg.python.org/cpython/rev/c4715c9f442f changeset: 87682:c4715c9f442f parent: 87679:84148898a606 parent: 87681:c6bb6f304f75 user: Alexandre Vassalotti date: Sun Dec 01 13:26:32 2013 -0800 summary: Issue #11480: Merge with 3.3. files: Lib/copy.py | 8 ++++++++ Lib/test/test_copy.py | 5 ++++- Misc/NEWS | 3 +++ 3 files changed, 15 insertions(+), 1 deletions(-) diff --git a/Lib/copy.py b/Lib/copy.py --- a/Lib/copy.py +++ b/Lib/copy.py @@ -76,6 +76,14 @@ if copier: return copier(x) + try: + issc = issubclass(cls, type) + except TypeError: # cls is not a class + issc = False + if issc: + # treat it as a regular class: + return _copy_immutable(x) + copier = getattr(cls, "__copy__", None) if copier: return copier(x) diff --git a/Lib/test/test_copy.py b/Lib/test/test_copy.py --- a/Lib/test/test_copy.py +++ b/Lib/test/test_copy.py @@ -3,6 +3,7 @@ import copy import copyreg import weakref +import abc from operator import le, lt, ge, gt, eq, ne import unittest @@ -93,9 +94,11 @@ pass def f(): pass + class WithMetaclass(metaclass=abc.ABCMeta): + pass tests = [None, 42, 2**100, 3.14, True, False, 1j, "hello", "hello\u1234", f.__code__, - NewStyle, range(10), Classic, max] + NewStyle, range(10), Classic, max, WithMetaclass] for x in tests: self.assertIs(copy.copy(x), x) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -33,6 +33,9 @@ - Fixed _pickle.Unpickler to not fail when loading empty strings as persistent IDs. +- Issue #11480: Fixed copy.copy to work with classes with custom metaclasses. + Patch by Daniel Urban. + - Issue #6477: Added support for pickling the types of built-in singletons (i.e., Ellipsis, NotImplemented, None). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 01:03:44 2013 From: python-checkins at python.org (gregory.p.smith) Date: Mon, 2 Dec 2013 01:03:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fixes_issue_=2315798=3A_su?= =?utf-8?q?bprocess=2EPopen=28=29_no_longer_fails_if_file?= Message-ID: <3dXmkm3TPNz7LnX@mail.python.org> http://hg.python.org/cpython/rev/2df5e1f537b0 changeset: 87683:2df5e1f537b0 user: Gregory P. Smith date: Sun Dec 01 16:03:24 2013 -0800 summary: Fixes issue #15798: subprocess.Popen() no longer fails if file descriptor 0, 1 or 2 is closed. (correct fix for 3.4 this time) files: Lib/subprocess.py | 7 +++++++ Lib/test/test_subprocess.py | 21 +++++++++++++++++++++ Misc/NEWS | 3 +++ 3 files changed, 31 insertions(+), 0 deletions(-) diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -1341,6 +1341,13 @@ # Data format: "exception name:hex errno:description" # Pickle is not used; it is complex and involves memory allocation. errpipe_read, errpipe_write = os.pipe() + # errpipe_write must not be in the standard io 0, 1, or 2 fd range. + low_fds_to_close = [] + while errpipe_write < 3: + low_fds_to_close.append(errpipe_write) + errpipe_write = os.dup(errpipe_write) + for low_fd in low_fds_to_close: + os.close(low_fd) try: try: # We must avoid complex work that could involve diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -1559,6 +1559,27 @@ # all standard fds closed. self.check_close_std_fds([0, 1, 2]) + def test_small_errpipe_write_fd(self): + """Issue #15798: Popen should work when stdio fds are available.""" + new_stdin = os.dup(0) + new_stdout = os.dup(1) + try: + os.close(0) + os.close(1) + + # Side test: if errpipe_write fails to have its CLOEXEC + # flag set this should cause the parent to think the exec + # failed. Extremely unlikely: everyone supports CLOEXEC. + subprocess.Popen([ + sys.executable, "-c", + "print('AssertionError:0:CLOEXEC failure.')"]).wait() + finally: + # Restore original stdin and stdout + os.dup2(new_stdin, 0) + os.dup2(new_stdout, 1) + os.close(new_stdin) + os.close(new_stdout) + def test_remapping_std_fds(self): # open up some temporary files temps = [mkstemp() for i in range(3)] diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,9 @@ Library ------- +- Issue #15798: Fixed subprocess.Popen() to no longer fail if file + descriptor 0, 1 or 2 is closed. + - Issue #17897: Optimized unpickle prefetching. - Issue #3693: Make the error message more helpful when the array.array() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 01:28:00 2013 From: python-checkins at python.org (alexandre.vassalotti) Date: Mon, 2 Dec 2013 01:28:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319754=3A_Make_pic?= =?utf-8?q?kletools=2Eoptimize_respect_the_frame_size_target=2E?= Message-ID: <3dXnGm2QZDz7Lp3@mail.python.org> http://hg.python.org/cpython/rev/bb71baa28f1b changeset: 87684:bb71baa28f1b user: Alexandre Vassalotti date: Sun Dec 01 16:27:46 2013 -0800 summary: Issue #19754: Make pickletools.optimize respect the frame size target. files: Lib/pickletools.py | 1 + Lib/test/pickletester.py | 13 +------------ Lib/test/test_pickletools.py | 2 -- 3 files changed, 2 insertions(+), 14 deletions(-) diff --git a/Lib/pickletools.py b/Lib/pickletools.py --- a/Lib/pickletools.py +++ b/Lib/pickletools.py @@ -2313,6 +2313,7 @@ buf.start_framing() for start, stop, putid in opcodes: if putid in gets: + #buf.commit_frame() buf.write(p[start:stop]) if proto >= 4: buf.end_framing() diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -1343,12 +1343,6 @@ pickled = self.dumps(obj, proto) unpickled = self.loads(pickled) self.assertEqual(obj, unpickled) - # Test the framing heuristic is sane, - # assuming a given frame size target. - if self.optimized: - # These assumptions are currently invalid for optimized - # pickles (see e.g. issue19754). - continue bytes_per_frame = (len(pickled) / count_opcode(pickle.FRAME, pickled)) self.assertGreater(bytes_per_frame, @@ -1365,12 +1359,7 @@ unpickled = self.loads(pickled) self.assertEqual(obj, unpickled) n_frames = count_opcode(pickle.FRAME, pickled) - if self.optimized: - # At least one frame was emitted (see issue19754). - self.assertGreaterEqual(n_frames, 1) - else: - # At least one frame was emitted per large bytes object. - self.assertGreaterEqual(n_frames, len(obj)) + self.assertGreaterEqual(n_frames, len(obj)) def test_optional_frames(self): if pickle.HIGHEST_PROTOCOL < 4: diff --git a/Lib/test/test_pickletools.py b/Lib/test/test_pickletools.py --- a/Lib/test/test_pickletools.py +++ b/Lib/test/test_pickletools.py @@ -6,8 +6,6 @@ class OptimizedPickleTests(AbstractPickleTests, AbstractPickleModuleTests): - optimized = True - def dumps(self, arg, proto=None): return pickletools.optimize(pickle.dumps(arg, proto)) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 01:32:00 2013 From: python-checkins at python.org (alexandre.vassalotti) Date: Mon, 2 Dec 2013 01:32:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319754=3A_Fix_typo?= =?utf-8?q?=2E?= Message-ID: <3dXnMN5wKyz7Lp3@mail.python.org> http://hg.python.org/cpython/rev/9feada79a411 changeset: 87685:9feada79a411 user: Alexandre Vassalotti date: Sun Dec 01 16:31:49 2013 -0800 summary: Issue #19754: Fix typo. files: Lib/pickletools.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/pickletools.py b/Lib/pickletools.py --- a/Lib/pickletools.py +++ b/Lib/pickletools.py @@ -2313,7 +2313,7 @@ buf.start_framing() for start, stop, putid in opcodes: if putid in gets: - #buf.commit_frame() + buf.commit_frame() buf.write(p[start:stop]) if proto >= 4: buf.end_framing() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 02:28:38 2013 From: python-checkins at python.org (gregory.p.smith) Date: Mon, 2 Dec 2013 02:28:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Fixes_issue_?= =?utf-8?q?=2315798=3A_subprocess=2EPopen=28=29_no_longer_fails_if_file?= Message-ID: <3dXpck1ZhGz7LqG@mail.python.org> http://hg.python.org/cpython/rev/07425df887b5 changeset: 87686:07425df887b5 branch: 3.3 parent: 87681:c6bb6f304f75 user: Gregory P. Smith date: Sun Dec 01 17:27:40 2013 -0800 summary: Fixes issue #15798: subprocess.Popen() no longer fails if file descriptor 0, 1 or 2 is closed. The errpipe_write fd will always be >= 3. files: Lib/test/test_subprocess.py | 21 ++++++++++ Misc/NEWS | 3 + Modules/_posixsubprocess.c | 48 +++++++++++++++++++++--- 3 files changed, 66 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -1559,6 +1559,27 @@ # all standard fds closed. self.check_close_std_fds([0, 1, 2]) + def test_small_errpipe_write_fd(self): + """Issue #15798: Popen should work when stdio fds are available.""" + new_stdin = os.dup(0) + new_stdout = os.dup(1) + try: + os.close(0) + os.close(1) + + # Side test: if errpipe_write fails to have its CLOEXEC + # flag set this should cause the parent to think the exec + # failed. Extremely unlikely: everyone supports CLOEXEC. + subprocess.Popen([ + sys.executable, "-c", + "print('AssertionError:0:CLOEXEC failure.')"]).wait() + finally: + # Restore original stdin and stdout + os.dup2(new_stdin, 0) + os.dup2(new_stdout, 1) + os.close(new_stdin) + os.close(new_stdout) + def test_remapping_std_fds(self): # open up some temporary files temps = [mkstemp() for i in range(3)] diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,9 @@ Library ------- +- Issue #15798: Fixed subprocess.Popen() to no longer fail if file + descriptor 0, 1 or 2 is closed. + - Issue #19088: Fixed incorrect caching of the copyreg module in object.__reduce__() and object.__reduce_ex__(). diff --git a/Modules/_posixsubprocess.c b/Modules/_posixsubprocess.c --- a/Modules/_posixsubprocess.c +++ b/Modules/_posixsubprocess.c @@ -723,26 +723,24 @@ PyDoc_STRVAR(subprocess_cloexec_pipe_doc, "cloexec_pipe() -> (read_end, write_end)\n\n\ -Create a pipe whose ends have the cloexec flag set."); +Create a pipe whose ends have the cloexec flag set; write_end will be >= 3."); static PyObject * subprocess_cloexec_pipe(PyObject *self, PyObject *noargs) { int fds[2]; - int res; + int res, saved_errno; + long oldflags; #ifdef HAVE_PIPE2 Py_BEGIN_ALLOW_THREADS res = pipe2(fds, O_CLOEXEC); Py_END_ALLOW_THREADS if (res != 0 && errno == ENOSYS) { - { #endif /* We hold the GIL which offers some protection from other code calling * fork() before the CLOEXEC flags have been set but we can't guarantee * anything without pipe2(). */ - long oldflags; - res = pipe(fds); if (res == 0) { @@ -759,9 +757,47 @@ if (res == 0) res = fcntl(fds[1], F_SETFD, oldflags | FD_CLOEXEC); #ifdef HAVE_PIPE2 - } } #endif + if (res == 0 && fds[1] < 3) { + /* We always want the write end of the pipe to avoid fds 0, 1 and 2 + * as our child may claim those for stdio connections. */ + int write_fd = fds[1]; + int fds_to_close[3] = {-1, -1, -1}; + int fds_to_close_idx = 0; +#ifdef F_DUPFD_CLOEXEC + fds_to_close[fds_to_close_idx++] = write_fd; + write_fd = fcntl(write_fd, F_DUPFD_CLOEXEC, 3); + if (write_fd < 0) /* We don't support F_DUPFD_CLOEXEC / other error */ +#endif + { + /* Use dup a few times until we get a desirable fd. */ + for (; fds_to_close_idx < 3; ++fds_to_close_idx) { + fds_to_close[fds_to_close_idx] = write_fd; + write_fd = dup(write_fd); + if (write_fd >= 3) + break; + /* We may dup a few extra times if it returns an error but + * that is okay. Repeat calls should return the same error. */ + } + if (write_fd < 0) res = write_fd; + if (res == 0) { + oldflags = fcntl(write_fd, F_GETFD, 0); + if (oldflags < 0) res = oldflags; + if (res == 0) + res = fcntl(write_fd, F_SETFD, oldflags | FD_CLOEXEC); + } + } + saved_errno = errno; + /* Close fds we tried for the write end that were too low. */ + for (fds_to_close_idx=0; fds_to_close_idx < 3; ++fds_to_close_idx) { + int temp_fd = fds_to_close[fds_to_close_idx]; + while (temp_fd >= 0 && close(temp_fd) < 0 && errno == EINTR); + } + errno = saved_errno; /* report dup or fcntl errors, not close. */ + fds[1] = write_fd; + } /* end if write fd was too small */ + if (res != 0) return PyErr_SetFromErrno(PyExc_OSError); return Py_BuildValue("(ii)", fds[0], fds[1]); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 02:28:39 2013 From: python-checkins at python.org (gregory.p.smith) Date: Mon, 2 Dec 2013 02:28:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_null_merge_=283=2E4_already_has_its_own_fix_for_this=29?= =?utf-8?q?=2E?= Message-ID: <3dXpcl3qXlz7LqC@mail.python.org> http://hg.python.org/cpython/rev/786a106fc6b7 changeset: 87687:786a106fc6b7 parent: 87685:9feada79a411 parent: 87686:07425df887b5 user: Gregory P. Smith date: Sun Dec 01 17:28:24 2013 -0800 summary: null merge (3.4 already has its own fix for this). files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 02:58:22 2013 From: python-checkins at python.org (christian.heimes) Date: Mon, 2 Dec 2013 02:58:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319509=3A_Add_SSLC?= =?utf-8?q?ontext=2Echeck=5Fhostname_to_match_the_peer=27s_certificate?= Message-ID: <3dXqH26mz0zLrD@mail.python.org> http://hg.python.org/cpython/rev/aa531135bc6b changeset: 87688:aa531135bc6b user: Christian Heimes date: Mon Dec 02 02:41:19 2013 +0100 summary: Issue #19509: Add SSLContext.check_hostname to match the peer's certificate with server_hostname on handshake. files: Doc/library/ssl.rst | 36 ++++++++++++++++++- Lib/ssl.py | 32 +++++++++++++-- Lib/test/test_ssl.py | 62 ++++++++++++++++++++++++++++++++ Misc/NEWS | 3 + Modules/_ssl.c | 35 ++++++++++++++++++ 5 files changed, 162 insertions(+), 6 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -773,6 +773,11 @@ Perform the SSL setup handshake. + .. versionchanged:: 3.4 + The handshake method also performce :func:`match_hostname` when the + :attr:`~SSLContext.check_hostname` attribute of the socket's + :attr:`~SSLSocket.context` is true. + .. method:: SSLSocket.getpeercert(binary_form=False) If there is no certificate for the peer on the other end of the connection, @@ -1182,6 +1187,33 @@ .. versionadded:: 3.4 +.. attribute:: SSLContext.check_hostname + + Wether to match the peer cert's hostname with :func:`match_hostname` in + :meth:`SSLSocket.do_handshake`. The context's + :attr:`~SSLContext.verify_mode` must be set to :data:`CERT_OPTIONAL` or + :data:`CERT_REQUIRED`, and you must pass *server_hostname* to + :meth:`~SSLContext.wrap_socket` in order to match the hostname. + + Example:: + + import socket, ssl + + context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + context.verify_mode = ssl.CERT_REQUIRED + context.check_hostname = True + context.load_default_certs() + + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + ssl_sock = context.wrap_socket(s, server_hostname='www.verisign.com'): + ssl_sock.connect(('www.verisign.com', 443)) + + .. versionadded:: 3.4 + + .. note:: + + This features requires OpenSSL 0.9.8f or newer. + .. attribute:: SSLContext.options An integer representing the set of SSL options enabled on this context. @@ -1596,7 +1628,9 @@ have to check that the server certificate, which can be obtained by calling :meth:`SSLSocket.getpeercert`, matches the desired service. For many protocols and applications, the service can be identified by the hostname; -in this case, the :func:`match_hostname` function can be used. +in this case, the :func:`match_hostname` function can be used. This common +check is automatically performed when :attr:`SSLContext.check_hostname` is +enabled. In server mode, if you want to authenticate your clients using the SSL layer (rather than using a higher-level authentication mechanism), you'll also have diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -148,6 +148,7 @@ from _ssl import enum_certificates, enum_crls from socket import getnameinfo as _getnameinfo +from socket import SHUT_RDWR as _SHUT_RDWR from socket import socket, AF_INET, SOCK_STREAM, create_connection import base64 # for DER-to-PEM translation import traceback @@ -235,7 +236,9 @@ returns nothing. """ if not cert: - raise ValueError("empty or no certificate") + raise ValueError("empty or no certificate, match_hostname needs a " + "SSL socket or SSL context with either " + "CERT_OPTIONAL or CERT_REQUIRED") dnsnames = [] san = cert.get('subjectAltName', ()) for key, value in san: @@ -387,9 +390,10 @@ context.options |= getattr(_ssl, "OP_NO_COMPRESSION", 0) # disallow ciphers with known vulnerabilities context.set_ciphers(_RESTRICTED_CIPHERS) - # verify certs in client mode + # verify certs and host name in client mode if purpose == Purpose.SERVER_AUTH: context.verify_mode = CERT_REQUIRED + context.check_hostname = True if cafile or capath or cadata: context.load_verify_locations(cafile, capath, cadata) elif context.verify_mode != CERT_NONE: @@ -480,6 +484,13 @@ if server_side and server_hostname: raise ValueError("server_hostname can only be specified " "in client mode") + if self._context.check_hostname and not server_hostname: + if HAS_SNI: + raise ValueError("check_hostname requires server_hostname") + else: + raise ValueError("check_hostname requires server_hostname, " + "but it's not supported by your OpenSSL " + "library") self.server_side = server_side self.server_hostname = server_hostname self.do_handshake_on_connect = do_handshake_on_connect @@ -522,9 +533,9 @@ raise ValueError("do_handshake_on_connect should not be specified for non-blocking sockets") self.do_handshake() - except OSError as x: + except (OSError, ValueError): self.close() - raise x + raise @property def context(self): @@ -751,6 +762,17 @@ finally: self.settimeout(timeout) + if self.context.check_hostname: + try: + if not self.server_hostname: + raise ValueError("check_hostname needs server_hostname " + "argument") + match_hostname(self.getpeercert(), self.server_hostname) + except Exception: + self.shutdown(_SHUT_RDWR) + self.close() + raise + def _real_connect(self, addr, connect_ex): if self.server_side: raise ValueError("can't connect in server-side mode") @@ -770,7 +792,7 @@ if self.do_handshake_on_connect: self.do_handshake() return rc - except OSError: + except (OSError, ValueError): self._sslobj = None raise diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1003,6 +1003,7 @@ ctx = ssl.create_default_context() self.assertEqual(ctx.protocol, ssl.PROTOCOL_TLSv1) self.assertEqual(ctx.verify_mode, ssl.CERT_REQUIRED) + self.assertTrue(ctx.check_hostname) self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) with open(SIGNING_CA) as f: @@ -1022,6 +1023,7 @@ ctx = ssl._create_stdlib_context() self.assertEqual(ctx.protocol, ssl.PROTOCOL_SSLv23) self.assertEqual(ctx.verify_mode, ssl.CERT_NONE) + self.assertFalse(ctx.check_hostname) self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) ctx = ssl._create_stdlib_context(ssl.PROTOCOL_TLSv1) @@ -1040,6 +1042,28 @@ self.assertEqual(ctx.verify_mode, ssl.CERT_NONE) self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) + def test_check_hostname(self): + ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + self.assertFalse(ctx.check_hostname) + + # Requires CERT_REQUIRED or CERT_OPTIONAL + with self.assertRaises(ValueError): + ctx.check_hostname = True + ctx.verify_mode = ssl.CERT_REQUIRED + self.assertFalse(ctx.check_hostname) + ctx.check_hostname = True + self.assertTrue(ctx.check_hostname) + + ctx.verify_mode = ssl.CERT_OPTIONAL + ctx.check_hostname = True + self.assertTrue(ctx.check_hostname) + + # Cannot set CERT_NONE with check_hostname enabled + with self.assertRaises(ValueError): + ctx.verify_mode = ssl.CERT_NONE + ctx.check_hostname = False + self.assertFalse(ctx.check_hostname) + class SSLErrorTests(unittest.TestCase): @@ -1930,6 +1954,44 @@ cert = s.getpeercert() self.assertTrue(cert, "Can't get peer certificate.") + def test_check_hostname(self): + if support.verbose: + sys.stdout.write("\n") + + server_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + server_context.load_cert_chain(SIGNED_CERTFILE) + + context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + context.verify_mode = ssl.CERT_REQUIRED + context.check_hostname = True + context.load_verify_locations(SIGNING_CA) + + # correct hostname should verify + server = ThreadedEchoServer(context=server_context, chatty=True) + with server: + with context.wrap_socket(socket.socket(), + server_hostname="localhost") as s: + s.connect((HOST, server.port)) + cert = s.getpeercert() + self.assertTrue(cert, "Can't get peer certificate.") + + # incorrect hostname should raise an exception + server = ThreadedEchoServer(context=server_context, chatty=True) + with server: + with context.wrap_socket(socket.socket(), + server_hostname="invalid") as s: + with self.assertRaisesRegex(ssl.CertificateError, + "hostname 'invalid' doesn't match 'localhost'"): + s.connect((HOST, server.port)) + + # missing server_hostname arg should cause an exception, too + server = ThreadedEchoServer(context=server_context, chatty=True) + with server: + with socket.socket() as s: + with self.assertRaisesRegex(ValueError, + "check_hostname requires server_hostname"): + context.wrap_socket(s) + def test_empty_cert(self): """Connecting with an empty cert file""" bad_cert_test(os.path.join(os.path.dirname(__file__) or os.curdir, diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,9 @@ Library ------- +- Issue #19509: Add SSLContext.check_hostname to match the peer's certificate + with server_hostname on handshake. + - Issue #15798: Fixed subprocess.Popen() to no longer fail if file descriptor 0, 1 or 2 is closed. diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -214,6 +214,7 @@ #ifndef OPENSSL_NO_TLSEXT PyObject *set_hostname; #endif + int check_hostname; } PySSLContext; typedef struct { @@ -2050,6 +2051,8 @@ #ifndef OPENSSL_NO_TLSEXT self->set_hostname = NULL; #endif + /* Don't check host name by default */ + self->check_hostname = 0; /* Defaults */ SSL_CTX_set_verify(self->ctx, SSL_VERIFY_NONE, NULL); SSL_CTX_set_options(self->ctx, @@ -2231,6 +2234,12 @@ "invalid value for verify_mode"); return -1; } + if (mode == SSL_VERIFY_NONE && self->check_hostname) { + PyErr_SetString(PyExc_ValueError, + "Cannot set verify_mode to CERT_NONE when " + "check_hostname is enabled."); + return -1; + } SSL_CTX_set_verify(self->ctx, mode, NULL); return 0; } @@ -2304,6 +2313,30 @@ return 0; } +static PyObject * +get_check_hostname(PySSLContext *self, void *c) +{ + return PyBool_FromLong(self->check_hostname); +} + +static int +set_check_hostname(PySSLContext *self, PyObject *arg, void *c) +{ + int check_hostname; + if (!PyArg_Parse(arg, "p", &check_hostname)) + return -1; + if (check_hostname && + SSL_CTX_get_verify_mode(self->ctx) == SSL_VERIFY_NONE) { + PyErr_SetString(PyExc_ValueError, + "check_hostname needs a SSL context with either " + "CERT_OPTIONAL or CERT_REQUIRED"); + return -1; + } + self->check_hostname = check_hostname; + return 0; +} + + typedef struct { PyThreadState *thread_state; PyObject *callable; @@ -3093,6 +3126,8 @@ static PyGetSetDef context_getsetlist[] = { + {"check_hostname", (getter) get_check_hostname, + (setter) set_check_hostname, NULL}, {"options", (getter) get_options, (setter) set_options, NULL}, #ifdef HAVE_OPENSSL_VERIFY_PARAM -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 02:58:24 2013 From: python-checkins at python.org (christian.heimes) Date: Mon, 2 Dec 2013 02:58:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319781=3A_ftplib_n?= =?utf-8?q?ow_supports_SSLContext=2Echeck=5Fhostname_and_server_name?= Message-ID: <3dXqH42mZZz7Lpx@mail.python.org> http://hg.python.org/cpython/rev/373797990f57 changeset: 87689:373797990f57 user: Christian Heimes date: Mon Dec 02 02:56:02 2013 +0100 summary: Issue #19781: ftplib now supports SSLContext.check_hostname and server name indication for TLS/SSL connections. files: Doc/library/ftplib.rst | 10 ++++++++ Lib/ftplib.py | 8 +++++- Lib/test/test_ftplib.py | 33 ++++++++++++++++++++++++++++- Misc/NEWS | 3 ++ 4 files changed, 51 insertions(+), 3 deletions(-) diff --git a/Doc/library/ftplib.rst b/Doc/library/ftplib.rst --- a/Doc/library/ftplib.rst +++ b/Doc/library/ftplib.rst @@ -94,6 +94,11 @@ .. versionchanged:: 3.3 *source_address* parameter was added. + .. versionchanged:: 3.4 + The class now supports hostname check with + :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see + :data:`~ssl.HAS_SNI`). + Here's a sample session using the :class:`FTP_TLS` class: >>> from ftplib import FTP_TLS @@ -427,6 +432,11 @@ Set up secure control connection by using TLS or SSL, depending on what specified in :meth:`ssl_version` attribute. + .. versionchanged:: 3.4 + The method now supports hostname check with + :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see + :data:`~ssl.HAS_SNI`). + .. method:: FTP_TLS.ccc() Revert control channel back to plaintext. This can be useful to take diff --git a/Lib/ftplib.py b/Lib/ftplib.py --- a/Lib/ftplib.py +++ b/Lib/ftplib.py @@ -748,7 +748,9 @@ resp = self.voidcmd('AUTH TLS') else: resp = self.voidcmd('AUTH SSL') - self.sock = self.context.wrap_socket(self.sock) + server_hostname = self.host if ssl.HAS_SNI else None + self.sock = self.context.wrap_socket(self.sock, + server_hostname=server_hostname) self.file = self.sock.makefile(mode='r', encoding=self.encoding) return resp @@ -787,7 +789,9 @@ def ntransfercmd(self, cmd, rest=None): conn, size = FTP.ntransfercmd(self, cmd, rest) if self._prot_p: - conn = self.context.wrap_socket(conn) + server_hostname = self.host if ssl.HAS_SNI else None + conn = self.context.wrap_socket(conn, + server_hostname=server_hostname) return conn, size def abort(self): diff --git a/Lib/test/test_ftplib.py b/Lib/test/test_ftplib.py --- a/Lib/test/test_ftplib.py +++ b/Lib/test/test_ftplib.py @@ -301,7 +301,8 @@ if ssl is not None: - CERTFILE = os.path.join(os.path.dirname(__file__), "keycert.pem") + CERTFILE = os.path.join(os.path.dirname(__file__), "keycert3.pem") + CAFILE = os.path.join(os.path.dirname(__file__), "pycacert.pem") class SSLConnection(asyncore.dispatcher): """An asyncore.dispatcher subclass supporting TLS/SSL.""" @@ -923,6 +924,36 @@ self.client.ccc() self.assertRaises(ValueError, self.client.sock.unwrap) + def test_check_hostname(self): + self.client.quit() + ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + ctx.verify_mode = ssl.CERT_REQUIRED + ctx.check_hostname = True + ctx.load_verify_locations(CAFILE) + self.client = ftplib.FTP_TLS(context=ctx, timeout=TIMEOUT) + + # 127.0.0.1 doesn't match SAN + self.client.connect(self.server.host, self.server.port) + with self.assertRaises(ssl.CertificateError): + self.client.auth() + # exception quits connection + + self.client.connect(self.server.host, self.server.port) + self.client.prot_p() + with self.assertRaises(ssl.CertificateError): + with self.client.transfercmd("list") as sock: + pass + self.client.quit() + + self.client.connect("localhost", self.server.port) + self.client.auth() + self.client.quit() + + self.client.connect("localhost", self.server.port) + self.client.prot_p() + with self.client.transfercmd("list") as sock: + pass + class TestTimeouts(TestCase): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,9 @@ Library ------- +- Issue #19781: ftplib now supports SSLContext.check_hostname and server name + indication for TLS/SSL connections. + - Issue #19509: Add SSLContext.check_hostname to match the peer's certificate with server_hostname on handshake. -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Mon Dec 2 09:40:35 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 02 Dec 2013 09:40:35 +0100 Subject: [Python-checkins] Daily reference leaks (373797990f57): sum=4 Message-ID: results for 373797990f57 on branch "default" -------------------------------------------- test_site leaked [2, 0, 0] references, sum=2 test_site leaked [2, 0, 0] memory blocks, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog0V3Uos', '-x'] From python-checkins at python.org Mon Dec 2 11:44:23 2013 From: python-checkins at python.org (walter.doerwald) Date: Mon, 2 Dec 2013 11:44:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogRml4IGlzc3VlICMx?= =?utf-8?q?9834=3A_Support_unpickling_of_exceptions_pickled_by_Python_2=2E?= Message-ID: <3dY2xz1wBwzSyK@mail.python.org> http://hg.python.org/cpython/rev/7d3297f127ae changeset: 87690:7d3297f127ae branch: 3.3 parent: 87686:07425df887b5 user: Walter Doerwald date: Mon Dec 02 11:41:01 2013 +0100 summary: Fix issue #19834: Support unpickling of exceptions pickled by Python 2. files: Lib/_compat_pickle.py | 56 +++++++++++++++++++ Lib/test/pickletester.py | 80 ++++++++++++++++++++++++++++ 2 files changed, 136 insertions(+), 0 deletions(-) diff --git a/Lib/_compat_pickle.py b/Lib/_compat_pickle.py --- a/Lib/_compat_pickle.py +++ b/Lib/_compat_pickle.py @@ -76,6 +76,62 @@ ('itertools', 'ifilterfalse'): ('itertools', 'filterfalse'), } +PYTHON2_EXCEPTIONS = ( + "ArithmeticError", + "AssertionError", + "AttributeError", + "BaseException", + "BufferError", + "BytesWarning", + "DeprecationWarning", + "EOFError", + "EnvironmentError", + "Exception", + "FloatingPointError", + "FutureWarning", + "GeneratorExit", + "IOError", + "ImportError", + "ImportWarning", + "IndentationError", + "IndexError", + "KeyError", + "KeyboardInterrupt", + "LookupError", + "MemoryError", + "NameError", + "NotImplementedError", + "OSError", + "OverflowError", + "PendingDeprecationWarning", + "ReferenceError", + "RuntimeError", + "RuntimeWarning", + # StandardError is gone in Python 3, so we map it to Exception + "StopIteration", + "SyntaxError", + "SyntaxWarning", + "SystemError", + "SystemExit", + "TabError", + "TypeError", + "UnboundLocalError", + "UnicodeDecodeError", + "UnicodeEncodeError", + "UnicodeError", + "UnicodeTranslateError", + "UnicodeWarning", + "UserWarning", + "ValueError", + "Warning", + "ZeroDivisionError", +) + +for excname in PYTHON2_EXCEPTIONS: + NAME_MAPPING[("exceptions", excname)] = ("builtins", excname) + +NAME_MAPPING[("exceptions", "StandardError")] = ("builtins", "Exception") + # Same, but for 3.x to 2.x REVERSE_IMPORT_MAPPING = dict((v, k) for (k, v) in IMPORT_MAPPING.items()) REVERSE_NAME_MAPPING = dict((v, k) for (k, v) in NAME_MAPPING.items()) diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -407,6 +407,71 @@ # set([3]) pickled from 2.x with protocol 2 DATA6 = b'\x80\x02c__builtin__\nset\nq\x00]q\x01K\x03a\x85q\x02Rq\x03.' +python2_exceptions_without_args = ( + ArithmeticError, + AssertionError, + AttributeError, + BaseException, + BufferError, + BytesWarning, + DeprecationWarning, + EOFError, + EnvironmentError, + Exception, + FloatingPointError, + FutureWarning, + GeneratorExit, + IOError, + ImportError, + ImportWarning, + IndentationError, + IndexError, + KeyError, + KeyboardInterrupt, + LookupError, + MemoryError, + NameError, + NotImplementedError, + OSError, + OverflowError, + PendingDeprecationWarning, + ReferenceError, + RuntimeError, + RuntimeWarning, + # StandardError is gone in Python 3, we map it to Exception + StopIteration, + SyntaxError, + SyntaxWarning, + SystemError, + SystemExit, + TabError, + TypeError, + UnboundLocalError, + UnicodeError, + UnicodeWarning, + UserWarning, + ValueError, + Warning, + ZeroDivisionError, +) + +exception_pickle = b'\x80\x02cexceptions\n?\nq\x00)Rq\x01.' + +# Exception objects without arguments pickled from 2.x with protocol 2 +DATA7 = { + exception : + exception_pickle.replace(b'?', exception.__name__.encode("ascii")) + for exception in python2_exceptions_without_args +} + +# StandardError is mapped to Exception, test that separately +DATA8 = exception_pickle.replace(b'?', b'StandardError') + +# UnicodeEncodeError object pickled from 2.x with protocol 2 +DATA9 = (b'\x80\x02cexceptions\nUnicodeEncodeError\n' + b'q\x00(U\x05asciiq\x01X\x03\x00\x00\x00fooq\x02K\x00K\x01' + b'U\x03badq\x03tq\x04Rq\x05.') + def create_data(): c = C() @@ -1160,6 +1225,21 @@ self.assertEqual(list(loaded.keys()), ["key"]) self.assertEqual(loaded["key"].value, "Set-Cookie: key=value") + for (exc, data) in DATA7.items(): + loaded = self.loads(data) + self.assertIs(type(loaded), exc) + + loaded = self.loads(DATA8) + self.assertIs(type(loaded), Exception) + + loaded = self.loads(DATA9) + self.assertIs(type(loaded), UnicodeEncodeError) + self.assertEqual(loaded.object, "foo") + self.assertEqual(loaded.encoding, "ascii") + self.assertEqual(loaded.start, 0) + self.assertEqual(loaded.end, 1) + self.assertEqual(loaded.reason, "bad") + def test_pickle_to_2x(self): # Pickle non-trivial data with protocol 2, expecting that it yields # the same result as Python 2.x did. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 11:44:24 2013 From: python-checkins at python.org (walter.doerwald) Date: Mon, 2 Dec 2013 11:44:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogRml4ICMxOTgzNDogbWVyZ2Ugd2l0aCAzLjMu?= Message-ID: <3dY2y04rKpz7LrZ@mail.python.org> http://hg.python.org/cpython/rev/9685c9d1d67f changeset: 87691:9685c9d1d67f parent: 87689:373797990f57 parent: 87690:7d3297f127ae user: Walter Doerwald date: Mon Dec 02 11:43:20 2013 +0100 summary: Fix #19834: merge with 3.3. files: Lib/_compat_pickle.py | 56 +++++++++++++++++++ Lib/test/pickletester.py | 80 ++++++++++++++++++++++++++++ 2 files changed, 136 insertions(+), 0 deletions(-) diff --git a/Lib/_compat_pickle.py b/Lib/_compat_pickle.py --- a/Lib/_compat_pickle.py +++ b/Lib/_compat_pickle.py @@ -76,6 +76,62 @@ ('itertools', 'ifilterfalse'): ('itertools', 'filterfalse'), } +PYTHON2_EXCEPTIONS = ( + "ArithmeticError", + "AssertionError", + "AttributeError", + "BaseException", + "BufferError", + "BytesWarning", + "DeprecationWarning", + "EOFError", + "EnvironmentError", + "Exception", + "FloatingPointError", + "FutureWarning", + "GeneratorExit", + "IOError", + "ImportError", + "ImportWarning", + "IndentationError", + "IndexError", + "KeyError", + "KeyboardInterrupt", + "LookupError", + "MemoryError", + "NameError", + "NotImplementedError", + "OSError", + "OverflowError", + "PendingDeprecationWarning", + "ReferenceError", + "RuntimeError", + "RuntimeWarning", + # StandardError is gone in Python 3, so we map it to Exception + "StopIteration", + "SyntaxError", + "SyntaxWarning", + "SystemError", + "SystemExit", + "TabError", + "TypeError", + "UnboundLocalError", + "UnicodeDecodeError", + "UnicodeEncodeError", + "UnicodeError", + "UnicodeTranslateError", + "UnicodeWarning", + "UserWarning", + "ValueError", + "Warning", + "ZeroDivisionError", +) + +for excname in PYTHON2_EXCEPTIONS: + NAME_MAPPING[("exceptions", excname)] = ("builtins", excname) + +NAME_MAPPING[("exceptions", "StandardError")] = ("builtins", "Exception") + # Same, but for 3.x to 2.x REVERSE_IMPORT_MAPPING = dict((v, k) for (k, v) in IMPORT_MAPPING.items()) REVERSE_NAME_MAPPING = dict((v, k) for (k, v) in NAME_MAPPING.items()) diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -413,6 +413,71 @@ # set([3]) pickled from 2.x with protocol 2 DATA6 = b'\x80\x02c__builtin__\nset\nq\x00]q\x01K\x03a\x85q\x02Rq\x03.' +python2_exceptions_without_args = ( + ArithmeticError, + AssertionError, + AttributeError, + BaseException, + BufferError, + BytesWarning, + DeprecationWarning, + EOFError, + EnvironmentError, + Exception, + FloatingPointError, + FutureWarning, + GeneratorExit, + IOError, + ImportError, + ImportWarning, + IndentationError, + IndexError, + KeyError, + KeyboardInterrupt, + LookupError, + MemoryError, + NameError, + NotImplementedError, + OSError, + OverflowError, + PendingDeprecationWarning, + ReferenceError, + RuntimeError, + RuntimeWarning, + # StandardError is gone in Python 3, we map it to Exception + StopIteration, + SyntaxError, + SyntaxWarning, + SystemError, + SystemExit, + TabError, + TypeError, + UnboundLocalError, + UnicodeError, + UnicodeWarning, + UserWarning, + ValueError, + Warning, + ZeroDivisionError, +) + +exception_pickle = b'\x80\x02cexceptions\n?\nq\x00)Rq\x01.' + +# Exception objects without arguments pickled from 2.x with protocol 2 +DATA7 = { + exception : + exception_pickle.replace(b'?', exception.__name__.encode("ascii")) + for exception in python2_exceptions_without_args +} + +# StandardError is mapped to Exception, test that separately +DATA8 = exception_pickle.replace(b'?', b'StandardError') + +# UnicodeEncodeError object pickled from 2.x with protocol 2 +DATA9 = (b'\x80\x02cexceptions\nUnicodeEncodeError\n' + b'q\x00(U\x05asciiq\x01X\x03\x00\x00\x00fooq\x02K\x00K\x01' + b'U\x03badq\x03tq\x04Rq\x05.') + def create_data(): c = C() @@ -1214,6 +1279,21 @@ self.assertEqual(list(loaded.keys()), ["key"]) self.assertEqual(loaded["key"].value, "Set-Cookie: key=value") + for (exc, data) in DATA7.items(): + loaded = self.loads(data) + self.assertIs(type(loaded), exc) + + loaded = self.loads(DATA8) + self.assertIs(type(loaded), Exception) + + loaded = self.loads(DATA9) + self.assertIs(type(loaded), UnicodeEncodeError) + self.assertEqual(loaded.object, "foo") + self.assertEqual(loaded.encoding, "ascii") + self.assertEqual(loaded.start, 0) + self.assertEqual(loaded.end, 1) + self.assertEqual(loaded.reason, "bad") + def test_pickle_to_2x(self): # Pickle non-trivial data with protocol 2, expecting that it yields # the same result as Python 2.x did. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 12:18:04 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 2 Dec 2013 12:18:04 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5NzI4?= =?utf-8?q?=3A_Fix_sys=2Egetfilesystemencoding=28=29_documentation?= Message-ID: <3dY3hr6BG1z7LrQ@mail.python.org> http://hg.python.org/cpython/rev/b231e0c3fd26 changeset: 87692:b231e0c3fd26 branch: 3.3 parent: 87690:7d3297f127ae user: Victor Stinner date: Mon Dec 02 12:16:46 2013 +0100 summary: Issue #19728: Fix sys.getfilesystemencoding() documentation files: Doc/library/sys.rst | 5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-) diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -409,7 +409,7 @@ * On Mac OS X, the encoding is ``'utf-8'``. * On Unix, the encoding is the user's preference according to the result of - nl_langinfo(CODESET), or ``'utf-8'`` if ``nl_langinfo(CODESET)`` failed. + nl_langinfo(CODESET). * On Windows NT+, file names are Unicode natively, so no conversion is performed. :func:`getfilesystemencoding` still returns ``'mbcs'``, as @@ -420,8 +420,7 @@ * On Windows 9x, the encoding is ``'mbcs'``. .. versionchanged:: 3.2 - On Unix, use ``'utf-8'`` instead of ``None`` if ``nl_langinfo(CODESET)`` - failed. :func:`getfilesystemencoding` result cannot be ``None``. + :func:`getfilesystemencoding` result cannot be ``None`` anymore. .. function:: getrefcount(object) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 12:18:06 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 2 Dec 2013 12:18:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E3=29_Issue_=2319728=3A_Fix_sys=2Egetfilesys?= =?utf-8?q?temencoding=28=29_documentation?= Message-ID: <3dY3ht0xVPz7Ls0@mail.python.org> http://hg.python.org/cpython/rev/e3c48bddf621 changeset: 87693:e3c48bddf621 parent: 87691:9685c9d1d67f parent: 87692:b231e0c3fd26 user: Victor Stinner date: Mon Dec 02 12:17:29 2013 +0100 summary: (Merge 3.3) Issue #19728: Fix sys.getfilesystemencoding() documentation files: Doc/library/sys.rst | 5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-) diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -425,7 +425,7 @@ * On Mac OS X, the encoding is ``'utf-8'``. * On Unix, the encoding is the user's preference according to the result of - nl_langinfo(CODESET), or ``'utf-8'`` if ``nl_langinfo(CODESET)`` failed. + nl_langinfo(CODESET). * On Windows NT+, file names are Unicode natively, so no conversion is performed. :func:`getfilesystemencoding` still returns ``'mbcs'``, as @@ -436,8 +436,7 @@ * On Windows 9x, the encoding is ``'mbcs'``. .. versionchanged:: 3.2 - On Unix, use ``'utf-8'`` instead of ``None`` if ``nl_langinfo(CODESET)`` - failed. :func:`getfilesystemencoding` result cannot be ``None``. + :func:`getfilesystemencoding` result cannot be ``None`` anymore. .. function:: getrefcount(object) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 12:42:13 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 2 Dec 2013 12:42:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319833=3A_asyncio_?= =?utf-8?q?doc=3A_add_class_name_to_methods?= Message-ID: <3dY4Dj2tsFz7LsT@mail.python.org> http://hg.python.org/cpython/rev/7b4d046dbf56 changeset: 87694:7b4d046dbf56 user: Victor Stinner date: Mon Dec 02 12:20:57 2013 +0100 summary: Issue #19833: asyncio doc: add class name to methods files: Doc/library/asyncio.rst | 68 ++++++++++++++-------------- 1 files changed, 34 insertions(+), 34 deletions(-) diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -87,12 +87,12 @@ implementation; ideally it is a monotonic clock. This will generally be a different clock than :func:`time.time`. -.. method:: time() +.. method:: BaseEventLoop.time() Return the current time, as a :class:`float` value, according to the event loop's internal clock. -.. method:: call_later(delay, callback, *args) +.. method:: BaseEventLoop.call_later(delay, callback, *args) Arrange for the *callback* to be called after the given *delay* seconds (either an int or float). @@ -108,7 +108,7 @@ is called. If you want the callback to be called with some named arguments, use a closure or :func:`functools.partial`. -.. method:: call_at(when, callback, *args) +.. method:: BaseEventLoop.call_at(when, callback, *args) Arrange for the *callback* to be called at the given absolute timestamp *when* (an int or float), using the same time reference as :meth:`time`. @@ -118,7 +118,7 @@ Creating connections ^^^^^^^^^^^^^^^^^^^^ -.. method:: create_connection(protocol_factory, host=None, port=None, **options) +.. method:: BaseEventLoop.create_connection(protocol_factory, host=None, port=None, **options) Create a streaming transport connection to a given Internet *host* and *port*. *protocol_factory* must be a callable returning a @@ -228,7 +228,7 @@ These callbacks may be called on :class:`Protocol` and :class:`SubprocessProtocol` instances: -.. method:: connection_made(transport) +.. method:: BaseProtocol.connection_made(transport) Called when a connection is made. @@ -236,7 +236,7 @@ connection. You are responsible for storing it somewhere (e.g. as an attribute) if you need to. -.. method:: connection_lost(exc) +.. method:: BaseProtocol.connection_lost(exc) Called when the connection is lost or closed. @@ -252,18 +252,18 @@ The following callbacks may be called only on :class:`SubprocessProtocol` instances: -.. method:: pipe_data_received(fd, data) +.. method:: SubprocessProtocol.pipe_data_received(fd, data) Called when the child process writes data into its stdout or stderr pipe. *fd* is the integer file descriptor of the pipe. *data* is a non-empty bytes object containing the data. -.. method:: pipe_connection_lost(fd, exc) +.. method:: SubprocessProtocol.pipe_connection_lost(fd, exc) Called when one of the pipes communicating with the child process is closed. *fd* is the integer file descriptor that was closed. -.. method:: process_exited() +.. method:: SubprocessProtocol.process_exited() Called when the child process has exited. @@ -276,7 +276,7 @@ The following callbacks are called on :class:`Protocol` instances: -.. method:: data_received(data) +.. method:: Protocol.data_received(data) Called when some data is received. *data* is a non-empty bytes object containing the incoming data. @@ -287,7 +287,7 @@ and instead make your parsing generic and flexible enough. However, data is always received in the correct order. -.. method:: eof_received() +.. method:: Protocol.eof_received() Calls when the other end signals it won't send any more data (for example by calling :meth:`write_eof`, if the other end also uses @@ -312,13 +312,13 @@ The following callbacks are called on :class:`DatagramProtocol` instances. -.. method:: datagram_received(data, addr) +.. method:: DatagramProtocol.datagram_received(data, addr) Called when a datagram is received. *data* is a bytes object containing the incoming data. *addr* is the address of the peer sending the data; the exact format depends on the transport. -.. method:: error_received(exc) +.. method:: DatagramProtocol.error_received(exc) Called when a previous send or receive operation raises an :class:`OSError`. *exc* is the :class:`OSError` instance. @@ -335,11 +335,11 @@ These callbacks may be called on :class:`Protocol` and :class:`SubprocessProtocol` instances: -.. method:: pause_writing() +.. method:: BaseProtocol.pause_writing() Called when the transport's buffer goes over the high-water mark. -.. method:: resume_writing() +.. method:: BaseProtocol.resume_writing() Called when the transport's buffer drains below the low-water mark. @@ -381,7 +381,7 @@ Methods common to all transports ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. method:: close(self) +.. method:: BaseTransport.close(self) Close the transport. If the transport has a buffer for outgoing data, buffered data will be flushed asynchronously. No more data @@ -390,7 +390,7 @@ :const:`None` as its argument. -.. method:: get_extra_info(name, default=None) +.. method:: BaseTransport.get_extra_info(name, default=None) Return optional transport information. *name* is a string representing the piece of transport-specific information to get, *default* is the @@ -402,13 +402,13 @@ Methods of readable streaming transports ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. method:: pause_reading() +.. method:: ReadTransport.pause_reading() Pause the receiving end of the transport. No data will be passed to the protocol's :meth:`data_received` method until meth:`resume_reading` is called. -.. method:: resume_reading() +.. method:: ReadTransport.resume_reading() Resume the receiving end. The protocol's :meth:`data_received` method will be called once again if some data is available for reading. @@ -416,20 +416,20 @@ Methods of writable streaming transports ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. method:: write(data) +.. method:: WriteTransport.write(data) Write some *data* bytes to the transport. This method does not block; it buffers the data and arranges for it to be sent out asynchronously. -.. method:: writelines(list_of_data) +.. method:: WriteTransport.writelines(list_of_data) Write a list (or any iterable) of data bytes to the transport. This is functionally equivalent to calling :meth:`write` on each element yielded by the iterable, but may be implemented more efficiently. -.. method:: write_eof() +.. method:: WriteTransport.write_eof() Close the write end of the transport after flushing buffered data. Data may still be received. @@ -437,19 +437,19 @@ This method can raise :exc:`NotImplementedError` if the transport (e.g. SSL) doesn't support half-closes. -.. method:: can_write_eof() +.. method:: WriteTransport.can_write_eof() Return :const:`True` if the transport supports :meth:`write_eof`, :const:`False` if not. -.. method:: abort() +.. method:: WriteTransport.abort() Close the transport immediately, without waiting for pending operations to complete. Buffered data will be lost. No more data will be received. The protocol's :meth:`connection_lost` method will eventually be called with :const:`None` as its argument. -.. method:: set_write_buffer_limits(high=None, low=None) +.. method:: WriteTransport.set_write_buffer_limits(high=None, low=None) Set the *high*- and *low*-water limits for write flow control. @@ -469,14 +469,14 @@ reduces opportunities for doing I/O and computation concurrently. -.. method:: get_write_buffer_size() +.. method:: WriteTransport.get_write_buffer_size() Return the current size of the output buffer used by the transport. Methods of datagram transports ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. method:: sendto(data, addr=None) +.. method:: DatagramTransport.sendto(data, addr=None) Send the *data* bytes to the remote peer given by *addr* (a transport-dependent target address). If *addr* is :const:`None`, the @@ -485,7 +485,7 @@ This method does not block; it buffers the data and arranges for it to be sent out asynchronously. -.. method:: abort() +.. method:: DatagramTransport.abort() Close the transport immediately, without waiting for pending operations to complete. Buffered data will be lost. No more data will be received. @@ -495,17 +495,17 @@ Methods of subprocess transports ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. method:: get_pid() +.. method:: BaseSubprocessTransport.get_pid() Return the subprocess process id as an integer. -.. method:: get_returncode() +.. method:: BaseSubprocessTransport.get_returncode() Return the subprocess returncode as an integer or :const:`None` if it hasn't returned, similarly to the :attr:`subprocess.Popen.returncode` attribute. -.. method:: get_pipe_transport(fd) +.. method:: BaseSubprocessTransport.get_pipe_transport(fd) Return the transport for the communication pipe correspondong to the integer file descriptor *fd*. The return value can be a readable or @@ -513,12 +513,12 @@ correspond to a pipe belonging to this transport, :const:`None` is returned. -.. method:: send_signal(signal) +.. method:: BaseSubprocessTransport.send_signal(signal) Send the *signal* number to the subprocess, as in :meth:`subprocess.Popen.send_signal`. -.. method:: terminate() +.. method:: BaseSubprocessTransport.terminate() Ask the subprocess to stop, as in :meth:`subprocess.Popen.terminate`. This method is an alias for the :meth:`close` method. @@ -527,7 +527,7 @@ On Windows, the Windows API function TerminateProcess() is called to stop the subprocess. -.. method:: kill(self) +.. method:: BaseSubprocessTransport.kill(self) Kill the subprocess, as in :meth:`subprocess.Popen.kill` -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 12:42:14 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 2 Dec 2013 12:42:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319833=3A_add_2_ex?= =?utf-8?q?amples_to_asyncio_doc_=28hello_world=29?= Message-ID: <3dY4Dk4YrLz7Lsm@mail.python.org> http://hg.python.org/cpython/rev/b25458022965 changeset: 87695:b25458022965 user: Victor Stinner date: Mon Dec 02 12:21:30 2013 +0100 summary: Issue #19833: add 2 examples to asyncio doc (hello world) files: Doc/library/asyncio.rst | 36 +++++++++++++++++++++++++++++ 1 files changed, 36 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -550,6 +550,42 @@ Examples -------- +Hello World (callback) +^^^^^^^^^^^^^^^^^^^^^^ + +Print ``Hello World`` every two seconds, using a callback:: + + import asyncio + + def print_and_repeat(loop): + print('Hello World') + loop.call_later(2, print_and_repeat, loop) + + loop = asyncio.get_event_loop() + print_and_repeat(loop) + loop.run_forever() + + +Hello World (callback) +^^^^^^^^^^^^^^^^^^^^^^ + +Print ``Hello World`` every two seconds, using a coroutine:: + + import asyncio + + @asyncio.coroutine + def greet_every_two_seconds(): + while True: + print('Hello World') + yield from asyncio.sleep(2) + + loop = asyncio.get_event_loop() + loop.run_until_complete(greet_every_two_seconds()) + + +Echo server +^^^^^^^^^^^ + A :class:`Protocol` implementing an echo server:: class EchoServer(asyncio.Protocol): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 12:42:16 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 2 Dec 2013 12:42:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319833=3A_Document?= =?utf-8?q?_more_asyncio=2EBaseEventLoop_methods?= Message-ID: <3dY4Dm0VbLz7LsQ@mail.python.org> http://hg.python.org/cpython/rev/5e4ea92f9a9b changeset: 87696:5e4ea92f9a9b user: Victor Stinner date: Mon Dec 02 12:41:54 2013 +0100 summary: Issue #19833: Document more asyncio.BaseEventLoop methods files: Doc/library/asyncio.rst | 145 +++++++++++++++++++++++++++- 1 files changed, 143 insertions(+), 2 deletions(-) diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -77,7 +77,76 @@ The easiest way to get an event loop is to call the :func:`get_event_loop` function. -.. XXX more docs +.. function:: get_event_loop() + + Get the event loop for current context. Returns an event loop object + implementing :class:`BaseEventLoop` interface, or raises an exception in case no + event loop has been set for the current context and the current policy does + not specify to create one. It should never return ``None``. + + +Run an event loop +^^^^^^^^^^^^^^^^^ + +.. method:: BaseEventLoop.run_forever() + + Run until :meth:`stop` is called. + +.. method:: BaseEventLoop.run_in_executor(executor, callback, \*args) + + XXX + +.. method:: BaseEventLoop.run_until_complete(future) + + Run until the :class:`Future` is done. + + If the argument is a coroutine, it is wrapped in a :class:`Task`. + + Return the Future's result, or raise its exception. + +.. method:: stop() + + Stop running the event loop. + + Every callback scheduled before :meth:`stop` is called will run. + Callback scheduled after :meth:`stop` is called won't. However, those + callbacks will run if :meth:`run_forever` is called again later. + +.. method:: BaseEventLoop.close() + + Close the event loop. + + This clears the queues and shuts down the executor, but does not wait for + the executor to finish. + +.. method:: BaseEventLoop.is_running() + + Returns running status of event loop. + + +Calls +^^^^^ + +.. method:: BaseEventLoop.call_soon(callback, \*args) + + Arrange for a callback to be called as soon as possible. + + This operates as a FIFO queue, callbacks are called in the order in + which they are registered. Each callback will be called exactly once. + + Any positional arguments after the callback will be passed to the + callback when it is called. + +.. method: BaseEventLoop.call_soon_threadsafe(callback, \*args) + + Like :meth:`call_soon`, but thread safe. + +.. method:: BaseEventLoop.set_default_executor(executor) + + XXX + + + Delayed calls ^^^^^^^^^^^^^ @@ -115,6 +184,38 @@ This method's behavior is the same as :meth:`call_later`. +.. method:: BaseEventLoop.time() + + Return the time according to the event loop's clock. + + The clock :func:`time.monotonic` is used by default. + + +Creating listening connections +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. method:: BaseEventLoop.create_server(protocol_factory, host=None, port=None, \*, family=socket.AF_UNSPEC, flags=socket.AI_PASSIVE, sock=None, backlog=100, ssl=None, reuse_address=None) + + XXX + + * *protocol_factory* + * *host*, *port* + * *family* + * *flags* + * *sock* + * *backlog* : the maximum number of queued connections and should be at + least ``0``; the maximum value is system-dependent (usually ``5``), + the minimum value is forced to ``0``. + * *ssl*: ``True`` or :class:`ssl.SSLContext` + * *reuse_address*: if ``True``, set :data:`socket.SO_REUSEADDR` option + on the listening socket. Default value: ``True`` on POSIX systems, + ``False`` on Windows. + +.. method:: BaseEventLoop.create_datagram_endpoint(protocol_factory, local_addr=None, remote_addr=None, \*, family=0, proto=0, flags=0) + + XXX + + Creating connections ^^^^^^^^^^^^^^^^^^^^ @@ -180,6 +281,46 @@ to bind the socket to locally. The *local_host* and *local_port* are looked up using getaddrinfo(), similarly to *host* and *port*. +.. method:: BaseEventLoop.connect_read_pipe(protocol_factory, pipe) + + XXX + +.. method:: BaseEventLoop.connect_write_pipe(protocol_factory, pipe) + + XXX + + +Resolve name +^^^^^^^^^^^^ + +.. method:: BaseEventLoop.getaddrinfo(host, port, \*, family=0, type=0, proto=0, flags=0) + + XXX + +.. method:: BaseEventLoop.getnameinfo(sockaddr, flags=0) + + XXX + + +Running subprocesses +^^^^^^^^^^^^^^^^^^^^ + +.. method:: BaseEventLoop.subprocess_shell(protocol_factory, cmd, \*, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=True, bufsize=0, \*\*kwargs) + + XXX + + See the constructor of the :class:`subprocess.Popen` class for parameters. + +.. method:: BaseEventLoop.subprocess_exec(protocol_factory, \*args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=False, bufsize=0, \*\*kwargs) + + XXX + + See the constructor of the :class:`subprocess.Popen` class for parameters. + +.. seealso:: + + The :mod:`subprocess` module. + .. _protocol: @@ -366,7 +507,7 @@ Transports are classed provided by :mod:`asyncio` in order to abstract various kinds of communication channels. You generally won't instantiate -a transport yourself; instead, you will call a :class:`EventLoop` method +a transport yourself; instead, you will call a :class:`BaseEventLoop` method which will create the transport and try to initiate the underlying communication channel, calling you back when it succeeds. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 13:04:34 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 2 Dec 2013 13:04:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_cleanup_doc?= Message-ID: <3dY4kV36LFz7Lsm@mail.python.org> http://hg.python.org/cpython/rev/a1e6c8390d33 changeset: 87697:a1e6c8390d33 user: Victor Stinner date: Mon Dec 02 13:04:25 2013 +0100 summary: asyncio: cleanup doc files: Doc/library/asyncio.rst | 66 +++++++++++++++++----------- 1 files changed, 40 insertions(+), 26 deletions(-) diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -92,10 +92,6 @@ Run until :meth:`stop` is called. -.. method:: BaseEventLoop.run_in_executor(executor, callback, \*args) - - XXX - .. method:: BaseEventLoop.run_until_complete(future) Run until the :class:`Future` is done. @@ -104,6 +100,10 @@ Return the Future's result, or raise its exception. +.. method:: BaseEventLoop.is_running() + + Returns running status of event loop. + .. method:: stop() Stop running the event loop. @@ -119,10 +119,6 @@ This clears the queues and shuts down the executor, but does not wait for the executor to finish. -.. method:: BaseEventLoop.is_running() - - Returns running status of event loop. - Calls ^^^^^ @@ -137,16 +133,10 @@ Any positional arguments after the callback will be passed to the callback when it is called. -.. method: BaseEventLoop.call_soon_threadsafe(callback, \*args) +.. method:: BaseEventLoop.call_soon_threadsafe(callback, \*args) Like :meth:`call_soon`, but thread safe. -.. method:: BaseEventLoop.set_default_executor(executor) - - XXX - - - Delayed calls ^^^^^^^^^^^^^ @@ -156,11 +146,6 @@ implementation; ideally it is a monotonic clock. This will generally be a different clock than :func:`time.time`. -.. method:: BaseEventLoop.time() - - Return the current time, as a :class:`float` value, according to the - event loop's internal clock. - .. method:: BaseEventLoop.call_later(delay, callback, *args) Arrange for the *callback* to be called after the given *delay* @@ -186,9 +171,27 @@ .. method:: BaseEventLoop.time() - Return the time according to the event loop's clock. + Return the current time, as a :class:`float` value, according to the + event loop's internal clock. - The clock :func:`time.monotonic` is used by default. + +Executor +^^^^^^^^ + +Call a function in an :class:`~concurrent.futures.Executor` (pool of threads or +pool of processes). By default, an event loop uses a thread pool executor +(:class:`~concurrent.futures.ThreadPoolExecutor`). + +.. method:: BaseEventLoop.run_in_executor(executor, callback, \*args) + + Arrange for a callback to be called in the specified executor. + + *executor* is a :class:`~concurrent.futures.Executor` instance, + the default executor is used if *executor* is ``None``. + +.. method:: BaseEventLoop.set_default_executor(executor) + + Set the default executor used by :meth:`run_in_executor`. Creating listening connections @@ -211,10 +214,15 @@ on the listening socket. Default value: ``True`` on POSIX systems, ``False`` on Windows. + This method returns a :ref:`coroutine `. + .. method:: BaseEventLoop.create_datagram_endpoint(protocol_factory, local_addr=None, remote_addr=None, \*, family=0, proto=0, flags=0) XXX + This method returns a :ref:`coroutine `. + + Creating connections ^^^^^^^^^^^^^^^^^^^^ @@ -285,10 +293,14 @@ XXX + This method returns a :ref:`coroutine `. + .. method:: BaseEventLoop.connect_write_pipe(protocol_factory, pipe) XXX + This method returns a :ref:`coroutine `. + Resolve name ^^^^^^^^^^^^ @@ -305,22 +317,24 @@ Running subprocesses ^^^^^^^^^^^^^^^^^^^^ +Run subprocesses asynchronously using the :mod:`subprocess` module. + .. method:: BaseEventLoop.subprocess_shell(protocol_factory, cmd, \*, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=True, bufsize=0, \*\*kwargs) XXX + This method returns a :ref:`coroutine `. + See the constructor of the :class:`subprocess.Popen` class for parameters. .. method:: BaseEventLoop.subprocess_exec(protocol_factory, \*args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=False, bufsize=0, \*\*kwargs) XXX + This method returns a :ref:`coroutine `. + See the constructor of the :class:`subprocess.Popen` class for parameters. -.. seealso:: - - The :mod:`subprocess` module. - .. _protocol: -- Repository URL: http://hg.python.org/cpython From ncoghlan at gmail.com Mon Dec 2 13:18:10 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 2 Dec 2013 22:18:10 +1000 Subject: [Python-checkins] cpython (3.3): Issue #19728: Fix sys.getfilesystemencoding() documentation In-Reply-To: <3dY3hr6BG1z7LrQ@mail.python.org> References: <3dY3hr6BG1z7LrQ@mail.python.org> Message-ID: On 2 Dec 2013 21:18, "victor.stinner" wrote: > > http://hg.python.org/cpython/rev/b231e0c3fd26 > changeset: 87692:b231e0c3fd26 > branch: 3.3 > parent: 87690:7d3297f127ae > user: Victor Stinner > date: Mon Dec 02 12:16:46 2013 +0100 > summary: > Issue #19728: Fix sys.getfilesystemencoding() documentation This doesn't appear to be the right issue number (ironically, this number refers to the PEP 453 Windows installer one I was trying to reference when I got a commit message wrong the other day and ended up referencing an asyncio bug instead). Cheers, Nick. > > files: > Doc/library/sys.rst | 5 ++--- > 1 files changed, 2 insertions(+), 3 deletions(-) > > > diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst > --- a/Doc/library/sys.rst > +++ b/Doc/library/sys.rst > @@ -409,7 +409,7 @@ > * On Mac OS X, the encoding is ``'utf-8'``. > > * On Unix, the encoding is the user's preference according to the result of > - nl_langinfo(CODESET), or ``'utf-8'`` if ``nl_langinfo(CODESET)`` failed. > + nl_langinfo(CODESET). > > * On Windows NT+, file names are Unicode natively, so no conversion is > performed. :func:`getfilesystemencoding` still returns ``'mbcs'``, as > @@ -420,8 +420,7 @@ > * On Windows 9x, the encoding is ``'mbcs'``. > > .. versionchanged:: 3.2 > - On Unix, use ``'utf-8'`` instead of ``None`` if ``nl_langinfo(CODESET)`` > - failed. :func:`getfilesystemencoding` result cannot be ``None``. > + :func:`getfilesystemencoding` result cannot be ``None`` anymore. > > > .. function:: getrefcount(object) > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-checkins at python.org Mon Dec 2 13:47:36 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 2 Dec 2013 13:47:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Document_Task_class_and_ta?= =?utf-8?q?sk_functions?= Message-ID: <3dY5h854tDz7Lnr@mail.python.org> http://hg.python.org/cpython/rev/72495938aaeb changeset: 87698:72495938aaeb user: Victor Stinner date: Mon Dec 02 13:47:23 2013 +0100 summary: Document Task class and task functions files: Doc/library/asyncio.rst | 200 ++++++++++++++++++++++++++++ 1 files changed, 200 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -690,11 +690,211 @@ On Windows, this method is an alias for :meth:`terminate`. +Task functions +-------------- + +.. function:: as_completed(fs, *, loop=None, timeout=None) + + Return an iterator whose values, when waited for, are + :class:`~concurrent.futures.Future` instances. + + Raises :exc:`TimeoutError` if the timeout occurs before all Futures are done. + + Example:: + + for f in as_completed(fs): + result = yield from f # The 'yield from' may raise + # Use result + + .. note:: + + The futures ``f`` are not necessarily members of fs. + +.. function:: async(coro_or_future, *, loop=None) + + Wrap a :ref:`coroutine ` in a future. + + If the argument is a :class:`~concurrent.futures.Future`, it is returned + directly. + +.. function:: gather(*coros_or_futures, loop=None, return_exceptions=False) + + Return a future aggregating results from the given coroutines or futures. + + All futures must share the same event loop. If all the tasks are done + successfully, the returned future's result is the list of results (in the + order of the original sequence, not necessarily the order of results + arrival). If *result_exception* is True, exceptions in the tasks are + treated the same as successful results, and gathered in the result list; + otherwise, the first raised exception will be immediately propagated to the + returned future. + + Cancellation: if the outer Future is cancelled, all children (that have not + completed yet) are also cancelled. If any child is cancelled, this is + treated as if it raised :exc:`~concurrent.futures.CancelledError` -- the + outer Future is *not* cancelled in this case. (This is to prevent the + cancellation of one child to cause other children to be cancelled.) + +.. function:: iscoroutinefunction(func) + + Return ``True`` if *func* is a decorated coroutine function. + +.. function:: iscoroutine(obj) + + Return ``True`` if *obj* is a coroutine object. + +.. function:: sleep(delay, result=None, *, loop=None) + + Create a :ref:`coroutine ` that completes after a given time + (in seconds). + +.. function:: shield(arg, *, loop=None) + + Wait for a future, shielding it from cancellation. + + The statement:: + + res = yield from shield(something()) + + is exactly equivalent to the statement:: + + res = yield from something() + + *except* that if the coroutine containing it is cancelled, the task running + in ``something()`` is not cancelled. From the point of view of + ``something()``, the cancellation did not happen. But its caller is still + cancelled, so the yield-from expression still raises + :exc:`~concurrent.futures.CancelledError`. Note: If ``something()`` is + cancelled by other means this will still cancel ``shield()``. + + If you want to completely ignore cancellation (not recommended) you can + combine ``shield()`` with a try/except clause, as follows:: + + try: + res = yield from shield(something()) + except CancelledError: + res = None + + +Task +---- + +.. class:: Task(coro, *, loop=None) + + A coroutine wrapped in a :class:`~concurrent.futures.Future`. + + .. classmethod:: all_tasks(loop=None) + + Return a set of all tasks for an event loop. + + By default all tasks for the current event loop are returned. + + .. method:: cancel() + + Cancel the task. + + .. method:: get_stack(self, *, limit=None) + + Return the list of stack frames for this task's coroutine. + + If the coroutine is active, this returns the stack where it is suspended. + If the coroutine has completed successfully or was cancelled, this + returns an empty list. If the coroutine was terminated by an exception, + this returns the list of traceback frames. + + The frames are always ordered from oldest to newest. + + The optional limit gives the maximum nummber of frames to return; by + default all available frames are returned. Its meaning differs depending + on whether a stack or a traceback is returned: the newest frames of a + stack are returned, but the oldest frames of a traceback are returned. + (This matches the behavior of the traceback module.) + + For reasons beyond our control, only one stack frame is returned for a + suspended coroutine. + + .. method:: print_stack(*, limit=None, file=None) + + Print the stack or traceback for this task's coroutine. + + This produces output similar to that of the traceback module, for the + frames retrieved by get_stack(). The limit argument is passed to + get_stack(). The file argument is an I/O stream to which the output + goes; by default it goes to sys.stderr. + + +Protocols +--------- + +:mod:`asyncio` provides base classes that you can subclass to implement +your network protocols. Those classes are used in conjunction with +:ref:`transports ` (see below): the protocol parses incoming +data and asks for the writing of outgoing data, while the transport is +responsible for the actual I/O and buffering. + +When subclassing a protocol class, it is recommended you override certain +methods. Those methods are callbacks: they will be called by the transport +on certain events (for example when some data is received); you shouldn't +call them yourself, unless you are implementing a transport. + +.. note:: + All callbacks have default implementations, which are empty. Therefore, + you only need to implement the callbacks for the events in which you + are interested. + + .. _coroutine: Coroutines ---------- +A coroutine is a generator that follows certain conventions. For +documentation purposes, all coroutines should be decorated with +``@asyncio.coroutine``, but this cannot be strictly enforced. + +Coroutines use the ``yield from`` syntax introduced in :pep:`380`, +instead of the original ``yield`` syntax. + +The word "coroutine", like the word "generator", is used for two +different (though related) concepts: + +- The function that defines a coroutine (a function definition + decorated with ``asyncio.coroutine``). If disambiguation is needed + we will call this a *coroutine function*. + +- The object obtained by calling a coroutine function. This object + represents a computation or an I/O operation (usually a combination) + that will complete eventually. If disambiguation is needed we will + call it a *coroutine object*. + +Things a coroutine can do: + +- ``result = yield from future`` -- suspends the coroutine until the + future is done, then returns the future's result, or raises an + exception, which will be propagated. (If the future is cancelled, + it will raise a ``CancelledError`` exception.) Note that tasks are + futures, and everything said about futures also applies to tasks. + +- ``result = yield from coroutine`` -- wait for another coroutine to + produce a result (or raise an exception, which will be propagated). + The ``coroutine`` expression must be a *call* to another coroutine. + +- ``return expression`` -- produce a result to the coroutine that is + waiting for this one using ``yield from``. + +- ``raise exception`` -- raise an exception in the coroutine that is + waiting for this one using ``yield from``. + +Calling a coroutine does not start its code running -- it is just a +generator, and the coroutine object returned by the call is really a +generator object, which doesn't do anything until you iterate over it. +In the case of a coroutine object, there are two basic ways to start +it running: call ``yield from coroutine`` from another coroutine +(assuming the other coroutine is already running!), or convert it to a +:class:`Task`. + +Coroutines (and tasks) can only run when the event loop is running. + .. _sync: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 14:07:13 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 2 Dec 2013 14:07:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_document_Queue_?= =?utf-8?q?classes?= Message-ID: <3dY66n75LWzMfZ@mail.python.org> http://hg.python.org/cpython/rev/5cf89bb64492 changeset: 87699:5cf89bb64492 user: Victor Stinner date: Mon Dec 02 14:06:03 2013 +0100 summary: asyncio: document Queue classes files: Doc/library/asyncio.rst | 108 ++++++++++++++++++++++++++++ 1 files changed, 108 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -901,6 +901,114 @@ Synchronization primitives -------------------------- +.. class:: Queue(maxsize=0, \*, loop=None) + + A queue, useful for coordinating producer and consumer coroutines. + + If *maxsize* is less than or equal to zero, the queue size is infinite. If + it is an integer greater than ``0``, then ``yield from put()`` will block + when the queue reaches *maxsize*, until an item is removed by :meth:`get`. + + Unlike the standard library :mod:`queue`, you can reliably know this Queue's + size with :meth:`qsize`, since your single-threaded Tulip application won't + be interrupted between calling :meth:`qsize` and doing an operation on the + Queue. + + .. method:: empty() + + Return ``True`` if the queue is empty, ``False`` otherwise. + + .. method:: full() + + Return ``True`` if there are maxsize items in the queue. + + .. note:: + + If the Queue was initialized with ``maxsize=0`` (the default), then + :meth:`full()` is never ``True``. + + .. method:: get() + + Remove and return an item from the queue. + + If you yield from :meth:`get()`, wait until a item is available. + + This method returns a :ref:`coroutine `. + + .. method:: get_nowait() + + Remove and return an item from the queue. + + Return an item if one is immediately available, else raise + :exc:`~queue.Empty`. + + .. method:: put(item) + + Put an item into the queue. + + If you yield from ``put()``, wait until a free slot is available before + adding item. + + This method returns a :ref:`coroutine `. + + .. method:: put_nowait(item) + + Put an item into the queue without blocking. + + If no free slot is immediately available, raise :exc:`~queue.Full`. + + .. method:: qsize() + + Number of items in the queue. + + .. attribute:: maxsize + + Number of items allowed in the queue. + +.. class:: PriorityQueue + + A subclass of :class:`Queue`; retrieves entries in priority order (lowest + first). + + Entries are typically tuples of the form: (priority number, data). + +.. class:: LifoQueue + + A subclass of :class:`Queue` that retrieves most recently added entries + first. + +.. class:: JoinableQueue + + A subclass of :class:`Queue` with :meth:`task_done` and :meth:`join` + methods. + + .. method:: task_done() + + Indicate that a formerly enqueued task is complete. + + Used by queue consumers. For each :meth:`~Queue.get` used to fetch a task, a + subsequent call to :meth:`task_done` tells the queue that the processing + on the task is complete. + + If a :meth:`join` is currently blocking, it will resume when all items + have been processed (meaning that a :meth:`task_done` call was received + for every item that had been :meth:`~Queue.put` into the queue). + + Raises :exc:`ValueError` if called more times than there were items + placed in the queue. + + .. method:: join() + + Block until all items in the queue have been gotten and processed. + + The count of unfinished tasks goes up whenever an item is added to the + queue. The count goes down whenever a consumer thread calls + :meth:`task_done` to indicate that the item was retrieved and all work on + it is complete. When the count of unfinished tasks drops to zero, + :meth:`join` unblocks. + + This method returns a :ref:`coroutine `. + Examples -------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 14:31:25 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 2 Dec 2013 14:31:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_document_locks?= Message-ID: <3dY6fj3N97z7Lk0@mail.python.org> http://hg.python.org/cpython/rev/7303cc4fd837 changeset: 87700:7303cc4fd837 user: Victor Stinner date: Mon Dec 02 14:31:16 2013 +0100 summary: asyncio: document locks files: Doc/library/asyncio.rst | 216 ++++++++++++++++++++++++++++ Lib/asyncio/locks.py | 6 +- 2 files changed, 219 insertions(+), 3 deletions(-) diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -901,6 +901,222 @@ Synchronization primitives -------------------------- +.. class:: Lock(\*, loop=None) + + Primitive lock objects. + + A primitive lock is a synchronization primitive that is not owned by a + particular coroutine when locked. A primitive lock is in one of two states, + 'locked' or 'unlocked'. + + It is created in the unlocked state. It has two basic methods, :meth:`acquire` + and :meth:`release`. When the state is unlocked, acquire() changes the state to + locked and returns immediately. When the state is locked, acquire() blocks + until a call to release() in another coroutine changes it to unlocked, then + the acquire() call resets it to locked and returns. The release() method + should only be called in the locked state; it changes the state to unlocked + and returns immediately. If an attempt is made to release an unlocked lock, + a :exc:`RuntimeError` will be raised. + + When more than one coroutine is blocked in acquire() waiting for the state + to turn to unlocked, only one coroutine proceeds when a release() call + resets the state to unlocked; first coroutine which is blocked in acquire() + is being processed. + + :meth:`acquire` is a coroutine and should be called with ``yield from``. + + Locks also support the context manager protocol. ``(yield from lock)`` + should be used as context manager expression. + + Usage:: + + lock = Lock() + ... + yield from lock + try: + ... + finally: + lock.release() + + Context manager usage:: + + lock = Lock() + ... + with (yield from lock): + ... + + Lock objects can be tested for locking state:: + + if not lock.locked(): + yield from lock + else: + # lock is acquired + ... + + .. method:: locked() + + Return ``True`` if lock is acquired. + + .. method:: acquire() + + Acquire a lock. + + This method blocks until the lock is unlocked, then sets it to locked and + returns ``True``. + + This method returns a :ref:`coroutine `. + + .. method:: release() + + Release a lock. + + When the lock is locked, reset it to unlocked, and return. If any other + coroutines are blocked waiting for the lock to become unlocked, allow + exactly one of them to proceed. + + When invoked on an unlocked lock, a :exc:`RuntimeError` is raised. + + There is no return value. + + +.. class:: Event(\*, loop=None) + + An Event implementation, asynchronous equivalent to :class:`threading.Event`. + + Class implementing event objects. An event manages a flag that can be set to + true with the :meth:`set` method and reset to false with the :meth:`clear` + method. The :meth:`wait` method blocks until the flag is true. The flag is + initially false. + + .. method:: is_set() + + Return ``True`` if and only if the internal flag is true. + + .. method:: set() + + Set the internal flag to true. All coroutines waiting for it to become + true are awakened. Coroutine that call :meth:`wait` once the flag is true + will not block at all. + + .. method:: clear() + + Reset the internal flag to false. Subsequently, coroutines calling + :meth:`wait` will block until :meth:`set` is called to set the internal + flag to true again. + + .. method:: wait() + + Block until the internal flag is true. + + If the internal flag is true on entry, return ``True`` immediately. + Otherwise, block until another coroutine calls :meth:`set` to set the + flag to true, then return ``True``. + + This method returns a :ref:`coroutine `. + + +.. class:: Condition(\*, loop=None) + + A Condition implementation, asynchronous equivalent to + :class:`threading.Condition`. + + This class implements condition variable objects. A condition variable + allows one or more coroutines to wait until they are notified by another + coroutine. + + A new :class:`Lock` object is created and used as the underlying lock. + + .. method:: wait() + + Wait until notified. + + If the calling coroutine has not acquired the lock when this method is + called, a :exc:`RuntimeError` is raised. + + This method releases the underlying lock, and then blocks until it is + awakened by a :meth:`notify` or :meth:`notify_all` call for the same + condition variable in another coroutine. Once awakened, it re-acquires + the lock and returns ``True``. + + This method returns a :ref:`coroutine `. + + .. method:: wait_for(predicate) + + Wait until a predicate becomes true. + + The predicate should be a callable which result will be interpreted as a + boolean value. The final predicate value is the return value. + + This method returns a :ref:`coroutine `. + + .. method:: notify(n=1) + + By default, wake up one coroutine waiting on this condition, if any. + If the calling coroutine has not acquired the lock when this method is + called, a :exc:`RuntimeError` is raised. + + This method wakes up at most *n* of the coroutines waiting for the + condition variable; it is a no-op if no coroutines are waiting. + + .. note:: + + An awakened coroutine does not actually return from its :meth:`wait` + call until it can reacquire the lock. Since :meth:`notify` does not + release the lock, its caller should. + + .. method:: notify_all() + + Wake up all threads waiting on this condition. This method acts like + :meth:`notify`, but wakes up all waiting threads instead of one. If the + calling thread has not acquired the lock when this method is called, a + :exc:`RuntimeError` is raised. + + +.. class:: Semaphore(value=1, \*, loop=None) + + A Semaphore implementation. + + A semaphore manages an internal counter which is decremented by each + :meth:`acquire` call and incremented by each :meth:`release` call. The + counter can never go below zero; when :meth:`acquire` finds that it is zero, + it blocks, waiting until some other thread calls :meth:`release`. + + Semaphores also support the context manager protocol. + + The optional argument gives the initial value for the internal counter; it + defaults to ``1``. If the value given is less than ``0``, :exc:`ValueError` + is raised. + + .. method:: locked() + + Returns ``True`` if semaphore can not be acquired immediately. + + .. method:: acquire() + + Acquire a semaphore. + + If the internal counter is larger than zero on entry, decrement it by one + and return ``True`` immediately. If it is zero on entry, block, waiting + until some other coroutine has called :meth:`release` to make it larger + than ``0``, and then return ``True``. + + This method returns a :ref:`coroutine `. + + .. method:: release() + + Release a semaphore, incrementing the internal counter by one. When it + was zero on entry and another coroutine is waiting for it to become + larger than zero again, wake up that coroutine. + + +.. class:: BoundedSemaphore(value=1, \*, loop=None) + + A bounded semaphore implementation. Inherit from :class:`Semaphore`. + + This raises :exc:`ValueError` in :meth:`~Semaphore.release` if it would + increase the value above the initial value. + + .. class:: Queue(maxsize=0, \*, loop=None) A queue, useful for coordinating producer and consumer coroutines. diff --git a/Lib/asyncio/locks.py b/Lib/asyncio/locks.py --- a/Lib/asyncio/locks.py +++ b/Lib/asyncio/locks.py @@ -79,7 +79,7 @@ return '<{} [{}]>'.format(res[1:-1], extra) def locked(self): - """Return true if lock is acquired.""" + """Return True if lock is acquired.""" return self._locked @tasks.coroutine @@ -138,7 +138,7 @@ class Event: - """An Event implementation, our equivalent to threading.Event. + """An Event implementation, asynchronous equivalent to threading.Event. Class implementing event objects. An event manages a flag that can be set to true with the set() method and reset to false with the clear() method. @@ -162,7 +162,7 @@ return '<{} [{}]>'.format(res[1:-1], extra) def is_set(self): - """Return true if and only if the internal flag is true.""" + """Return True if and only if the internal flag is true.""" return self._value def set(self): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 14:31:53 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 2 Dec 2013 14:31:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_replace_our_wit?= =?utf-8?q?h_asynchronous_in_docstring?= Message-ID: <3dY6gF5Yfsz7Ljd@mail.python.org> http://hg.python.org/cpython/rev/b7c1977dbe16 changeset: 87701:b7c1977dbe16 user: Victor Stinner date: Mon Dec 02 14:31:43 2013 +0100 summary: asyncio: replace our with asynchronous in docstring files: Lib/asyncio/locks.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/locks.py b/Lib/asyncio/locks.py --- a/Lib/asyncio/locks.py +++ b/Lib/asyncio/locks.py @@ -204,7 +204,7 @@ class Condition: - """A Condition implementation, our equivalent to threading.Condition. + """A Condition implementation, asynchronous equivalent to threading.Condition. This class implements condition variable objects. A condition variable allows one or more coroutines to wait until they are notified by another -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 14:50:51 2013 From: python-checkins at python.org (eli.bendersky) Date: Mon, 2 Dec 2013 14:50:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5ODE0?= =?utf-8?q?=3A_Clarify_argparse=27s_docs_w=2Er=2Et_prefix_matching?= Message-ID: <3dY7574QFnz7LjW@mail.python.org> http://hg.python.org/cpython/rev/c457bd1935f9 changeset: 87702:c457bd1935f9 branch: 3.3 parent: 87692:b231e0c3fd26 user: Eli Bendersky date: Mon Dec 02 05:49:54 2013 -0800 summary: Issue #19814: Clarify argparse's docs w.r.t prefix matching files: Doc/library/argparse.rst | 14 +++++++++++--- 1 files changed, 11 insertions(+), 3 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1371,12 +1371,14 @@ >>> parser.parse_args(['--', '-f']) Namespace(foo='-f', one=None) +.. _prefix-matching: -Argument abbreviations -^^^^^^^^^^^^^^^^^^^^^^ +Argument abbreviations (prefix matching) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The :meth:`~ArgumentParser.parse_args` method allows long options to be -abbreviated if the abbreviation is unambiguous:: +abbreviated to a prefix, if the abbreviation is unambiguous (the prefix matches +a unique option):: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('-bacon') @@ -1852,6 +1854,12 @@ >>> parser.parse_known_args(['--foo', '--badger', 'BAR', 'spam']) (Namespace(bar='BAR', foo=True), ['--badger', 'spam']) +.. warning:: + :ref:`Prefix matching ` rules apply to + :meth:`parse_known_args`. The parser may consume an option even if it's just + a prefix of one of its known options, instead of leaving it in the remaining + arguments list. + Customizing file parsing ^^^^^^^^^^^^^^^^^^^^^^^^ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 14:50:52 2013 From: python-checkins at python.org (eli.bendersky) Date: Mon, 2 Dec 2013 14:50:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319814=3A_Clarify_argparse=27s_docs_w=2Er=2Et_pr?= =?utf-8?q?efix_matching?= Message-ID: <3dY7586z5Yz7Ljw@mail.python.org> http://hg.python.org/cpython/rev/46976bd44bfc changeset: 87703:46976bd44bfc parent: 87701:b7c1977dbe16 parent: 87702:c457bd1935f9 user: Eli Bendersky date: Mon Dec 02 05:50:48 2013 -0800 summary: Issue #19814: Clarify argparse's docs w.r.t prefix matching files: Doc/library/argparse.rst | 14 +++++++++++--- 1 files changed, 11 insertions(+), 3 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1371,12 +1371,14 @@ >>> parser.parse_args(['--', '-f']) Namespace(foo='-f', one=None) +.. _prefix-matching: -Argument abbreviations -^^^^^^^^^^^^^^^^^^^^^^ +Argument abbreviations (prefix matching) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The :meth:`~ArgumentParser.parse_args` method allows long options to be -abbreviated if the abbreviation is unambiguous:: +abbreviated to a prefix, if the abbreviation is unambiguous (the prefix matches +a unique option):: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('-bacon') @@ -1854,6 +1856,12 @@ >>> parser.parse_known_args(['--foo', '--badger', 'BAR', 'spam']) (Namespace(bar='BAR', foo=True), ['--badger', 'spam']) +.. warning:: + :ref:`Prefix matching ` rules apply to + :meth:`parse_known_args`. The parser may consume an option even if it's just + a prefix of one of its known options, instead of leaving it in the remaining + arguments list. + Customizing file parsing ^^^^^^^^^^^^^^^^^^^^^^^^ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 14:53:37 2013 From: python-checkins at python.org (eli.bendersky) Date: Mon, 2 Dec 2013 14:53:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5ODE0?= =?utf-8?q?=3A_Clarify_argparse=27s_docs_w=2Er=2Et_prefix_matching?= Message-ID: <3dY78K02TGz7LjW@mail.python.org> http://hg.python.org/cpython/rev/181ced5bf0be changeset: 87704:181ced5bf0be branch: 2.7 parent: 87680:02d186e3af09 user: Eli Bendersky date: Mon Dec 02 05:53:35 2013 -0800 summary: Issue #19814: Clarify argparse's docs w.r.t prefix matching files: Doc/library/argparse.rst | 14 +++++++++++--- 1 files changed, 11 insertions(+), 3 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1351,12 +1351,14 @@ >>> parser.parse_args(['--', '-f']) Namespace(foo='-f', one=None) +.. _prefix-matching: -Argument abbreviations -^^^^^^^^^^^^^^^^^^^^^^ +Argument abbreviations (prefix matching) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The :meth:`~ArgumentParser.parse_args` method allows long options to be -abbreviated if the abbreviation is unambiguous:: +abbreviated to a prefix, if the abbreviation is unambiguous (the prefix matches +a unique option):: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('-bacon') @@ -1822,6 +1824,12 @@ >>> parser.parse_known_args(['--foo', '--badger', 'BAR', 'spam']) (Namespace(bar='BAR', foo=True), ['--badger', 'spam']) +.. warning:: + :ref:`Prefix matching ` rules apply to + :meth:`parse_known_args`. The parser may consume an option even if it's just + a prefix of one of its known options, instead of leaving it in the remaining + arguments list. + Customizing file parsing ^^^^^^^^^^^^^^^^^^^^^^^^ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 17:21:30 2013 From: python-checkins at python.org (walter.doerwald) Date: Mon, 2 Dec 2013 17:21:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Add_NEWS_entry?= =?utf-8?q?_for_issue_=2319834=2E?= Message-ID: <3dYBQy4y25z7LlQ@mail.python.org> http://hg.python.org/cpython/rev/a270370ec487 changeset: 87705:a270370ec487 branch: 3.3 parent: 87702:c457bd1935f9 user: Walter Doerwald date: Mon Dec 02 17:19:00 2013 +0100 summary: Add NEWS entry for issue #19834. files: Misc/NEWS | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,8 @@ Library ------- +- Issue #19834: Support unpickling of exceptions pickled by Python 2. + - Issue #15798: Fixed subprocess.Popen() to no longer fail if file descriptor 0, 1 or 2 is closed. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 17:21:32 2013 From: python-checkins at python.org (walter.doerwald) Date: Mon, 2 Dec 2013 17:21:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Add_NEWS_entry_for_issue_=2319834=3A_merge_with_3=2E3=2E?= Message-ID: <3dYBR00k42z7Lmb@mail.python.org> http://hg.python.org/cpython/rev/ef6ad7172d50 changeset: 87706:ef6ad7172d50 parent: 87703:46976bd44bfc parent: 87705:a270370ec487 user: Walter Doerwald date: Mon Dec 02 17:20:50 2013 +0100 summary: Add NEWS entry for issue #19834: merge with 3.3. files: Misc/NEWS | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,8 @@ Library ------- +- Issue #19834: Support unpickling of exceptions pickled by Python 2. + - Issue #19781: ftplib now supports SSLContext.check_hostname and server name indication for TLS/SSL connections. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 17:28:49 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 2 Dec 2013 17:28:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_document_networ?= =?utf-8?q?k_functions=2C_and_stream_reader/writer?= Message-ID: <3dYBbP4kcJz7LjS@mail.python.org> http://hg.python.org/cpython/rev/d36bbda433af changeset: 87707:d36bbda433af user: Victor Stinner date: Mon Dec 02 17:28:32 2013 +0100 summary: asyncio: document network functions, and stream reader/writer files: Doc/library/asyncio.rst | 153 +++++++++++++++++++++++++++- 1 files changed, 152 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -94,7 +94,7 @@ .. method:: BaseEventLoop.run_until_complete(future) - Run until the :class:`Future` is done. + Run until the :class:`~concurrent.futures.Future` is done. If the argument is a coroutine, it is wrapped in a :class:`Task`. @@ -336,6 +336,58 @@ See the constructor of the :class:`subprocess.Popen` class for parameters. +Network functions +----------------- + +.. function:: open_connection(host=None, port=None, *, loop=None, limit=_DEFAULT_LIMIT, **kwds) + + A wrapper for create_connection() returning a (reader, writer) pair. + + The reader returned is a StreamReader instance; the writer is a + :class:`Transport`. + + The arguments are all the usual arguments to + :meth:`BaseEventLoop.create_connection` except *protocol_factory*; most + common are positional host and port, with various optional keyword arguments + following. + + Additional optional keyword arguments are *loop* (to set the event loop + instance to use) and *limit* (to set the buffer limit passed to the + StreamReader). + + (If you want to customize the :class:`StreamReader` and/or + :class:`StreamReaderProtocol` classes, just copy the code -- there's really + nothing special here except some convenience.) + + This function returns a :ref:`coroutine `. + +.. function:: start_server(client_connected_cb, host=None, port=None, *, loop=None, limit=_DEFAULT_LIMIT, **kwds) + + Start a socket server, call back for each client connected. + + The first parameter, *client_connected_cb*, takes two parameters: + *client_reader*, *client_writer*. *client_reader* is a + :class:`StreamReader` object, while *client_writer* is a + :class:`StreamWriter` object. This parameter can either be a plain callback + function or a :ref:`coroutine `; if it is a coroutine, it will be + automatically converted into a :class:`Task`. + + The rest of the arguments are all the usual arguments to + :meth:`~BaseEventLoop.create_server()` except *protocol_factory*; most + common are positional host and port, with various optional keyword arguments + following. The return value is the same as + :meth:`~BaseEventLoop.create_server()`. + + Additional optional keyword arguments are *loop* (to set the event loop + instance to use) and *limit* (to set the buffer limit passed to the + :class:`StreamReader`). + + The return value is the same as :meth:`~BaseEventLoop.create_server()`, i.e. + a Server object which can be used to stop the service. + + This function returns a :ref:`coroutine `. + + .. _protocol: Protocols @@ -843,6 +895,105 @@ are interested. +Stream reader and writer +------------------------ + +.. class:: StreamWriter(transport, protocol, reader, loop) + + Wraps a Transport. + + This exposes :meth:`write`, :meth:`writelines`, :meth:`can_write_eof()`, :meth:`write_eof`, :meth:`get_extra_info` and + :meth:`close`. It adds :meth:`drain` which returns an optional :class:`~concurrent.futures.Future` on which you can + wait for flow control. It also adds a transport attribute which references + the :class:`Transport` directly. + + .. attribute:: transport + + Transport. + + .. method:: write(data) + + XXX + + .. method:: writelines(data) + + XXX + + .. method:: write_eof() + + XXX + + .. method:: can_write_eof() + + XXX + + .. method:: close() + + XXX + + .. method:: get_extra_info(name, default=None) + + XXX + + .. method:: drain() + + This method has an unusual return value. + + The intended use is to write:: + + w.write(data) + yield from w.drain() + + When there's nothing to wait for, :meth:`drain()` returns ``()``, and the + yield-from continues immediately. When the transport buffer is full (the + protocol is paused), :meth:`drain` creates and returns a + :class:`~concurrent.futures.Future` and the yield-from will block until + that Future is completed, which will happen when the buffer is + (partially) drained and the protocol is resumed. + + +.. class:: StreamReader(limit=_DEFAULT_LIMIT, loop=None) + + .. method:: exception() + + Get the exception. + + .. method:: set_exception(exc) + + Set the exception. + + .. method:: set_transport(transport) + + Set the transport. + + .. method:: feed_eof() + + XXX + + .. method:: feed_data(data) + + XXX + + .. method:: read(n=-1) + + XXX + + This method returns a :ref:`coroutine `. + + .. method:: readexactly(n) + + XXX + + This method returns a :ref:`coroutine `. + + .. method:: readline() + + XXX + + This method returns a :ref:`coroutine `. + + + .. _coroutine: Coroutines -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 17:52:43 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 2 Dec 2013 17:52:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_more_documentat?= =?utf-8?q?ion?= Message-ID: <3dYC6z2w0mz7Ljl@mail.python.org> http://hg.python.org/cpython/rev/c1ecdc4c140f changeset: 87708:c1ecdc4c140f user: Victor Stinner date: Mon Dec 02 17:40:48 2013 +0100 summary: asyncio: more documentation files: Doc/library/asyncio.rst | 102 ++++++++++++++++++++++----- 1 files changed, 83 insertions(+), 19 deletions(-) diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -71,8 +71,8 @@ * Delegating costly function calls to a pool of threads -Getting an event loop -^^^^^^^^^^^^^^^^^^^^^ +Event loop functions +^^^^^^^^^^^^^^^^^^^^ The easiest way to get an event loop is to call the :func:`get_event_loop` function. @@ -84,6 +84,26 @@ event loop has been set for the current context and the current policy does not specify to create one. It should never return ``None``. +.. function:: set_event_loop(loop) + + XXX + +.. function:: new_event_loop() + + XXX + + +Event loop policy +^^^^^^^^^^^^^^^^^ + +.. function:: get_event_loop_policy() + + XXX + +.. function:: set_event_loop_policy(policy) + + XXX + Run an event loop ^^^^^^^^^^^^^^^^^ @@ -114,11 +134,14 @@ .. method:: BaseEventLoop.close() - Close the event loop. + Close the event loop. The loop should not be running. This clears the queues and shuts down the executor, but does not wait for the executor to finish. + This is idempotent and irreversible. No other methods should be called after + this one. + Calls ^^^^^ @@ -199,26 +222,41 @@ .. method:: BaseEventLoop.create_server(protocol_factory, host=None, port=None, \*, family=socket.AF_UNSPEC, flags=socket.AI_PASSIVE, sock=None, backlog=100, ssl=None, reuse_address=None) - XXX + A :ref:`coroutine ` which creates a TCP server bound to host and + port. - * *protocol_factory* - * *host*, *port* - * *family* - * *flags* - * *sock* - * *backlog* : the maximum number of queued connections and should be at - least ``0``; the maximum value is system-dependent (usually ``5``), - the minimum value is forced to ``0``. - * *ssl*: ``True`` or :class:`ssl.SSLContext` - * *reuse_address*: if ``True``, set :data:`socket.SO_REUSEADDR` option - on the listening socket. Default value: ``True`` on POSIX systems, - ``False`` on Windows. + The return value is a Server object which can be used to stop + the service. + + If *host* is an empty string or None all interfaces are assumed + and a list of multiple sockets will be returned (most likely + one for IPv4 and another one for IPv6). + + *family* can be set to either :data:`~socket.AF_INET` or + :data:`~socket.AF_INET6` to force the socket to use IPv4 or IPv6. If not set + it will be determined from host (defaults to :data:`~socket.AF_UNSPEC`). + + *flags* is a bitmask for :meth:`getaddrinfo`. + + *sock* can optionally be specified in order to use a preexisting + socket object. + + *backlog* is the maximum number of queued connections passed to + :meth:`~socket.socket.listen` (defaults to 100). + + ssl can be set to an :class:`~ssl.SSLContext` to enable SSL over the + accepted connections. + + *reuse_address* tells the kernel to reuse a local socket in + TIME_WAIT state, without waiting for its natural timeout to + expire. If not specified will automatically be set to True on + UNIX. This method returns a :ref:`coroutine `. .. method:: BaseEventLoop.create_datagram_endpoint(protocol_factory, local_addr=None, remote_addr=None, \*, family=0, proto=0, flags=0) - XXX + Create datagram connection. This method returns a :ref:`coroutine `. @@ -291,13 +329,23 @@ .. method:: BaseEventLoop.connect_read_pipe(protocol_factory, pipe) - XXX + Register read pipe in eventloop. + + *protocol_factory* should instantiate object with :class:`Protocol` + interface. pipe is file-like object already switched to nonblocking. + Return pair (transport, protocol), where transport support + :class:`ReadTransport` interface. This method returns a :ref:`coroutine `. .. method:: BaseEventLoop.connect_write_pipe(protocol_factory, pipe) - XXX + Register write pipe in eventloop. + + *protocol_factory* should instantiate object with :class:`BaseProtocol` + interface. Pipe is file-like object already switched to nonblocking. + Return pair (transport, protocol), where transport support + :class:`WriteTransport` interface. This method returns a :ref:`coroutine `. @@ -1047,6 +1095,22 @@ Coroutines (and tasks) can only run when the event loop is running. +Server +------ + +.. class:: AbstractServer + + Abstract server returned by create_service(). + + .. method:: close() + + Stop serving. This leaves existing connections open. + + .. method:: wait_closed() + + Coroutine to wait until service is closed. + + .. _sync: Synchronization primitives -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 17:52:44 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 2 Dec 2013 17:52:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_group_trans?= =?utf-8?q?port_method_by_classes?= Message-ID: <3dYC706wwrz7Ljy@mail.python.org> http://hg.python.org/cpython/rev/1d3fbfdc94ce changeset: 87709:1d3fbfdc94ce user: Victor Stinner date: Mon Dec 02 17:46:04 2013 +0100 summary: asyncio doc: group transport method by classes Declare classes because they are mentionned in documentation of other functions files: Doc/library/asyncio.rst | 207 +++++++++++++++------------ 1 files changed, 113 insertions(+), 94 deletions(-) diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -633,100 +633,116 @@ subprocess pipes. The methods available on a transport depend on the transport's kind. -Methods common to all transports -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. method:: BaseTransport.close(self) +Methods common to all transports: BaseTransport +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Close the transport. If the transport has a buffer for outgoing - data, buffered data will be flushed asynchronously. No more data - will be received. After all buffered data is flushed, the - protocol's :meth:`connection_lost` method will be called with - :const:`None` as its argument. +.. class:: BaseTransport + Base class for transports. -.. method:: BaseTransport.get_extra_info(name, default=None) + .. method:: close(self) - Return optional transport information. *name* is a string representing - the piece of transport-specific information to get, *default* is the - value to return if the information doesn't exist. + Close the transport. If the transport has a buffer for outgoing + data, buffered data will be flushed asynchronously. No more data + will be received. After all buffered data is flushed, the + protocol's :meth:`connection_lost` method will be called with + :const:`None` as its argument. - This method allows transport implementations to easily expose - channel-specific information. -Methods of readable streaming transports -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + .. method:: get_extra_info(name, default=None) -.. method:: ReadTransport.pause_reading() + Return optional transport information. *name* is a string representing + the piece of transport-specific information to get, *default* is the + value to return if the information doesn't exist. - Pause the receiving end of the transport. No data will be passed to - the protocol's :meth:`data_received` method until meth:`resume_reading` - is called. + This method allows transport implementations to easily expose + channel-specific information. -.. method:: ReadTransport.resume_reading() - Resume the receiving end. The protocol's :meth:`data_received` method - will be called once again if some data is available for reading. +Methods of readable streaming transports: ReadTransport +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Methods of writable streaming transports -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.. class:: ReadTransport -.. method:: WriteTransport.write(data) + Interface for read-only transports. - Write some *data* bytes to the transport. + .. method:: pause_reading() - This method does not block; it buffers the data and arranges for it - to be sent out asynchronously. + Pause the receiving end of the transport. No data will be passed to + the protocol's :meth:`data_received` method until meth:`resume_reading` + is called. -.. method:: WriteTransport.writelines(list_of_data) + .. method:: resume_reading() - Write a list (or any iterable) of data bytes to the transport. - This is functionally equivalent to calling :meth:`write` on each - element yielded by the iterable, but may be implemented more efficiently. + Resume the receiving end. The protocol's :meth:`data_received` method + will be called once again if some data is available for reading. -.. method:: WriteTransport.write_eof() - Close the write end of the transport after flushing buffered data. - Data may still be received. +Methods of writable streaming transports: WriteTransport +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - This method can raise :exc:`NotImplementedError` if the transport - (e.g. SSL) doesn't support half-closes. +.. class:: WriteTransport -.. method:: WriteTransport.can_write_eof() + Interface for write-only transports. - Return :const:`True` if the transport supports :meth:`write_eof`, - :const:`False` if not. + .. method:: write(data) -.. method:: WriteTransport.abort() + Write some *data* bytes to the transport. - Close the transport immediately, without waiting for pending operations - to complete. Buffered data will be lost. No more data will be received. - The protocol's :meth:`connection_lost` method will eventually be - called with :const:`None` as its argument. + This method does not block; it buffers the data and arranges for it + to be sent out asynchronously. -.. method:: WriteTransport.set_write_buffer_limits(high=None, low=None) + .. method:: writelines(list_of_data) - Set the *high*- and *low*-water limits for write flow control. + Write a list (or any iterable) of data bytes to the transport. + This is functionally equivalent to calling :meth:`write` on each + element yielded by the iterable, but may be implemented more efficiently. - These two values control when call the protocol's - :meth:`pause_writing` and :meth:`resume_writing` methods are called. - If specified, the low-water limit must be less than or equal to the - high-water limit. Neither *high* nor *low* can be negative. + .. method:: write_eof() - The defaults are implementation-specific. If only the - high-water limit is given, the low-water limit defaults to a - implementation-specific value less than or equal to the - high-water limit. Setting *high* to zero forces *low* to zero as - well, and causes :meth:`pause_writing` to be called whenever the - buffer becomes non-empty. Setting *low* to zero causes - :meth:`resume_writing` to be called only once the buffer is empty. - Use of zero for either limit is generally sub-optimal as it - reduces opportunities for doing I/O and computation - concurrently. + Close the write end of the transport after flushing buffered data. + Data may still be received. -.. method:: WriteTransport.get_write_buffer_size() + This method can raise :exc:`NotImplementedError` if the transport + (e.g. SSL) doesn't support half-closes. - Return the current size of the output buffer used by the transport. + .. method:: can_write_eof() + + Return :const:`True` if the transport supports :meth:`write_eof`, + :const:`False` if not. + + .. method:: abort() + + Close the transport immediately, without waiting for pending operations + to complete. Buffered data will be lost. No more data will be received. + The protocol's :meth:`connection_lost` method will eventually be + called with :const:`None` as its argument. + + .. method:: set_write_buffer_limits(high=None, low=None) + + Set the *high*- and *low*-water limits for write flow control. + + These two values control when call the protocol's + :meth:`pause_writing` and :meth:`resume_writing` methods are called. + If specified, the low-water limit must be less than or equal to the + high-water limit. Neither *high* nor *low* can be negative. + + The defaults are implementation-specific. If only the + high-water limit is given, the low-water limit defaults to a + implementation-specific value less than or equal to the + high-water limit. Setting *high* to zero forces *low* to zero as + well, and causes :meth:`pause_writing` to be called whenever the + buffer becomes non-empty. Setting *low* to zero causes + :meth:`resume_writing` to be called only once the buffer is empty. + Use of zero for either limit is generally sub-optimal as it + reduces opportunities for doing I/O and computation + concurrently. + + .. method:: get_write_buffer_size() + + Return the current size of the output buffer used by the transport. + Methods of datagram transports ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -747,47 +763,50 @@ The protocol's :meth:`connection_lost` method will eventually be called with :const:`None` as its argument. + Methods of subprocess transports ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. method:: BaseSubprocessTransport.get_pid() +.. class:: BaseSubprocessTransport - Return the subprocess process id as an integer. + .. method:: get_pid() -.. method:: BaseSubprocessTransport.get_returncode() + Return the subprocess process id as an integer. - Return the subprocess returncode as an integer or :const:`None` - if it hasn't returned, similarly to the - :attr:`subprocess.Popen.returncode` attribute. + .. method:: get_returncode() -.. method:: BaseSubprocessTransport.get_pipe_transport(fd) + Return the subprocess returncode as an integer or :const:`None` + if it hasn't returned, similarly to the + :attr:`subprocess.Popen.returncode` attribute. - Return the transport for the communication pipe correspondong to the - integer file descriptor *fd*. The return value can be a readable or - writable streaming transport, depending on the *fd*. If *fd* doesn't - correspond to a pipe belonging to this transport, :const:`None` is - returned. + .. method:: get_pipe_transport(fd) -.. method:: BaseSubprocessTransport.send_signal(signal) + Return the transport for the communication pipe correspondong to the + integer file descriptor *fd*. The return value can be a readable or + writable streaming transport, depending on the *fd*. If *fd* doesn't + correspond to a pipe belonging to this transport, :const:`None` is + returned. - Send the *signal* number to the subprocess, as in - :meth:`subprocess.Popen.send_signal`. + .. method:: send_signal(signal) -.. method:: BaseSubprocessTransport.terminate() + Send the *signal* number to the subprocess, as in + :meth:`subprocess.Popen.send_signal`. - Ask the subprocess to stop, as in :meth:`subprocess.Popen.terminate`. - This method is an alias for the :meth:`close` method. + .. method:: terminate() - On POSIX systems, this method sends SIGTERM to the subprocess. - On Windows, the Windows API function TerminateProcess() is called to - stop the subprocess. + Ask the subprocess to stop, as in :meth:`subprocess.Popen.terminate`. + This method is an alias for the :meth:`close` method. -.. method:: BaseSubprocessTransport.kill(self) + On POSIX systems, this method sends SIGTERM to the subprocess. + On Windows, the Windows API function TerminateProcess() is called to + stop the subprocess. - Kill the subprocess, as in :meth:`subprocess.Popen.kill` + .. method:: kill(self) - On POSIX systems, the function sends SIGKILL to the subprocess. - On Windows, this method is an alias for :meth:`terminate`. + Kill the subprocess, as in :meth:`subprocess.Popen.kill` + + On POSIX systems, the function sends SIGKILL to the subprocess. + On Windows, this method is an alias for :meth:`terminate`. Task functions @@ -843,12 +862,12 @@ Return ``True`` if *obj* is a coroutine object. -.. function:: sleep(delay, result=None, *, loop=None) +.. function:: sleep(delay, result=None, \*, loop=None) Create a :ref:`coroutine ` that completes after a given time (in seconds). -.. function:: shield(arg, *, loop=None) +.. function:: shield(arg, \*, loop=None) Wait for a future, shielding it from cancellation. @@ -879,7 +898,7 @@ Task ---- -.. class:: Task(coro, *, loop=None) +.. class:: Task(coro, \*, loop=None) A coroutine wrapped in a :class:`~concurrent.futures.Future`. @@ -893,7 +912,7 @@ Cancel the task. - .. method:: get_stack(self, *, limit=None) + .. method:: get_stack(self, \*, limit=None) Return the list of stack frames for this task's coroutine. @@ -913,7 +932,7 @@ For reasons beyond our control, only one stack frame is returned for a suspended coroutine. - .. method:: print_stack(*, limit=None, file=None) + .. method:: print_stack(\*, limit=None, file=None) Print the stack or traceback for this task's coroutine. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 17:52:46 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 2 Dec 2013 17:52:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_sort_some_metho?= =?utf-8?q?ds?= Message-ID: <3dYC723CB0z7Lk5@mail.python.org> http://hg.python.org/cpython/rev/74bd800edb90 changeset: 87710:74bd800edb90 user: Victor Stinner date: Mon Dec 02 17:52:31 2013 +0100 summary: asyncio: sort some methods files: Doc/library/asyncio.rst | 162 ++++++++++++++------------- 1 files changed, 85 insertions(+), 77 deletions(-) diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -367,7 +367,7 @@ Run subprocesses asynchronously using the :mod:`subprocess` module. -.. method:: BaseEventLoop.subprocess_shell(protocol_factory, cmd, \*, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=True, bufsize=0, \*\*kwargs) +.. method:: BaseEventLoop.subprocess_exec(protocol_factory, \*args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=False, bufsize=0, \*\*kwargs) XXX @@ -375,7 +375,7 @@ See the constructor of the :class:`subprocess.Popen` class for parameters. -.. method:: BaseEventLoop.subprocess_exec(protocol_factory, \*args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=False, bufsize=0, \*\*kwargs) +.. method:: BaseEventLoop.subprocess_shell(protocol_factory, cmd, \*, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=True, bufsize=0, \*\*kwargs) XXX @@ -978,29 +978,9 @@ Transport. - .. method:: write(data) - - XXX - - .. method:: writelines(data) - - XXX - - .. method:: write_eof() - - XXX - - .. method:: can_write_eof() - - XXX - .. method:: close() - XXX - - .. method:: get_extra_info(name, default=None) - - XXX + Close the transport: see :meth:`BaseTransport.close`. .. method:: drain() @@ -1018,6 +998,31 @@ that Future is completed, which will happen when the buffer is (partially) drained and the protocol is resumed. + .. method:: get_extra_info(name, default=None) + + Return optional transport information: see + :meth:`BaseTransport.get_extra_info`. + + .. method:: write(data) + + Write some *data* bytes to the transport: see + :meth:`WriteTransport.write`. + + .. method:: writelines(data) + + Write a list (or any iterable) of data bytes to the transport: + see :meth:`WriteTransport.writelines`. + + .. method:: can_write_eof() + + Return :const:`True` if the transport supports :meth:`write_eof`, + :const:`False` if not. See :meth:`WriteTransport.can_write_eof`. + + .. method:: write_eof() + + Close the write end of the transport after flushing buffered data: + see :meth:`WriteTransport.write_eof`. + .. class:: StreamReader(limit=_DEFAULT_LIMIT, loop=None) @@ -1025,6 +1030,14 @@ Get the exception. + .. method:: feed_eof() + + XXX + + .. method:: feed_data(data) + + XXX + .. method:: set_exception(exc) Set the exception. @@ -1033,27 +1046,19 @@ Set the transport. - .. method:: feed_eof() - - XXX - - .. method:: feed_data(data) - - XXX - .. method:: read(n=-1) XXX This method returns a :ref:`coroutine `. - .. method:: readexactly(n) + .. method:: readline() XXX This method returns a :ref:`coroutine `. - .. method:: readline() + .. method:: readexactly(n) XXX @@ -1222,6 +1227,12 @@ method. The :meth:`wait` method blocks until the flag is true. The flag is initially false. + .. method:: clear() + + Reset the internal flag to false. Subsequently, coroutines calling + :meth:`wait` will block until :meth:`set` is called to set the internal + flag to true again. + .. method:: is_set() Return ``True`` if and only if the internal flag is true. @@ -1232,12 +1243,6 @@ true are awakened. Coroutine that call :meth:`wait` once the flag is true will not block at all. - .. method:: clear() - - Reset the internal flag to false. Subsequently, coroutines calling - :meth:`wait` will block until :meth:`set` is called to set the internal - flag to true again. - .. method:: wait() Block until the internal flag is true. @@ -1260,6 +1265,28 @@ A new :class:`Lock` object is created and used as the underlying lock. + .. method:: notify(n=1) + + By default, wake up one coroutine waiting on this condition, if any. + If the calling coroutine has not acquired the lock when this method is + called, a :exc:`RuntimeError` is raised. + + This method wakes up at most *n* of the coroutines waiting for the + condition variable; it is a no-op if no coroutines are waiting. + + .. note:: + + An awakened coroutine does not actually return from its :meth:`wait` + call until it can reacquire the lock. Since :meth:`notify` does not + release the lock, its caller should. + + .. method:: notify_all() + + Wake up all threads waiting on this condition. This method acts like + :meth:`notify`, but wakes up all waiting threads instead of one. If the + calling thread has not acquired the lock when this method is called, a + :exc:`RuntimeError` is raised. + .. method:: wait() Wait until notified. @@ -1283,28 +1310,6 @@ This method returns a :ref:`coroutine `. - .. method:: notify(n=1) - - By default, wake up one coroutine waiting on this condition, if any. - If the calling coroutine has not acquired the lock when this method is - called, a :exc:`RuntimeError` is raised. - - This method wakes up at most *n* of the coroutines waiting for the - condition variable; it is a no-op if no coroutines are waiting. - - .. note:: - - An awakened coroutine does not actually return from its :meth:`wait` - call until it can reacquire the lock. Since :meth:`notify` does not - release the lock, its caller should. - - .. method:: notify_all() - - Wake up all threads waiting on this condition. This method acts like - :meth:`notify`, but wakes up all waiting threads instead of one. If the - calling thread has not acquired the lock when this method is called, a - :exc:`RuntimeError` is raised. - .. class:: Semaphore(value=1, \*, loop=None) @@ -1321,10 +1326,6 @@ defaults to ``1``. If the value given is less than ``0``, :exc:`ValueError` is raised. - .. method:: locked() - - Returns ``True`` if semaphore can not be acquired immediately. - .. method:: acquire() Acquire a semaphore. @@ -1336,6 +1337,10 @@ This method returns a :ref:`coroutine `. + .. method:: locked() + + Returns ``True`` if semaphore can not be acquired immediately. + .. method:: release() Release a semaphore, incrementing the internal counter by one. When it @@ -1415,6 +1420,7 @@ Number of items allowed in the queue. + .. class:: PriorityQueue A subclass of :class:`Queue`; retrieves entries in priority order (lowest @@ -1422,16 +1428,30 @@ Entries are typically tuples of the form: (priority number, data). + .. class:: LifoQueue A subclass of :class:`Queue` that retrieves most recently added entries first. + .. class:: JoinableQueue A subclass of :class:`Queue` with :meth:`task_done` and :meth:`join` methods. + .. method:: join() + + Block until all items in the queue have been gotten and processed. + + The count of unfinished tasks goes up whenever an item is added to the + queue. The count goes down whenever a consumer thread calls + :meth:`task_done` to indicate that the item was retrieved and all work on + it is complete. When the count of unfinished tasks drops to zero, + :meth:`join` unblocks. + + This method returns a :ref:`coroutine `. + .. method:: task_done() Indicate that a formerly enqueued task is complete. @@ -1447,18 +1467,6 @@ Raises :exc:`ValueError` if called more times than there were items placed in the queue. - .. method:: join() - - Block until all items in the queue have been gotten and processed. - - The count of unfinished tasks goes up whenever an item is added to the - queue. The count goes down whenever a consumer thread calls - :meth:`task_done` to indicate that the item was retrieved and all work on - it is complete. When the count of unfinished tasks drops to zero, - :meth:`join` unblocks. - - This method returns a :ref:`coroutine `. - Examples -------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 19:23:19 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 2 Dec 2013 19:23:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fixed_regex_match_represen?= =?utf-8?q?tation_in_an_example=2E?= Message-ID: <3dYF7W5Kv4zRkL@mail.python.org> http://hg.python.org/cpython/rev/0348647113e8 changeset: 87711:0348647113e8 user: Serhiy Storchaka date: Mon Dec 02 20:23:19 2013 +0200 summary: Fixed regex match representation in an example. files: Doc/library/re.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/re.rst b/Doc/library/re.rst --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -801,7 +801,7 @@ >>> pattern.fullmatch("dog") # No match as "o" is not at the start of "dog". >>> pattern.fullmatch("ogre") # No match as not the full string matches. >>> pattern.fullmatch("doggie", 1, 3) # Matches within given limits. - <_sre.SRE_Match object at ...> + <_sre.SRE_Match object; span=(1, 3), match='og'> .. versionadded:: 3.4 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 19:31:06 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 2 Dec 2013 19:31:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fixed_formatting_=28was_pa?= =?utf-8?q?rsed_as_description_list=29=2E?= Message-ID: <3dYFJV44Xdz7LjT@mail.python.org> http://hg.python.org/cpython/rev/93e75bbd3ab8 changeset: 87712:93e75bbd3ab8 user: Serhiy Storchaka date: Mon Dec 02 20:31:00 2013 +0200 summary: Fixed formatting (was parsed as description list). files: Doc/whatsnew/3.4.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -742,7 +742,7 @@ cert store. (Contributed by Christian Heimes in :issue:`18143`, :issue:`18147` and - :issue:`17134`.) +:issue:`17134`.) Support for server-side SNI using the new :meth:`ssl.SSLContext.set_servername_callback` method. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 20:02:17 2013 From: python-checkins at python.org (christian.heimes) Date: Mon, 2 Dec 2013 20:02:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319782=3A_imaplib_?= =?utf-8?q?now_supports_SSLContext=2Echeck=5Fhostname_and_server_name?= Message-ID: <3dYG0T4pGkz7Lk9@mail.python.org> http://hg.python.org/cpython/rev/23001d915b39 changeset: 87713:23001d915b39 parent: 87710:74bd800edb90 user: Christian Heimes date: Mon Dec 02 20:01:29 2013 +0100 summary: Issue #19782: imaplib now supports SSLContext.check_hostname and server name indication for TLS/SSL connections. files: Doc/library/imaplib.rst | 8 ++++++++ Lib/imaplib.py | 8 ++++++-- Lib/test/test_imaplib.py | 28 ++++++++++++++++++++++++++-- Misc/NEWS | 3 +++ 4 files changed, 43 insertions(+), 4 deletions(-) diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst --- a/Doc/library/imaplib.rst +++ b/Doc/library/imaplib.rst @@ -80,6 +80,10 @@ .. versionchanged:: 3.3 *ssl_context* parameter added. + .. versionchanged:: 3.4 + The class now supports hostname check with + :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see + :data:`~ssl.HAS_SNI`). The second subclass allows for connections created by a child process: @@ -437,6 +441,10 @@ .. versionadded:: 3.2 + .. versionchanged:: 3.4 + The method now supports hostname check with + :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see + :data:`~ssl.HAS_SNI`). .. method:: IMAP4.status(mailbox, names) diff --git a/Lib/imaplib.py b/Lib/imaplib.py --- a/Lib/imaplib.py +++ b/Lib/imaplib.py @@ -745,7 +745,9 @@ ssl_context = ssl._create_stdlib_context() typ, dat = self._simple_command(name) if typ == 'OK': - self.sock = ssl_context.wrap_socket(self.sock) + server_hostname = self.host if ssl.HAS_SNI else None + self.sock = ssl_context.wrap_socket(self.sock, + server_hostname=server_hostname) self.file = self.sock.makefile('rb') self._tls_established = True self._get_capabilities() @@ -1216,7 +1218,9 @@ def _create_socket(self): sock = IMAP4._create_socket(self) - return self.ssl_context.wrap_socket(sock) + server_hostname = self.host if ssl.HAS_SNI else None + return self.ssl_context.wrap_socket(sock, + server_hostname=server_hostname) def open(self, host='', port=IMAP4_SSL_PORT): """Setup connection to remote server on "host:port". diff --git a/Lib/test/test_imaplib.py b/Lib/test/test_imaplib.py --- a/Lib/test/test_imaplib.py +++ b/Lib/test/test_imaplib.py @@ -20,6 +20,7 @@ ssl = None CERTFILE = None +CAFILE = None class TestImaplib(unittest.TestCase): @@ -348,6 +349,25 @@ server_class = SecureTCPServer imap_class = IMAP4_SSL + @reap_threads + def test_ssl_verified(self): + ssl_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + ssl_context.verify_mode = ssl.CERT_REQUIRED + ssl_context.check_hostname = True + ssl_context.load_verify_locations(CAFILE) + + with self.assertRaisesRegex(ssl.CertificateError, + "hostname '127.0.0.1' doesn't match 'localhost'"): + with self.reaped_server(SimpleIMAPHandler) as server: + client = self.imap_class(*server.server_address, + ssl_context=ssl_context) + client.shutdown() + + with self.reaped_server(SimpleIMAPHandler) as server: + client = self.imap_class("localhost", server.server_address[1], + ssl_context=ssl_context) + client.shutdown() + class RemoteIMAPTest(unittest.TestCase): host = 'cyrus.andrew.cmu.edu' @@ -460,11 +480,15 @@ if support.is_resource_enabled('network'): if ssl: - global CERTFILE + global CERTFILE, CAFILE CERTFILE = os.path.join(os.path.dirname(__file__) or os.curdir, - "keycert.pem") + "keycert3.pem") if not os.path.exists(CERTFILE): raise support.TestFailed("Can't read certificate files!") + CAFILE = os.path.join(os.path.dirname(__file__) or os.curdir, + "pycacert.pem") + if not os.path.exists(CAFILE): + raise support.TestFailed("Can't read CA file!") tests.extend([ ThreadedNetworkedTests, ThreadedNetworkedTestsSSL, RemoteIMAPTest, RemoteIMAP_SSLTest, RemoteIMAP_STARTTLSTest, diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,9 @@ Library ------- +- Issue #19782: imaplib now supports SSLContext.check_hostname and server name + indication for TLS/SSL connections. + - Issue #19834: Support unpickling of exceptions pickled by Python 2. - Issue #19781: ftplib now supports SSLContext.check_hostname and server name -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 20:02:18 2013 From: python-checkins at python.org (christian.heimes) Date: Mon, 2 Dec 2013 20:02:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?q?=29=3A_merge?= Message-ID: <3dYG0V6rQ9z7Ll3@mail.python.org> http://hg.python.org/cpython/rev/88272c8f233e changeset: 87714:88272c8f233e parent: 87713:23001d915b39 parent: 87712:93e75bbd3ab8 user: Christian Heimes date: Mon Dec 02 20:02:04 2013 +0100 summary: merge files: Doc/library/re.rst | 2 +- Doc/whatsnew/3.4.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/re.rst b/Doc/library/re.rst --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -801,7 +801,7 @@ >>> pattern.fullmatch("dog") # No match as "o" is not at the start of "dog". >>> pattern.fullmatch("ogre") # No match as not the full string matches. >>> pattern.fullmatch("doggie", 1, 3) # Matches within given limits. - <_sre.SRE_Match object at ...> + <_sre.SRE_Match object; span=(1, 3), match='og'> .. versionadded:: 3.4 diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -742,7 +742,7 @@ cert store. (Contributed by Christian Heimes in :issue:`18143`, :issue:`18147` and - :issue:`17134`.) +:issue:`17134`.) Support for server-side SNI using the new :meth:`ssl.SSLContext.set_servername_callback` method. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 20:11:01 2013 From: python-checkins at python.org (christian.heimes) Date: Mon, 2 Dec 2013 20:11:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319784=3A_poplib_n?= =?utf-8?q?ow_supports_SSLContext=2Echeck=5Fhostname_and_server_name?= Message-ID: <3dYGBY0GZdzSDW@mail.python.org> http://hg.python.org/cpython/rev/1ce752754280 changeset: 87715:1ce752754280 user: Christian Heimes date: Mon Dec 02 20:10:50 2013 +0100 summary: Issue #19784: poplib now supports SSLContext.check_hostname and server name indication for TLS/SSL connections. files: Doc/library/poplib.rst | 9 +++++++++ Lib/poplib.py | 8 ++++++-- Lib/test/test_poplib.py | 9 ++++++++- Misc/NEWS | 3 +++ 4 files changed, 26 insertions(+), 3 deletions(-) diff --git a/Doc/library/poplib.rst b/Doc/library/poplib.rst --- a/Doc/library/poplib.rst +++ b/Doc/library/poplib.rst @@ -53,6 +53,10 @@ .. versionchanged:: 3.2 *context* parameter added. + .. versionchanged:: 3.4 + The class now supports hostname check with + :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see + :data:`~ssl.HAS_SNI`). One exception is defined as an attribute of the :mod:`poplib` module: @@ -198,6 +202,11 @@ .. versionadded:: 3.4 + .. versionchanged:: 3.4 + The method now supports hostname check with + :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see + :data:`~ssl.HAS_SNI`). + Instances of :class:`POP3_SSL` have no additional methods. The interface of this subclass is identical to its parent. diff --git a/Lib/poplib.py b/Lib/poplib.py --- a/Lib/poplib.py +++ b/Lib/poplib.py @@ -387,7 +387,9 @@ if context is None: context = ssl._create_stdlib_context() resp = self._shortcmd('STLS') - self.sock = context.wrap_socket(self.sock) + server_hostname = self.host if ssl.HAS_SNI else None + self.sock = context.wrap_socket(self.sock, + server_hostname=server_hostname) self.file = self.sock.makefile('rb') self._tls_established = True return resp @@ -428,7 +430,9 @@ def _create_socket(self, timeout): sock = POP3._create_socket(self, timeout) - sock = self.context.wrap_socket(sock) + server_hostname = self.host if ssl.HAS_SNI else None + sock = self.context.wrap_socket(sock, + server_hostname=server_hostname) return sock def stls(self, keyfile=None, certfile=None, context=None): diff --git a/Lib/test/test_poplib.py b/Lib/test/test_poplib.py --- a/Lib/test/test_poplib.py +++ b/Lib/test/test_poplib.py @@ -23,7 +23,8 @@ import ssl SUPPORTS_SSL = True - CERTFILE = os.path.join(os.path.dirname(__file__) or os.curdir, "keycert.pem") + CERTFILE = os.path.join(os.path.dirname(__file__) or os.curdir, "keycert3.pem") + CAFILE = os.path.join(os.path.dirname(__file__) or os.curdir, "pycacert.pem") requires_ssl = skipUnless(SUPPORTS_SSL, 'SSL not supported') # the dummy data returned by server when LIST and RETR commands are issued @@ -332,6 +333,12 @@ def test_stls_context(self): expected = b'+OK Begin TLS negotiation' ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + ctx.load_verify_locations(CAFILE) + ctx.verify_mode = ssl.CERT_REQUIRED + ctx.check_hostname = True + with self.assertRaises(ssl.CertificateError): + resp = self.client.stls(context=ctx) + self.client = poplib.POP3("localhost", self.server.port, timeout=3) resp = self.client.stls(context=ctx) self.assertEqual(resp, expected) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,9 @@ Library ------- +- Issue #19784: poplib now supports SSLContext.check_hostname and server name + indication for TLS/SSL connections. + - Issue #19782: imaplib now supports SSLContext.check_hostname and server name indication for TLS/SSL connections. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 20:20:21 2013 From: python-checkins at python.org (christian.heimes) Date: Mon, 2 Dec 2013 20:20:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319783=3A_nntplib_?= =?utf-8?q?now_supports_SSLContext=2Echeck=5Fhostname_and_server_name?= Message-ID: <3dYGPK0lKSz7LjX@mail.python.org> http://hg.python.org/cpython/rev/42a6919ee7e5 changeset: 87716:42a6919ee7e5 user: Christian Heimes date: Mon Dec 02 20:20:11 2013 +0100 summary: Issue #19783: nntplib now supports SSLContext.check_hostname and server name indication for TLS/SSL connections. files: Doc/library/nntplib.rst | 8 ++++++++ Lib/nntplib.py | 9 +++++---- Misc/NEWS | 3 +++ 3 files changed, 16 insertions(+), 4 deletions(-) diff --git a/Doc/library/nntplib.rst b/Doc/library/nntplib.rst --- a/Doc/library/nntplib.rst +++ b/Doc/library/nntplib.rst @@ -102,6 +102,10 @@ .. versionadded:: 3.2 + .. versionchanged:: 3.4 + The class now supports hostname check with + :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see + :data:`~ssl.HAS_SNI`). .. exception:: NNTPError @@ -241,6 +245,10 @@ .. versionadded:: 3.2 + .. versionchanged:: 3.4 + The method now supports hostname check with + :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see + :data:`~ssl.HAS_SNI`). .. method:: NNTP.newgroups(date, *, file=None) diff --git a/Lib/nntplib.py b/Lib/nntplib.py --- a/Lib/nntplib.py +++ b/Lib/nntplib.py @@ -279,7 +279,7 @@ if _have_ssl: - def _encrypt_on(sock, context): + def _encrypt_on(sock, context, hostname): """Wrap a socket in SSL/TLS. Arguments: - sock: Socket to wrap - context: SSL context to use for the encrypted connection @@ -289,7 +289,8 @@ # Generate a default SSL context if none was passed. if context is None: context = ssl._create_stdlib_context() - return context.wrap_socket(sock) + server_hostname = hostname if ssl.HAS_SNI else None + return context.wrap_socket(sock, server_hostname=server_hostname) # The classes themselves @@ -1005,7 +1006,7 @@ resp = self._shortcmd('STARTTLS') if resp.startswith('382'): self.file.close() - self.sock = _encrypt_on(self.sock, context) + self.sock = _encrypt_on(self.sock, context, self.host) self.file = self.sock.makefile("rwb") self.tls_on = True # Capabilities may change after TLS starts up, so ask for them @@ -1065,7 +1066,7 @@ in default port and the `ssl_context` argument for SSL connections. """ self.sock = socket.create_connection((host, port), timeout) - self.sock = _encrypt_on(self.sock, ssl_context) + self.sock = _encrypt_on(self.sock, ssl_context, host) file = self.sock.makefile("rwb") _NNTPBase.__init__(self, file, host, readermode=readermode, timeout=timeout) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -21,6 +21,9 @@ - Issue #19784: poplib now supports SSLContext.check_hostname and server name indication for TLS/SSL connections. +- Issue #19783: nntplib now supports SSLContext.check_hostname and server name + indication for TLS/SSL connections. + - Issue #19782: imaplib now supports SSLContext.check_hostname and server name indication for TLS/SSL connections. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 20:44:27 2013 From: python-checkins at python.org (christian.heimes) Date: Mon, 2 Dec 2013 20:44:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319785=3A_smtplib_?= =?utf-8?q?now_supports_SSLContext=2Echeck=5Fhostname_and_server_name?= Message-ID: <3dYGx72PVQz7LjQ@mail.python.org> http://hg.python.org/cpython/rev/c0079358e791 changeset: 87717:c0079358e791 user: Christian Heimes date: Mon Dec 02 20:44:17 2013 +0100 summary: Issue #19785: smtplib now supports SSLContext.check_hostname and server name indication for TLS/SSL connections. files: Doc/library/smtplib.rst | 9 ++++++++ Lib/smtplib.py | 9 ++++++- Lib/test/test_smtpnet.py | 30 ++++++++++++++++++++++++--- Misc/NEWS | 3 ++ 4 files changed, 45 insertions(+), 6 deletions(-) diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -90,6 +90,10 @@ .. versionchanged:: 3.3 source_address argument was added. + .. versionchanged:: 3.4 + The class now supports hostname check with + :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see + :data:`~ssl.HAS_SNI`). .. class:: LMTP(host='', port=LMTP_PORT, local_hostname=None, source_address=None) @@ -316,6 +320,11 @@ .. versionchanged:: 3.3 *context* was added. + .. versionchanged:: 3.4 + The method now supports hostname check with + :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see + :data:`~ssl.HAS_SNI`). + .. method:: SMTP.sendmail(from_addr, to_addrs, msg, mail_options=[], rcpt_options=[]) diff --git a/Lib/smtplib.py b/Lib/smtplib.py --- a/Lib/smtplib.py +++ b/Lib/smtplib.py @@ -232,6 +232,7 @@ will be used. """ + self._host = host self.timeout = timeout self.esmtp_features = {} self.source_address = source_address @@ -667,7 +668,9 @@ if context is None: context = ssl._create_stdlib_context(certfile=certfile, keyfile=keyfile) - self.sock = context.wrap_socket(self.sock) + server_hostname = self._host if ssl.HAS_SNI else None + self.sock = context.wrap_socket(self.sock, + server_hostname=server_hostname) self.file = None # RFC 3207: # The client MUST discard any knowledge obtained from @@ -892,7 +895,9 @@ print('connect:', (host, port), file=stderr) new_socket = socket.create_connection((host, port), timeout, self.source_address) - new_socket = self.context.wrap_socket(new_socket) + server_hostname = self._host if ssl.HAS_SNI else None + new_socket = self.context.wrap_socket(new_socket, + server_hostname=server_hostname) return new_socket __all__.append("SMTP_SSL") diff --git a/Lib/test/test_smtpnet.py b/Lib/test/test_smtpnet.py --- a/Lib/test/test_smtpnet.py +++ b/Lib/test/test_smtpnet.py @@ -3,23 +3,35 @@ import unittest from test import support import smtplib +import socket ssl = support.import_module("ssl") support.requires("network") +def check_ssl_verifiy(host, port): + context = ssl.create_default_context() + with socket.create_connection((host, port)) as sock: + try: + sock = context.wrap_socket(sock, server_hostname=host) + except Exception: + return False + else: + sock.close() + return True + class SmtpTest(unittest.TestCase): testServer = 'smtp.gmail.com' remotePort = 25 - context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) def test_connect_starttls(self): support.get_attribute(smtplib, 'SMTP_SSL') + context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) with support.transient_internet(self.testServer): server = smtplib.SMTP(self.testServer, self.remotePort) try: - server.starttls(context=self.context) + server.starttls(context=context) except smtplib.SMTPException as e: if e.args[0] == 'STARTTLS extension not supported by server.': unittest.skip(e.args[0]) @@ -32,7 +44,7 @@ class SmtpSSLTest(unittest.TestCase): testServer = 'smtp.gmail.com' remotePort = 465 - context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + can_verify = check_ssl_verifiy(testServer, remotePort) def test_connect(self): support.get_attribute(smtplib, 'SMTP_SSL') @@ -49,9 +61,19 @@ server.quit() def test_connect_using_sslcontext(self): + context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) support.get_attribute(smtplib, 'SMTP_SSL') with support.transient_internet(self.testServer): - server = smtplib.SMTP_SSL(self.testServer, self.remotePort, context=self.context) + server = smtplib.SMTP_SSL(self.testServer, self.remotePort, context=context) + server.ehlo() + server.quit() + + @unittest.skipUnless(can_verify, "SSL certificate can't be verified") + def test_connect_using_sslcontext_verified(self): + support.get_attribute(smtplib, 'SMTP_SSL') + context = ssl.create_default_context() + with support.transient_internet(self.testServer): + server = smtplib.SMTP_SSL(self.testServer, self.remotePort, context=context) server.ehlo() server.quit() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,9 @@ Library ------- +- Issue #19785: smtplib now supports SSLContext.check_hostname and server name + indication for TLS/SSL connections. + - Issue #19784: poplib now supports SSLContext.check_hostname and server name indication for TLS/SSL connections. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 20:59:38 2013 From: python-checkins at python.org (christian.heimes) Date: Mon, 2 Dec 2013 20:59:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_add_check=5Fhostname_arg_t?= =?utf-8?b?byBzc2wuX2NyZWF0ZV9zdGRsaWJfY29udGV4dCgp?= Message-ID: <3dYHGf20LHz7LjR@mail.python.org> http://hg.python.org/cpython/rev/ba1cc1a6de01 changeset: 87718:ba1cc1a6de01 user: Christian Heimes date: Mon Dec 02 20:59:28 2013 +0100 summary: add check_hostname arg to ssl._create_stdlib_context() files: Lib/ssl.py | 3 ++- Lib/test/test_ssl.py | 4 +++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -405,7 +405,7 @@ def _create_stdlib_context(protocol=PROTOCOL_SSLv23, *, cert_reqs=None, - purpose=Purpose.SERVER_AUTH, + check_hostname=False, purpose=Purpose.SERVER_AUTH, certfile=None, keyfile=None, cafile=None, capath=None, cadata=None): """Create a SSLContext object for Python stdlib modules @@ -424,6 +424,7 @@ if cert_reqs is not None: context.verify_mode = cert_reqs + context.check_hostname = check_hostname if keyfile and not certfile: raise ValueError("certfile must be specified") diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1032,9 +1032,11 @@ self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) ctx = ssl._create_stdlib_context(ssl.PROTOCOL_TLSv1, - cert_reqs=ssl.CERT_REQUIRED) + cert_reqs=ssl.CERT_REQUIRED, + check_hostname=True) self.assertEqual(ctx.protocol, ssl.PROTOCOL_TLSv1) self.assertEqual(ctx.verify_mode, ssl.CERT_REQUIRED) + self.assertTrue(ctx.check_hostname) self.assertEqual(ctx.options & ssl.OP_NO_SSLv2, ssl.OP_NO_SSLv2) ctx = ssl._create_stdlib_context(purpose=ssl.Purpose.CLIENT_AUTH) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 2 21:25:25 2013 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 2 Dec 2013 21:25:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319852=3A_move_Pat?= =?utf-8?q?h=2E=5Fraw=5Fopen=28=29_around=2C_as_it_is_now_a_private_method?= =?utf-8?q?=2E?= Message-ID: <3dYHrP5YBlz7Ljr@mail.python.org> http://hg.python.org/cpython/rev/cd8230437f40 changeset: 87719:cd8230437f40 user: Antoine Pitrou date: Mon Dec 02 21:25:18 2013 +0100 summary: Issue #19852: move Path._raw_open() around, as it is now a private method. Patch by Vajrasky Kok. files: Lib/pathlib.py | 18 +++++++++--------- 1 files changed, 9 insertions(+), 9 deletions(-) diff --git a/Lib/pathlib.py b/Lib/pathlib.py --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -939,6 +939,15 @@ # A stub for the opener argument to built-in open() return self._accessor.open(self, flags, mode) + def _raw_open(self, flags, mode=0o777): + """ + Open the file pointed by this path and return a file descriptor, + as os.open() does. + """ + if self._closed: + self._raise_closed() + return self._accessor.open(self, flags, mode) + # Public API @classmethod @@ -1045,15 +1054,6 @@ import grp return grp.getgrgid(self.stat().st_gid).gr_name - def _raw_open(self, flags, mode=0o777): - """ - Open the file pointed by this path and return a file descriptor, - as os.open() does. - """ - if self._closed: - self._raise_closed() - return self._accessor.open(self, flags, mode) - def open(self, mode='r', buffering=-1, encoding=None, errors=None, newline=None): """ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 00:33:19 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 3 Dec 2013 00:33:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc?= Message-ID: <3dYN1C1jxDzS9j@mail.python.org> http://hg.python.org/cpython/rev/94f251c04541 changeset: 87720:94f251c04541 user: Victor Stinner date: Tue Dec 03 00:32:48 2013 +0100 summary: asyncio doc * complete create_connection() prototype * link to Server * document get_extra_info() keys files: Doc/library/asyncio.rst | 42 ++++++++++++++++++++++++---- 1 files changed, 35 insertions(+), 7 deletions(-) diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -225,7 +225,7 @@ A :ref:`coroutine ` which creates a TCP server bound to host and port. - The return value is a Server object which can be used to stop + The return value is a :class:`AbstractServer` object which can be used to stop the service. If *host* is an empty string or None all interfaces are assumed @@ -265,7 +265,7 @@ Creating connections ^^^^^^^^^^^^^^^^^^^^ -.. method:: BaseEventLoop.create_connection(protocol_factory, host=None, port=None, **options) +.. method:: BaseEventLoop.create_connection(protocol_factory, host=None, port=None, \*, ssl=None, family=0, proto=0, flags=0, sock=None, local_addr=None, server_hostname=None) Create a streaming transport connection to a given Internet *host* and *port*. *protocol_factory* must be a callable returning a @@ -296,8 +296,7 @@ a class. For example, if you want to use a pre-created protocol instance, you can pass ``lambda: my_protocol``. - *options* are optional named arguments allowing to change how the - connection is created: + Options allowing to change how the connection is created: * *ssl*: if given and not false, a SSL/TLS transport is created (by default a plain TCP transport is created). If *ssl* is @@ -431,7 +430,7 @@ :class:`StreamReader`). The return value is the same as :meth:`~BaseEventLoop.create_server()`, i.e. - a Server object which can be used to stop the service. + a :class:`AbstractServer` object which can be used to stop the service. This function returns a :ref:`coroutine `. @@ -659,6 +658,35 @@ This method allows transport implementations to easily expose channel-specific information. + * socket: + + - ``'peername'``: the remote address to which the socket is connected, + result of :meth:`socket.socket.getpeername` (``None`` on error) + - ``'socket'``: :class:`socket.socket` instance + - ``'sockname'``: the socket's own address, + result of :meth:`socket.socket.getsockname` + + * SSL socket: + + - ``'compression'``: the compression algorithm being used as a string, + or ``None`` if the connection isn't compressed; result of + :meth:`ssl.SSLSocket.compression` + - ``'cipher'``: a three-value tuple containing the name of the cipher + being used, the version of the SSL protocol that defines its use, and + the number of secret bits being used; result of + :meth:`ssl.SSLSocket.cipher` + - ``'peercert'``: peer certificate; result of + :meth:`ssl.SSLSocket.getpeercert` + - ``'sslcontext'``: :class:`ssl.SSLContext` instance + + * pipe: + + - ``'pipe'``: pipe object + + * subprocess: + + - ``'subprocess'``: :class:`subprocess.Popen` instance + Methods of readable streaming transports: ReadTransport ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -854,11 +882,11 @@ outer Future is *not* cancelled in this case. (This is to prevent the cancellation of one child to cause other children to be cancelled.) -.. function:: iscoroutinefunction(func) +.. function:: tasks.iscoroutinefunction(func) Return ``True`` if *func* is a decorated coroutine function. -.. function:: iscoroutine(obj) +.. function:: tasks.iscoroutine(obj) Return ``True`` if *obj* is a coroutine object. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 00:49:37 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 3 Dec 2013 00:49:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_document_wait?= =?utf-8?q?=28=29_function?= Message-ID: <3dYNN157Vdz7LjM@mail.python.org> http://hg.python.org/cpython/rev/1c6b08fd7a86 changeset: 87721:1c6b08fd7a86 user: Victor Stinner date: Tue Dec 03 00:42:59 2013 +0100 summary: asyncio: document wait() function files: Doc/library/asyncio.rst | 42 +++++++++++++++++++++++++++++ 1 files changed, 42 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -922,6 +922,48 @@ except CancelledError: res = None +.. function:: wait(fs, \*, loop=None, timeout=None, return_when=ALL_COMPLETED) + + Wait for the Futures and coroutines given by fs to complete. Coroutines will + be wrapped in Tasks. Returns two sets of + :class:`~concurrent.futures.Future`: (done, pending). + + *timeout* can be used to control the maximum number of seconds to wait before + returning. *timeout* can be an int or float. If *timeout* is not specified + or ``None``, there is no limit to the wait time. + + *return_when* indicates when this function should return. It must be one of + the following constants of the :mod`concurrent.futures` module: + + .. tabularcolumns:: |l|L| + + +-----------------------------+----------------------------------------+ + | Constant | Description | + +=============================+========================================+ + | :const:`FIRST_COMPLETED` | The function will return when any | + | | future finishes or is cancelled. | + +-----------------------------+----------------------------------------+ + | :const:`FIRST_EXCEPTION` | The function will return when any | + | | future finishes by raising an | + | | exception. If no future raises an | + | | exception then it is equivalent to | + | | :const:`ALL_COMPLETED`. | + +-----------------------------+----------------------------------------+ + | :const:`ALL_COMPLETED` | The function will return when all | + | | futures finish or are cancelled. | + +-----------------------------+----------------------------------------+ + + This function returns a :ref:`coroutine `. + + Usage:: + + done, pending = yield from asyncio.wait(fs) + + .. note:: + + This does not raise :exc:`TimeoutError`! Futures that aren't done when + the timeout occurs are returned in the second set. + Task ---- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 00:49:39 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 3 Dec 2013 00:49:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_reorder_sec?= =?utf-8?q?tions?= Message-ID: <3dYNN33fx7z7Ljg@mail.python.org> http://hg.python.org/cpython/rev/9ac097756dba changeset: 87722:9ac097756dba user: Victor Stinner date: Tue Dec 03 00:49:26 2013 +0100 summary: asyncio doc: reorder sections files: Doc/library/asyncio.rst | 1072 +++++++++++++------------- 1 files changed, 536 insertions(+), 536 deletions(-) diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -326,6 +326,40 @@ to bind the socket to locally. The *local_host* and *local_port* are looked up using getaddrinfo(), similarly to *host* and *port*. + +Resolve name +^^^^^^^^^^^^ + +.. method:: BaseEventLoop.getaddrinfo(host, port, \*, family=0, type=0, proto=0, flags=0) + + XXX + +.. method:: BaseEventLoop.getnameinfo(sockaddr, flags=0) + + XXX + + +Running subprocesses +^^^^^^^^^^^^^^^^^^^^ + +Run subprocesses asynchronously using the :mod:`subprocess` module. + +.. method:: BaseEventLoop.subprocess_exec(protocol_factory, \*args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=False, bufsize=0, \*\*kwargs) + + XXX + + This method returns a :ref:`coroutine `. + + See the constructor of the :class:`subprocess.Popen` class for parameters. + +.. method:: BaseEventLoop.subprocess_shell(protocol_factory, cmd, \*, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=True, bufsize=0, \*\*kwargs) + + XXX + + This method returns a :ref:`coroutine `. + + See the constructor of the :class:`subprocess.Popen` class for parameters. + .. method:: BaseEventLoop.connect_read_pipe(protocol_factory, pipe) Register read pipe in eventloop. @@ -349,492 +383,104 @@ This method returns a :ref:`coroutine `. -Resolve name -^^^^^^^^^^^^ +.. _coroutine: -.. method:: BaseEventLoop.getaddrinfo(host, port, \*, family=0, type=0, proto=0, flags=0) - - XXX - -.. method:: BaseEventLoop.getnameinfo(sockaddr, flags=0) - - XXX - - -Running subprocesses -^^^^^^^^^^^^^^^^^^^^ - -Run subprocesses asynchronously using the :mod:`subprocess` module. - -.. method:: BaseEventLoop.subprocess_exec(protocol_factory, \*args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=False, bufsize=0, \*\*kwargs) - - XXX - - This method returns a :ref:`coroutine `. - - See the constructor of the :class:`subprocess.Popen` class for parameters. - -.. method:: BaseEventLoop.subprocess_shell(protocol_factory, cmd, \*, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=True, bufsize=0, \*\*kwargs) - - XXX - - This method returns a :ref:`coroutine `. - - See the constructor of the :class:`subprocess.Popen` class for parameters. - - -Network functions ------------------ - -.. function:: open_connection(host=None, port=None, *, loop=None, limit=_DEFAULT_LIMIT, **kwds) - - A wrapper for create_connection() returning a (reader, writer) pair. - - The reader returned is a StreamReader instance; the writer is a - :class:`Transport`. - - The arguments are all the usual arguments to - :meth:`BaseEventLoop.create_connection` except *protocol_factory*; most - common are positional host and port, with various optional keyword arguments - following. - - Additional optional keyword arguments are *loop* (to set the event loop - instance to use) and *limit* (to set the buffer limit passed to the - StreamReader). - - (If you want to customize the :class:`StreamReader` and/or - :class:`StreamReaderProtocol` classes, just copy the code -- there's really - nothing special here except some convenience.) - - This function returns a :ref:`coroutine `. - -.. function:: start_server(client_connected_cb, host=None, port=None, *, loop=None, limit=_DEFAULT_LIMIT, **kwds) - - Start a socket server, call back for each client connected. - - The first parameter, *client_connected_cb*, takes two parameters: - *client_reader*, *client_writer*. *client_reader* is a - :class:`StreamReader` object, while *client_writer* is a - :class:`StreamWriter` object. This parameter can either be a plain callback - function or a :ref:`coroutine `; if it is a coroutine, it will be - automatically converted into a :class:`Task`. - - The rest of the arguments are all the usual arguments to - :meth:`~BaseEventLoop.create_server()` except *protocol_factory*; most - common are positional host and port, with various optional keyword arguments - following. The return value is the same as - :meth:`~BaseEventLoop.create_server()`. - - Additional optional keyword arguments are *loop* (to set the event loop - instance to use) and *limit* (to set the buffer limit passed to the - :class:`StreamReader`). - - The return value is the same as :meth:`~BaseEventLoop.create_server()`, i.e. - a :class:`AbstractServer` object which can be used to stop the service. - - This function returns a :ref:`coroutine `. - - -.. _protocol: - -Protocols ---------- - -:mod:`asyncio` provides base classes that you can subclass to implement -your network protocols. Those classes are used in conjunction with -:ref:`transports ` (see below): the protocol parses incoming -data and asks for the writing of outgoing data, while the transport is -responsible for the actual I/O and buffering. - -When subclassing a protocol class, it is recommended you override certain -methods. Those methods are callbacks: they will be called by the transport -on certain events (for example when some data is received); you shouldn't -call them yourself, unless you are implementing a transport. - -.. note:: - All callbacks have default implementations, which are empty. Therefore, - you only need to implement the callbacks for the events in which you - are interested. - - -Protocol classes -^^^^^^^^^^^^^^^^ - -.. class:: Protocol - - The base class for implementing streaming protocols (for use with - e.g. TCP and SSL transports). - -.. class:: DatagramProtocol - - The base class for implementing datagram protocols (for use with - e.g. UDP transports). - -.. class:: SubprocessProtocol - - The base class for implementing protocols communicating with child - processes (through a set of unidirectional pipes). - - -Connection callbacks -^^^^^^^^^^^^^^^^^^^^ - -These callbacks may be called on :class:`Protocol` and -:class:`SubprocessProtocol` instances: - -.. method:: BaseProtocol.connection_made(transport) - - Called when a connection is made. - - The *transport* argument is the transport representing the - connection. You are responsible for storing it somewhere - (e.g. as an attribute) if you need to. - -.. method:: BaseProtocol.connection_lost(exc) - - Called when the connection is lost or closed. - - The argument is either an exception object or :const:`None`. - The latter means a regular EOF is received, or the connection was - aborted or closed by this side of the connection. - -:meth:`connection_made` and :meth:`connection_lost` are called exactly once -per successful connection. All other callbacks will be called between those -two methods, which allows for easier resource management in your protocol -implementation. - -The following callbacks may be called only on :class:`SubprocessProtocol` -instances: - -.. method:: SubprocessProtocol.pipe_data_received(fd, data) - - Called when the child process writes data into its stdout or stderr pipe. - *fd* is the integer file descriptor of the pipe. *data* is a non-empty - bytes object containing the data. - -.. method:: SubprocessProtocol.pipe_connection_lost(fd, exc) - - Called when one of the pipes communicating with the child process - is closed. *fd* is the integer file descriptor that was closed. - -.. method:: SubprocessProtocol.process_exited() - - Called when the child process has exited. - - -Data reception callbacks -^^^^^^^^^^^^^^^^^^^^^^^^ - -Streaming protocols -""""""""""""""""""" - -The following callbacks are called on :class:`Protocol` instances: - -.. method:: Protocol.data_received(data) - - Called when some data is received. *data* is a non-empty bytes object - containing the incoming data. - - .. note:: - Whether the data is buffered, chunked or reassembled depends on - the transport. In general, you shouldn't rely on specific semantics - and instead make your parsing generic and flexible enough. However, - data is always received in the correct order. - -.. method:: Protocol.eof_received() - - Calls when the other end signals it won't send any more data - (for example by calling :meth:`write_eof`, if the other end also uses - asyncio). - - This method may return a false value (including None), in which case - the transport will close itself. Conversely, if this method returns a - true value, closing the transport is up to the protocol. Since the - default implementation returns None, it implicitly closes the connection. - - .. note:: - Some transports such as SSL don't support half-closed connections, - in which case returning true from this method will not prevent closing - the connection. - -:meth:`data_received` can be called an arbitrary number of times during -a connection. However, :meth:`eof_received` is called at most once -and, if called, :meth:`data_received` won't be called after it. - -Datagram protocols -"""""""""""""""""" - -The following callbacks are called on :class:`DatagramProtocol` instances. - -.. method:: DatagramProtocol.datagram_received(data, addr) - - Called when a datagram is received. *data* is a bytes object containing - the incoming data. *addr* is the address of the peer sending the data; - the exact format depends on the transport. - -.. method:: DatagramProtocol.error_received(exc) - - Called when a previous send or receive operation raises an - :class:`OSError`. *exc* is the :class:`OSError` instance. - - This method is called in rare conditions, when the transport (e.g. UDP) - detects that a datagram couldn't be delivered to its recipient. - In many conditions though, undeliverable datagrams will be silently - dropped. - - -Flow control callbacks -^^^^^^^^^^^^^^^^^^^^^^ - -These callbacks may be called on :class:`Protocol` and -:class:`SubprocessProtocol` instances: - -.. method:: BaseProtocol.pause_writing() - - Called when the transport's buffer goes over the high-water mark. - -.. method:: BaseProtocol.resume_writing() - - Called when the transport's buffer drains below the low-water mark. - - -:meth:`pause_writing` and :meth:`resume_writing` calls are paired -- -:meth:`pause_writing` is called once when the buffer goes strictly over -the high-water mark (even if subsequent writes increases the buffer size -even more), and eventually :meth:`resume_writing` is called once when the -buffer size reaches the low-water mark. - -.. note:: - If the buffer size equals the high-water mark, - :meth:`pause_writing` is not called -- it must go strictly over. - Conversely, :meth:`resume_writing` is called when the buffer size is - equal or lower than the low-water mark. These end conditions - are important to ensure that things go as expected when either - mark is zero. - - -.. _transport: - -Transports +Coroutines ---------- -Transports are classed provided by :mod:`asyncio` in order to abstract -various kinds of communication channels. You generally won't instantiate -a transport yourself; instead, you will call a :class:`BaseEventLoop` method -which will create the transport and try to initiate the underlying -communication channel, calling you back when it succeeds. +A coroutine is a generator that follows certain conventions. For +documentation purposes, all coroutines should be decorated with +``@asyncio.coroutine``, but this cannot be strictly enforced. -Once the communication channel is established, a transport is always -paired with a :ref:`protocol ` instance. The protocol can -then call the transport's methods for various purposes. +Coroutines use the ``yield from`` syntax introduced in :pep:`380`, +instead of the original ``yield`` syntax. -:mod:`asyncio` currently implements transports for TCP, UDP, SSL, and -subprocess pipes. The methods available on a transport depend on -the transport's kind. +The word "coroutine", like the word "generator", is used for two +different (though related) concepts: +- The function that defines a coroutine (a function definition + decorated with ``asyncio.coroutine``). If disambiguation is needed + we will call this a *coroutine function*. -Methods common to all transports: BaseTransport -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +- The object obtained by calling a coroutine function. This object + represents a computation or an I/O operation (usually a combination) + that will complete eventually. If disambiguation is needed we will + call it a *coroutine object*. -.. class:: BaseTransport +Things a coroutine can do: - Base class for transports. +- ``result = yield from future`` -- suspends the coroutine until the + future is done, then returns the future's result, or raises an + exception, which will be propagated. (If the future is cancelled, + it will raise a ``CancelledError`` exception.) Note that tasks are + futures, and everything said about futures also applies to tasks. - .. method:: close(self) +- ``result = yield from coroutine`` -- wait for another coroutine to + produce a result (or raise an exception, which will be propagated). + The ``coroutine`` expression must be a *call* to another coroutine. - Close the transport. If the transport has a buffer for outgoing - data, buffered data will be flushed asynchronously. No more data - will be received. After all buffered data is flushed, the - protocol's :meth:`connection_lost` method will be called with - :const:`None` as its argument. +- ``return expression`` -- produce a result to the coroutine that is + waiting for this one using ``yield from``. +- ``raise exception`` -- raise an exception in the coroutine that is + waiting for this one using ``yield from``. - .. method:: get_extra_info(name, default=None) +Calling a coroutine does not start its code running -- it is just a +generator, and the coroutine object returned by the call is really a +generator object, which doesn't do anything until you iterate over it. +In the case of a coroutine object, there are two basic ways to start +it running: call ``yield from coroutine`` from another coroutine +(assuming the other coroutine is already running!), or convert it to a +:class:`Task`. - Return optional transport information. *name* is a string representing - the piece of transport-specific information to get, *default* is the - value to return if the information doesn't exist. +Coroutines (and tasks) can only run when the event loop is running. - This method allows transport implementations to easily expose - channel-specific information. - * socket: +Task +---- - - ``'peername'``: the remote address to which the socket is connected, - result of :meth:`socket.socket.getpeername` (``None`` on error) - - ``'socket'``: :class:`socket.socket` instance - - ``'sockname'``: the socket's own address, - result of :meth:`socket.socket.getsockname` +.. class:: Task(coro, \*, loop=None) - * SSL socket: + A coroutine wrapped in a :class:`~concurrent.futures.Future`. - - ``'compression'``: the compression algorithm being used as a string, - or ``None`` if the connection isn't compressed; result of - :meth:`ssl.SSLSocket.compression` - - ``'cipher'``: a three-value tuple containing the name of the cipher - being used, the version of the SSL protocol that defines its use, and - the number of secret bits being used; result of - :meth:`ssl.SSLSocket.cipher` - - ``'peercert'``: peer certificate; result of - :meth:`ssl.SSLSocket.getpeercert` - - ``'sslcontext'``: :class:`ssl.SSLContext` instance + .. classmethod:: all_tasks(loop=None) - * pipe: + Return a set of all tasks for an event loop. - - ``'pipe'``: pipe object + By default all tasks for the current event loop are returned. - * subprocess: + .. method:: cancel() - - ``'subprocess'``: :class:`subprocess.Popen` instance + Cancel the task. + .. method:: get_stack(self, \*, limit=None) -Methods of readable streaming transports: ReadTransport -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + Return the list of stack frames for this task's coroutine. -.. class:: ReadTransport + If the coroutine is active, this returns the stack where it is suspended. + If the coroutine has completed successfully or was cancelled, this + returns an empty list. If the coroutine was terminated by an exception, + this returns the list of traceback frames. - Interface for read-only transports. + The frames are always ordered from oldest to newest. - .. method:: pause_reading() + The optional limit gives the maximum nummber of frames to return; by + default all available frames are returned. Its meaning differs depending + on whether a stack or a traceback is returned: the newest frames of a + stack are returned, but the oldest frames of a traceback are returned. + (This matches the behavior of the traceback module.) - Pause the receiving end of the transport. No data will be passed to - the protocol's :meth:`data_received` method until meth:`resume_reading` - is called. + For reasons beyond our control, only one stack frame is returned for a + suspended coroutine. - .. method:: resume_reading() + .. method:: print_stack(\*, limit=None, file=None) - Resume the receiving end. The protocol's :meth:`data_received` method - will be called once again if some data is available for reading. + Print the stack or traceback for this task's coroutine. - -Methods of writable streaming transports: WriteTransport -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. class:: WriteTransport - - Interface for write-only transports. - - .. method:: write(data) - - Write some *data* bytes to the transport. - - This method does not block; it buffers the data and arranges for it - to be sent out asynchronously. - - .. method:: writelines(list_of_data) - - Write a list (or any iterable) of data bytes to the transport. - This is functionally equivalent to calling :meth:`write` on each - element yielded by the iterable, but may be implemented more efficiently. - - .. method:: write_eof() - - Close the write end of the transport after flushing buffered data. - Data may still be received. - - This method can raise :exc:`NotImplementedError` if the transport - (e.g. SSL) doesn't support half-closes. - - .. method:: can_write_eof() - - Return :const:`True` if the transport supports :meth:`write_eof`, - :const:`False` if not. - - .. method:: abort() - - Close the transport immediately, without waiting for pending operations - to complete. Buffered data will be lost. No more data will be received. - The protocol's :meth:`connection_lost` method will eventually be - called with :const:`None` as its argument. - - .. method:: set_write_buffer_limits(high=None, low=None) - - Set the *high*- and *low*-water limits for write flow control. - - These two values control when call the protocol's - :meth:`pause_writing` and :meth:`resume_writing` methods are called. - If specified, the low-water limit must be less than or equal to the - high-water limit. Neither *high* nor *low* can be negative. - - The defaults are implementation-specific. If only the - high-water limit is given, the low-water limit defaults to a - implementation-specific value less than or equal to the - high-water limit. Setting *high* to zero forces *low* to zero as - well, and causes :meth:`pause_writing` to be called whenever the - buffer becomes non-empty. Setting *low* to zero causes - :meth:`resume_writing` to be called only once the buffer is empty. - Use of zero for either limit is generally sub-optimal as it - reduces opportunities for doing I/O and computation - concurrently. - - .. method:: get_write_buffer_size() - - Return the current size of the output buffer used by the transport. - - -Methods of datagram transports -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. method:: DatagramTransport.sendto(data, addr=None) - - Send the *data* bytes to the remote peer given by *addr* (a - transport-dependent target address). If *addr* is :const:`None`, the - data is sent to the target address given on transport creation. - - This method does not block; it buffers the data and arranges for it - to be sent out asynchronously. - -.. method:: DatagramTransport.abort() - - Close the transport immediately, without waiting for pending operations - to complete. Buffered data will be lost. No more data will be received. - The protocol's :meth:`connection_lost` method will eventually be - called with :const:`None` as its argument. - - -Methods of subprocess transports -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. class:: BaseSubprocessTransport - - .. method:: get_pid() - - Return the subprocess process id as an integer. - - .. method:: get_returncode() - - Return the subprocess returncode as an integer or :const:`None` - if it hasn't returned, similarly to the - :attr:`subprocess.Popen.returncode` attribute. - - .. method:: get_pipe_transport(fd) - - Return the transport for the communication pipe correspondong to the - integer file descriptor *fd*. The return value can be a readable or - writable streaming transport, depending on the *fd*. If *fd* doesn't - correspond to a pipe belonging to this transport, :const:`None` is - returned. - - .. method:: send_signal(signal) - - Send the *signal* number to the subprocess, as in - :meth:`subprocess.Popen.send_signal`. - - .. method:: terminate() - - Ask the subprocess to stop, as in :meth:`subprocess.Popen.terminate`. - This method is an alias for the :meth:`close` method. - - On POSIX systems, this method sends SIGTERM to the subprocess. - On Windows, the Windows API function TerminateProcess() is called to - stop the subprocess. - - .. method:: kill(self) - - Kill the subprocess, as in :meth:`subprocess.Popen.kill` - - On POSIX systems, the function sends SIGKILL to the subprocess. - On Windows, this method is an alias for :meth:`terminate`. + This produces output similar to that of the traceback module, for the + frames retrieved by get_stack(). The limit argument is passed to + get_stack(). The file argument is an I/O stream to which the output + goes; by default it goes to sys.stderr. Task functions @@ -965,71 +611,228 @@ the timeout occurs are returned in the second set. -Task ----- +.. _transport: -.. class:: Task(coro, \*, loop=None) +Transports +---------- - A coroutine wrapped in a :class:`~concurrent.futures.Future`. +Transports are classed provided by :mod:`asyncio` in order to abstract +various kinds of communication channels. You generally won't instantiate +a transport yourself; instead, you will call a :class:`BaseEventLoop` method +which will create the transport and try to initiate the underlying +communication channel, calling you back when it succeeds. - .. classmethod:: all_tasks(loop=None) +Once the communication channel is established, a transport is always +paired with a :ref:`protocol ` instance. The protocol can +then call the transport's methods for various purposes. - Return a set of all tasks for an event loop. +:mod:`asyncio` currently implements transports for TCP, UDP, SSL, and +subprocess pipes. The methods available on a transport depend on +the transport's kind. - By default all tasks for the current event loop are returned. - .. method:: cancel() +Methods common to all transports: BaseTransport +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Cancel the task. +.. class:: BaseTransport - .. method:: get_stack(self, \*, limit=None) + Base class for transports. - Return the list of stack frames for this task's coroutine. + .. method:: close(self) - If the coroutine is active, this returns the stack where it is suspended. - If the coroutine has completed successfully or was cancelled, this - returns an empty list. If the coroutine was terminated by an exception, - this returns the list of traceback frames. + Close the transport. If the transport has a buffer for outgoing + data, buffered data will be flushed asynchronously. No more data + will be received. After all buffered data is flushed, the + protocol's :meth:`connection_lost` method will be called with + :const:`None` as its argument. - The frames are always ordered from oldest to newest. - The optional limit gives the maximum nummber of frames to return; by - default all available frames are returned. Its meaning differs depending - on whether a stack or a traceback is returned: the newest frames of a - stack are returned, but the oldest frames of a traceback are returned. - (This matches the behavior of the traceback module.) + .. method:: get_extra_info(name, default=None) - For reasons beyond our control, only one stack frame is returned for a - suspended coroutine. + Return optional transport information. *name* is a string representing + the piece of transport-specific information to get, *default* is the + value to return if the information doesn't exist. - .. method:: print_stack(\*, limit=None, file=None) + This method allows transport implementations to easily expose + channel-specific information. - Print the stack or traceback for this task's coroutine. + * socket: - This produces output similar to that of the traceback module, for the - frames retrieved by get_stack(). The limit argument is passed to - get_stack(). The file argument is an I/O stream to which the output - goes; by default it goes to sys.stderr. + - ``'peername'``: the remote address to which the socket is connected, + result of :meth:`socket.socket.getpeername` (``None`` on error) + - ``'socket'``: :class:`socket.socket` instance + - ``'sockname'``: the socket's own address, + result of :meth:`socket.socket.getsockname` + * SSL socket: -Protocols ---------- + - ``'compression'``: the compression algorithm being used as a string, + or ``None`` if the connection isn't compressed; result of + :meth:`ssl.SSLSocket.compression` + - ``'cipher'``: a three-value tuple containing the name of the cipher + being used, the version of the SSL protocol that defines its use, and + the number of secret bits being used; result of + :meth:`ssl.SSLSocket.cipher` + - ``'peercert'``: peer certificate; result of + :meth:`ssl.SSLSocket.getpeercert` + - ``'sslcontext'``: :class:`ssl.SSLContext` instance -:mod:`asyncio` provides base classes that you can subclass to implement -your network protocols. Those classes are used in conjunction with -:ref:`transports ` (see below): the protocol parses incoming -data and asks for the writing of outgoing data, while the transport is -responsible for the actual I/O and buffering. + * pipe: -When subclassing a protocol class, it is recommended you override certain -methods. Those methods are callbacks: they will be called by the transport -on certain events (for example when some data is received); you shouldn't -call them yourself, unless you are implementing a transport. + - ``'pipe'``: pipe object -.. note:: - All callbacks have default implementations, which are empty. Therefore, - you only need to implement the callbacks for the events in which you - are interested. + * subprocess: + + - ``'subprocess'``: :class:`subprocess.Popen` instance + + +Methods of readable streaming transports: ReadTransport +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. class:: ReadTransport + + Interface for read-only transports. + + .. method:: pause_reading() + + Pause the receiving end of the transport. No data will be passed to + the protocol's :meth:`data_received` method until meth:`resume_reading` + is called. + + .. method:: resume_reading() + + Resume the receiving end. The protocol's :meth:`data_received` method + will be called once again if some data is available for reading. + + +Methods of writable streaming transports: WriteTransport +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. class:: WriteTransport + + Interface for write-only transports. + + .. method:: abort() + + Close the transport immediately, without waiting for pending operations + to complete. Buffered data will be lost. No more data will be received. + The protocol's :meth:`connection_lost` method will eventually be + called with :const:`None` as its argument. + + .. method:: can_write_eof() + + Return :const:`True` if the transport supports :meth:`write_eof`, + :const:`False` if not. + + .. method:: get_write_buffer_size() + + Return the current size of the output buffer used by the transport. + + .. method:: set_write_buffer_limits(high=None, low=None) + + Set the *high*- and *low*-water limits for write flow control. + + These two values control when call the protocol's + :meth:`pause_writing` and :meth:`resume_writing` methods are called. + If specified, the low-water limit must be less than or equal to the + high-water limit. Neither *high* nor *low* can be negative. + + The defaults are implementation-specific. If only the + high-water limit is given, the low-water limit defaults to a + implementation-specific value less than or equal to the + high-water limit. Setting *high* to zero forces *low* to zero as + well, and causes :meth:`pause_writing` to be called whenever the + buffer becomes non-empty. Setting *low* to zero causes + :meth:`resume_writing` to be called only once the buffer is empty. + Use of zero for either limit is generally sub-optimal as it + reduces opportunities for doing I/O and computation + concurrently. + + .. method:: write(data) + + Write some *data* bytes to the transport. + + This method does not block; it buffers the data and arranges for it + to be sent out asynchronously. + + .. method:: writelines(list_of_data) + + Write a list (or any iterable) of data bytes to the transport. + This is functionally equivalent to calling :meth:`write` on each + element yielded by the iterable, but may be implemented more efficiently. + + .. method:: write_eof() + + Close the write end of the transport after flushing buffered data. + Data may still be received. + + This method can raise :exc:`NotImplementedError` if the transport + (e.g. SSL) doesn't support half-closes. + + +Methods of datagram transports +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. method:: DatagramTransport.sendto(data, addr=None) + + Send the *data* bytes to the remote peer given by *addr* (a + transport-dependent target address). If *addr* is :const:`None`, the + data is sent to the target address given on transport creation. + + This method does not block; it buffers the data and arranges for it + to be sent out asynchronously. + +.. method:: DatagramTransport.abort() + + Close the transport immediately, without waiting for pending operations + to complete. Buffered data will be lost. No more data will be received. + The protocol's :meth:`connection_lost` method will eventually be + called with :const:`None` as its argument. + + +Methods of subprocess transports +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. class:: BaseSubprocessTransport + + .. method:: get_pid() + + Return the subprocess process id as an integer. + + .. method:: get_returncode() + + Return the subprocess returncode as an integer or :const:`None` + if it hasn't returned, similarly to the + :attr:`subprocess.Popen.returncode` attribute. + + .. method:: get_pipe_transport(fd) + + Return the transport for the communication pipe correspondong to the + integer file descriptor *fd*. The return value can be a readable or + writable streaming transport, depending on the *fd*. If *fd* doesn't + correspond to a pipe belonging to this transport, :const:`None` is + returned. + + .. method:: send_signal(signal) + + Send the *signal* number to the subprocess, as in + :meth:`subprocess.Popen.send_signal`. + + .. method:: terminate() + + Ask the subprocess to stop, as in :meth:`subprocess.Popen.terminate`. + This method is an alias for the :meth:`close` method. + + On POSIX systems, this method sends SIGTERM to the subprocess. + On Windows, the Windows API function TerminateProcess() is called to + stop the subprocess. + + .. method:: kill(self) + + Kill the subprocess, as in :meth:`subprocess.Popen.kill` + + On POSIX systems, the function sends SIGKILL to the subprocess. + On Windows, this method is an alias for :meth:`terminate`. Stream reader and writer @@ -1136,57 +939,202 @@ -.. _coroutine: +.. _protocol: -Coroutines ----------- +Protocols +--------- -A coroutine is a generator that follows certain conventions. For -documentation purposes, all coroutines should be decorated with -``@asyncio.coroutine``, but this cannot be strictly enforced. +:mod:`asyncio` provides base classes that you can subclass to implement +your network protocols. Those classes are used in conjunction with +:ref:`transports ` (see below): the protocol parses incoming +data and asks for the writing of outgoing data, while the transport is +responsible for the actual I/O and buffering. -Coroutines use the ``yield from`` syntax introduced in :pep:`380`, -instead of the original ``yield`` syntax. +When subclassing a protocol class, it is recommended you override certain +methods. Those methods are callbacks: they will be called by the transport +on certain events (for example when some data is received); you shouldn't +call them yourself, unless you are implementing a transport. -The word "coroutine", like the word "generator", is used for two -different (though related) concepts: +.. note:: + All callbacks have default implementations, which are empty. Therefore, + you only need to implement the callbacks for the events in which you + are interested. -- The function that defines a coroutine (a function definition - decorated with ``asyncio.coroutine``). If disambiguation is needed - we will call this a *coroutine function*. -- The object obtained by calling a coroutine function. This object - represents a computation or an I/O operation (usually a combination) - that will complete eventually. If disambiguation is needed we will - call it a *coroutine object*. +Protocol classes +^^^^^^^^^^^^^^^^ -Things a coroutine can do: +.. class:: Protocol -- ``result = yield from future`` -- suspends the coroutine until the - future is done, then returns the future's result, or raises an - exception, which will be propagated. (If the future is cancelled, - it will raise a ``CancelledError`` exception.) Note that tasks are - futures, and everything said about futures also applies to tasks. + The base class for implementing streaming protocols (for use with + e.g. TCP and SSL transports). -- ``result = yield from coroutine`` -- wait for another coroutine to - produce a result (or raise an exception, which will be propagated). - The ``coroutine`` expression must be a *call* to another coroutine. +.. class:: DatagramProtocol -- ``return expression`` -- produce a result to the coroutine that is - waiting for this one using ``yield from``. + The base class for implementing datagram protocols (for use with + e.g. UDP transports). -- ``raise exception`` -- raise an exception in the coroutine that is - waiting for this one using ``yield from``. +.. class:: SubprocessProtocol -Calling a coroutine does not start its code running -- it is just a -generator, and the coroutine object returned by the call is really a -generator object, which doesn't do anything until you iterate over it. -In the case of a coroutine object, there are two basic ways to start -it running: call ``yield from coroutine`` from another coroutine -(assuming the other coroutine is already running!), or convert it to a -:class:`Task`. + The base class for implementing protocols communicating with child + processes (through a set of unidirectional pipes). -Coroutines (and tasks) can only run when the event loop is running. + +Connection callbacks +^^^^^^^^^^^^^^^^^^^^ + +These callbacks may be called on :class:`Protocol` and +:class:`SubprocessProtocol` instances: + +.. method:: BaseProtocol.connection_made(transport) + + Called when a connection is made. + + The *transport* argument is the transport representing the + connection. You are responsible for storing it somewhere + (e.g. as an attribute) if you need to. + +.. method:: BaseProtocol.connection_lost(exc) + + Called when the connection is lost or closed. + + The argument is either an exception object or :const:`None`. + The latter means a regular EOF is received, or the connection was + aborted or closed by this side of the connection. + +:meth:`connection_made` and :meth:`connection_lost` are called exactly once +per successful connection. All other callbacks will be called between those +two methods, which allows for easier resource management in your protocol +implementation. + +The following callbacks may be called only on :class:`SubprocessProtocol` +instances: + +.. method:: SubprocessProtocol.pipe_data_received(fd, data) + + Called when the child process writes data into its stdout or stderr pipe. + *fd* is the integer file descriptor of the pipe. *data* is a non-empty + bytes object containing the data. + +.. method:: SubprocessProtocol.pipe_connection_lost(fd, exc) + + Called when one of the pipes communicating with the child process + is closed. *fd* is the integer file descriptor that was closed. + +.. method:: SubprocessProtocol.process_exited() + + Called when the child process has exited. + + +Data reception callbacks +^^^^^^^^^^^^^^^^^^^^^^^^ + +Streaming protocols +""""""""""""""""""" + +The following callbacks are called on :class:`Protocol` instances: + +.. method:: Protocol.data_received(data) + + Called when some data is received. *data* is a non-empty bytes object + containing the incoming data. + + .. note:: + Whether the data is buffered, chunked or reassembled depends on + the transport. In general, you shouldn't rely on specific semantics + and instead make your parsing generic and flexible enough. However, + data is always received in the correct order. + +.. method:: Protocol.eof_received() + + Calls when the other end signals it won't send any more data + (for example by calling :meth:`write_eof`, if the other end also uses + asyncio). + + This method may return a false value (including None), in which case + the transport will close itself. Conversely, if this method returns a + true value, closing the transport is up to the protocol. Since the + default implementation returns None, it implicitly closes the connection. + + .. note:: + Some transports such as SSL don't support half-closed connections, + in which case returning true from this method will not prevent closing + the connection. + +:meth:`data_received` can be called an arbitrary number of times during +a connection. However, :meth:`eof_received` is called at most once +and, if called, :meth:`data_received` won't be called after it. + +Datagram protocols +"""""""""""""""""" + +The following callbacks are called on :class:`DatagramProtocol` instances. + +.. method:: DatagramProtocol.datagram_received(data, addr) + + Called when a datagram is received. *data* is a bytes object containing + the incoming data. *addr* is the address of the peer sending the data; + the exact format depends on the transport. + +.. method:: DatagramProtocol.error_received(exc) + + Called when a previous send or receive operation raises an + :class:`OSError`. *exc* is the :class:`OSError` instance. + + This method is called in rare conditions, when the transport (e.g. UDP) + detects that a datagram couldn't be delivered to its recipient. + In many conditions though, undeliverable datagrams will be silently + dropped. + + +Flow control callbacks +^^^^^^^^^^^^^^^^^^^^^^ + +These callbacks may be called on :class:`Protocol` and +:class:`SubprocessProtocol` instances: + +.. method:: BaseProtocol.pause_writing() + + Called when the transport's buffer goes over the high-water mark. + +.. method:: BaseProtocol.resume_writing() + + Called when the transport's buffer drains below the low-water mark. + + +:meth:`pause_writing` and :meth:`resume_writing` calls are paired -- +:meth:`pause_writing` is called once when the buffer goes strictly over +the high-water mark (even if subsequent writes increases the buffer size +even more), and eventually :meth:`resume_writing` is called once when the +buffer size reaches the low-water mark. + +.. note:: + If the buffer size equals the high-water mark, + :meth:`pause_writing` is not called -- it must go strictly over. + Conversely, :meth:`resume_writing` is called when the buffer size is + equal or lower than the low-water mark. These end conditions + are important to ensure that things go as expected when either + mark is zero. + + +Protocols +--------- + +:mod:`asyncio` provides base classes that you can subclass to implement +your network protocols. Those classes are used in conjunction with +:ref:`transports ` (see below): the protocol parses incoming +data and asks for the writing of outgoing data, while the transport is +responsible for the actual I/O and buffering. + +When subclassing a protocol class, it is recommended you override certain +methods. Those methods are callbacks: they will be called by the transport +on certain events (for example when some data is received); you shouldn't +call them yourself, unless you are implementing a transport. + +.. note:: + All callbacks have default implementations, which are empty. Therefore, + you only need to implement the callbacks for the events in which you + are interested. Server @@ -1205,6 +1153,58 @@ Coroutine to wait until service is closed. +Network functions +----------------- + +.. function:: open_connection(host=None, port=None, *, loop=None, limit=_DEFAULT_LIMIT, **kwds) + + A wrapper for create_connection() returning a (reader, writer) pair. + + The reader returned is a StreamReader instance; the writer is a + :class:`Transport`. + + The arguments are all the usual arguments to + :meth:`BaseEventLoop.create_connection` except *protocol_factory*; most + common are positional host and port, with various optional keyword arguments + following. + + Additional optional keyword arguments are *loop* (to set the event loop + instance to use) and *limit* (to set the buffer limit passed to the + StreamReader). + + (If you want to customize the :class:`StreamReader` and/or + :class:`StreamReaderProtocol` classes, just copy the code -- there's really + nothing special here except some convenience.) + + This function returns a :ref:`coroutine `. + +.. function:: start_server(client_connected_cb, host=None, port=None, *, loop=None, limit=_DEFAULT_LIMIT, **kwds) + + Start a socket server, call back for each client connected. + + The first parameter, *client_connected_cb*, takes two parameters: + *client_reader*, *client_writer*. *client_reader* is a + :class:`StreamReader` object, while *client_writer* is a + :class:`StreamWriter` object. This parameter can either be a plain callback + function or a :ref:`coroutine `; if it is a coroutine, it will be + automatically converted into a :class:`Task`. + + The rest of the arguments are all the usual arguments to + :meth:`~BaseEventLoop.create_server()` except *protocol_factory*; most + common are positional host and port, with various optional keyword arguments + following. The return value is the same as + :meth:`~BaseEventLoop.create_server()`. + + Additional optional keyword arguments are *loop* (to set the event loop + instance to use) and *limit* (to set the buffer limit passed to the + :class:`StreamReader`). + + The return value is the same as :meth:`~BaseEventLoop.create_server()`, i.e. + a :class:`AbstractServer` object which can be used to stop the service. + + This function returns a :ref:`coroutine `. + + .. _sync: Synchronization primitives -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 00:56:37 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 3 Dec 2013 00:56:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_change_leve?= =?utf-8?q?l_of_titles?= Message-ID: <3dYNX54wVqz7LjS@mail.python.org> http://hg.python.org/cpython/rev/f0ae0508e5d5 changeset: 87723:f0ae0508e5d5 user: Victor Stinner date: Tue Dec 03 00:56:27 2013 +0100 summary: asyncio doc: change level of titles Remove also a duplicated Protocols section! files: Doc/library/asyncio.rst | 224 +++++++++++++-------------- 1 files changed, 109 insertions(+), 115 deletions(-) diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -42,7 +42,7 @@ Disclaimer ----------- +========== Full documentation is not yet ready; we hope to have it written before Python 3.4 leaves beta. Until then, the best reference is @@ -52,11 +52,10 @@ .. XXX should the asyncio documentation come in several pages, as for logging? - .. _event-loop: Event loops ------------ +=========== The event loop is the central execution device provided by :mod:`asyncio`. It provides multiple facilities, amongst which: @@ -72,7 +71,7 @@ * Delegating costly function calls to a pool of threads Event loop functions -^^^^^^^^^^^^^^^^^^^^ +-------------------- The easiest way to get an event loop is to call the :func:`get_event_loop` function. @@ -94,7 +93,7 @@ Event loop policy -^^^^^^^^^^^^^^^^^ +----------------- .. function:: get_event_loop_policy() @@ -106,7 +105,7 @@ Run an event loop -^^^^^^^^^^^^^^^^^ +----------------- .. method:: BaseEventLoop.run_forever() @@ -144,7 +143,7 @@ Calls -^^^^^ +----- .. method:: BaseEventLoop.call_soon(callback, \*args) @@ -162,7 +161,7 @@ Delayed calls -^^^^^^^^^^^^^ +------------- The event loop has its own internal clock for computing timeouts. Which clock is used depends on the (platform-specific) event loop @@ -198,70 +197,6 @@ event loop's internal clock. -Executor -^^^^^^^^ - -Call a function in an :class:`~concurrent.futures.Executor` (pool of threads or -pool of processes). By default, an event loop uses a thread pool executor -(:class:`~concurrent.futures.ThreadPoolExecutor`). - -.. method:: BaseEventLoop.run_in_executor(executor, callback, \*args) - - Arrange for a callback to be called in the specified executor. - - *executor* is a :class:`~concurrent.futures.Executor` instance, - the default executor is used if *executor* is ``None``. - -.. method:: BaseEventLoop.set_default_executor(executor) - - Set the default executor used by :meth:`run_in_executor`. - - -Creating listening connections -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. method:: BaseEventLoop.create_server(protocol_factory, host=None, port=None, \*, family=socket.AF_UNSPEC, flags=socket.AI_PASSIVE, sock=None, backlog=100, ssl=None, reuse_address=None) - - A :ref:`coroutine ` which creates a TCP server bound to host and - port. - - The return value is a :class:`AbstractServer` object which can be used to stop - the service. - - If *host* is an empty string or None all interfaces are assumed - and a list of multiple sockets will be returned (most likely - one for IPv4 and another one for IPv6). - - *family* can be set to either :data:`~socket.AF_INET` or - :data:`~socket.AF_INET6` to force the socket to use IPv4 or IPv6. If not set - it will be determined from host (defaults to :data:`~socket.AF_UNSPEC`). - - *flags* is a bitmask for :meth:`getaddrinfo`. - - *sock* can optionally be specified in order to use a preexisting - socket object. - - *backlog* is the maximum number of queued connections passed to - :meth:`~socket.socket.listen` (defaults to 100). - - ssl can be set to an :class:`~ssl.SSLContext` to enable SSL over the - accepted connections. - - *reuse_address* tells the kernel to reuse a local socket in - TIME_WAIT state, without waiting for its natural timeout to - expire. If not specified will automatically be set to True on - UNIX. - - This method returns a :ref:`coroutine `. - -.. method:: BaseEventLoop.create_datagram_endpoint(protocol_factory, local_addr=None, remote_addr=None, \*, family=0, proto=0, flags=0) - - Create datagram connection. - - This method returns a :ref:`coroutine `. - - - Creating connections ^^^^^^^^^^^^^^^^^^^^ @@ -327,8 +262,53 @@ are looked up using getaddrinfo(), similarly to *host* and *port*. +Creating listening connections +------------------------------ + +.. method:: BaseEventLoop.create_server(protocol_factory, host=None, port=None, \*, family=socket.AF_UNSPEC, flags=socket.AI_PASSIVE, sock=None, backlog=100, ssl=None, reuse_address=None) + + A :ref:`coroutine ` which creates a TCP server bound to host and + port. + + The return value is a :class:`AbstractServer` object which can be used to stop + the service. + + If *host* is an empty string or None all interfaces are assumed + and a list of multiple sockets will be returned (most likely + one for IPv4 and another one for IPv6). + + *family* can be set to either :data:`~socket.AF_INET` or + :data:`~socket.AF_INET6` to force the socket to use IPv4 or IPv6. If not set + it will be determined from host (defaults to :data:`~socket.AF_UNSPEC`). + + *flags* is a bitmask for :meth:`getaddrinfo`. + + *sock* can optionally be specified in order to use a preexisting + socket object. + + *backlog* is the maximum number of queued connections passed to + :meth:`~socket.socket.listen` (defaults to 100). + + ssl can be set to an :class:`~ssl.SSLContext` to enable SSL over the + accepted connections. + + *reuse_address* tells the kernel to reuse a local socket in + TIME_WAIT state, without waiting for its natural timeout to + expire. If not specified will automatically be set to True on + UNIX. + + This method returns a :ref:`coroutine `. + +.. method:: BaseEventLoop.create_datagram_endpoint(protocol_factory, local_addr=None, remote_addr=None, \*, family=0, proto=0, flags=0) + + Create datagram connection. + + This method returns a :ref:`coroutine `. + + + Resolve name -^^^^^^^^^^^^ +------------ .. method:: BaseEventLoop.getaddrinfo(host, port, \*, family=0, type=0, proto=0, flags=0) @@ -340,7 +320,7 @@ Running subprocesses -^^^^^^^^^^^^^^^^^^^^ +-------------------- Run subprocesses asynchronously using the :mod:`subprocess` module. @@ -383,6 +363,28 @@ This method returns a :ref:`coroutine `. +Executor +-------- + +Call a function in an :class:`~concurrent.futures.Executor` (pool of threads or +pool of processes). By default, an event loop uses a thread pool executor +(:class:`~concurrent.futures.ThreadPoolExecutor`). + +.. method:: BaseEventLoop.run_in_executor(executor, callback, \*args) + + Arrange for a callback to be called in the specified executor. + + *executor* is a :class:`~concurrent.futures.Executor` instance, + the default executor is used if *executor* is ``None``. + +.. method:: BaseEventLoop.set_default_executor(executor) + + Set the default executor used by :meth:`run_in_executor`. + + +Tasks and coroutines +==================== + .. _coroutine: Coroutines @@ -614,7 +616,7 @@ .. _transport: Transports ----------- +========== Transports are classed provided by :mod:`asyncio` in order to abstract various kinds of communication channels. You generally won't instantiate @@ -631,8 +633,8 @@ the transport's kind. -Methods common to all transports: BaseTransport -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +BaseTransport: Methods common to all transports +----------------------------------------------- .. class:: BaseTransport @@ -686,8 +688,8 @@ - ``'subprocess'``: :class:`subprocess.Popen` instance -Methods of readable streaming transports: ReadTransport -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +ReadTransport: Methods of readable streaming transports +------------------------------------------------------- .. class:: ReadTransport @@ -705,8 +707,8 @@ will be called once again if some data is available for reading. -Methods of writable streaming transports: WriteTransport -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +WriteTransport: Methods of writable streaming transports +-------------------------------------------------------- .. class:: WriteTransport @@ -770,8 +772,8 @@ (e.g. SSL) doesn't support half-closes. -Methods of datagram transports -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +DatagramTransport: Methods of datagram transports +------------------------------------------------- .. method:: DatagramTransport.sendto(data, addr=None) @@ -791,7 +793,7 @@ Methods of subprocess transports -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +-------------------------------- .. class:: BaseSubprocessTransport @@ -835,8 +837,8 @@ On Windows, this method is an alias for :meth:`terminate`. -Stream reader and writer ------------------------- +Stream reader +------------- .. class:: StreamWriter(transport, protocol, reader, loop) @@ -897,6 +899,9 @@ see :meth:`WriteTransport.write_eof`. +Stream writer +------------- + .. class:: StreamReader(limit=_DEFAULT_LIMIT, loop=None) .. method:: exception() @@ -942,7 +947,7 @@ .. _protocol: Protocols ---------- +========= :mod:`asyncio` provides base classes that you can subclass to implement your network protocols. Those classes are used in conjunction with @@ -962,7 +967,7 @@ Protocol classes -^^^^^^^^^^^^^^^^ +---------------- .. class:: Protocol @@ -981,7 +986,7 @@ Connection callbacks -^^^^^^^^^^^^^^^^^^^^ +-------------------- These callbacks may be called on :class:`Protocol` and :class:`SubprocessProtocol` instances: @@ -1027,10 +1032,10 @@ Data reception callbacks -^^^^^^^^^^^^^^^^^^^^^^^^ +------------------------ Streaming protocols -""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^ The following callbacks are called on :class:`Protocol` instances: @@ -1066,7 +1071,7 @@ and, if called, :meth:`data_received` won't be called after it. Datagram protocols -"""""""""""""""""" +^^^^^^^^^^^^^^^^^^ The following callbacks are called on :class:`DatagramProtocol` instances. @@ -1088,7 +1093,7 @@ Flow control callbacks -^^^^^^^^^^^^^^^^^^^^^^ +---------------------- These callbacks may be called on :class:`Protocol` and :class:`SubprocessProtocol` instances: @@ -1117,26 +1122,6 @@ mark is zero. -Protocols ---------- - -:mod:`asyncio` provides base classes that you can subclass to implement -your network protocols. Those classes are used in conjunction with -:ref:`transports ` (see below): the protocol parses incoming -data and asks for the writing of outgoing data, while the transport is -responsible for the actual I/O and buffering. - -When subclassing a protocol class, it is recommended you override certain -methods. Those methods are callbacks: they will be called by the transport -on certain events (for example when some data is received); you shouldn't -call them yourself, unless you are implementing a transport. - -.. note:: - All callbacks have default implementations, which are empty. Therefore, - you only need to implement the callbacks for the events in which you - are interested. - - Server ------ @@ -1154,7 +1139,7 @@ Network functions ------------------ +================= .. function:: open_connection(host=None, port=None, *, loop=None, limit=_DEFAULT_LIMIT, **kwds) @@ -1208,7 +1193,10 @@ .. _sync: Synchronization primitives --------------------------- +========================== + +Locks +----- .. class:: Lock(\*, loop=None) @@ -1381,6 +1369,9 @@ This method returns a :ref:`coroutine `. +Semaphores +---------- + .. class:: Semaphore(value=1, \*, loop=None) A Semaphore implementation. @@ -1426,6 +1417,9 @@ increase the value above the initial value. +Queues +------ + .. class:: Queue(maxsize=0, \*, loop=None) A queue, useful for coordinating producer and consumer coroutines. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 01:08:12 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 3 Dec 2013 01:08:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Split_asyncio_documentatio?= =?utf-8?q?n_into_subfiles?= Message-ID: <3dYNnS75W4z7LjN@mail.python.org> http://hg.python.org/cpython/rev/551dc67158bc changeset: 87724:551dc67158bc user: Victor Stinner date: Tue Dec 03 01:08:00 2013 +0100 summary: Split asyncio documentation into subfiles files: Doc/library/asyncio-eventloop.rst | 367 +++ Doc/library/asyncio-protocol.rst | 615 ++++++ Doc/library/asyncio-sync.rst | 341 +++ Doc/library/asyncio-task.rst | 232 ++ Doc/library/asyncio.rst | 1561 +---------------- 5 files changed, 1562 insertions(+), 1554 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst new file mode 100644 --- /dev/null +++ b/Doc/library/asyncio-eventloop.rst @@ -0,0 +1,367 @@ +.. module:: asyncio + +.. _event-loop: + +Event loops +=========== + +The event loop is the central execution device provided by :mod:`asyncio`. +It provides multiple facilities, amongst which: + +* Registering, executing and cancelling delayed calls (timeouts) + +* Creating client and server :ref:`transports ` for various + kinds of communication + +* Launching subprocesses and the associated :ref:`transports ` + for communication with an external program + +* Delegating costly function calls to a pool of threads + +Event loop functions +-------------------- + +The easiest way to get an event loop is to call the :func:`get_event_loop` +function. + +.. function:: get_event_loop() + + Get the event loop for current context. Returns an event loop object + implementing :class:`BaseEventLoop` interface, or raises an exception in case no + event loop has been set for the current context and the current policy does + not specify to create one. It should never return ``None``. + +.. function:: set_event_loop(loop) + + XXX + +.. function:: new_event_loop() + + XXX + + +Event loop policy +----------------- + +.. function:: get_event_loop_policy() + + XXX + +.. function:: set_event_loop_policy(policy) + + XXX + + +Run an event loop +----------------- + +.. method:: BaseEventLoop.run_forever() + + Run until :meth:`stop` is called. + +.. method:: BaseEventLoop.run_until_complete(future) + + Run until the :class:`~concurrent.futures.Future` is done. + + If the argument is a coroutine, it is wrapped in a :class:`Task`. + + Return the Future's result, or raise its exception. + +.. method:: BaseEventLoop.is_running() + + Returns running status of event loop. + +.. method:: stop() + + Stop running the event loop. + + Every callback scheduled before :meth:`stop` is called will run. + Callback scheduled after :meth:`stop` is called won't. However, those + callbacks will run if :meth:`run_forever` is called again later. + +.. method:: BaseEventLoop.close() + + Close the event loop. The loop should not be running. + + This clears the queues and shuts down the executor, but does not wait for + the executor to finish. + + This is idempotent and irreversible. No other methods should be called after + this one. + + +Calls +----- + +.. method:: BaseEventLoop.call_soon(callback, \*args) + + Arrange for a callback to be called as soon as possible. + + This operates as a FIFO queue, callbacks are called in the order in + which they are registered. Each callback will be called exactly once. + + Any positional arguments after the callback will be passed to the + callback when it is called. + +.. method:: BaseEventLoop.call_soon_threadsafe(callback, \*args) + + Like :meth:`call_soon`, but thread safe. + + +Delayed calls +------------- + +The event loop has its own internal clock for computing timeouts. +Which clock is used depends on the (platform-specific) event loop +implementation; ideally it is a monotonic clock. This will generally be +a different clock than :func:`time.time`. + +.. method:: BaseEventLoop.call_later(delay, callback, *args) + + Arrange for the *callback* to be called after the given *delay* + seconds (either an int or float). + + A "handle" is returned: an opaque object with a :meth:`cancel` method + that can be used to cancel the call. + + *callback* will be called exactly once per call to :meth:`call_later`. + If two callbacks are scheduled for exactly the same time, it is + undefined which will be called first. + + The optional positional *args* will be passed to the callback when it + is called. If you want the callback to be called with some named + arguments, use a closure or :func:`functools.partial`. + +.. method:: BaseEventLoop.call_at(when, callback, *args) + + Arrange for the *callback* to be called at the given absolute timestamp + *when* (an int or float), using the same time reference as :meth:`time`. + + This method's behavior is the same as :meth:`call_later`. + +.. method:: BaseEventLoop.time() + + Return the current time, as a :class:`float` value, according to the + event loop's internal clock. + + +Creating connections +^^^^^^^^^^^^^^^^^^^^ + +.. method:: BaseEventLoop.create_connection(protocol_factory, host=None, port=None, \*, ssl=None, family=0, proto=0, flags=0, sock=None, local_addr=None, server_hostname=None) + + Create a streaming transport connection to a given Internet *host* and + *port*. *protocol_factory* must be a callable returning a + :ref:`protocol ` instance. + + This method returns a :ref:`coroutine ` which will try to + establish the connection in the background. When successful, the + coroutine returns a ``(transport, protocol)`` pair. + + The chronological synopsis of the underlying operation is as follows: + + #. The connection is established, and a :ref:`transport ` + is created to represent it. + + #. *protocol_factory* is called without arguments and must return a + :ref:`protocol ` instance. + + #. The protocol instance is tied to the transport, and its + :meth:`connection_made` method is called. + + #. The coroutine returns successfully with the ``(transport, protocol)`` + pair. + + The created transport is an implementation-dependent bidirectional stream. + + .. note:: + *protocol_factory* can be any kind of callable, not necessarily + a class. For example, if you want to use a pre-created + protocol instance, you can pass ``lambda: my_protocol``. + + Options allowing to change how the connection is created: + + * *ssl*: if given and not false, a SSL/TLS transport is created + (by default a plain TCP transport is created). If *ssl* is + a :class:`ssl.SSLContext` object, this context is used to create + the transport; if *ssl* is :const:`True`, a context with some + unspecified default settings is used. + + * *server_hostname*, is only for use together with *ssl*, + and sets or overrides the hostname that the target server's certificate + will be matched against. By default the value of the *host* argument + is used. If *host* is empty, there is no default and you must pass a + value for *server_hostname*. If *server_hostname* is an empty + string, hostname matching is disabled (which is a serious security + risk, allowing for man-in-the-middle-attacks). + + * *family*, *proto*, *flags* are the optional address family, protocol + and flags to be passed through to getaddrinfo() for *host* resolution. + If given, these should all be integers from the corresponding + :mod:`socket` module constants. + + * *sock*, if given, should be an existing, already connected + :class:`socket.socket` object to be used by the transport. + If *sock* is given, none of *host*, *port*, *family*, *proto*, *flags* + and *local_addr* should be specified. + + * *local_addr*, if given, is a ``(local_host, local_port)`` tuple used + to bind the socket to locally. The *local_host* and *local_port* + are looked up using getaddrinfo(), similarly to *host* and *port*. + + +Creating listening connections +------------------------------ + +.. method:: BaseEventLoop.create_server(protocol_factory, host=None, port=None, \*, family=socket.AF_UNSPEC, flags=socket.AI_PASSIVE, sock=None, backlog=100, ssl=None, reuse_address=None) + + A :ref:`coroutine ` which creates a TCP server bound to host and + port. + + The return value is a :class:`AbstractServer` object which can be used to stop + the service. + + If *host* is an empty string or None all interfaces are assumed + and a list of multiple sockets will be returned (most likely + one for IPv4 and another one for IPv6). + + *family* can be set to either :data:`~socket.AF_INET` or + :data:`~socket.AF_INET6` to force the socket to use IPv4 or IPv6. If not set + it will be determined from host (defaults to :data:`~socket.AF_UNSPEC`). + + *flags* is a bitmask for :meth:`getaddrinfo`. + + *sock* can optionally be specified in order to use a preexisting + socket object. + + *backlog* is the maximum number of queued connections passed to + :meth:`~socket.socket.listen` (defaults to 100). + + ssl can be set to an :class:`~ssl.SSLContext` to enable SSL over the + accepted connections. + + *reuse_address* tells the kernel to reuse a local socket in + TIME_WAIT state, without waiting for its natural timeout to + expire. If not specified will automatically be set to True on + UNIX. + + This method returns a :ref:`coroutine `. + +.. method:: BaseEventLoop.create_datagram_endpoint(protocol_factory, local_addr=None, remote_addr=None, \*, family=0, proto=0, flags=0) + + Create datagram connection. + + This method returns a :ref:`coroutine `. + + + +Resolve name +------------ + +.. method:: BaseEventLoop.getaddrinfo(host, port, \*, family=0, type=0, proto=0, flags=0) + + XXX + +.. method:: BaseEventLoop.getnameinfo(sockaddr, flags=0) + + XXX + + +Running subprocesses +-------------------- + +Run subprocesses asynchronously using the :mod:`subprocess` module. + +.. method:: BaseEventLoop.subprocess_exec(protocol_factory, \*args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=False, bufsize=0, \*\*kwargs) + + XXX + + This method returns a :ref:`coroutine `. + + See the constructor of the :class:`subprocess.Popen` class for parameters. + +.. method:: BaseEventLoop.subprocess_shell(protocol_factory, cmd, \*, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=True, bufsize=0, \*\*kwargs) + + XXX + + This method returns a :ref:`coroutine `. + + See the constructor of the :class:`subprocess.Popen` class for parameters. + +.. method:: BaseEventLoop.connect_read_pipe(protocol_factory, pipe) + + Register read pipe in eventloop. + + *protocol_factory* should instantiate object with :class:`Protocol` + interface. pipe is file-like object already switched to nonblocking. + Return pair (transport, protocol), where transport support + :class:`ReadTransport` interface. + + This method returns a :ref:`coroutine `. + +.. method:: BaseEventLoop.connect_write_pipe(protocol_factory, pipe) + + Register write pipe in eventloop. + + *protocol_factory* should instantiate object with :class:`BaseProtocol` + interface. Pipe is file-like object already switched to nonblocking. + Return pair (transport, protocol), where transport support + :class:`WriteTransport` interface. + + This method returns a :ref:`coroutine `. + + +Executor +-------- + +Call a function in an :class:`~concurrent.futures.Executor` (pool of threads or +pool of processes). By default, an event loop uses a thread pool executor +(:class:`~concurrent.futures.ThreadPoolExecutor`). + +.. method:: BaseEventLoop.run_in_executor(executor, callback, \*args) + + Arrange for a callback to be called in the specified executor. + + *executor* is a :class:`~concurrent.futures.Executor` instance, + the default executor is used if *executor* is ``None``. + +.. method:: BaseEventLoop.set_default_executor(executor) + + Set the default executor used by :meth:`run_in_executor`. + + +Examples +-------- + +Hello World (callback) +^^^^^^^^^^^^^^^^^^^^^^ + +Print ``Hello World`` every two seconds, using a callback:: + + import asyncio + + def print_and_repeat(loop): + print('Hello World') + loop.call_later(2, print_and_repeat, loop) + + loop = asyncio.get_event_loop() + print_and_repeat(loop) + loop.run_forever() + + +Hello World (coroutine) +^^^^^^^^^^^^^^^^^^^^^^^ + +Print ``Hello World`` every two seconds, using a coroutine:: + + import asyncio + + @asyncio.coroutine + def greet_every_two_seconds(): + while True: + print('Hello World') + yield from asyncio.sleep(2) + + loop = asyncio.get_event_loop() + loop.run_until_complete(greet_every_two_seconds()) + diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst new file mode 100644 --- /dev/null +++ b/Doc/library/asyncio-protocol.rst @@ -0,0 +1,615 @@ +.. module:: asyncio + +.. _transport: + +Transports +========== + +Transports are classed provided by :mod:`asyncio` in order to abstract +various kinds of communication channels. You generally won't instantiate +a transport yourself; instead, you will call a :class:`BaseEventLoop` method +which will create the transport and try to initiate the underlying +communication channel, calling you back when it succeeds. + +Once the communication channel is established, a transport is always +paired with a :ref:`protocol ` instance. The protocol can +then call the transport's methods for various purposes. + +:mod:`asyncio` currently implements transports for TCP, UDP, SSL, and +subprocess pipes. The methods available on a transport depend on +the transport's kind. + + +BaseTransport: Methods common to all transports +----------------------------------------------- + +.. class:: BaseTransport + + Base class for transports. + + .. method:: close(self) + + Close the transport. If the transport has a buffer for outgoing + data, buffered data will be flushed asynchronously. No more data + will be received. After all buffered data is flushed, the + protocol's :meth:`connection_lost` method will be called with + :const:`None` as its argument. + + + .. method:: get_extra_info(name, default=None) + + Return optional transport information. *name* is a string representing + the piece of transport-specific information to get, *default* is the + value to return if the information doesn't exist. + + This method allows transport implementations to easily expose + channel-specific information. + + * socket: + + - ``'peername'``: the remote address to which the socket is connected, + result of :meth:`socket.socket.getpeername` (``None`` on error) + - ``'socket'``: :class:`socket.socket` instance + - ``'sockname'``: the socket's own address, + result of :meth:`socket.socket.getsockname` + + * SSL socket: + + - ``'compression'``: the compression algorithm being used as a string, + or ``None`` if the connection isn't compressed; result of + :meth:`ssl.SSLSocket.compression` + - ``'cipher'``: a three-value tuple containing the name of the cipher + being used, the version of the SSL protocol that defines its use, and + the number of secret bits being used; result of + :meth:`ssl.SSLSocket.cipher` + - ``'peercert'``: peer certificate; result of + :meth:`ssl.SSLSocket.getpeercert` + - ``'sslcontext'``: :class:`ssl.SSLContext` instance + + * pipe: + + - ``'pipe'``: pipe object + + * subprocess: + + - ``'subprocess'``: :class:`subprocess.Popen` instance + + +ReadTransport: Methods of readable streaming transports +------------------------------------------------------- + +.. class:: ReadTransport + + Interface for read-only transports. + + .. method:: pause_reading() + + Pause the receiving end of the transport. No data will be passed to + the protocol's :meth:`data_received` method until meth:`resume_reading` + is called. + + .. method:: resume_reading() + + Resume the receiving end. The protocol's :meth:`data_received` method + will be called once again if some data is available for reading. + + +WriteTransport: Methods of writable streaming transports +-------------------------------------------------------- + +.. class:: WriteTransport + + Interface for write-only transports. + + .. method:: abort() + + Close the transport immediately, without waiting for pending operations + to complete. Buffered data will be lost. No more data will be received. + The protocol's :meth:`connection_lost` method will eventually be + called with :const:`None` as its argument. + + .. method:: can_write_eof() + + Return :const:`True` if the transport supports :meth:`write_eof`, + :const:`False` if not. + + .. method:: get_write_buffer_size() + + Return the current size of the output buffer used by the transport. + + .. method:: set_write_buffer_limits(high=None, low=None) + + Set the *high*- and *low*-water limits for write flow control. + + These two values control when call the protocol's + :meth:`pause_writing` and :meth:`resume_writing` methods are called. + If specified, the low-water limit must be less than or equal to the + high-water limit. Neither *high* nor *low* can be negative. + + The defaults are implementation-specific. If only the + high-water limit is given, the low-water limit defaults to a + implementation-specific value less than or equal to the + high-water limit. Setting *high* to zero forces *low* to zero as + well, and causes :meth:`pause_writing` to be called whenever the + buffer becomes non-empty. Setting *low* to zero causes + :meth:`resume_writing` to be called only once the buffer is empty. + Use of zero for either limit is generally sub-optimal as it + reduces opportunities for doing I/O and computation + concurrently. + + .. method:: write(data) + + Write some *data* bytes to the transport. + + This method does not block; it buffers the data and arranges for it + to be sent out asynchronously. + + .. method:: writelines(list_of_data) + + Write a list (or any iterable) of data bytes to the transport. + This is functionally equivalent to calling :meth:`write` on each + element yielded by the iterable, but may be implemented more efficiently. + + .. method:: write_eof() + + Close the write end of the transport after flushing buffered data. + Data may still be received. + + This method can raise :exc:`NotImplementedError` if the transport + (e.g. SSL) doesn't support half-closes. + + +DatagramTransport: Methods of datagram transports +------------------------------------------------- + +.. method:: DatagramTransport.sendto(data, addr=None) + + Send the *data* bytes to the remote peer given by *addr* (a + transport-dependent target address). If *addr* is :const:`None`, the + data is sent to the target address given on transport creation. + + This method does not block; it buffers the data and arranges for it + to be sent out asynchronously. + +.. method:: DatagramTransport.abort() + + Close the transport immediately, without waiting for pending operations + to complete. Buffered data will be lost. No more data will be received. + The protocol's :meth:`connection_lost` method will eventually be + called with :const:`None` as its argument. + + +Methods of subprocess transports +-------------------------------- + +.. class:: BaseSubprocessTransport + + .. method:: get_pid() + + Return the subprocess process id as an integer. + + .. method:: get_returncode() + + Return the subprocess returncode as an integer or :const:`None` + if it hasn't returned, similarly to the + :attr:`subprocess.Popen.returncode` attribute. + + .. method:: get_pipe_transport(fd) + + Return the transport for the communication pipe correspondong to the + integer file descriptor *fd*. The return value can be a readable or + writable streaming transport, depending on the *fd*. If *fd* doesn't + correspond to a pipe belonging to this transport, :const:`None` is + returned. + + .. method:: send_signal(signal) + + Send the *signal* number to the subprocess, as in + :meth:`subprocess.Popen.send_signal`. + + .. method:: terminate() + + Ask the subprocess to stop, as in :meth:`subprocess.Popen.terminate`. + This method is an alias for the :meth:`close` method. + + On POSIX systems, this method sends SIGTERM to the subprocess. + On Windows, the Windows API function TerminateProcess() is called to + stop the subprocess. + + .. method:: kill(self) + + Kill the subprocess, as in :meth:`subprocess.Popen.kill` + + On POSIX systems, the function sends SIGKILL to the subprocess. + On Windows, this method is an alias for :meth:`terminate`. + + +Stream reader +------------- + +.. class:: StreamWriter(transport, protocol, reader, loop) + + Wraps a Transport. + + This exposes :meth:`write`, :meth:`writelines`, :meth:`can_write_eof()`, :meth:`write_eof`, :meth:`get_extra_info` and + :meth:`close`. It adds :meth:`drain` which returns an optional :class:`~concurrent.futures.Future` on which you can + wait for flow control. It also adds a transport attribute which references + the :class:`Transport` directly. + + .. attribute:: transport + + Transport. + + .. method:: close() + + Close the transport: see :meth:`BaseTransport.close`. + + .. method:: drain() + + This method has an unusual return value. + + The intended use is to write:: + + w.write(data) + yield from w.drain() + + When there's nothing to wait for, :meth:`drain()` returns ``()``, and the + yield-from continues immediately. When the transport buffer is full (the + protocol is paused), :meth:`drain` creates and returns a + :class:`~concurrent.futures.Future` and the yield-from will block until + that Future is completed, which will happen when the buffer is + (partially) drained and the protocol is resumed. + + .. method:: get_extra_info(name, default=None) + + Return optional transport information: see + :meth:`BaseTransport.get_extra_info`. + + .. method:: write(data) + + Write some *data* bytes to the transport: see + :meth:`WriteTransport.write`. + + .. method:: writelines(data) + + Write a list (or any iterable) of data bytes to the transport: + see :meth:`WriteTransport.writelines`. + + .. method:: can_write_eof() + + Return :const:`True` if the transport supports :meth:`write_eof`, + :const:`False` if not. See :meth:`WriteTransport.can_write_eof`. + + .. method:: write_eof() + + Close the write end of the transport after flushing buffered data: + see :meth:`WriteTransport.write_eof`. + + +Stream writer +------------- + +.. class:: StreamReader(limit=_DEFAULT_LIMIT, loop=None) + + .. method:: exception() + + Get the exception. + + .. method:: feed_eof() + + XXX + + .. method:: feed_data(data) + + XXX + + .. method:: set_exception(exc) + + Set the exception. + + .. method:: set_transport(transport) + + Set the transport. + + .. method:: read(n=-1) + + XXX + + This method returns a :ref:`coroutine `. + + .. method:: readline() + + XXX + + This method returns a :ref:`coroutine `. + + .. method:: readexactly(n) + + XXX + + This method returns a :ref:`coroutine `. + + + +.. _protocol: + +Protocols +========= + +:mod:`asyncio` provides base classes that you can subclass to implement +your network protocols. Those classes are used in conjunction with +:ref:`transports ` (see below): the protocol parses incoming +data and asks for the writing of outgoing data, while the transport is +responsible for the actual I/O and buffering. + +When subclassing a protocol class, it is recommended you override certain +methods. Those methods are callbacks: they will be called by the transport +on certain events (for example when some data is received); you shouldn't +call them yourself, unless you are implementing a transport. + +.. note:: + All callbacks have default implementations, which are empty. Therefore, + you only need to implement the callbacks for the events in which you + are interested. + + +Protocol classes +---------------- + +.. class:: Protocol + + The base class for implementing streaming protocols (for use with + e.g. TCP and SSL transports). + +.. class:: DatagramProtocol + + The base class for implementing datagram protocols (for use with + e.g. UDP transports). + +.. class:: SubprocessProtocol + + The base class for implementing protocols communicating with child + processes (through a set of unidirectional pipes). + + +Connection callbacks +-------------------- + +These callbacks may be called on :class:`Protocol` and +:class:`SubprocessProtocol` instances: + +.. method:: BaseProtocol.connection_made(transport) + + Called when a connection is made. + + The *transport* argument is the transport representing the + connection. You are responsible for storing it somewhere + (e.g. as an attribute) if you need to. + +.. method:: BaseProtocol.connection_lost(exc) + + Called when the connection is lost or closed. + + The argument is either an exception object or :const:`None`. + The latter means a regular EOF is received, or the connection was + aborted or closed by this side of the connection. + +:meth:`connection_made` and :meth:`connection_lost` are called exactly once +per successful connection. All other callbacks will be called between those +two methods, which allows for easier resource management in your protocol +implementation. + +The following callbacks may be called only on :class:`SubprocessProtocol` +instances: + +.. method:: SubprocessProtocol.pipe_data_received(fd, data) + + Called when the child process writes data into its stdout or stderr pipe. + *fd* is the integer file descriptor of the pipe. *data* is a non-empty + bytes object containing the data. + +.. method:: SubprocessProtocol.pipe_connection_lost(fd, exc) + + Called when one of the pipes communicating with the child process + is closed. *fd* is the integer file descriptor that was closed. + +.. method:: SubprocessProtocol.process_exited() + + Called when the child process has exited. + + +Data reception callbacks +------------------------ + +Streaming protocols +^^^^^^^^^^^^^^^^^^^ + +The following callbacks are called on :class:`Protocol` instances: + +.. method:: Protocol.data_received(data) + + Called when some data is received. *data* is a non-empty bytes object + containing the incoming data. + + .. note:: + Whether the data is buffered, chunked or reassembled depends on + the transport. In general, you shouldn't rely on specific semantics + and instead make your parsing generic and flexible enough. However, + data is always received in the correct order. + +.. method:: Protocol.eof_received() + + Calls when the other end signals it won't send any more data + (for example by calling :meth:`write_eof`, if the other end also uses + asyncio). + + This method may return a false value (including None), in which case + the transport will close itself. Conversely, if this method returns a + true value, closing the transport is up to the protocol. Since the + default implementation returns None, it implicitly closes the connection. + + .. note:: + Some transports such as SSL don't support half-closed connections, + in which case returning true from this method will not prevent closing + the connection. + +:meth:`data_received` can be called an arbitrary number of times during +a connection. However, :meth:`eof_received` is called at most once +and, if called, :meth:`data_received` won't be called after it. + +Datagram protocols +^^^^^^^^^^^^^^^^^^ + +The following callbacks are called on :class:`DatagramProtocol` instances. + +.. method:: DatagramProtocol.datagram_received(data, addr) + + Called when a datagram is received. *data* is a bytes object containing + the incoming data. *addr* is the address of the peer sending the data; + the exact format depends on the transport. + +.. method:: DatagramProtocol.error_received(exc) + + Called when a previous send or receive operation raises an + :class:`OSError`. *exc* is the :class:`OSError` instance. + + This method is called in rare conditions, when the transport (e.g. UDP) + detects that a datagram couldn't be delivered to its recipient. + In many conditions though, undeliverable datagrams will be silently + dropped. + + +Flow control callbacks +---------------------- + +These callbacks may be called on :class:`Protocol` and +:class:`SubprocessProtocol` instances: + +.. method:: BaseProtocol.pause_writing() + + Called when the transport's buffer goes over the high-water mark. + +.. method:: BaseProtocol.resume_writing() + + Called when the transport's buffer drains below the low-water mark. + + +:meth:`pause_writing` and :meth:`resume_writing` calls are paired -- +:meth:`pause_writing` is called once when the buffer goes strictly over +the high-water mark (even if subsequent writes increases the buffer size +even more), and eventually :meth:`resume_writing` is called once when the +buffer size reaches the low-water mark. + +.. note:: + If the buffer size equals the high-water mark, + :meth:`pause_writing` is not called -- it must go strictly over. + Conversely, :meth:`resume_writing` is called when the buffer size is + equal or lower than the low-water mark. These end conditions + are important to ensure that things go as expected when either + mark is zero. + + +Server +------ + +.. class:: AbstractServer + + Abstract server returned by create_service(). + + .. method:: close() + + Stop serving. This leaves existing connections open. + + .. method:: wait_closed() + + Coroutine to wait until service is closed. + + +Network functions +================= + +.. function:: open_connection(host=None, port=None, *, loop=None, limit=_DEFAULT_LIMIT, **kwds) + + A wrapper for create_connection() returning a (reader, writer) pair. + + The reader returned is a StreamReader instance; the writer is a + :class:`Transport`. + + The arguments are all the usual arguments to + :meth:`BaseEventLoop.create_connection` except *protocol_factory*; most + common are positional host and port, with various optional keyword arguments + following. + + Additional optional keyword arguments are *loop* (to set the event loop + instance to use) and *limit* (to set the buffer limit passed to the + StreamReader). + + (If you want to customize the :class:`StreamReader` and/or + :class:`StreamReaderProtocol` classes, just copy the code -- there's really + nothing special here except some convenience.) + + This function returns a :ref:`coroutine `. + +.. function:: start_server(client_connected_cb, host=None, port=None, *, loop=None, limit=_DEFAULT_LIMIT, **kwds) + + Start a socket server, call back for each client connected. + + The first parameter, *client_connected_cb*, takes two parameters: + *client_reader*, *client_writer*. *client_reader* is a + :class:`StreamReader` object, while *client_writer* is a + :class:`StreamWriter` object. This parameter can either be a plain callback + function or a :ref:`coroutine `; if it is a coroutine, it will be + automatically converted into a :class:`Task`. + + The rest of the arguments are all the usual arguments to + :meth:`~BaseEventLoop.create_server()` except *protocol_factory*; most + common are positional host and port, with various optional keyword arguments + following. The return value is the same as + :meth:`~BaseEventLoop.create_server()`. + + Additional optional keyword arguments are *loop* (to set the event loop + instance to use) and *limit* (to set the buffer limit passed to the + :class:`StreamReader`). + + The return value is the same as :meth:`~BaseEventLoop.create_server()`, i.e. + a :class:`AbstractServer` object which can be used to stop the service. + + This function returns a :ref:`coroutine `. + +Example: Echo server +-------------------- + +A :class:`Protocol` implementing an echo server:: + + class EchoServer(asyncio.Protocol): + + TIMEOUT = 5.0 + + def timeout(self): + print('connection timeout, closing.') + self.transport.close() + + def connection_made(self, transport): + print('connection made') + self.transport = transport + + # start 5 seconds timeout timer + self.h_timeout = asyncio.get_event_loop().call_later( + self.TIMEOUT, self.timeout) + + def data_received(self, data): + print('data received: ', data.decode()) + self.transport.write(b'Re: ' + data) + + # restart timeout timer + self.h_timeout.cancel() + self.h_timeout = asyncio.get_event_loop().call_later( + self.TIMEOUT, self.timeout) + + def eof_received(self): + pass + + def connection_lost(self, exc): + print('connection lost:', exc) + self.h_timeout.cancel() + diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst new file mode 100644 --- /dev/null +++ b/Doc/library/asyncio-sync.rst @@ -0,0 +1,341 @@ +.. _sync: + +Synchronization primitives +========================== + +Locks +----- + +.. class:: Lock(\*, loop=None) + + Primitive lock objects. + + A primitive lock is a synchronization primitive that is not owned by a + particular coroutine when locked. A primitive lock is in one of two states, + 'locked' or 'unlocked'. + + It is created in the unlocked state. It has two basic methods, :meth:`acquire` + and :meth:`release`. When the state is unlocked, acquire() changes the state to + locked and returns immediately. When the state is locked, acquire() blocks + until a call to release() in another coroutine changes it to unlocked, then + the acquire() call resets it to locked and returns. The release() method + should only be called in the locked state; it changes the state to unlocked + and returns immediately. If an attempt is made to release an unlocked lock, + a :exc:`RuntimeError` will be raised. + + When more than one coroutine is blocked in acquire() waiting for the state + to turn to unlocked, only one coroutine proceeds when a release() call + resets the state to unlocked; first coroutine which is blocked in acquire() + is being processed. + + :meth:`acquire` is a coroutine and should be called with ``yield from``. + + Locks also support the context manager protocol. ``(yield from lock)`` + should be used as context manager expression. + + Usage:: + + lock = Lock() + ... + yield from lock + try: + ... + finally: + lock.release() + + Context manager usage:: + + lock = Lock() + ... + with (yield from lock): + ... + + Lock objects can be tested for locking state:: + + if not lock.locked(): + yield from lock + else: + # lock is acquired + ... + + .. method:: locked() + + Return ``True`` if lock is acquired. + + .. method:: acquire() + + Acquire a lock. + + This method blocks until the lock is unlocked, then sets it to locked and + returns ``True``. + + This method returns a :ref:`coroutine `. + + .. method:: release() + + Release a lock. + + When the lock is locked, reset it to unlocked, and return. If any other + coroutines are blocked waiting for the lock to become unlocked, allow + exactly one of them to proceed. + + When invoked on an unlocked lock, a :exc:`RuntimeError` is raised. + + There is no return value. + + +.. class:: Event(\*, loop=None) + + An Event implementation, asynchronous equivalent to :class:`threading.Event`. + + Class implementing event objects. An event manages a flag that can be set to + true with the :meth:`set` method and reset to false with the :meth:`clear` + method. The :meth:`wait` method blocks until the flag is true. The flag is + initially false. + + .. method:: clear() + + Reset the internal flag to false. Subsequently, coroutines calling + :meth:`wait` will block until :meth:`set` is called to set the internal + flag to true again. + + .. method:: is_set() + + Return ``True`` if and only if the internal flag is true. + + .. method:: set() + + Set the internal flag to true. All coroutines waiting for it to become + true are awakened. Coroutine that call :meth:`wait` once the flag is true + will not block at all. + + .. method:: wait() + + Block until the internal flag is true. + + If the internal flag is true on entry, return ``True`` immediately. + Otherwise, block until another coroutine calls :meth:`set` to set the + flag to true, then return ``True``. + + This method returns a :ref:`coroutine `. + + +.. class:: Condition(\*, loop=None) + + A Condition implementation, asynchronous equivalent to + :class:`threading.Condition`. + + This class implements condition variable objects. A condition variable + allows one or more coroutines to wait until they are notified by another + coroutine. + + A new :class:`Lock` object is created and used as the underlying lock. + + .. method:: notify(n=1) + + By default, wake up one coroutine waiting on this condition, if any. + If the calling coroutine has not acquired the lock when this method is + called, a :exc:`RuntimeError` is raised. + + This method wakes up at most *n* of the coroutines waiting for the + condition variable; it is a no-op if no coroutines are waiting. + + .. note:: + + An awakened coroutine does not actually return from its :meth:`wait` + call until it can reacquire the lock. Since :meth:`notify` does not + release the lock, its caller should. + + .. method:: notify_all() + + Wake up all threads waiting on this condition. This method acts like + :meth:`notify`, but wakes up all waiting threads instead of one. If the + calling thread has not acquired the lock when this method is called, a + :exc:`RuntimeError` is raised. + + .. method:: wait() + + Wait until notified. + + If the calling coroutine has not acquired the lock when this method is + called, a :exc:`RuntimeError` is raised. + + This method releases the underlying lock, and then blocks until it is + awakened by a :meth:`notify` or :meth:`notify_all` call for the same + condition variable in another coroutine. Once awakened, it re-acquires + the lock and returns ``True``. + + This method returns a :ref:`coroutine `. + + .. method:: wait_for(predicate) + + Wait until a predicate becomes true. + + The predicate should be a callable which result will be interpreted as a + boolean value. The final predicate value is the return value. + + This method returns a :ref:`coroutine `. + + +Semaphores +---------- + +.. class:: Semaphore(value=1, \*, loop=None) + + A Semaphore implementation. + + A semaphore manages an internal counter which is decremented by each + :meth:`acquire` call and incremented by each :meth:`release` call. The + counter can never go below zero; when :meth:`acquire` finds that it is zero, + it blocks, waiting until some other thread calls :meth:`release`. + + Semaphores also support the context manager protocol. + + The optional argument gives the initial value for the internal counter; it + defaults to ``1``. If the value given is less than ``0``, :exc:`ValueError` + is raised. + + .. method:: acquire() + + Acquire a semaphore. + + If the internal counter is larger than zero on entry, decrement it by one + and return ``True`` immediately. If it is zero on entry, block, waiting + until some other coroutine has called :meth:`release` to make it larger + than ``0``, and then return ``True``. + + This method returns a :ref:`coroutine `. + + .. method:: locked() + + Returns ``True`` if semaphore can not be acquired immediately. + + .. method:: release() + + Release a semaphore, incrementing the internal counter by one. When it + was zero on entry and another coroutine is waiting for it to become + larger than zero again, wake up that coroutine. + + +.. class:: BoundedSemaphore(value=1, \*, loop=None) + + A bounded semaphore implementation. Inherit from :class:`Semaphore`. + + This raises :exc:`ValueError` in :meth:`~Semaphore.release` if it would + increase the value above the initial value. + + +Queues +------ + +.. class:: Queue(maxsize=0, \*, loop=None) + + A queue, useful for coordinating producer and consumer coroutines. + + If *maxsize* is less than or equal to zero, the queue size is infinite. If + it is an integer greater than ``0``, then ``yield from put()`` will block + when the queue reaches *maxsize*, until an item is removed by :meth:`get`. + + Unlike the standard library :mod:`queue`, you can reliably know this Queue's + size with :meth:`qsize`, since your single-threaded Tulip application won't + be interrupted between calling :meth:`qsize` and doing an operation on the + Queue. + + .. method:: empty() + + Return ``True`` if the queue is empty, ``False`` otherwise. + + .. method:: full() + + Return ``True`` if there are maxsize items in the queue. + + .. note:: + + If the Queue was initialized with ``maxsize=0`` (the default), then + :meth:`full()` is never ``True``. + + .. method:: get() + + Remove and return an item from the queue. + + If you yield from :meth:`get()`, wait until a item is available. + + This method returns a :ref:`coroutine `. + + .. method:: get_nowait() + + Remove and return an item from the queue. + + Return an item if one is immediately available, else raise + :exc:`~queue.Empty`. + + .. method:: put(item) + + Put an item into the queue. + + If you yield from ``put()``, wait until a free slot is available before + adding item. + + This method returns a :ref:`coroutine `. + + .. method:: put_nowait(item) + + Put an item into the queue without blocking. + + If no free slot is immediately available, raise :exc:`~queue.Full`. + + .. method:: qsize() + + Number of items in the queue. + + .. attribute:: maxsize + + Number of items allowed in the queue. + + +.. class:: PriorityQueue + + A subclass of :class:`Queue`; retrieves entries in priority order (lowest + first). + + Entries are typically tuples of the form: (priority number, data). + + +.. class:: LifoQueue + + A subclass of :class:`Queue` that retrieves most recently added entries + first. + + +.. class:: JoinableQueue + + A subclass of :class:`Queue` with :meth:`task_done` and :meth:`join` + methods. + + .. method:: join() + + Block until all items in the queue have been gotten and processed. + + The count of unfinished tasks goes up whenever an item is added to the + queue. The count goes down whenever a consumer thread calls + :meth:`task_done` to indicate that the item was retrieved and all work on + it is complete. When the count of unfinished tasks drops to zero, + :meth:`join` unblocks. + + This method returns a :ref:`coroutine `. + + .. method:: task_done() + + Indicate that a formerly enqueued task is complete. + + Used by queue consumers. For each :meth:`~Queue.get` used to fetch a task, a + subsequent call to :meth:`task_done` tells the queue that the processing + on the task is complete. + + If a :meth:`join` is currently blocking, it will resume when all items + have been processed (meaning that a :meth:`task_done` call was received + for every item that had been :meth:`~Queue.put` into the queue). + + Raises :exc:`ValueError` if called more times than there were items + placed in the queue. + diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst new file mode 100644 --- /dev/null +++ b/Doc/library/asyncio-task.rst @@ -0,0 +1,232 @@ +.. module:: asyncio + +Tasks and coroutines +==================== + +.. _coroutine: + +Coroutines +---------- + +A coroutine is a generator that follows certain conventions. For +documentation purposes, all coroutines should be decorated with +``@asyncio.coroutine``, but this cannot be strictly enforced. + +Coroutines use the ``yield from`` syntax introduced in :pep:`380`, +instead of the original ``yield`` syntax. + +The word "coroutine", like the word "generator", is used for two +different (though related) concepts: + +- The function that defines a coroutine (a function definition + decorated with ``asyncio.coroutine``). If disambiguation is needed + we will call this a *coroutine function*. + +- The object obtained by calling a coroutine function. This object + represents a computation or an I/O operation (usually a combination) + that will complete eventually. If disambiguation is needed we will + call it a *coroutine object*. + +Things a coroutine can do: + +- ``result = yield from future`` -- suspends the coroutine until the + future is done, then returns the future's result, or raises an + exception, which will be propagated. (If the future is cancelled, + it will raise a ``CancelledError`` exception.) Note that tasks are + futures, and everything said about futures also applies to tasks. + +- ``result = yield from coroutine`` -- wait for another coroutine to + produce a result (or raise an exception, which will be propagated). + The ``coroutine`` expression must be a *call* to another coroutine. + +- ``return expression`` -- produce a result to the coroutine that is + waiting for this one using ``yield from``. + +- ``raise exception`` -- raise an exception in the coroutine that is + waiting for this one using ``yield from``. + +Calling a coroutine does not start its code running -- it is just a +generator, and the coroutine object returned by the call is really a +generator object, which doesn't do anything until you iterate over it. +In the case of a coroutine object, there are two basic ways to start +it running: call ``yield from coroutine`` from another coroutine +(assuming the other coroutine is already running!), or convert it to a +:class:`Task`. + +Coroutines (and tasks) can only run when the event loop is running. + + +Task +---- + +.. class:: Task(coro, \*, loop=None) + + A coroutine wrapped in a :class:`~concurrent.futures.Future`. + + .. classmethod:: all_tasks(loop=None) + + Return a set of all tasks for an event loop. + + By default all tasks for the current event loop are returned. + + .. method:: cancel() + + Cancel the task. + + .. method:: get_stack(self, \*, limit=None) + + Return the list of stack frames for this task's coroutine. + + If the coroutine is active, this returns the stack where it is suspended. + If the coroutine has completed successfully or was cancelled, this + returns an empty list. If the coroutine was terminated by an exception, + this returns the list of traceback frames. + + The frames are always ordered from oldest to newest. + + The optional limit gives the maximum nummber of frames to return; by + default all available frames are returned. Its meaning differs depending + on whether a stack or a traceback is returned: the newest frames of a + stack are returned, but the oldest frames of a traceback are returned. + (This matches the behavior of the traceback module.) + + For reasons beyond our control, only one stack frame is returned for a + suspended coroutine. + + .. method:: print_stack(\*, limit=None, file=None) + + Print the stack or traceback for this task's coroutine. + + This produces output similar to that of the traceback module, for the + frames retrieved by get_stack(). The limit argument is passed to + get_stack(). The file argument is an I/O stream to which the output + goes; by default it goes to sys.stderr. + + +Task functions +-------------- + +.. function:: as_completed(fs, *, loop=None, timeout=None) + + Return an iterator whose values, when waited for, are + :class:`~concurrent.futures.Future` instances. + + Raises :exc:`TimeoutError` if the timeout occurs before all Futures are done. + + Example:: + + for f in as_completed(fs): + result = yield from f # The 'yield from' may raise + # Use result + + .. note:: + + The futures ``f`` are not necessarily members of fs. + +.. function:: async(coro_or_future, *, loop=None) + + Wrap a :ref:`coroutine ` in a future. + + If the argument is a :class:`~concurrent.futures.Future`, it is returned + directly. + +.. function:: gather(*coros_or_futures, loop=None, return_exceptions=False) + + Return a future aggregating results from the given coroutines or futures. + + All futures must share the same event loop. If all the tasks are done + successfully, the returned future's result is the list of results (in the + order of the original sequence, not necessarily the order of results + arrival). If *result_exception* is True, exceptions in the tasks are + treated the same as successful results, and gathered in the result list; + otherwise, the first raised exception will be immediately propagated to the + returned future. + + Cancellation: if the outer Future is cancelled, all children (that have not + completed yet) are also cancelled. If any child is cancelled, this is + treated as if it raised :exc:`~concurrent.futures.CancelledError` -- the + outer Future is *not* cancelled in this case. (This is to prevent the + cancellation of one child to cause other children to be cancelled.) + +.. function:: tasks.iscoroutinefunction(func) + + Return ``True`` if *func* is a decorated coroutine function. + +.. function:: tasks.iscoroutine(obj) + + Return ``True`` if *obj* is a coroutine object. + +.. function:: sleep(delay, result=None, \*, loop=None) + + Create a :ref:`coroutine ` that completes after a given time + (in seconds). + +.. function:: shield(arg, \*, loop=None) + + Wait for a future, shielding it from cancellation. + + The statement:: + + res = yield from shield(something()) + + is exactly equivalent to the statement:: + + res = yield from something() + + *except* that if the coroutine containing it is cancelled, the task running + in ``something()`` is not cancelled. From the point of view of + ``something()``, the cancellation did not happen. But its caller is still + cancelled, so the yield-from expression still raises + :exc:`~concurrent.futures.CancelledError`. Note: If ``something()`` is + cancelled by other means this will still cancel ``shield()``. + + If you want to completely ignore cancellation (not recommended) you can + combine ``shield()`` with a try/except clause, as follows:: + + try: + res = yield from shield(something()) + except CancelledError: + res = None + +.. function:: wait(fs, \*, loop=None, timeout=None, return_when=ALL_COMPLETED) + + Wait for the Futures and coroutines given by fs to complete. Coroutines will + be wrapped in Tasks. Returns two sets of + :class:`~concurrent.futures.Future`: (done, pending). + + *timeout* can be used to control the maximum number of seconds to wait before + returning. *timeout* can be an int or float. If *timeout* is not specified + or ``None``, there is no limit to the wait time. + + *return_when* indicates when this function should return. It must be one of + the following constants of the :mod`concurrent.futures` module: + + .. tabularcolumns:: |l|L| + + +-----------------------------+----------------------------------------+ + | Constant | Description | + +=============================+========================================+ + | :const:`FIRST_COMPLETED` | The function will return when any | + | | future finishes or is cancelled. | + +-----------------------------+----------------------------------------+ + | :const:`FIRST_EXCEPTION` | The function will return when any | + | | future finishes by raising an | + | | exception. If no future raises an | + | | exception then it is equivalent to | + | | :const:`ALL_COMPLETED`. | + +-----------------------------+----------------------------------------+ + | :const:`ALL_COMPLETED` | The function will return when all | + | | futures finish or are cancelled. | + +-----------------------------+----------------------------------------+ + + This function returns a :ref:`coroutine `. + + Usage:: + + done, pending = yield from asyncio.wait(fs) + + .. note:: + + This does not raise :exc:`TimeoutError`! Futures that aren't done when + the timeout occurs are returned in the second set. + diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -49,1559 +49,12 @@ :PEP:`3156`. For a motivational primer on transports and protocols, see :PEP:`3153`. +.. toctree:: + :maxdepth: 3 + :numbered: -.. XXX should the asyncio documentation come in several pages, as for logging? + asyncio-eventloop.rst + asyncio-task.rst + asyncio-protocol.rst + asyncio-sync.rst -.. _event-loop: - -Event loops -=========== - -The event loop is the central execution device provided by :mod:`asyncio`. -It provides multiple facilities, amongst which: - -* Registering, executing and cancelling delayed calls (timeouts) - -* Creating client and server :ref:`transports ` for various - kinds of communication - -* Launching subprocesses and the associated :ref:`transports ` - for communication with an external program - -* Delegating costly function calls to a pool of threads - -Event loop functions --------------------- - -The easiest way to get an event loop is to call the :func:`get_event_loop` -function. - -.. function:: get_event_loop() - - Get the event loop for current context. Returns an event loop object - implementing :class:`BaseEventLoop` interface, or raises an exception in case no - event loop has been set for the current context and the current policy does - not specify to create one. It should never return ``None``. - -.. function:: set_event_loop(loop) - - XXX - -.. function:: new_event_loop() - - XXX - - -Event loop policy ------------------ - -.. function:: get_event_loop_policy() - - XXX - -.. function:: set_event_loop_policy(policy) - - XXX - - -Run an event loop ------------------ - -.. method:: BaseEventLoop.run_forever() - - Run until :meth:`stop` is called. - -.. method:: BaseEventLoop.run_until_complete(future) - - Run until the :class:`~concurrent.futures.Future` is done. - - If the argument is a coroutine, it is wrapped in a :class:`Task`. - - Return the Future's result, or raise its exception. - -.. method:: BaseEventLoop.is_running() - - Returns running status of event loop. - -.. method:: stop() - - Stop running the event loop. - - Every callback scheduled before :meth:`stop` is called will run. - Callback scheduled after :meth:`stop` is called won't. However, those - callbacks will run if :meth:`run_forever` is called again later. - -.. method:: BaseEventLoop.close() - - Close the event loop. The loop should not be running. - - This clears the queues and shuts down the executor, but does not wait for - the executor to finish. - - This is idempotent and irreversible. No other methods should be called after - this one. - - -Calls ------ - -.. method:: BaseEventLoop.call_soon(callback, \*args) - - Arrange for a callback to be called as soon as possible. - - This operates as a FIFO queue, callbacks are called in the order in - which they are registered. Each callback will be called exactly once. - - Any positional arguments after the callback will be passed to the - callback when it is called. - -.. method:: BaseEventLoop.call_soon_threadsafe(callback, \*args) - - Like :meth:`call_soon`, but thread safe. - - -Delayed calls -------------- - -The event loop has its own internal clock for computing timeouts. -Which clock is used depends on the (platform-specific) event loop -implementation; ideally it is a monotonic clock. This will generally be -a different clock than :func:`time.time`. - -.. method:: BaseEventLoop.call_later(delay, callback, *args) - - Arrange for the *callback* to be called after the given *delay* - seconds (either an int or float). - - A "handle" is returned: an opaque object with a :meth:`cancel` method - that can be used to cancel the call. - - *callback* will be called exactly once per call to :meth:`call_later`. - If two callbacks are scheduled for exactly the same time, it is - undefined which will be called first. - - The optional positional *args* will be passed to the callback when it - is called. If you want the callback to be called with some named - arguments, use a closure or :func:`functools.partial`. - -.. method:: BaseEventLoop.call_at(when, callback, *args) - - Arrange for the *callback* to be called at the given absolute timestamp - *when* (an int or float), using the same time reference as :meth:`time`. - - This method's behavior is the same as :meth:`call_later`. - -.. method:: BaseEventLoop.time() - - Return the current time, as a :class:`float` value, according to the - event loop's internal clock. - - -Creating connections -^^^^^^^^^^^^^^^^^^^^ - -.. method:: BaseEventLoop.create_connection(protocol_factory, host=None, port=None, \*, ssl=None, family=0, proto=0, flags=0, sock=None, local_addr=None, server_hostname=None) - - Create a streaming transport connection to a given Internet *host* and - *port*. *protocol_factory* must be a callable returning a - :ref:`protocol ` instance. - - This method returns a :ref:`coroutine ` which will try to - establish the connection in the background. When successful, the - coroutine returns a ``(transport, protocol)`` pair. - - The chronological synopsis of the underlying operation is as follows: - - #. The connection is established, and a :ref:`transport ` - is created to represent it. - - #. *protocol_factory* is called without arguments and must return a - :ref:`protocol ` instance. - - #. The protocol instance is tied to the transport, and its - :meth:`connection_made` method is called. - - #. The coroutine returns successfully with the ``(transport, protocol)`` - pair. - - The created transport is an implementation-dependent bidirectional stream. - - .. note:: - *protocol_factory* can be any kind of callable, not necessarily - a class. For example, if you want to use a pre-created - protocol instance, you can pass ``lambda: my_protocol``. - - Options allowing to change how the connection is created: - - * *ssl*: if given and not false, a SSL/TLS transport is created - (by default a plain TCP transport is created). If *ssl* is - a :class:`ssl.SSLContext` object, this context is used to create - the transport; if *ssl* is :const:`True`, a context with some - unspecified default settings is used. - - * *server_hostname*, is only for use together with *ssl*, - and sets or overrides the hostname that the target server's certificate - will be matched against. By default the value of the *host* argument - is used. If *host* is empty, there is no default and you must pass a - value for *server_hostname*. If *server_hostname* is an empty - string, hostname matching is disabled (which is a serious security - risk, allowing for man-in-the-middle-attacks). - - * *family*, *proto*, *flags* are the optional address family, protocol - and flags to be passed through to getaddrinfo() for *host* resolution. - If given, these should all be integers from the corresponding - :mod:`socket` module constants. - - * *sock*, if given, should be an existing, already connected - :class:`socket.socket` object to be used by the transport. - If *sock* is given, none of *host*, *port*, *family*, *proto*, *flags* - and *local_addr* should be specified. - - * *local_addr*, if given, is a ``(local_host, local_port)`` tuple used - to bind the socket to locally. The *local_host* and *local_port* - are looked up using getaddrinfo(), similarly to *host* and *port*. - - -Creating listening connections ------------------------------- - -.. method:: BaseEventLoop.create_server(protocol_factory, host=None, port=None, \*, family=socket.AF_UNSPEC, flags=socket.AI_PASSIVE, sock=None, backlog=100, ssl=None, reuse_address=None) - - A :ref:`coroutine ` which creates a TCP server bound to host and - port. - - The return value is a :class:`AbstractServer` object which can be used to stop - the service. - - If *host* is an empty string or None all interfaces are assumed - and a list of multiple sockets will be returned (most likely - one for IPv4 and another one for IPv6). - - *family* can be set to either :data:`~socket.AF_INET` or - :data:`~socket.AF_INET6` to force the socket to use IPv4 or IPv6. If not set - it will be determined from host (defaults to :data:`~socket.AF_UNSPEC`). - - *flags* is a bitmask for :meth:`getaddrinfo`. - - *sock* can optionally be specified in order to use a preexisting - socket object. - - *backlog* is the maximum number of queued connections passed to - :meth:`~socket.socket.listen` (defaults to 100). - - ssl can be set to an :class:`~ssl.SSLContext` to enable SSL over the - accepted connections. - - *reuse_address* tells the kernel to reuse a local socket in - TIME_WAIT state, without waiting for its natural timeout to - expire. If not specified will automatically be set to True on - UNIX. - - This method returns a :ref:`coroutine `. - -.. method:: BaseEventLoop.create_datagram_endpoint(protocol_factory, local_addr=None, remote_addr=None, \*, family=0, proto=0, flags=0) - - Create datagram connection. - - This method returns a :ref:`coroutine `. - - - -Resolve name ------------- - -.. method:: BaseEventLoop.getaddrinfo(host, port, \*, family=0, type=0, proto=0, flags=0) - - XXX - -.. method:: BaseEventLoop.getnameinfo(sockaddr, flags=0) - - XXX - - -Running subprocesses --------------------- - -Run subprocesses asynchronously using the :mod:`subprocess` module. - -.. method:: BaseEventLoop.subprocess_exec(protocol_factory, \*args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=False, bufsize=0, \*\*kwargs) - - XXX - - This method returns a :ref:`coroutine `. - - See the constructor of the :class:`subprocess.Popen` class for parameters. - -.. method:: BaseEventLoop.subprocess_shell(protocol_factory, cmd, \*, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False, shell=True, bufsize=0, \*\*kwargs) - - XXX - - This method returns a :ref:`coroutine `. - - See the constructor of the :class:`subprocess.Popen` class for parameters. - -.. method:: BaseEventLoop.connect_read_pipe(protocol_factory, pipe) - - Register read pipe in eventloop. - - *protocol_factory* should instantiate object with :class:`Protocol` - interface. pipe is file-like object already switched to nonblocking. - Return pair (transport, protocol), where transport support - :class:`ReadTransport` interface. - - This method returns a :ref:`coroutine `. - -.. method:: BaseEventLoop.connect_write_pipe(protocol_factory, pipe) - - Register write pipe in eventloop. - - *protocol_factory* should instantiate object with :class:`BaseProtocol` - interface. Pipe is file-like object already switched to nonblocking. - Return pair (transport, protocol), where transport support - :class:`WriteTransport` interface. - - This method returns a :ref:`coroutine `. - - -Executor --------- - -Call a function in an :class:`~concurrent.futures.Executor` (pool of threads or -pool of processes). By default, an event loop uses a thread pool executor -(:class:`~concurrent.futures.ThreadPoolExecutor`). - -.. method:: BaseEventLoop.run_in_executor(executor, callback, \*args) - - Arrange for a callback to be called in the specified executor. - - *executor* is a :class:`~concurrent.futures.Executor` instance, - the default executor is used if *executor* is ``None``. - -.. method:: BaseEventLoop.set_default_executor(executor) - - Set the default executor used by :meth:`run_in_executor`. - - -Tasks and coroutines -==================== - -.. _coroutine: - -Coroutines ----------- - -A coroutine is a generator that follows certain conventions. For -documentation purposes, all coroutines should be decorated with -``@asyncio.coroutine``, but this cannot be strictly enforced. - -Coroutines use the ``yield from`` syntax introduced in :pep:`380`, -instead of the original ``yield`` syntax. - -The word "coroutine", like the word "generator", is used for two -different (though related) concepts: - -- The function that defines a coroutine (a function definition - decorated with ``asyncio.coroutine``). If disambiguation is needed - we will call this a *coroutine function*. - -- The object obtained by calling a coroutine function. This object - represents a computation or an I/O operation (usually a combination) - that will complete eventually. If disambiguation is needed we will - call it a *coroutine object*. - -Things a coroutine can do: - -- ``result = yield from future`` -- suspends the coroutine until the - future is done, then returns the future's result, or raises an - exception, which will be propagated. (If the future is cancelled, - it will raise a ``CancelledError`` exception.) Note that tasks are - futures, and everything said about futures also applies to tasks. - -- ``result = yield from coroutine`` -- wait for another coroutine to - produce a result (or raise an exception, which will be propagated). - The ``coroutine`` expression must be a *call* to another coroutine. - -- ``return expression`` -- produce a result to the coroutine that is - waiting for this one using ``yield from``. - -- ``raise exception`` -- raise an exception in the coroutine that is - waiting for this one using ``yield from``. - -Calling a coroutine does not start its code running -- it is just a -generator, and the coroutine object returned by the call is really a -generator object, which doesn't do anything until you iterate over it. -In the case of a coroutine object, there are two basic ways to start -it running: call ``yield from coroutine`` from another coroutine -(assuming the other coroutine is already running!), or convert it to a -:class:`Task`. - -Coroutines (and tasks) can only run when the event loop is running. - - -Task ----- - -.. class:: Task(coro, \*, loop=None) - - A coroutine wrapped in a :class:`~concurrent.futures.Future`. - - .. classmethod:: all_tasks(loop=None) - - Return a set of all tasks for an event loop. - - By default all tasks for the current event loop are returned. - - .. method:: cancel() - - Cancel the task. - - .. method:: get_stack(self, \*, limit=None) - - Return the list of stack frames for this task's coroutine. - - If the coroutine is active, this returns the stack where it is suspended. - If the coroutine has completed successfully or was cancelled, this - returns an empty list. If the coroutine was terminated by an exception, - this returns the list of traceback frames. - - The frames are always ordered from oldest to newest. - - The optional limit gives the maximum nummber of frames to return; by - default all available frames are returned. Its meaning differs depending - on whether a stack or a traceback is returned: the newest frames of a - stack are returned, but the oldest frames of a traceback are returned. - (This matches the behavior of the traceback module.) - - For reasons beyond our control, only one stack frame is returned for a - suspended coroutine. - - .. method:: print_stack(\*, limit=None, file=None) - - Print the stack or traceback for this task's coroutine. - - This produces output similar to that of the traceback module, for the - frames retrieved by get_stack(). The limit argument is passed to - get_stack(). The file argument is an I/O stream to which the output - goes; by default it goes to sys.stderr. - - -Task functions --------------- - -.. function:: as_completed(fs, *, loop=None, timeout=None) - - Return an iterator whose values, when waited for, are - :class:`~concurrent.futures.Future` instances. - - Raises :exc:`TimeoutError` if the timeout occurs before all Futures are done. - - Example:: - - for f in as_completed(fs): - result = yield from f # The 'yield from' may raise - # Use result - - .. note:: - - The futures ``f`` are not necessarily members of fs. - -.. function:: async(coro_or_future, *, loop=None) - - Wrap a :ref:`coroutine ` in a future. - - If the argument is a :class:`~concurrent.futures.Future`, it is returned - directly. - -.. function:: gather(*coros_or_futures, loop=None, return_exceptions=False) - - Return a future aggregating results from the given coroutines or futures. - - All futures must share the same event loop. If all the tasks are done - successfully, the returned future's result is the list of results (in the - order of the original sequence, not necessarily the order of results - arrival). If *result_exception* is True, exceptions in the tasks are - treated the same as successful results, and gathered in the result list; - otherwise, the first raised exception will be immediately propagated to the - returned future. - - Cancellation: if the outer Future is cancelled, all children (that have not - completed yet) are also cancelled. If any child is cancelled, this is - treated as if it raised :exc:`~concurrent.futures.CancelledError` -- the - outer Future is *not* cancelled in this case. (This is to prevent the - cancellation of one child to cause other children to be cancelled.) - -.. function:: tasks.iscoroutinefunction(func) - - Return ``True`` if *func* is a decorated coroutine function. - -.. function:: tasks.iscoroutine(obj) - - Return ``True`` if *obj* is a coroutine object. - -.. function:: sleep(delay, result=None, \*, loop=None) - - Create a :ref:`coroutine ` that completes after a given time - (in seconds). - -.. function:: shield(arg, \*, loop=None) - - Wait for a future, shielding it from cancellation. - - The statement:: - - res = yield from shield(something()) - - is exactly equivalent to the statement:: - - res = yield from something() - - *except* that if the coroutine containing it is cancelled, the task running - in ``something()`` is not cancelled. From the point of view of - ``something()``, the cancellation did not happen. But its caller is still - cancelled, so the yield-from expression still raises - :exc:`~concurrent.futures.CancelledError`. Note: If ``something()`` is - cancelled by other means this will still cancel ``shield()``. - - If you want to completely ignore cancellation (not recommended) you can - combine ``shield()`` with a try/except clause, as follows:: - - try: - res = yield from shield(something()) - except CancelledError: - res = None - -.. function:: wait(fs, \*, loop=None, timeout=None, return_when=ALL_COMPLETED) - - Wait for the Futures and coroutines given by fs to complete. Coroutines will - be wrapped in Tasks. Returns two sets of - :class:`~concurrent.futures.Future`: (done, pending). - - *timeout* can be used to control the maximum number of seconds to wait before - returning. *timeout* can be an int or float. If *timeout* is not specified - or ``None``, there is no limit to the wait time. - - *return_when* indicates when this function should return. It must be one of - the following constants of the :mod`concurrent.futures` module: - - .. tabularcolumns:: |l|L| - - +-----------------------------+----------------------------------------+ - | Constant | Description | - +=============================+========================================+ - | :const:`FIRST_COMPLETED` | The function will return when any | - | | future finishes or is cancelled. | - +-----------------------------+----------------------------------------+ - | :const:`FIRST_EXCEPTION` | The function will return when any | - | | future finishes by raising an | - | | exception. If no future raises an | - | | exception then it is equivalent to | - | | :const:`ALL_COMPLETED`. | - +-----------------------------+----------------------------------------+ - | :const:`ALL_COMPLETED` | The function will return when all | - | | futures finish or are cancelled. | - +-----------------------------+----------------------------------------+ - - This function returns a :ref:`coroutine `. - - Usage:: - - done, pending = yield from asyncio.wait(fs) - - .. note:: - - This does not raise :exc:`TimeoutError`! Futures that aren't done when - the timeout occurs are returned in the second set. - - -.. _transport: - -Transports -========== - -Transports are classed provided by :mod:`asyncio` in order to abstract -various kinds of communication channels. You generally won't instantiate -a transport yourself; instead, you will call a :class:`BaseEventLoop` method -which will create the transport and try to initiate the underlying -communication channel, calling you back when it succeeds. - -Once the communication channel is established, a transport is always -paired with a :ref:`protocol ` instance. The protocol can -then call the transport's methods for various purposes. - -:mod:`asyncio` currently implements transports for TCP, UDP, SSL, and -subprocess pipes. The methods available on a transport depend on -the transport's kind. - - -BaseTransport: Methods common to all transports ------------------------------------------------ - -.. class:: BaseTransport - - Base class for transports. - - .. method:: close(self) - - Close the transport. If the transport has a buffer for outgoing - data, buffered data will be flushed asynchronously. No more data - will be received. After all buffered data is flushed, the - protocol's :meth:`connection_lost` method will be called with - :const:`None` as its argument. - - - .. method:: get_extra_info(name, default=None) - - Return optional transport information. *name* is a string representing - the piece of transport-specific information to get, *default* is the - value to return if the information doesn't exist. - - This method allows transport implementations to easily expose - channel-specific information. - - * socket: - - - ``'peername'``: the remote address to which the socket is connected, - result of :meth:`socket.socket.getpeername` (``None`` on error) - - ``'socket'``: :class:`socket.socket` instance - - ``'sockname'``: the socket's own address, - result of :meth:`socket.socket.getsockname` - - * SSL socket: - - - ``'compression'``: the compression algorithm being used as a string, - or ``None`` if the connection isn't compressed; result of - :meth:`ssl.SSLSocket.compression` - - ``'cipher'``: a three-value tuple containing the name of the cipher - being used, the version of the SSL protocol that defines its use, and - the number of secret bits being used; result of - :meth:`ssl.SSLSocket.cipher` - - ``'peercert'``: peer certificate; result of - :meth:`ssl.SSLSocket.getpeercert` - - ``'sslcontext'``: :class:`ssl.SSLContext` instance - - * pipe: - - - ``'pipe'``: pipe object - - * subprocess: - - - ``'subprocess'``: :class:`subprocess.Popen` instance - - -ReadTransport: Methods of readable streaming transports -------------------------------------------------------- - -.. class:: ReadTransport - - Interface for read-only transports. - - .. method:: pause_reading() - - Pause the receiving end of the transport. No data will be passed to - the protocol's :meth:`data_received` method until meth:`resume_reading` - is called. - - .. method:: resume_reading() - - Resume the receiving end. The protocol's :meth:`data_received` method - will be called once again if some data is available for reading. - - -WriteTransport: Methods of writable streaming transports --------------------------------------------------------- - -.. class:: WriteTransport - - Interface for write-only transports. - - .. method:: abort() - - Close the transport immediately, without waiting for pending operations - to complete. Buffered data will be lost. No more data will be received. - The protocol's :meth:`connection_lost` method will eventually be - called with :const:`None` as its argument. - - .. method:: can_write_eof() - - Return :const:`True` if the transport supports :meth:`write_eof`, - :const:`False` if not. - - .. method:: get_write_buffer_size() - - Return the current size of the output buffer used by the transport. - - .. method:: set_write_buffer_limits(high=None, low=None) - - Set the *high*- and *low*-water limits for write flow control. - - These two values control when call the protocol's - :meth:`pause_writing` and :meth:`resume_writing` methods are called. - If specified, the low-water limit must be less than or equal to the - high-water limit. Neither *high* nor *low* can be negative. - - The defaults are implementation-specific. If only the - high-water limit is given, the low-water limit defaults to a - implementation-specific value less than or equal to the - high-water limit. Setting *high* to zero forces *low* to zero as - well, and causes :meth:`pause_writing` to be called whenever the - buffer becomes non-empty. Setting *low* to zero causes - :meth:`resume_writing` to be called only once the buffer is empty. - Use of zero for either limit is generally sub-optimal as it - reduces opportunities for doing I/O and computation - concurrently. - - .. method:: write(data) - - Write some *data* bytes to the transport. - - This method does not block; it buffers the data and arranges for it - to be sent out asynchronously. - - .. method:: writelines(list_of_data) - - Write a list (or any iterable) of data bytes to the transport. - This is functionally equivalent to calling :meth:`write` on each - element yielded by the iterable, but may be implemented more efficiently. - - .. method:: write_eof() - - Close the write end of the transport after flushing buffered data. - Data may still be received. - - This method can raise :exc:`NotImplementedError` if the transport - (e.g. SSL) doesn't support half-closes. - - -DatagramTransport: Methods of datagram transports -------------------------------------------------- - -.. method:: DatagramTransport.sendto(data, addr=None) - - Send the *data* bytes to the remote peer given by *addr* (a - transport-dependent target address). If *addr* is :const:`None`, the - data is sent to the target address given on transport creation. - - This method does not block; it buffers the data and arranges for it - to be sent out asynchronously. - -.. method:: DatagramTransport.abort() - - Close the transport immediately, without waiting for pending operations - to complete. Buffered data will be lost. No more data will be received. - The protocol's :meth:`connection_lost` method will eventually be - called with :const:`None` as its argument. - - -Methods of subprocess transports --------------------------------- - -.. class:: BaseSubprocessTransport - - .. method:: get_pid() - - Return the subprocess process id as an integer. - - .. method:: get_returncode() - - Return the subprocess returncode as an integer or :const:`None` - if it hasn't returned, similarly to the - :attr:`subprocess.Popen.returncode` attribute. - - .. method:: get_pipe_transport(fd) - - Return the transport for the communication pipe correspondong to the - integer file descriptor *fd*. The return value can be a readable or - writable streaming transport, depending on the *fd*. If *fd* doesn't - correspond to a pipe belonging to this transport, :const:`None` is - returned. - - .. method:: send_signal(signal) - - Send the *signal* number to the subprocess, as in - :meth:`subprocess.Popen.send_signal`. - - .. method:: terminate() - - Ask the subprocess to stop, as in :meth:`subprocess.Popen.terminate`. - This method is an alias for the :meth:`close` method. - - On POSIX systems, this method sends SIGTERM to the subprocess. - On Windows, the Windows API function TerminateProcess() is called to - stop the subprocess. - - .. method:: kill(self) - - Kill the subprocess, as in :meth:`subprocess.Popen.kill` - - On POSIX systems, the function sends SIGKILL to the subprocess. - On Windows, this method is an alias for :meth:`terminate`. - - -Stream reader -------------- - -.. class:: StreamWriter(transport, protocol, reader, loop) - - Wraps a Transport. - - This exposes :meth:`write`, :meth:`writelines`, :meth:`can_write_eof()`, :meth:`write_eof`, :meth:`get_extra_info` and - :meth:`close`. It adds :meth:`drain` which returns an optional :class:`~concurrent.futures.Future` on which you can - wait for flow control. It also adds a transport attribute which references - the :class:`Transport` directly. - - .. attribute:: transport - - Transport. - - .. method:: close() - - Close the transport: see :meth:`BaseTransport.close`. - - .. method:: drain() - - This method has an unusual return value. - - The intended use is to write:: - - w.write(data) - yield from w.drain() - - When there's nothing to wait for, :meth:`drain()` returns ``()``, and the - yield-from continues immediately. When the transport buffer is full (the - protocol is paused), :meth:`drain` creates and returns a - :class:`~concurrent.futures.Future` and the yield-from will block until - that Future is completed, which will happen when the buffer is - (partially) drained and the protocol is resumed. - - .. method:: get_extra_info(name, default=None) - - Return optional transport information: see - :meth:`BaseTransport.get_extra_info`. - - .. method:: write(data) - - Write some *data* bytes to the transport: see - :meth:`WriteTransport.write`. - - .. method:: writelines(data) - - Write a list (or any iterable) of data bytes to the transport: - see :meth:`WriteTransport.writelines`. - - .. method:: can_write_eof() - - Return :const:`True` if the transport supports :meth:`write_eof`, - :const:`False` if not. See :meth:`WriteTransport.can_write_eof`. - - .. method:: write_eof() - - Close the write end of the transport after flushing buffered data: - see :meth:`WriteTransport.write_eof`. - - -Stream writer -------------- - -.. class:: StreamReader(limit=_DEFAULT_LIMIT, loop=None) - - .. method:: exception() - - Get the exception. - - .. method:: feed_eof() - - XXX - - .. method:: feed_data(data) - - XXX - - .. method:: set_exception(exc) - - Set the exception. - - .. method:: set_transport(transport) - - Set the transport. - - .. method:: read(n=-1) - - XXX - - This method returns a :ref:`coroutine `. - - .. method:: readline() - - XXX - - This method returns a :ref:`coroutine `. - - .. method:: readexactly(n) - - XXX - - This method returns a :ref:`coroutine `. - - - -.. _protocol: - -Protocols -========= - -:mod:`asyncio` provides base classes that you can subclass to implement -your network protocols. Those classes are used in conjunction with -:ref:`transports ` (see below): the protocol parses incoming -data and asks for the writing of outgoing data, while the transport is -responsible for the actual I/O and buffering. - -When subclassing a protocol class, it is recommended you override certain -methods. Those methods are callbacks: they will be called by the transport -on certain events (for example when some data is received); you shouldn't -call them yourself, unless you are implementing a transport. - -.. note:: - All callbacks have default implementations, which are empty. Therefore, - you only need to implement the callbacks for the events in which you - are interested. - - -Protocol classes ----------------- - -.. class:: Protocol - - The base class for implementing streaming protocols (for use with - e.g. TCP and SSL transports). - -.. class:: DatagramProtocol - - The base class for implementing datagram protocols (for use with - e.g. UDP transports). - -.. class:: SubprocessProtocol - - The base class for implementing protocols communicating with child - processes (through a set of unidirectional pipes). - - -Connection callbacks --------------------- - -These callbacks may be called on :class:`Protocol` and -:class:`SubprocessProtocol` instances: - -.. method:: BaseProtocol.connection_made(transport) - - Called when a connection is made. - - The *transport* argument is the transport representing the - connection. You are responsible for storing it somewhere - (e.g. as an attribute) if you need to. - -.. method:: BaseProtocol.connection_lost(exc) - - Called when the connection is lost or closed. - - The argument is either an exception object or :const:`None`. - The latter means a regular EOF is received, or the connection was - aborted or closed by this side of the connection. - -:meth:`connection_made` and :meth:`connection_lost` are called exactly once -per successful connection. All other callbacks will be called between those -two methods, which allows for easier resource management in your protocol -implementation. - -The following callbacks may be called only on :class:`SubprocessProtocol` -instances: - -.. method:: SubprocessProtocol.pipe_data_received(fd, data) - - Called when the child process writes data into its stdout or stderr pipe. - *fd* is the integer file descriptor of the pipe. *data* is a non-empty - bytes object containing the data. - -.. method:: SubprocessProtocol.pipe_connection_lost(fd, exc) - - Called when one of the pipes communicating with the child process - is closed. *fd* is the integer file descriptor that was closed. - -.. method:: SubprocessProtocol.process_exited() - - Called when the child process has exited. - - -Data reception callbacks ------------------------- - -Streaming protocols -^^^^^^^^^^^^^^^^^^^ - -The following callbacks are called on :class:`Protocol` instances: - -.. method:: Protocol.data_received(data) - - Called when some data is received. *data* is a non-empty bytes object - containing the incoming data. - - .. note:: - Whether the data is buffered, chunked or reassembled depends on - the transport. In general, you shouldn't rely on specific semantics - and instead make your parsing generic and flexible enough. However, - data is always received in the correct order. - -.. method:: Protocol.eof_received() - - Calls when the other end signals it won't send any more data - (for example by calling :meth:`write_eof`, if the other end also uses - asyncio). - - This method may return a false value (including None), in which case - the transport will close itself. Conversely, if this method returns a - true value, closing the transport is up to the protocol. Since the - default implementation returns None, it implicitly closes the connection. - - .. note:: - Some transports such as SSL don't support half-closed connections, - in which case returning true from this method will not prevent closing - the connection. - -:meth:`data_received` can be called an arbitrary number of times during -a connection. However, :meth:`eof_received` is called at most once -and, if called, :meth:`data_received` won't be called after it. - -Datagram protocols -^^^^^^^^^^^^^^^^^^ - -The following callbacks are called on :class:`DatagramProtocol` instances. - -.. method:: DatagramProtocol.datagram_received(data, addr) - - Called when a datagram is received. *data* is a bytes object containing - the incoming data. *addr* is the address of the peer sending the data; - the exact format depends on the transport. - -.. method:: DatagramProtocol.error_received(exc) - - Called when a previous send or receive operation raises an - :class:`OSError`. *exc* is the :class:`OSError` instance. - - This method is called in rare conditions, when the transport (e.g. UDP) - detects that a datagram couldn't be delivered to its recipient. - In many conditions though, undeliverable datagrams will be silently - dropped. - - -Flow control callbacks ----------------------- - -These callbacks may be called on :class:`Protocol` and -:class:`SubprocessProtocol` instances: - -.. method:: BaseProtocol.pause_writing() - - Called when the transport's buffer goes over the high-water mark. - -.. method:: BaseProtocol.resume_writing() - - Called when the transport's buffer drains below the low-water mark. - - -:meth:`pause_writing` and :meth:`resume_writing` calls are paired -- -:meth:`pause_writing` is called once when the buffer goes strictly over -the high-water mark (even if subsequent writes increases the buffer size -even more), and eventually :meth:`resume_writing` is called once when the -buffer size reaches the low-water mark. - -.. note:: - If the buffer size equals the high-water mark, - :meth:`pause_writing` is not called -- it must go strictly over. - Conversely, :meth:`resume_writing` is called when the buffer size is - equal or lower than the low-water mark. These end conditions - are important to ensure that things go as expected when either - mark is zero. - - -Server ------- - -.. class:: AbstractServer - - Abstract server returned by create_service(). - - .. method:: close() - - Stop serving. This leaves existing connections open. - - .. method:: wait_closed() - - Coroutine to wait until service is closed. - - -Network functions -================= - -.. function:: open_connection(host=None, port=None, *, loop=None, limit=_DEFAULT_LIMIT, **kwds) - - A wrapper for create_connection() returning a (reader, writer) pair. - - The reader returned is a StreamReader instance; the writer is a - :class:`Transport`. - - The arguments are all the usual arguments to - :meth:`BaseEventLoop.create_connection` except *protocol_factory*; most - common are positional host and port, with various optional keyword arguments - following. - - Additional optional keyword arguments are *loop* (to set the event loop - instance to use) and *limit* (to set the buffer limit passed to the - StreamReader). - - (If you want to customize the :class:`StreamReader` and/or - :class:`StreamReaderProtocol` classes, just copy the code -- there's really - nothing special here except some convenience.) - - This function returns a :ref:`coroutine `. - -.. function:: start_server(client_connected_cb, host=None, port=None, *, loop=None, limit=_DEFAULT_LIMIT, **kwds) - - Start a socket server, call back for each client connected. - - The first parameter, *client_connected_cb*, takes two parameters: - *client_reader*, *client_writer*. *client_reader* is a - :class:`StreamReader` object, while *client_writer* is a - :class:`StreamWriter` object. This parameter can either be a plain callback - function or a :ref:`coroutine `; if it is a coroutine, it will be - automatically converted into a :class:`Task`. - - The rest of the arguments are all the usual arguments to - :meth:`~BaseEventLoop.create_server()` except *protocol_factory*; most - common are positional host and port, with various optional keyword arguments - following. The return value is the same as - :meth:`~BaseEventLoop.create_server()`. - - Additional optional keyword arguments are *loop* (to set the event loop - instance to use) and *limit* (to set the buffer limit passed to the - :class:`StreamReader`). - - The return value is the same as :meth:`~BaseEventLoop.create_server()`, i.e. - a :class:`AbstractServer` object which can be used to stop the service. - - This function returns a :ref:`coroutine `. - - -.. _sync: - -Synchronization primitives -========================== - -Locks ------ - -.. class:: Lock(\*, loop=None) - - Primitive lock objects. - - A primitive lock is a synchronization primitive that is not owned by a - particular coroutine when locked. A primitive lock is in one of two states, - 'locked' or 'unlocked'. - - It is created in the unlocked state. It has two basic methods, :meth:`acquire` - and :meth:`release`. When the state is unlocked, acquire() changes the state to - locked and returns immediately. When the state is locked, acquire() blocks - until a call to release() in another coroutine changes it to unlocked, then - the acquire() call resets it to locked and returns. The release() method - should only be called in the locked state; it changes the state to unlocked - and returns immediately. If an attempt is made to release an unlocked lock, - a :exc:`RuntimeError` will be raised. - - When more than one coroutine is blocked in acquire() waiting for the state - to turn to unlocked, only one coroutine proceeds when a release() call - resets the state to unlocked; first coroutine which is blocked in acquire() - is being processed. - - :meth:`acquire` is a coroutine and should be called with ``yield from``. - - Locks also support the context manager protocol. ``(yield from lock)`` - should be used as context manager expression. - - Usage:: - - lock = Lock() - ... - yield from lock - try: - ... - finally: - lock.release() - - Context manager usage:: - - lock = Lock() - ... - with (yield from lock): - ... - - Lock objects can be tested for locking state:: - - if not lock.locked(): - yield from lock - else: - # lock is acquired - ... - - .. method:: locked() - - Return ``True`` if lock is acquired. - - .. method:: acquire() - - Acquire a lock. - - This method blocks until the lock is unlocked, then sets it to locked and - returns ``True``. - - This method returns a :ref:`coroutine `. - - .. method:: release() - - Release a lock. - - When the lock is locked, reset it to unlocked, and return. If any other - coroutines are blocked waiting for the lock to become unlocked, allow - exactly one of them to proceed. - - When invoked on an unlocked lock, a :exc:`RuntimeError` is raised. - - There is no return value. - - -.. class:: Event(\*, loop=None) - - An Event implementation, asynchronous equivalent to :class:`threading.Event`. - - Class implementing event objects. An event manages a flag that can be set to - true with the :meth:`set` method and reset to false with the :meth:`clear` - method. The :meth:`wait` method blocks until the flag is true. The flag is - initially false. - - .. method:: clear() - - Reset the internal flag to false. Subsequently, coroutines calling - :meth:`wait` will block until :meth:`set` is called to set the internal - flag to true again. - - .. method:: is_set() - - Return ``True`` if and only if the internal flag is true. - - .. method:: set() - - Set the internal flag to true. All coroutines waiting for it to become - true are awakened. Coroutine that call :meth:`wait` once the flag is true - will not block at all. - - .. method:: wait() - - Block until the internal flag is true. - - If the internal flag is true on entry, return ``True`` immediately. - Otherwise, block until another coroutine calls :meth:`set` to set the - flag to true, then return ``True``. - - This method returns a :ref:`coroutine `. - - -.. class:: Condition(\*, loop=None) - - A Condition implementation, asynchronous equivalent to - :class:`threading.Condition`. - - This class implements condition variable objects. A condition variable - allows one or more coroutines to wait until they are notified by another - coroutine. - - A new :class:`Lock` object is created and used as the underlying lock. - - .. method:: notify(n=1) - - By default, wake up one coroutine waiting on this condition, if any. - If the calling coroutine has not acquired the lock when this method is - called, a :exc:`RuntimeError` is raised. - - This method wakes up at most *n* of the coroutines waiting for the - condition variable; it is a no-op if no coroutines are waiting. - - .. note:: - - An awakened coroutine does not actually return from its :meth:`wait` - call until it can reacquire the lock. Since :meth:`notify` does not - release the lock, its caller should. - - .. method:: notify_all() - - Wake up all threads waiting on this condition. This method acts like - :meth:`notify`, but wakes up all waiting threads instead of one. If the - calling thread has not acquired the lock when this method is called, a - :exc:`RuntimeError` is raised. - - .. method:: wait() - - Wait until notified. - - If the calling coroutine has not acquired the lock when this method is - called, a :exc:`RuntimeError` is raised. - - This method releases the underlying lock, and then blocks until it is - awakened by a :meth:`notify` or :meth:`notify_all` call for the same - condition variable in another coroutine. Once awakened, it re-acquires - the lock and returns ``True``. - - This method returns a :ref:`coroutine `. - - .. method:: wait_for(predicate) - - Wait until a predicate becomes true. - - The predicate should be a callable which result will be interpreted as a - boolean value. The final predicate value is the return value. - - This method returns a :ref:`coroutine `. - - -Semaphores ----------- - -.. class:: Semaphore(value=1, \*, loop=None) - - A Semaphore implementation. - - A semaphore manages an internal counter which is decremented by each - :meth:`acquire` call and incremented by each :meth:`release` call. The - counter can never go below zero; when :meth:`acquire` finds that it is zero, - it blocks, waiting until some other thread calls :meth:`release`. - - Semaphores also support the context manager protocol. - - The optional argument gives the initial value for the internal counter; it - defaults to ``1``. If the value given is less than ``0``, :exc:`ValueError` - is raised. - - .. method:: acquire() - - Acquire a semaphore. - - If the internal counter is larger than zero on entry, decrement it by one - and return ``True`` immediately. If it is zero on entry, block, waiting - until some other coroutine has called :meth:`release` to make it larger - than ``0``, and then return ``True``. - - This method returns a :ref:`coroutine `. - - .. method:: locked() - - Returns ``True`` if semaphore can not be acquired immediately. - - .. method:: release() - - Release a semaphore, incrementing the internal counter by one. When it - was zero on entry and another coroutine is waiting for it to become - larger than zero again, wake up that coroutine. - - -.. class:: BoundedSemaphore(value=1, \*, loop=None) - - A bounded semaphore implementation. Inherit from :class:`Semaphore`. - - This raises :exc:`ValueError` in :meth:`~Semaphore.release` if it would - increase the value above the initial value. - - -Queues ------- - -.. class:: Queue(maxsize=0, \*, loop=None) - - A queue, useful for coordinating producer and consumer coroutines. - - If *maxsize* is less than or equal to zero, the queue size is infinite. If - it is an integer greater than ``0``, then ``yield from put()`` will block - when the queue reaches *maxsize*, until an item is removed by :meth:`get`. - - Unlike the standard library :mod:`queue`, you can reliably know this Queue's - size with :meth:`qsize`, since your single-threaded Tulip application won't - be interrupted between calling :meth:`qsize` and doing an operation on the - Queue. - - .. method:: empty() - - Return ``True`` if the queue is empty, ``False`` otherwise. - - .. method:: full() - - Return ``True`` if there are maxsize items in the queue. - - .. note:: - - If the Queue was initialized with ``maxsize=0`` (the default), then - :meth:`full()` is never ``True``. - - .. method:: get() - - Remove and return an item from the queue. - - If you yield from :meth:`get()`, wait until a item is available. - - This method returns a :ref:`coroutine `. - - .. method:: get_nowait() - - Remove and return an item from the queue. - - Return an item if one is immediately available, else raise - :exc:`~queue.Empty`. - - .. method:: put(item) - - Put an item into the queue. - - If you yield from ``put()``, wait until a free slot is available before - adding item. - - This method returns a :ref:`coroutine `. - - .. method:: put_nowait(item) - - Put an item into the queue without blocking. - - If no free slot is immediately available, raise :exc:`~queue.Full`. - - .. method:: qsize() - - Number of items in the queue. - - .. attribute:: maxsize - - Number of items allowed in the queue. - - -.. class:: PriorityQueue - - A subclass of :class:`Queue`; retrieves entries in priority order (lowest - first). - - Entries are typically tuples of the form: (priority number, data). - - -.. class:: LifoQueue - - A subclass of :class:`Queue` that retrieves most recently added entries - first. - - -.. class:: JoinableQueue - - A subclass of :class:`Queue` with :meth:`task_done` and :meth:`join` - methods. - - .. method:: join() - - Block until all items in the queue have been gotten and processed. - - The count of unfinished tasks goes up whenever an item is added to the - queue. The count goes down whenever a consumer thread calls - :meth:`task_done` to indicate that the item was retrieved and all work on - it is complete. When the count of unfinished tasks drops to zero, - :meth:`join` unblocks. - - This method returns a :ref:`coroutine `. - - .. method:: task_done() - - Indicate that a formerly enqueued task is complete. - - Used by queue consumers. For each :meth:`~Queue.get` used to fetch a task, a - subsequent call to :meth:`task_done` tells the queue that the processing - on the task is complete. - - If a :meth:`join` is currently blocking, it will resume when all items - have been processed (meaning that a :meth:`task_done` call was received - for every item that had been :meth:`~Queue.put` into the queue). - - Raises :exc:`ValueError` if called more times than there were items - placed in the queue. - - -Examples --------- - -Hello World (callback) -^^^^^^^^^^^^^^^^^^^^^^ - -Print ``Hello World`` every two seconds, using a callback:: - - import asyncio - - def print_and_repeat(loop): - print('Hello World') - loop.call_later(2, print_and_repeat, loop) - - loop = asyncio.get_event_loop() - print_and_repeat(loop) - loop.run_forever() - - -Hello World (callback) -^^^^^^^^^^^^^^^^^^^^^^ - -Print ``Hello World`` every two seconds, using a coroutine:: - - import asyncio - - @asyncio.coroutine - def greet_every_two_seconds(): - while True: - print('Hello World') - yield from asyncio.sleep(2) - - loop = asyncio.get_event_loop() - loop.run_until_complete(greet_every_two_seconds()) - - -Echo server -^^^^^^^^^^^ - -A :class:`Protocol` implementing an echo server:: - - class EchoServer(asyncio.Protocol): - - TIMEOUT = 5.0 - - def timeout(self): - print('connection timeout, closing.') - self.transport.close() - - def connection_made(self, transport): - print('connection made') - self.transport = transport - - # start 5 seconds timeout timer - self.h_timeout = asyncio.get_event_loop().call_later( - self.TIMEOUT, self.timeout) - - def data_received(self, data): - print('data received: ', data.decode()) - self.transport.write(b'Re: ' + data) - - # restart timeout timer - self.h_timeout.cancel() - self.h_timeout = asyncio.get_event_loop().call_later( - self.TIMEOUT, self.timeout) - - def eof_received(self): - pass - - def connection_lost(self, exc): - print('connection lost:', exc) - self.h_timeout.cancel() - -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 01:47:07 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 3 Dec 2013 01:47:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_move_corout?= =?utf-8?q?ine_example_to_the_Task_page?= Message-ID: <3dYPfM3G9mzNjC@mail.python.org> http://hg.python.org/cpython/rev/18ba60ac4174 changeset: 87725:18ba60ac4174 user: Victor Stinner date: Tue Dec 03 01:22:06 2013 +0100 summary: asyncio doc: move coroutine example to the Task page files: Doc/library/asyncio-eventloop.rst | 28 ++++++------------ Doc/library/asyncio-task.rst | 23 +++++++++++++++ 2 files changed, 32 insertions(+), 19 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -144,6 +144,10 @@ Return the current time, as a :class:`float` value, according to the event loop's internal clock. +.. seealso:: + + The :func:`asyncio.sleep` function. + Creating connections ^^^^^^^^^^^^^^^^^^^^ @@ -330,11 +334,10 @@ Set the default executor used by :meth:`run_in_executor`. -Examples --------- +.. _asyncio-hello-world-callback: -Hello World (callback) -^^^^^^^^^^^^^^^^^^^^^^ +Example: Hello World (callback) +------------------------------- Print ``Hello World`` every two seconds, using a callback:: @@ -348,20 +351,7 @@ print_and_repeat(loop) loop.run_forever() +.. seealso:: -Hello World (coroutine) -^^^^^^^^^^^^^^^^^^^^^^^ + :ref:`Hello World example using a coroutine `. -Print ``Hello World`` every two seconds, using a coroutine:: - - import asyncio - - @asyncio.coroutine - def greet_every_two_seconds(): - while True: - print('Hello World') - yield from asyncio.sleep(2) - - loop = asyncio.get_event_loop() - loop.run_until_complete(greet_every_two_seconds()) - diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -230,3 +230,26 @@ This does not raise :exc:`TimeoutError`! Futures that aren't done when the timeout occurs are returned in the second set. + +.. _asyncio-hello-world-coroutine: + +Example: Hello World (coroutine) +-------------------------------- + +Print ``Hello World`` every two seconds, using a coroutine:: + + import asyncio + + @asyncio.coroutine + def greet_every_two_seconds(): + while True: + print('Hello World') + yield from asyncio.sleep(2) + + loop = asyncio.get_event_loop() + loop.run_until_complete(greet_every_two_seconds()) + + +.. seealso:: + + :ref:`Hello World example using a callback `. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 01:47:08 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 3 Dec 2013 01:47:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_write_a_com?= =?utf-8?q?plete_TCP_echo_example=2C_client_and_server?= Message-ID: <3dYPfN6753zNjC@mail.python.org> http://hg.python.org/cpython/rev/c17a04f85c0e changeset: 87726:c17a04f85c0e user: Victor Stinner date: Tue Dec 03 01:46:39 2013 +0100 summary: asyncio doc: write a complete TCP echo example, client and server Example based on tcp_echo.py example from Tulip source code. files: Doc/library/asyncio-eventloop.rst | 2 +- Doc/library/asyncio-protocol.rst | 116 +++++++++++------ 2 files changed, 72 insertions(+), 46 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -150,7 +150,7 @@ Creating connections -^^^^^^^^^^^^^^^^^^^^ +-------------------- .. method:: BaseEventLoop.create_connection(protocol_factory, host=None, port=None, \*, ssl=None, family=0, proto=0, flags=0, sock=None, local_addr=None, server_hostname=None) diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -20,8 +20,8 @@ the transport's kind. -BaseTransport: Methods common to all transports ------------------------------------------------ +BaseTransport +------------- .. class:: BaseTransport @@ -75,8 +75,8 @@ - ``'subprocess'``: :class:`subprocess.Popen` instance -ReadTransport: Methods of readable streaming transports -------------------------------------------------------- +ReadTransport +------------- .. class:: ReadTransport @@ -94,8 +94,8 @@ will be called once again if some data is available for reading. -WriteTransport: Methods of writable streaming transports --------------------------------------------------------- +WriteTransport +-------------- .. class:: WriteTransport @@ -159,8 +159,8 @@ (e.g. SSL) doesn't support half-closes. -DatagramTransport: Methods of datagram transports -------------------------------------------------- +DatagramTransport +----------------- .. method:: DatagramTransport.sendto(data, addr=None) @@ -179,8 +179,8 @@ called with :const:`None` as its argument. -Methods of subprocess transports --------------------------------- +BaseSubprocessTransport +----------------------- .. class:: BaseSubprocessTransport @@ -224,8 +224,8 @@ On Windows, this method is an alias for :meth:`terminate`. -Stream reader -------------- +StreamWriter +------------ .. class:: StreamWriter(transport, protocol, reader, loop) @@ -286,8 +286,8 @@ see :meth:`WriteTransport.write_eof`. -Stream writer -------------- +StreamReader +------------ .. class:: StreamReader(limit=_DEFAULT_LIMIT, loop=None) @@ -418,11 +418,8 @@ Called when the child process has exited. -Data reception callbacks ------------------------- - Streaming protocols -^^^^^^^^^^^^^^^^^^^ +------------------- The following callbacks are called on :class:`Protocol` instances: @@ -458,7 +455,7 @@ and, if called, :meth:`data_received` won't be called after it. Datagram protocols -^^^^^^^^^^^^^^^^^^ +------------------ The following callbacks are called on :class:`DatagramProtocol` instances. @@ -576,40 +573,69 @@ This function returns a :ref:`coroutine `. -Example: Echo server --------------------- -A :class:`Protocol` implementing an echo server:: +Protocol example: TCP echo client and server +============================================ - class EchoServer(asyncio.Protocol): +Echo server +----------- - TIMEOUT = 5.0 +TCP echo server example:: - def timeout(self): - print('connection timeout, closing.') - self.transport.close() + import asyncio - def connection_made(self, transport): - print('connection made') - self.transport = transport + class EchoServer(asyncio.Protocol): + def timeout(self): + print('connection timeout, closing.') + self.transport.close() - # start 5 seconds timeout timer - self.h_timeout = asyncio.get_event_loop().call_later( - self.TIMEOUT, self.timeout) + def connection_made(self, transport): + print('connection made') + self.transport = transport - def data_received(self, data): - print('data received: ', data.decode()) - self.transport.write(b'Re: ' + data) + # close the client connection after 2 seconds + asyncio.get_event_loop().call_later(2.0, self.timeout) - # restart timeout timer - self.h_timeout.cancel() - self.h_timeout = asyncio.get_event_loop().call_later( - self.TIMEOUT, self.timeout) + def data_received(self, data): + print('data received:', data.decode()) + self.transport.write(data) - def eof_received(self): - pass + def connection_lost(self, exc): + print('connection lost') - def connection_lost(self, exc): - print('connection lost:', exc) - self.h_timeout.cancel() + loop = asyncio.get_event_loop() + f = loop.create_server(EchoServer, '127.0.0.1', 8888) + s = loop.run_until_complete(f) + print('serving on', s.sockets[0].getsockname()) + loop.run_forever() + + +Echo client +----------- + +TCP echo client example:: + + import asyncio + + class EchoClient(asyncio.Protocol): + message = 'This is the message. It will be echoed.' + + def connection_made(self, transport): + self.transport = transport + self.transport.write(self.message.encode()) + print('data sent:', self.message) + + def data_received(self, data): + print('data received:', data.decode()) + + def connection_lost(self, exc): + print('connection lost') + asyncio.get_event_loop().stop() + + loop = asyncio.get_event_loop() + task = loop.create_connection(EchoClient, '127.0.0.1', 8888) + loop.run_until_complete(task) + loop.run_forever() + loop.close() + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 01:49:52 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 3 Dec 2013 01:49:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_io=3A_group_transp?= =?utf-8?q?orts_and_protocols_in_a_new_title?= Message-ID: <3dYPjX3zwJz7LkG@mail.python.org> http://hg.python.org/cpython/rev/b76e0416deb8 changeset: 87727:b76e0416deb8 user: Victor Stinner date: Tue Dec 03 01:49:43 2013 +0100 summary: asyncio io: group transports and protocols in a new title files: Doc/library/asyncio-protocol.rst | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -1,5 +1,9 @@ .. module:: asyncio +++++++++++++++++++++++++ +Transports and protocols +++++++++++++++++++++++++ + .. _transport: Transports -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 01:59:47 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 3 Dec 2013 01:59:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_reorder_met?= =?utf-8?q?hods=3B_typo?= Message-ID: <3dYPwz3V3wz7LjR@mail.python.org> http://hg.python.org/cpython/rev/14bbeae989fa changeset: 87728:14bbeae989fa user: Victor Stinner date: Tue Dec 03 01:59:38 2013 +0100 summary: asyncio doc: reorder methods; typo files: Doc/library/asyncio-protocol.rst | 26 ++++++++++---------- Doc/library/asyncio-task.rst | 2 +- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -192,12 +192,6 @@ Return the subprocess process id as an integer. - .. method:: get_returncode() - - Return the subprocess returncode as an integer or :const:`None` - if it hasn't returned, similarly to the - :attr:`subprocess.Popen.returncode` attribute. - .. method:: get_pipe_transport(fd) Return the transport for the communication pipe correspondong to the @@ -206,6 +200,19 @@ correspond to a pipe belonging to this transport, :const:`None` is returned. + .. method:: get_returncode() + + Return the subprocess returncode as an integer or :const:`None` + if it hasn't returned, similarly to the + :attr:`subprocess.Popen.returncode` attribute. + + .. method:: kill(self) + + Kill the subprocess, as in :meth:`subprocess.Popen.kill` + + On POSIX systems, the function sends SIGKILL to the subprocess. + On Windows, this method is an alias for :meth:`terminate`. + .. method:: send_signal(signal) Send the *signal* number to the subprocess, as in @@ -220,13 +227,6 @@ On Windows, the Windows API function TerminateProcess() is called to stop the subprocess. - .. method:: kill(self) - - Kill the subprocess, as in :meth:`subprocess.Popen.kill` - - On POSIX systems, the function sends SIGKILL to the subprocess. - On Windows, this method is an alias for :meth:`terminate`. - StreamWriter ------------ diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -199,7 +199,7 @@ or ``None``, there is no limit to the wait time. *return_when* indicates when this function should return. It must be one of - the following constants of the :mod`concurrent.futures` module: + the following constants of the :mod:`concurrent.futures` module: .. tabularcolumns:: |l|L| -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 02:05:51 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 3 Dec 2013 02:05:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_fix_definit?= =?utf-8?q?ion_of_stop=28=29_method?= Message-ID: <3dYQ3z4J4qz7LjR@mail.python.org> http://hg.python.org/cpython/rev/29d67ac7a9d5 changeset: 87729:29d67ac7a9d5 user: Victor Stinner date: Tue Dec 03 02:05:42 2013 +0100 summary: asyncio doc: fix definition of stop() method files: Doc/library/asyncio-eventloop.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -71,7 +71,7 @@ Returns running status of event loop. -.. method:: stop() +.. method:: BaseEventLoop.stop() Stop running the event loop. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 03:36:39 2013 From: python-checkins at python.org (guido.van.rossum) Date: Tue, 3 Dec 2013 03:36:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_Improve_default?= =?utf-8?q?_writelines=28=29=2E?= Message-ID: <3dYS4l5x28z7LjY@mail.python.org> http://hg.python.org/cpython/rev/9283a9c5d0ce changeset: 87730:9283a9c5d0ce user: Guido van Rossum date: Mon Dec 02 18:36:30 2013 -0800 summary: asyncio: Improve default writelines(). files: Lib/asyncio/transports.py | 16 +++++++++--- Lib/test/test_asyncio/test_transports.py | 10 ++++++- 2 files changed, 20 insertions(+), 6 deletions(-) diff --git a/Lib/asyncio/transports.py b/Lib/asyncio/transports.py --- a/Lib/asyncio/transports.py +++ b/Lib/asyncio/transports.py @@ -1,5 +1,9 @@ """Abstract Transport class.""" +import sys + +PY34 = sys.version_info >= (3, 4) + __all__ = ['ReadTransport', 'WriteTransport', 'Transport'] @@ -85,11 +89,15 @@ def writelines(self, list_of_data): """Write a list (or any iterable) of data bytes to the transport. - The default implementation just calls write() for each item in - the list/iterable. + The default implementation concatenates the arguments and + calls write() on the result. """ - for data in list_of_data: - self.write(data) + if not PY34: + # In Python 3.3, bytes.join() doesn't handle memoryview. + list_of_data = ( + bytes(data) if isinstance(data, memoryview) else data + for data in list_of_data) + self.write(b''.join(list_of_data)) def write_eof(self): """Close the write end after flushing buffered data. diff --git a/Lib/test/test_asyncio/test_transports.py b/Lib/test/test_asyncio/test_transports.py --- a/Lib/test/test_asyncio/test_transports.py +++ b/Lib/test/test_asyncio/test_transports.py @@ -24,12 +24,18 @@ transport = transports.Transport() transport.write = unittest.mock.Mock() - transport.writelines(['line1', 'line2', 'line3']) - self.assertEqual(3, transport.write.call_count) + transport.writelines([b'line1', + bytearray(b'line2'), + memoryview(b'line3')]) + self.assertEqual(1, transport.write.call_count) + transport.write.assert_called_with(b'line1line2line3') def test_not_implemented(self): transport = transports.Transport() + self.assertRaises(NotImplementedError, + transport.set_write_buffer_limits) + self.assertRaises(NotImplementedError, transport.get_write_buffer_size) self.assertRaises(NotImplementedError, transport.write, 'data') self.assertRaises(NotImplementedError, transport.write_eof) self.assertRaises(NotImplementedError, transport.can_write_eof) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 09:41:42 2013 From: python-checkins at python.org (antoine.pitrou) Date: Tue, 3 Dec 2013 09:41:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319872=3A_remove_u?= =?utf-8?q?nused_imports_in_pathlib=2E__Patch_by_Vajrasky_Kok=2E?= Message-ID: <3dYc9y6v52z7Ljs@mail.python.org> http://hg.python.org/cpython/rev/a6245b10e8b6 changeset: 87731:a6245b10e8b6 user: Antoine Pitrou date: Tue Dec 03 09:41:35 2013 +0100 summary: Issue #19872: remove unused imports in pathlib. Patch by Vajrasky Kok. files: Lib/pathlib.py | 11 ++--------- 1 files changed, 2 insertions(+), 9 deletions(-) diff --git a/Lib/pathlib.py b/Lib/pathlib.py --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -6,19 +6,12 @@ import posixpath import re import sys -import weakref -try: - import threading -except ImportError: - import dummy_threading as threading - -from collections import Sequence, defaultdict +from collections import Sequence from contextlib import contextmanager from errno import EINVAL, ENOENT -from itertools import chain, count from operator import attrgetter from stat import S_ISDIR, S_ISLNK, S_ISREG, S_ISSOCK, S_ISBLK, S_ISCHR, S_ISFIFO -from urllib.parse import quote as urlquote, quote_from_bytes as urlquote_from_bytes +from urllib.parse import quote_from_bytes as urlquote_from_bytes supports_symlinks = True -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Tue Dec 3 09:42:28 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 03 Dec 2013 09:42:28 +0100 Subject: [Python-checkins] Daily reference leaks (29d67ac7a9d5): sum=4 Message-ID: results for 29d67ac7a9d5 on branch "default" -------------------------------------------- test_asyncio leaked [0, 0, 4] memory blocks, sum=4 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/refloggcqpSK', '-x'] From python-checkins at python.org Tue Dec 3 09:59:35 2013 From: python-checkins at python.org (antoine.pitrou) Date: Tue, 3 Dec 2013 09:59:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319800=3A_make_the?= =?utf-8?q?_pickle_framing_tests_more_precise=2E?= Message-ID: <3dYcZb1mW8z7LjR@mail.python.org> http://hg.python.org/cpython/rev/1c04427fff07 changeset: 87732:1c04427fff07 user: Antoine Pitrou date: Tue Dec 03 09:51:40 2013 +0100 summary: Issue #19800: make the pickle framing tests more precise. files: Lib/test/pickletester.py | 22 ++++++++++++++++++++++ 1 files changed, 22 insertions(+), 0 deletions(-) diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -1416,6 +1416,26 @@ FRAME_SIZE_TARGET = 64 * 1024 + def check_frame_opcodes(self, pickled): + """ + Check the arguments of FRAME opcodes in a protocol 4+ pickle. + """ + frame_opcode_size = 9 + last_arg = last_pos = None + for op, arg, pos in pickletools.genops(pickled): + if op.name != 'FRAME': + continue + if last_pos is not None: + # The previous frame's size should be equal to the number + # of bytes up to the current frame. + frame_size = pos - last_pos - frame_opcode_size + self.assertEqual(frame_size, last_arg) + last_arg, last_pos = arg, pos + # The last frame's size should be equal to the number of bytes up + # to the pickle's end. + frame_size = len(pickled) - last_pos - frame_opcode_size + self.assertEqual(frame_size, last_arg) + def test_framing_many_objects(self): obj = list(range(10**5)) for proto in range(4, pickle.HIGHEST_PROTOCOL + 1): @@ -1429,6 +1449,7 @@ self.FRAME_SIZE_TARGET / 2) self.assertLessEqual(bytes_per_frame, self.FRAME_SIZE_TARGET * 1) + self.check_frame_opcodes(pickled) def test_framing_large_objects(self): N = 1024 * 1024 @@ -1440,6 +1461,7 @@ self.assertEqual(obj, unpickled) n_frames = count_opcode(pickle.FRAME, pickled) self.assertGreaterEqual(n_frames, len(obj)) + self.check_frame_opcodes(pickled) def test_optional_frames(self): if pickle.HIGHEST_PROTOCOL < 4: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 11:01:16 2013 From: python-checkins at python.org (antoine.pitrou) Date: Tue, 3 Dec 2013 11:01:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_a_test_for_complex_sym?= =?utf-8?q?links=2E?= Message-ID: <3dYdxm72xVz7LjW@mail.python.org> http://hg.python.org/cpython/rev/6e09ca4d61df changeset: 87733:6e09ca4d61df user: Antoine Pitrou date: Tue Dec 03 11:01:08 2013 +0100 summary: Add a test for complex symlinks. files: Lib/test/test_pathlib.py | 31 ++++++++++++++++++--------- 1 files changed, 21 insertions(+), 10 deletions(-) diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -1093,20 +1093,21 @@ with open(join('dirC', 'dirD', 'fileD'), 'wb') as f: f.write(b"this is file D\n") if not symlink_skip_reason: - if os.name == 'nt': - # Workaround for http://bugs.python.org/issue13772 - def dirlink(src, dest): - os.symlink(src, dest, target_is_directory=True) - else: - def dirlink(src, dest): - os.symlink(src, dest) # Relative symlinks os.symlink('fileA', join('linkA')) os.symlink('non-existing', join('brokenLink')) - dirlink('dirB', join('linkB')) - dirlink(os.path.join('..', 'dirB'), join('dirA', 'linkC')) + self.dirlink('dirB', join('linkB')) + self.dirlink(os.path.join('..', 'dirB'), join('dirA', 'linkC')) # This one goes upwards but doesn't create a loop - dirlink(os.path.join('..', 'dirB'), join('dirB', 'linkD')) + self.dirlink(os.path.join('..', 'dirB'), join('dirB', 'linkD')) + + if os.name == 'nt': + # Workaround for http://bugs.python.org/issue13772 + def dirlink(self, src, dest): + os.symlink(src, dest, target_is_directory=True) + else: + def dirlink(self, src, dest): + os.symlink(src, dest) def assertSame(self, path_a, path_b): self.assertTrue(os.path.samefile(str(path_a), str(path_b)), @@ -1269,6 +1270,16 @@ p = P(BASE, 'dirA', 'linkX', 'linkY', 'fileB') self._check_resolve_absolute(p, P(BASE, 'dirB', 'fileB')) + @with_symlinks + def test_resolve_dot(self): + # See https://bitbucket.org/pitrou/pathlib/issue/9/pathresolve-fails-on-complex-symlinks + p = self.cls(BASE) + self.dirlink('.', join('0')) + self.dirlink('0/0', join('1')) + self.dirlink('1/1', join('2')) + q = p / '2' + self.assertEqual(q.resolve(), p) + def test_with(self): p = self.cls(BASE) it = p.iterdir() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 12:30:07 2013 From: python-checkins at python.org (vinay.sajip) Date: Tue, 3 Dec 2013 12:30:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5NjY1?= =?utf-8?q?=3A_Increased_timeout_for_SMTPHandler_test=2E?= Message-ID: <3dYgwH3MPYz7Lk4@mail.python.org> http://hg.python.org/cpython/rev/4c5fc9f08b4d changeset: 87734:4c5fc9f08b4d branch: 3.3 parent: 87705:a270370ec487 user: Vinay Sajip date: Tue Dec 03 11:28:55 2013 +0000 summary: Issue #19665: Increased timeout for SMTPHandler test. files: Lib/test/test_logging.py | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -959,19 +959,21 @@ @unittest.skipUnless(threading, 'Threading required for this test.') class SMTPHandlerTest(BaseTest): + TIMEOUT = 8.0 def test_basic(self): sockmap = {} server = TestSMTPServer(('localhost', 0), self.process_message, 0.001, sockmap) server.start() addr = ('localhost', server.port) - h = logging.handlers.SMTPHandler(addr, 'me', 'you', 'Log', timeout=5.0) + h = logging.handlers.SMTPHandler(addr, 'me', 'you', 'Log', + timeout=self.TIMEOUT) self.assertEqual(h.toaddrs, ['you']) self.messages = [] r = logging.makeLogRecord({'msg': 'Hello'}) self.handled = threading.Event() h.handle(r) - self.handled.wait(5.0) # 14314: don't wait forever + self.handled.wait(self.TIMEOUT) # 14314: don't wait forever server.stop() self.assertTrue(self.handled.is_set()) self.assertEqual(len(self.messages), 1) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 12:30:08 2013 From: python-checkins at python.org (vinay.sajip) Date: Tue, 3 Dec 2013 12:30:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Closes_=2319665=3A_Merged_fi_from_3=2E3=2E?= Message-ID: <3dYgwJ5GMZz7Ll1@mail.python.org> http://hg.python.org/cpython/rev/bfd45dc46569 changeset: 87735:bfd45dc46569 parent: 87733:6e09ca4d61df parent: 87734:4c5fc9f08b4d user: Vinay Sajip date: Tue Dec 03 11:29:45 2013 +0000 summary: Closes #19665: Merged fi from 3.3. files: Lib/test/test_logging.py | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -912,19 +912,21 @@ @unittest.skipUnless(threading, 'Threading required for this test.') class SMTPHandlerTest(BaseTest): + TIMEOUT = 8.0 def test_basic(self): sockmap = {} server = TestSMTPServer(('localhost', 0), self.process_message, 0.001, sockmap) server.start() addr = ('localhost', server.port) - h = logging.handlers.SMTPHandler(addr, 'me', 'you', 'Log', timeout=5.0) + h = logging.handlers.SMTPHandler(addr, 'me', 'you', 'Log', + timeout=self.TIMEOUT) self.assertEqual(h.toaddrs, ['you']) self.messages = [] r = logging.makeLogRecord({'msg': 'Hello'}) self.handled = threading.Event() h.handle(r) - self.handled.wait(5.0) # 14314: don't wait forever + self.handled.wait(self.TIMEOUT) # 14314: don't wait forever server.stop() self.assertTrue(self.handled.is_set()) self.assertEqual(len(self.messages), 1) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 13:31:37 2013 From: python-checkins at python.org (vinay.sajip) Date: Tue, 3 Dec 2013 13:31:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Added_some_diagnostics_to_?= =?utf-8?q?help_with_=2319690=2E?= Message-ID: <3dYjHF4yY8z7Ljs@mail.python.org> http://hg.python.org/cpython/rev/8fe3022af4b3 changeset: 87736:8fe3022af4b3 user: Vinay Sajip date: Tue Dec 03 12:31:23 2013 +0000 summary: Added some diagnostics to help with #19690. files: Lib/test/test_logging.py | 13 ++++++++++++- 1 files changed, 12 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -590,6 +590,7 @@ for _ in range(tries): try: os.unlink(fname) + self.deletion_time = time.time() except OSError: pass time.sleep(0.004 * random.randint(0, 4)) @@ -597,6 +598,9 @@ del_count = 500 log_count = 500 + self.handle_time = None + self.deletion_time = None + for delay in (False, True): fd, fn = tempfile.mkstemp('.log', 'test_logging-3-') os.close(fd) @@ -610,7 +614,14 @@ for _ in range(log_count): time.sleep(0.005) r = logging.makeLogRecord({'msg': 'testing' }) - h.handle(r) + try: + self.handle_time = time.time() + h.handle(r) + except Exception: + print('Deleted at %s, ' + 'opened at %s' % (self.deletion_time, + self.handle_time)) + raise finally: remover.join() h.close() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 13:55:55 2013 From: python-checkins at python.org (stefan.krah) Date: Tue, 3 Dec 2013 13:55:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=239709=3A_Stop_addi?= =?utf-8?q?ng_PyInit=5F=22_+_module=5Fname=27_to_export=5Fsymbols=2E__This?= =?utf-8?q?_is?= Message-ID: <3dYjqH3dPWz7Llr@mail.python.org> http://hg.python.org/cpython/rev/97fb852c5c26 changeset: 87737:97fb852c5c26 user: Stefan Krah date: Tue Dec 03 13:55:20 2013 +0100 summary: Issue #9709: Stop adding PyInit_" + module_name' to export_symbols. This is already done by PyMODINIT_FUNC. files: Lib/distutils/command/build_ext.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/distutils/command/build_ext.py b/Lib/distutils/command/build_ext.py --- a/Lib/distutils/command/build_ext.py +++ b/Lib/distutils/command/build_ext.py @@ -538,7 +538,7 @@ library_dirs=ext.library_dirs, runtime_library_dirs=ext.runtime_library_dirs, extra_postargs=extra_args, - export_symbols=self.get_export_symbols(ext), + export_symbols=ext.export_symbols, debug=self.debug, build_temp=self.build_temp, target_lang=language) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 14:34:20 2013 From: python-checkins at python.org (stefan.krah) Date: Tue, 3 Dec 2013 14:34:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Make_a_couple_of_parameter?= =?utf-8?q?s_constant=2E?= Message-ID: <3dYkgc21xDz7Lk4@mail.python.org> http://hg.python.org/cpython/rev/98b6a1194a68 changeset: 87738:98b6a1194a68 user: Stefan Krah date: Tue Dec 03 14:33:46 2013 +0100 summary: Make a couple of parameters constant. files: Modules/_decimal/libmpdec/mpdecimal.c | 14 +++++++------- Modules/_decimal/libmpdec/mpdecimal.h | 14 +++++++------- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -392,42 +392,42 @@ /* Dynamic decimal */ ALWAYS_INLINE int -mpd_isdynamic(mpd_t *dec) +mpd_isdynamic(const mpd_t *dec) { return !(dec->flags & MPD_STATIC); } /* Static decimal */ ALWAYS_INLINE int -mpd_isstatic(mpd_t *dec) +mpd_isstatic(const mpd_t *dec) { return dec->flags & MPD_STATIC; } /* Data of decimal is dynamic */ ALWAYS_INLINE int -mpd_isdynamic_data(mpd_t *dec) +mpd_isdynamic_data(const mpd_t *dec) { return !(dec->flags & MPD_DATAFLAGS); } /* Data of decimal is static */ ALWAYS_INLINE int -mpd_isstatic_data(mpd_t *dec) +mpd_isstatic_data(const mpd_t *dec) { return dec->flags & MPD_STATIC_DATA; } /* Data of decimal is shared */ ALWAYS_INLINE int -mpd_isshared_data(mpd_t *dec) +mpd_isshared_data(const mpd_t *dec) { return dec->flags & MPD_SHARED_DATA; } /* Data of decimal is const */ ALWAYS_INLINE int -mpd_isconst_data(mpd_t *dec) +mpd_isconst_data(const mpd_t *dec) { return dec->flags & MPD_CONST_DATA; } @@ -597,7 +597,7 @@ /* Copy sign from another decimal */ ALWAYS_INLINE void -mpd_signcpy(mpd_t *result, mpd_t *a) +mpd_signcpy(mpd_t *result, const mpd_t *a) { uint8_t sign = a->flags&MPD_NEG; diff --git a/Modules/_decimal/libmpdec/mpdecimal.h b/Modules/_decimal/libmpdec/mpdecimal.h --- a/Modules/_decimal/libmpdec/mpdecimal.h +++ b/Modules/_decimal/libmpdec/mpdecimal.h @@ -752,12 +752,12 @@ /* 1 if dec is positive, -1 if dec is negative */ EXTINLINE int mpd_arith_sign(const mpd_t *dec); EXTINLINE long mpd_radix(void); -EXTINLINE int mpd_isdynamic(mpd_t *dec); -EXTINLINE int mpd_isstatic(mpd_t *dec); -EXTINLINE int mpd_isdynamic_data(mpd_t *dec); -EXTINLINE int mpd_isstatic_data(mpd_t *dec); -EXTINLINE int mpd_isshared_data(mpd_t *dec); -EXTINLINE int mpd_isconst_data(mpd_t *dec); +EXTINLINE int mpd_isdynamic(const mpd_t *dec); +EXTINLINE int mpd_isstatic(const mpd_t *dec); +EXTINLINE int mpd_isdynamic_data(const mpd_t *dec); +EXTINLINE int mpd_isstatic_data(const mpd_t *dec); +EXTINLINE int mpd_isshared_data(const mpd_t *dec); +EXTINLINE int mpd_isconst_data(const mpd_t *dec); EXTINLINE mpd_ssize_t mpd_trail_zeros(const mpd_t *dec); @@ -769,7 +769,7 @@ EXTINLINE void mpd_setdigits(mpd_t *result); EXTINLINE void mpd_set_sign(mpd_t *result, uint8_t sign); /* copy sign from another decimal */ -EXTINLINE void mpd_signcpy(mpd_t *result, mpd_t *a); +EXTINLINE void mpd_signcpy(mpd_t *result, const mpd_t *a); EXTINLINE void mpd_set_infinity(mpd_t *result); EXTINLINE void mpd_set_qnan(mpd_t *result); EXTINLINE void mpd_set_snan(mpd_t *result); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 14:39:32 2013 From: python-checkins at python.org (stefan.krah) Date: Tue, 3 Dec 2013 14:39:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Make_a_couple_?= =?utf-8?q?of_parameters_constant=2E?= Message-ID: <3dYknc62Bcz7LkS@mail.python.org> http://hg.python.org/cpython/rev/afc68547cad6 changeset: 87739:afc68547cad6 branch: 3.3 parent: 87734:4c5fc9f08b4d user: Stefan Krah date: Tue Dec 03 14:33:46 2013 +0100 summary: Make a couple of parameters constant. files: Modules/_decimal/libmpdec/mpdecimal.c | 14 +++++++------- Modules/_decimal/libmpdec/mpdecimal.h | 14 +++++++------- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -392,42 +392,42 @@ /* Dynamic decimal */ ALWAYS_INLINE int -mpd_isdynamic(mpd_t *dec) +mpd_isdynamic(const mpd_t *dec) { return !(dec->flags & MPD_STATIC); } /* Static decimal */ ALWAYS_INLINE int -mpd_isstatic(mpd_t *dec) +mpd_isstatic(const mpd_t *dec) { return dec->flags & MPD_STATIC; } /* Data of decimal is dynamic */ ALWAYS_INLINE int -mpd_isdynamic_data(mpd_t *dec) +mpd_isdynamic_data(const mpd_t *dec) { return !(dec->flags & MPD_DATAFLAGS); } /* Data of decimal is static */ ALWAYS_INLINE int -mpd_isstatic_data(mpd_t *dec) +mpd_isstatic_data(const mpd_t *dec) { return dec->flags & MPD_STATIC_DATA; } /* Data of decimal is shared */ ALWAYS_INLINE int -mpd_isshared_data(mpd_t *dec) +mpd_isshared_data(const mpd_t *dec) { return dec->flags & MPD_SHARED_DATA; } /* Data of decimal is const */ ALWAYS_INLINE int -mpd_isconst_data(mpd_t *dec) +mpd_isconst_data(const mpd_t *dec) { return dec->flags & MPD_CONST_DATA; } @@ -597,7 +597,7 @@ /* Copy sign from another decimal */ ALWAYS_INLINE void -mpd_signcpy(mpd_t *result, mpd_t *a) +mpd_signcpy(mpd_t *result, const mpd_t *a) { uint8_t sign = a->flags&MPD_NEG; diff --git a/Modules/_decimal/libmpdec/mpdecimal.h b/Modules/_decimal/libmpdec/mpdecimal.h --- a/Modules/_decimal/libmpdec/mpdecimal.h +++ b/Modules/_decimal/libmpdec/mpdecimal.h @@ -752,12 +752,12 @@ /* 1 if dec is positive, -1 if dec is negative */ EXTINLINE int mpd_arith_sign(const mpd_t *dec); EXTINLINE long mpd_radix(void); -EXTINLINE int mpd_isdynamic(mpd_t *dec); -EXTINLINE int mpd_isstatic(mpd_t *dec); -EXTINLINE int mpd_isdynamic_data(mpd_t *dec); -EXTINLINE int mpd_isstatic_data(mpd_t *dec); -EXTINLINE int mpd_isshared_data(mpd_t *dec); -EXTINLINE int mpd_isconst_data(mpd_t *dec); +EXTINLINE int mpd_isdynamic(const mpd_t *dec); +EXTINLINE int mpd_isstatic(const mpd_t *dec); +EXTINLINE int mpd_isdynamic_data(const mpd_t *dec); +EXTINLINE int mpd_isstatic_data(const mpd_t *dec); +EXTINLINE int mpd_isshared_data(const mpd_t *dec); +EXTINLINE int mpd_isconst_data(const mpd_t *dec); EXTINLINE mpd_ssize_t mpd_trail_zeros(const mpd_t *dec); @@ -769,7 +769,7 @@ EXTINLINE void mpd_setdigits(mpd_t *result); EXTINLINE void mpd_set_sign(mpd_t *result, uint8_t sign); /* copy sign from another decimal */ -EXTINLINE void mpd_signcpy(mpd_t *result, mpd_t *a); +EXTINLINE void mpd_signcpy(mpd_t *result, const mpd_t *a); EXTINLINE void mpd_set_infinity(mpd_t *result); EXTINLINE void mpd_set_qnan(mpd_t *result); EXTINLINE void mpd_set_snan(mpd_t *result); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 14:39:34 2013 From: python-checkins at python.org (stefan.krah) Date: Tue, 3 Dec 2013 14:39:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Null_merge=2E?= Message-ID: <3dYknf0Xcjz7LkS@mail.python.org> http://hg.python.org/cpython/rev/d435d744109d changeset: 87740:d435d744109d parent: 87738:98b6a1194a68 parent: 87739:afc68547cad6 user: Stefan Krah date: Tue Dec 03 14:38:56 2013 +0100 summary: Null merge. files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 15:06:05 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 3 Dec 2013 15:06:05 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_simplify_pi?= =?utf-8?q?ng_example=2C_remove_the_useless_timeout?= Message-ID: <3dYlNF5tz3z7Ljs@mail.python.org> http://hg.python.org/cpython/rev/9fc428162cbd changeset: 87741:9fc428162cbd user: Victor Stinner date: Tue Dec 03 15:04:18 2013 +0100 summary: asyncio doc: simplify ping example, remove the useless timeout files: Doc/library/asyncio-protocol.rst | 10 +++------- 1 files changed, 3 insertions(+), 7 deletions(-) diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -589,25 +589,21 @@ import asyncio class EchoServer(asyncio.Protocol): - def timeout(self): - print('connection timeout, closing.') - self.transport.close() - def connection_made(self, transport): print('connection made') self.transport = transport - # close the client connection after 2 seconds - asyncio.get_event_loop().call_later(2.0, self.timeout) def data_received(self, data): print('data received:', data.decode()) self.transport.write(data) + # close the socket + self.transport.close() + def connection_lost(self, exc): print('connection lost') - loop = asyncio.get_event_loop() f = loop.create_server(EchoServer, '127.0.0.1', 8888) s = loop.run_until_complete(f) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 15:06:08 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 3 Dec 2013 15:06:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_ayncio=3A_replace_the_disc?= =?utf-8?q?lamer_with_a_seealso_section?= Message-ID: <3dYlNJ17t4z7Llt@mail.python.org> http://hg.python.org/cpython/rev/60d90f8ad38d changeset: 87742:60d90f8ad38d user: Victor Stinner date: Tue Dec 03 15:04:36 2013 +0100 summary: ayncio: replace the disclamer with a seealso section files: Doc/library/asyncio.rst | 15 ++++++--------- 1 files changed, 6 insertions(+), 9 deletions(-) diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -13,7 +13,6 @@ This module provides infrastructure for writing single-threaded concurrent code using coroutines, multiplexing I/O access over sockets and other resources, running network clients and servers, and other related primitives. - Here is a more detailed list of the package contents: * a pluggable :ref:`event loop ` with various system-specific @@ -40,14 +39,7 @@ you absolutely, positively have to use a library that makes blocking I/O calls. - -Disclaimer -========== - -Full documentation is not yet ready; we hope to have it written -before Python 3.4 leaves beta. Until then, the best reference is -:PEP:`3156`. For a motivational primer on transports and protocols, -see :PEP:`3153`. +Table of content: .. toctree:: :maxdepth: 3 @@ -58,3 +50,8 @@ asyncio-protocol.rst asyncio-sync.rst +.. seealso:: + + The :mod:`asyncio` module was designed in the :PEP:`3156`. For a + motivational primer on transports and protocols, see :PEP:`3153`. + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 17:13:21 2013 From: python-checkins at python.org (antoine.pitrou) Date: Tue, 3 Dec 2013 17:13:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319877=3A_fix_regr?= =?utf-8?q?ession_in_test=5Fpathlib_when_Windows_has_symlink_support?= Message-ID: <3dYpC53Fgnz7LlQ@mail.python.org> http://hg.python.org/cpython/rev/076824de7650 changeset: 87743:076824de7650 user: Antoine Pitrou date: Tue Dec 03 17:13:13 2013 +0100 summary: Issue #19877: fix regression in test_pathlib when Windows has symlink support available (i.e. in administrator mode). Patch by Vajrasky Kok. files: Lib/test/test_pathlib.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -1275,8 +1275,8 @@ # See https://bitbucket.org/pitrou/pathlib/issue/9/pathresolve-fails-on-complex-symlinks p = self.cls(BASE) self.dirlink('.', join('0')) - self.dirlink('0/0', join('1')) - self.dirlink('1/1', join('2')) + self.dirlink(os.path.join('0', '0'), join('1')) + self.dirlink(os.path.join('1', '1'), join('2')) q = p / '2' self.assertEqual(q.resolve(), p) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 17:38:04 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 3 Dec 2013 17:38:04 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_add_one_mor?= =?utf-8?q?e_example_of_coroutines?= Message-ID: <3dYplc3XHkz7Lkk@mail.python.org> http://hg.python.org/cpython/rev/486e784d5fc2 changeset: 87744:486e784d5fc2 user: Victor Stinner date: Tue Dec 03 17:37:31 2013 +0100 summary: asyncio doc: add one more example of coroutines files: Doc/library/asyncio-task.rst | 66 +++++++++++++++++++++++- 1 files changed, 65 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -231,10 +231,14 @@ the timeout occurs are returned in the second set. +Examples +-------- + + .. _asyncio-hello-world-coroutine: Example: Hello World (coroutine) --------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Print ``Hello World`` every two seconds, using a coroutine:: @@ -253,3 +257,63 @@ .. seealso:: :ref:`Hello World example using a callback `. + +Example: Chains coroutines and parallel execution +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Example chaining coroutines and executing multiple coroutines in parallel:: + + import asyncio + + @asyncio.coroutine + def compute(x, y): + print("Start computing %s + %s" % (x, y)) + yield from asyncio.sleep(3.0) + return x + y + + @asyncio.coroutine + def print_sum(x, y): + result = yield from compute(x, y) + print("%s + %s = %s" % (x, y, result)) + + @asyncio.coroutine + def wait_task(task): + while 1: + done, pending = yield from asyncio.wait([task], timeout=1.0) + if done: + break + print("Compute in progress...") + asyncio.get_event_loop().stop() + + print("Schedule tasks") + task = asyncio.async(print_sum(1, 2)) + asyncio.async(wait_task(task)) + + print("Execute tasks") + loop = asyncio.get_event_loop() + loop.run_forever() + loop.close() + + + +Output:: + + Schedule tasks + Execute tasks + Start computing 1 + 2 + Compute in progress... + Compute in progress... + 1 + 2 = 3 + +Details: + +* ``compute()`` is chained to ``print_sum()``: ``print_sum()`` coroutine waits + until ``compute()`` is complete. Coroutines are executed in parallel: + ``wait_task()`` is executed while ``compute()`` is blocked in + ``asyncio.sleep(3.0)``. + +* Coroutines are not executed before the loop is running: ``"Execute tasks"`` + is written before ``"Start computing 1 + 2"``. + +* ``wait_task()`` stops the event loop when ``print_sum()`` is done. + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 18:24:15 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 3 Dec 2013 18:24:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_typo_in_asyncio=2EAbst?= =?utf-8?q?ractServer_documentation?= Message-ID: <3dYqmv0591z7Lkt@mail.python.org> http://hg.python.org/cpython/rev/c061dbfd6842 changeset: 87745:c061dbfd6842 user: Victor Stinner date: Tue Dec 03 18:23:52 2013 +0100 summary: Fix typo in asyncio.AbstractServer documentation files: Doc/library/asyncio-protocol.rst | 2 +- Lib/asyncio/events.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -515,7 +515,7 @@ .. class:: AbstractServer - Abstract server returned by create_service(). + Abstract server returned by :func:`BaseEventLoop.create_server`. .. method:: close() diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py --- a/Lib/asyncio/events.py +++ b/Lib/asyncio/events.py @@ -99,7 +99,7 @@ class AbstractServer: - """Abstract server returned by create_service().""" + """Abstract server returned by create_server().""" def close(self): """Stop serving. This leaves existing connections open.""" -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 3 19:17:42 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 3 Dec 2013 19:17:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_oh=2C_async?= =?utf-8?q?io_module_has_its_own_Future_class?= Message-ID: <3dYryZ3ZQ9z7Ll3@mail.python.org> http://hg.python.org/cpython/rev/f2c7f771bf6a changeset: 87746:f2c7f771bf6a user: Victor Stinner date: Tue Dec 03 19:17:25 2013 +0100 summary: asyncio doc: oh, asyncio module has its own Future class Improve also wait() documentation: mention that the first parameter is a sequence files: Doc/library/asyncio-eventloop.rst | 2 +- Doc/library/asyncio-protocol.rst | 7 +- Doc/library/asyncio-task.rst | 115 ++++++++++++++++- Doc/library/asyncio.rst | 4 +- 4 files changed, 110 insertions(+), 18 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -61,7 +61,7 @@ .. method:: BaseEventLoop.run_until_complete(future) - Run until the :class:`~concurrent.futures.Future` is done. + Run until the :class:`Future` is done. If the argument is a coroutine, it is wrapped in a :class:`Task`. diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -235,8 +235,9 @@ Wraps a Transport. - This exposes :meth:`write`, :meth:`writelines`, :meth:`can_write_eof()`, :meth:`write_eof`, :meth:`get_extra_info` and - :meth:`close`. It adds :meth:`drain` which returns an optional :class:`~concurrent.futures.Future` on which you can + This exposes :meth:`write`, :meth:`writelines`, :meth:`can_write_eof()`, + :meth:`write_eof`, :meth:`get_extra_info` and :meth:`close`. It adds + :meth:`drain` which returns an optional :class:`Future` on which you can wait for flow control. It also adds a transport attribute which references the :class:`Transport` directly. @@ -260,7 +261,7 @@ When there's nothing to wait for, :meth:`drain()` returns ``()``, and the yield-from continues immediately. When the transport buffer is full (the protocol is paused), :meth:`drain` creates and returns a - :class:`~concurrent.futures.Future` and the yield-from will block until + :class:`Future` and the yield-from will block until that Future is completed, which will happen when the buffer is (partially) drained and the protocol is resumed. diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -56,12 +56,104 @@ Coroutines (and tasks) can only run when the event loop is running. +InvalidStateError +----------------- + +.. exception:: InvalidStateError + + The operation is not allowed in this state. + + +Future +------ + +.. class:: Future(\*, loop=None) + + This class is *almost* compatible with :class:`concurrent.futures.Future`. + + Differences: + + - :meth:`result` and :meth:`exception` do not take a timeout argument and + raise an exception when the future isn't done yet. + + - Callbacks registered with :meth:`add_done_callback` are always called + via the event loop's :meth:`~BaseEventLoop.call_soon_threadsafe`. + + - This class is not compatible with the :func:`~concurrent.futures.wait` and + :func:`~concurrent.futures.as_completed` functions in the + :mod:`concurrent.futures` package. + + .. method:: cancel() + + Cancel the future and schedule callbacks. + + If the future is already done or cancelled, return ``False``. Otherwise, + change the future's state to cancelled, schedule the callbacks and return + ``True``. + + .. method:: cancelled() + + Return ``True`` if the future was cancelled. + + .. method:: done() + + Return True if the future is done. + + Done means either that a result / exception are available, or that the + future was cancelled. + + .. method:: result() + + Return the result this future represents. + + If the future has been cancelled, raises :exc:`CancelledError`. If the + future's result isn't yet available, raises :exc:`InvalidStateError`. If + the future is done and has an exception set, this exception is raised. + + .. method:: exception() + + Return the exception that was set on this future. + + The exception (or ``None`` if no exception was set) is returned only if + the future is done. If the future has been cancelled, raises + :exc:`CancelledError`. If the future isn't done yet, raises + :exc:`InvalidStateError`. + + .. method:: add_done_callback(fn) + + Add a callback to be run when the future becomes done. + + The callback is called with a single argument - the future object. If the + future is already done when this is called, the callback is scheduled + with :meth:`~BaseEventLoop.call_soon`. + + .. method:: remove_done_callback(fn) + + Remove all instances of a callback from the "call when done" list. + + Returns the number of callbacks removed. + + .. method:: set_result(result) + + Mark the future done and set its result. + + If the future is already done when this method is called, raises + :exc:`InvalidStateError`. + + .. method:: set_exception(exception) + + Mark the future done and set an exception. + + If the future is already done when this method is called, raises + :exc:`InvalidStateError`. + + Task ---- .. class:: Task(coro, \*, loop=None) - A coroutine wrapped in a :class:`~concurrent.futures.Future`. + A coroutine wrapped in a :class:`Future`. Subclass of :class:`Future`. .. classmethod:: all_tasks(loop=None) @@ -106,10 +198,10 @@ Task functions -------------- -.. function:: as_completed(fs, *, loop=None, timeout=None) +.. function:: as_completed(fs, \*, loop=None, timeout=None) - Return an iterator whose values, when waited for, are - :class:`~concurrent.futures.Future` instances. + Return an iterator whose values, when waited for, are :class:`Future` + instances. Raises :exc:`TimeoutError` if the timeout occurs before all Futures are done. @@ -123,14 +215,13 @@ The futures ``f`` are not necessarily members of fs. -.. function:: async(coro_or_future, *, loop=None) +.. function:: async(coro_or_future, \*, loop=None) Wrap a :ref:`coroutine ` in a future. - If the argument is a :class:`~concurrent.futures.Future`, it is returned - directly. + If the argument is a :class:`Future`, it is returned directly. -.. function:: gather(*coros_or_futures, loop=None, return_exceptions=False) +.. function:: gather(\*coros_or_futures, loop=None, return_exceptions=False) Return a future aggregating results from the given coroutines or futures. @@ -188,11 +279,11 @@ except CancelledError: res = None -.. function:: wait(fs, \*, loop=None, timeout=None, return_when=ALL_COMPLETED) +.. function:: wait(futures, \*, loop=None, timeout=None, return_when=ALL_COMPLETED) - Wait for the Futures and coroutines given by fs to complete. Coroutines will - be wrapped in Tasks. Returns two sets of - :class:`~concurrent.futures.Future`: (done, pending). + Wait for the Futures and coroutines given by the sequence *futures* to + complete. Coroutines will be wrapped in Tasks. Returns two sets of + :class:`Future`: (done, pending). *timeout* can be used to control the maximum number of seconds to wait before returning. *timeout* can be an int or float. If *timeout* is not specified diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -24,8 +24,8 @@ * concrete support for TCP, UDP, SSL, subprocess pipes, delayed calls, and others (some may be system-dependent); -* a Future class that mimicks the one in the :mod:`concurrent.futures` module, - but adapted for use with the event loop; +* a :class:`Future` class that mimicks the one in the :mod:`concurrent.futures` + module, but adapted for use with the event loop; * coroutines and tasks based on ``yield from`` (:PEP:`380`), to help write concurrent code in a sequential fashion; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 00:45:19 2013 From: python-checkins at python.org (victor.stinner) Date: Wed, 4 Dec 2013 00:45:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Close_=2319827=3A_On_UNIX?= =?utf-8?q?=2C_setblocking=28=29_and_settimeout=28=29_methods_of_socket=2E?= =?utf-8?q?socket?= Message-ID: <3dZ0Db5cHdz7LjV@mail.python.org> http://hg.python.org/cpython/rev/5f0d1aad7322 changeset: 87747:5f0d1aad7322 user: Victor Stinner date: Wed Dec 04 00:41:24 2013 +0100 summary: Close #19827: On UNIX, setblocking() and settimeout() methods of socket.socket can now avoid a second syscall if the ioctl() function can be used, or if the non-blocking flag of the socket is unchanged. files: Misc/NEWS | 4 ++++ Modules/socketmodule.c | 18 ++++++++++-------- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,10 @@ Library ------- +- Issue #19827: On UNIX, setblocking() and settimeout() methods of + socket.socket can now avoid a second syscall if the ioctl() function can be + used, or if the non-blocking flag of the socket is unchanged. + - Issue #19785: smtplib now supports SSLContext.check_hostname and server name indication for TLS/SSL connections. diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -585,8 +585,9 @@ static int internal_setblocking(PySocketSockObject *s, int block) { -#ifndef MS_WINDOWS - int delay_flag; +#if !defined(MS_WINDOWS) \ + && !((defined(HAVE_SYS_IOCTL_H) && defined(FIONBIO)) || defined(__VMS)) + int delay_flag, new_delay_flag; #endif #ifdef SOCK_NONBLOCK if (block) @@ -597,17 +598,18 @@ Py_BEGIN_ALLOW_THREADS #ifndef MS_WINDOWS -#if defined(__VMS) +#if (defined(HAVE_SYS_IOCTL_H) && defined(FIONBIO)) || defined(__VMS) block = !block; ioctl(s->sock_fd, FIONBIO, (unsigned int *)&block); -#else /* !__VMS */ +#else delay_flag = fcntl(s->sock_fd, F_GETFL, 0); if (block) - delay_flag &= (~O_NONBLOCK); + new_delay_flag = delay_flag & (~O_NONBLOCK); else - delay_flag |= O_NONBLOCK; - fcntl(s->sock_fd, F_SETFL, delay_flag); -#endif /* !__VMS */ + new_delay_flag = delay_flag | O_NONBLOCK; + if (new_delay_flag != delay_flag) + fcntl(s->sock_fd, F_SETFL, new_delay_flag); +#endif #else /* MS_WINDOWS */ block = !block; ioctlsocket(s->sock_fd, FIONBIO, (u_long*)&block); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 01:29:49 2013 From: python-checkins at python.org (victor.stinner) Date: Wed, 4 Dec 2013 01:29:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Close_=2319757=3A_Cleanup_?= =?utf-8?q?tracemalloc=2C_move?= Message-ID: <3dZ1Cx5H1Yz7LjS@mail.python.org> http://hg.python.org/cpython/rev/194f74044537 changeset: 87748:194f74044537 user: Victor Stinner date: Wed Dec 04 01:29:35 2013 +0100 summary: Close #19757: Cleanup tracemalloc, move PyGILState_Ensure()/PyGILState_Release() calls to the raw wrappers to simplify the code. Rename also tracemalloc_log_alloc/log_free() to tracemalloc_add_trace/remove_trace(). files: Modules/_tracemalloc.c | 231 ++++++++++++++++------------ 1 files changed, 131 insertions(+), 100 deletions(-) diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -439,7 +439,7 @@ } static int -tracemalloc_log_alloc(void *ptr, size_t size) +tracemalloc_add_trace(void *ptr, size_t size) { traceback_t *traceback; trace_t trace; @@ -470,7 +470,7 @@ } static void -tracemalloc_log_free(void *ptr) +tracemalloc_remove_trace(void *ptr) { trace_t trace; @@ -483,118 +483,57 @@ } static void* -tracemalloc_malloc(void *ctx, size_t size, int gil_held) +tracemalloc_malloc(void *ctx, size_t size) { PyMemAllocator *alloc = (PyMemAllocator *)ctx; -#if defined(TRACE_RAW_MALLOC) && defined(WITH_THREAD) - PyGILState_STATE gil_state; -#endif void *ptr; - if (get_reentrant()) { - return alloc->malloc(alloc->ctx, size); + ptr = alloc->malloc(alloc->ctx, size); + if (ptr == NULL) + return NULL; + + if (tracemalloc_add_trace(ptr, size) < 0) { + /* Failed to allocate a trace for the new memory block */ + alloc->free(alloc->ctx, ptr); + return NULL; } - - /* Ignore reentrant call. PyObjet_Malloc() calls PyMem_Malloc() - for allocations larger than 512 bytes. PyGILState_Ensure() may call - PyMem_RawMalloc() indirectly which would call PyGILState_Ensure() if - reentrant are not disabled. */ - set_reentrant(1); -#ifdef WITH_THREAD -#ifdef TRACE_RAW_MALLOC - if (!gil_held) - gil_state = PyGILState_Ensure(); -#else - assert(gil_held); -#endif -#endif - ptr = alloc->malloc(alloc->ctx, size); - - if (ptr != NULL) { - if (tracemalloc_log_alloc(ptr, size) < 0) { - /* Memory allocation failed */ - alloc->free(alloc->ctx, ptr); - ptr = NULL; - } - } - set_reentrant(0); - -#if defined(TRACE_RAW_MALLOC) && defined(WITH_THREAD) - if (!gil_held) - PyGILState_Release(gil_state); -#endif - return ptr; } static void* -tracemalloc_realloc(void *ctx, void *ptr, size_t new_size, int gil_held) +tracemalloc_realloc(void *ctx, void *ptr, size_t new_size) { PyMemAllocator *alloc = (PyMemAllocator *)ctx; -#if defined(TRACE_RAW_MALLOC) && defined(WITH_THREAD) - PyGILState_STATE gil_state; -#endif void *ptr2; - if (get_reentrant()) { - /* Reentrant call to PyMem_Realloc() and PyMem_RawRealloc(). - Example: PyMem_RawRealloc() is called internally by pymalloc - (_PyObject_Malloc() and _PyObject_Realloc()) to allocate a new - arena (new_arena()). */ - ptr2 = alloc->realloc(alloc->ctx, ptr, new_size); + ptr2 = alloc->realloc(alloc->ctx, ptr, new_size); + if (ptr2 == NULL) + return NULL; - if (ptr2 != NULL && ptr != NULL) - tracemalloc_log_free(ptr); + if (ptr != NULL) { + /* an existing memory block has been resized */ - return ptr2; - } + tracemalloc_remove_trace(ptr); - /* Ignore reentrant call. PyObjet_Realloc() calls PyMem_Realloc() for - allocations larger than 512 bytes. PyGILState_Ensure() may call - PyMem_RawMalloc() indirectly which would call PyGILState_Ensure() if - reentrant are not disabled. */ - set_reentrant(1); -#ifdef WITH_THREAD -#ifdef TRACE_RAW_MALLOC - if (!gil_held) - gil_state = PyGILState_Ensure(); -#else - assert(gil_held); -#endif -#endif - ptr2 = alloc->realloc(alloc->ctx, ptr, new_size); + if (tracemalloc_add_trace(ptr2, new_size) < 0) { + /* Memory allocation failed. The error cannot be reported to + the caller, because realloc() may already have shrinked the + memory block and so removed bytes. - if (ptr2 != NULL) { - if (ptr != NULL) { - /* resize */ - tracemalloc_log_free(ptr); - - if (tracemalloc_log_alloc(ptr2, new_size) < 0) { - /* Memory allocation failed. The error cannot be reported to - the caller, because realloc() may already have shrinked the - memory block and so removed bytes. - - This case is very unlikely since we just released an hash - entry, so we have enough free bytes to allocate the new - entry. */ - } - } - else { - /* new allocation */ - if (tracemalloc_log_alloc(ptr2, new_size) < 0) { - /* Memory allocation failed */ - alloc->free(alloc->ctx, ptr2); - ptr2 = NULL; - } + This case is very unlikely since we just released an hash + entry, so we have enough free bytes to allocate the new + entry. */ } } - set_reentrant(0); + else { + /* new allocation */ -#if defined(TRACE_RAW_MALLOC) && defined(WITH_THREAD) - if (!gil_held) - PyGILState_Release(gil_state); -#endif - + if (tracemalloc_add_trace(ptr2, new_size) < 0) { + /* Failed to allocate a trace for the new memory block */ + alloc->free(alloc->ctx, ptr2); + return NULL; + } + } return ptr2; } @@ -610,34 +549,126 @@ a deadlock in PyThreadState_DeleteCurrent(). */ alloc->free(alloc->ctx, ptr); - tracemalloc_log_free(ptr); + tracemalloc_remove_trace(ptr); } static void* tracemalloc_malloc_gil(void *ctx, size_t size) { - return tracemalloc_malloc(ctx, size, 1); + void *ptr; + + if (get_reentrant()) { + PyMemAllocator *alloc = (PyMemAllocator *)ctx; + return alloc->malloc(alloc->ctx, size); + } + + /* Ignore reentrant call. PyObjet_Malloc() calls PyMem_Malloc() for + allocations larger than 512 bytes, don't trace the same memory + allocation twice. */ + set_reentrant(1); + + ptr = tracemalloc_malloc(ctx, size); + + set_reentrant(0); + return ptr; } static void* tracemalloc_realloc_gil(void *ctx, void *ptr, size_t new_size) { - return tracemalloc_realloc(ctx, ptr, new_size, 1); + void *ptr2; + + if (get_reentrant()) { + /* Reentrant call to PyMem_Realloc() and PyMem_RawRealloc(). + Example: PyMem_RawRealloc() is called internally by pymalloc + (_PyObject_Malloc() and _PyObject_Realloc()) to allocate a new + arena (new_arena()). */ + PyMemAllocator *alloc = (PyMemAllocator *)ctx; + + ptr2 = alloc->realloc(alloc->ctx, ptr, new_size); + if (ptr2 != NULL && ptr != NULL) + tracemalloc_remove_trace(ptr); + return ptr2; + } + + /* Ignore reentrant call. PyObjet_Realloc() calls PyMem_Realloc() for + allocations larger than 512 bytes. Don't trace the same memory + allocation twice. */ + set_reentrant(1); + + ptr2 = tracemalloc_realloc(ctx, ptr, new_size); + + set_reentrant(0); + return ptr2; } #ifdef TRACE_RAW_MALLOC static void* tracemalloc_raw_malloc(void *ctx, size_t size) { - return tracemalloc_malloc(ctx, size, 0); +#ifdef WITH_THREAD + PyGILState_STATE gil_state; +#endif + void *ptr; + + if (get_reentrant()) { + PyMemAllocator *alloc = (PyMemAllocator *)ctx; + return alloc->malloc(alloc->ctx, size); + } + + /* Ignore reentrant call. PyGILState_Ensure() may call PyMem_RawMalloc() + indirectly which would call PyGILState_Ensure() if reentrant are not + disabled. */ + set_reentrant(1); + +#ifdef WITH_THREAD + gil_state = PyGILState_Ensure(); + ptr = tracemalloc_malloc(ctx, size); + PyGILState_Release(gil_state); +#else + ptr = tracemalloc_malloc(ctx, size); +#endif + + set_reentrant(0); + return ptr; } static void* tracemalloc_raw_realloc(void *ctx, void *ptr, size_t new_size) { - return tracemalloc_realloc(ctx, ptr, new_size, 0); +#ifdef WITH_THREAD + PyGILState_STATE gil_state; +#endif + void *ptr2; + + if (get_reentrant()) { + /* Reentrant call to PyMem_RawRealloc(). */ + PyMemAllocator *alloc = (PyMemAllocator *)ctx; + + ptr2 = alloc->realloc(alloc->ctx, ptr, new_size); + if (ptr2 != NULL && ptr != NULL) + tracemalloc_remove_trace(ptr); + + return ptr2; + } + + /* Ignore reentrant call. PyGILState_Ensure() may call PyMem_RawMalloc() + indirectly which would call PyGILState_Ensure() if reentrant calls are + not disabled. */ + set_reentrant(1); + +#ifdef WITH_THREAD + gil_state = PyGILState_Ensure(); + ptr2 = tracemalloc_realloc(ctx, ptr, new_size); + PyGILState_Release(gil_state); +#else + ptr2 = tracemalloc_realloc(ctx, ptr, new_size); +#endif + + set_reentrant(0); + return ptr2; } -#endif +#endif /* TRACE_RAW_MALLOC */ static int tracemalloc_clear_filename(_Py_hashtable_entry_t *entry, void *user_data) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 01:48:04 2013 From: python-checkins at python.org (victor.stinner) Date: Wed, 4 Dec 2013 01:48:04 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Close_=2319741=3A_tracemal?= =?utf-8?q?loc=5Frealloc=28=29_does_not_release_the_table_lock_anymore?= Message-ID: <3dZ1d01N21z7LjP@mail.python.org> http://hg.python.org/cpython/rev/3c34ab550358 changeset: 87749:3c34ab550358 user: Victor Stinner date: Wed Dec 04 01:47:46 2013 +0100 summary: Close #19741: tracemalloc_realloc() does not release the table lock anymore between tracemalloc_remove_trace() and tracemalloc_add_trace() to reduce the risk of race condition. tracemalloc_add_trace() cannot fail anymore in tracemalloc_realloc() when tracemalloc_realloc() resizes a memory block. files: Modules/_tracemalloc.c | 36 +++++++++++++++++++++-------- 1 files changed, 26 insertions(+), 10 deletions(-) diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -456,7 +456,6 @@ trace.size = size; trace.traceback = traceback; - TABLES_LOCK(); res = _Py_HASHTABLE_SET(tracemalloc_traces, ptr, trace); if (res == 0) { assert(tracemalloc_traced_memory <= PY_SIZE_MAX - size); @@ -464,7 +463,6 @@ if (tracemalloc_traced_memory > tracemalloc_peak_traced_memory) tracemalloc_peak_traced_memory = tracemalloc_traced_memory; } - TABLES_UNLOCK(); return res; } @@ -474,12 +472,10 @@ { trace_t trace; - TABLES_LOCK(); if (_Py_hashtable_pop(tracemalloc_traces, ptr, &trace, sizeof(trace))) { assert(tracemalloc_traced_memory >= trace.size); tracemalloc_traced_memory -= trace.size; } - TABLES_UNLOCK(); } static void* @@ -492,11 +488,14 @@ if (ptr == NULL) return NULL; + TABLES_LOCK(); if (tracemalloc_add_trace(ptr, size) < 0) { /* Failed to allocate a trace for the new memory block */ + TABLES_UNLOCK(); alloc->free(alloc->ctx, ptr); return NULL; } + TABLES_UNLOCK(); return ptr; } @@ -513,6 +512,7 @@ if (ptr != NULL) { /* an existing memory block has been resized */ + TABLES_LOCK(); tracemalloc_remove_trace(ptr); if (tracemalloc_add_trace(ptr2, new_size) < 0) { @@ -520,19 +520,26 @@ the caller, because realloc() may already have shrinked the memory block and so removed bytes. - This case is very unlikely since we just released an hash - entry, so we have enough free bytes to allocate the new - entry. */ + This case is very unlikely: an hash entry has just been + released, so the hash table should have at least one free entry. + + The GIL and the table lock ensures that only one thread is + allocating memory. */ + assert(0 && "should never happen"); } + TABLES_UNLOCK(); } else { /* new allocation */ + TABLES_LOCK(); if (tracemalloc_add_trace(ptr2, new_size) < 0) { /* Failed to allocate a trace for the new memory block */ + TABLES_UNLOCK(); alloc->free(alloc->ctx, ptr2); return NULL; } + TABLES_UNLOCK(); } return ptr2; } @@ -549,7 +556,10 @@ a deadlock in PyThreadState_DeleteCurrent(). */ alloc->free(alloc->ctx, ptr); + + TABLES_LOCK(); tracemalloc_remove_trace(ptr); + TABLES_UNLOCK(); } static void* @@ -586,8 +596,11 @@ PyMemAllocator *alloc = (PyMemAllocator *)ctx; ptr2 = alloc->realloc(alloc->ctx, ptr, new_size); - if (ptr2 != NULL && ptr != NULL) + if (ptr2 != NULL && ptr != NULL) { + TABLES_LOCK(); tracemalloc_remove_trace(ptr); + TABLES_UNLOCK(); + } return ptr2; } @@ -646,9 +659,12 @@ PyMemAllocator *alloc = (PyMemAllocator *)ctx; ptr2 = alloc->realloc(alloc->ctx, ptr, new_size); - if (ptr2 != NULL && ptr != NULL) + + if (ptr2 != NULL && ptr != NULL) { + TABLES_LOCK(); tracemalloc_remove_trace(ptr); - + TABLES_UNLOCK(); + } return ptr2; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 04:02:19 2013 From: python-checkins at python.org (tim.peters) Date: Wed, 4 Dec 2013 04:02:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319138=3A_doctest?= =?utf-8?q?=27s_IGNORE=5FEXCEPTION=5FDETAIL_now_allows_no_detail_at_all=2E?= Message-ID: <3dZ4bv3Jvhz7Lkq@mail.python.org> http://hg.python.org/cpython/rev/c80083ad142d changeset: 87750:c80083ad142d user: Tim Peters date: Tue Dec 03 21:02:05 2013 -0600 summary: Issue #19138: doctest's IGNORE_EXCEPTION_DETAIL now allows no detail at all. files: Lib/doctest.py | 33 ++++++++++++++++++++++++--- Lib/test/test_doctest.py | 27 ++++++++++++++++++++++ Misc/NEWS | 4 +++ 3 files changed, 60 insertions(+), 4 deletions(-) diff --git a/Lib/doctest.py b/Lib/doctest.py --- a/Lib/doctest.py +++ b/Lib/doctest.py @@ -318,6 +318,32 @@ else: return '#' +def _strip_exception_details(msg): + # Support for IGNORE_EXCEPTION_DETAIL. + # Get rid of everything except the exception name; in particular, drop + # the possibly dotted module path (if any) and the exception message (if + # any). We assume that a colon is never part of a dotted name, or of an + # exception name. + # E.g., given + # "foo.bar.MyError: la di da" + # return "MyError" + # Or for "abc.def" or "abc.def:\n" return "def". + + start, end = 0, len(msg) + # The exception name must appear on the first line. + i = msg.find("\n") + if i >= 0: + end = i + # retain up to the first colon (if any) + i = msg.find(':', 0, end) + if i >= 0: + end = i + # retain just the exception name + i = msg.rfind('.', 0, end) + if i >= 0: + start = i+1 + return msg[start: end] + class _OutputRedirectingPdb(pdb.Pdb): """ A specialized version of the python debugger that redirects stdout @@ -1325,10 +1351,9 @@ # Another chance if they didn't care about the detail. elif self.optionflags & IGNORE_EXCEPTION_DETAIL: - m1 = re.match(r'(?:[^:]*\.)?([^:]*:)', example.exc_msg) - m2 = re.match(r'(?:[^:]*\.)?([^:]*:)', exc_msg) - if m1 and m2 and check(m1.group(1), m2.group(1), - self.optionflags): + if check(_strip_exception_details(example.exc_msg), + _strip_exception_details(exc_msg), + self.optionflags): outcome = SUCCESS # Report the outcome. diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py --- a/Lib/test/test_doctest.py +++ b/Lib/test/test_doctest.py @@ -1054,6 +1054,33 @@ ValueError: message TestResults(failed=1, attempted=1) +If the exception does not have a message, you can still use +IGNORE_EXCEPTION_DETAIL to normalize the modules between Python 2 and 3: + + >>> def f(x): + ... r''' + ... >>> from http.client import HTTPException + ... >>> raise HTTPException() #doctest: +IGNORE_EXCEPTION_DETAIL + ... Traceback (most recent call last): + ... foo.bar.HTTPException + ... ''' + >>> test = doctest.DocTestFinder().find(f)[0] + >>> doctest.DocTestRunner(verbose=False).run(test) + TestResults(failed=0, attempted=2) + +Note that a trailing colon doesn't matter either: + + >>> def f(x): + ... r''' + ... >>> from http.client import HTTPException + ... >>> raise HTTPException() #doctest: +IGNORE_EXCEPTION_DETAIL + ... Traceback (most recent call last): + ... foo.bar.HTTPException: + ... ''' + >>> test = doctest.DocTestFinder().find(f)[0] + >>> doctest.DocTestRunner(verbose=False).run(test) + TestResults(failed=0, attempted=2) + If an exception is raised but not expected, then it is reported as an unexpected exception: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,10 @@ Library ------- +- Issue #19138: doctest's IGNORE_EXCEPTION_DETAIL now allows a match when + no exception detail exists (no colon following the exception's name, or + a colon does follow but no text follows the colon). + - Issue #19827: On UNIX, setblocking() and settimeout() methods of socket.socket can now avoid a second syscall if the ioctl() function can be used, or if the non-blocking flag of the socket is unchanged. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 04:21:40 2013 From: python-checkins at python.org (tim.peters) Date: Wed, 4 Dec 2013 04:21:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5MTM4?= =?utf-8?q?=3A_doctest=27s_IGNORE=5FEXCEPTION=5FDETAIL_now_allows_no_detai?= =?utf-8?q?l_at_all=2E?= Message-ID: <3dZ52D282Cz7Lkq@mail.python.org> http://hg.python.org/cpython/rev/5e14a3435f52 changeset: 87751:5e14a3435f52 branch: 2.7 parent: 87704:181ced5bf0be user: Tim Peters date: Tue Dec 03 21:02:05 2013 -0600 summary: Issue #19138: doctest's IGNORE_EXCEPTION_DETAIL now allows no detail at all. Grafted from c80083ad142d. files: Lib/doctest.py | 33 ++++++++++++++++++++++++--- Lib/test/test_doctest.py | 27 ++++++++++++++++++++++ Misc/NEWS | 4 +++ 3 files changed, 60 insertions(+), 4 deletions(-) diff --git a/Lib/doctest.py b/Lib/doctest.py --- a/Lib/doctest.py +++ b/Lib/doctest.py @@ -326,6 +326,32 @@ else: return '#' +def _strip_exception_details(msg): + # Support for IGNORE_EXCEPTION_DETAIL. + # Get rid of everything except the exception name; in particular, drop + # the possibly dotted module path (if any) and the exception message (if + # any). We assume that a colon is never part of a dotted name, or of an + # exception name. + # E.g., given + # "foo.bar.MyError: la di da" + # return "MyError" + # Or for "abc.def" or "abc.def:\n" return "def". + + start, end = 0, len(msg) + # The exception name must appear on the first line. + i = msg.find("\n") + if i >= 0: + end = i + # retain up to the first colon (if any) + i = msg.find(':', 0, end) + if i >= 0: + end = i + # retain just the exception name + i = msg.rfind('.', 0, end) + if i >= 0: + start = i+1 + return msg[start: end] + class _OutputRedirectingPdb(pdb.Pdb): """ A specialized version of the python debugger that redirects stdout @@ -1323,10 +1349,9 @@ # Another chance if they didn't care about the detail. elif self.optionflags & IGNORE_EXCEPTION_DETAIL: - m1 = re.match(r'(?:[^:]*\.)?([^:]*:)', example.exc_msg) - m2 = re.match(r'(?:[^:]*\.)?([^:]*:)', exc_msg) - if m1 and m2 and check(m1.group(1), m2.group(1), - self.optionflags): + if check(_strip_exception_details(example.exc_msg), + _strip_exception_details(exc_msg), + self.optionflags): outcome = SUCCESS # Report the outcome. diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py --- a/Lib/test/test_doctest.py +++ b/Lib/test/test_doctest.py @@ -1019,6 +1019,33 @@ ValueError: message TestResults(failed=1, attempted=1) +If the exception does not have a message, you can still use +IGNORE_EXCEPTION_DETAIL to normalize the modules between Python 2 and 3: + + >>> def f(x): + ... r''' + ... >>> from http.client import HTTPException + ... >>> raise HTTPException() #doctest: +IGNORE_EXCEPTION_DETAIL + ... Traceback (most recent call last): + ... foo.bar.HTTPException + ... ''' + >>> test = doctest.DocTestFinder().find(f)[0] + >>> doctest.DocTestRunner(verbose=False).run(test) + TestResults(failed=0, attempted=2) + +Note that a trailing colon doesn't matter either: + + >>> def f(x): + ... r''' + ... >>> from http.client import HTTPException + ... >>> raise HTTPException() #doctest: +IGNORE_EXCEPTION_DETAIL + ... Traceback (most recent call last): + ... foo.bar.HTTPException: + ... ''' + >>> test = doctest.DocTestFinder().find(f)[0] + >>> doctest.DocTestRunner(verbose=False).run(test) + TestResults(failed=0, attempted=2) + If an exception is raised but not expected, then it is reported as an unexpected exception: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,10 @@ Library ------- +- Issue #19138: doctest's IGNORE_EXCEPTION_DETAIL now allows a match when + no exception detail exists (no colon following the exception's name, or + a colon does follow but no text follows the colon). + - Issue #16231: Fixed pickle.Pickler to only fallback to its default pickling behaviour when Pickler.persistent_id returns None, but not for any other false values. This allows false values other than None to be used as -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 04:30:53 2013 From: python-checkins at python.org (tim.peters) Date: Wed, 4 Dec 2013 04:30:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5MTM4?= =?utf-8?q?=3A_doctest=27s_IGNORE=5FEXCEPTION=5FDETAIL_now_allows_no_detai?= =?utf-8?q?l_at_all=2E?= Message-ID: <3dZ5Ds1SRzz7LkH@mail.python.org> http://hg.python.org/cpython/rev/017d7c27a4f6 changeset: 87752:017d7c27a4f6 branch: 3.3 parent: 87739:afc68547cad6 user: Tim Peters date: Tue Dec 03 21:02:05 2013 -0600 summary: Issue #19138: doctest's IGNORE_EXCEPTION_DETAIL now allows no detail at all. (grafted from c80083ad142db2939507800c755082293a87f2de) files: Lib/doctest.py | 33 ++++++++++++++++++++++++--- Lib/test/test_doctest.py | 27 ++++++++++++++++++++++ Misc/NEWS | 4 +++ 3 files changed, 60 insertions(+), 4 deletions(-) diff --git a/Lib/doctest.py b/Lib/doctest.py --- a/Lib/doctest.py +++ b/Lib/doctest.py @@ -314,6 +314,32 @@ else: return '#' +def _strip_exception_details(msg): + # Support for IGNORE_EXCEPTION_DETAIL. + # Get rid of everything except the exception name; in particular, drop + # the possibly dotted module path (if any) and the exception message (if + # any). We assume that a colon is never part of a dotted name, or of an + # exception name. + # E.g., given + # "foo.bar.MyError: la di da" + # return "MyError" + # Or for "abc.def" or "abc.def:\n" return "def". + + start, end = 0, len(msg) + # The exception name must appear on the first line. + i = msg.find("\n") + if i >= 0: + end = i + # retain up to the first colon (if any) + i = msg.find(':', 0, end) + if i >= 0: + end = i + # retain just the exception name + i = msg.rfind('.', 0, end) + if i >= 0: + start = i+1 + return msg[start: end] + class _OutputRedirectingPdb(pdb.Pdb): """ A specialized version of the python debugger that redirects stdout @@ -1320,10 +1346,9 @@ # Another chance if they didn't care about the detail. elif self.optionflags & IGNORE_EXCEPTION_DETAIL: - m1 = re.match(r'(?:[^:]*\.)?([^:]*:)', example.exc_msg) - m2 = re.match(r'(?:[^:]*\.)?([^:]*:)', exc_msg) - if m1 and m2 and check(m1.group(1), m2.group(1), - self.optionflags): + if check(_strip_exception_details(example.exc_msg), + _strip_exception_details(exc_msg), + self.optionflags): outcome = SUCCESS # Report the outcome. diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py --- a/Lib/test/test_doctest.py +++ b/Lib/test/test_doctest.py @@ -1020,6 +1020,33 @@ ValueError: message TestResults(failed=1, attempted=1) +If the exception does not have a message, you can still use +IGNORE_EXCEPTION_DETAIL to normalize the modules between Python 2 and 3: + + >>> def f(x): + ... r''' + ... >>> from http.client import HTTPException + ... >>> raise HTTPException() #doctest: +IGNORE_EXCEPTION_DETAIL + ... Traceback (most recent call last): + ... foo.bar.HTTPException + ... ''' + >>> test = doctest.DocTestFinder().find(f)[0] + >>> doctest.DocTestRunner(verbose=False).run(test) + TestResults(failed=0, attempted=2) + +Note that a trailing colon doesn't matter either: + + >>> def f(x): + ... r''' + ... >>> from http.client import HTTPException + ... >>> raise HTTPException() #doctest: +IGNORE_EXCEPTION_DETAIL + ... Traceback (most recent call last): + ... foo.bar.HTTPException: + ... ''' + >>> test = doctest.DocTestFinder().find(f)[0] + >>> doctest.DocTestRunner(verbose=False).run(test) + TestResults(failed=0, attempted=2) + If an exception is raised but not expected, then it is reported as an unexpected exception: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,10 @@ Library ------- +- Issue #19138: doctest's IGNORE_EXCEPTION_DETAIL now allows a match when + no exception detail exists (no colon following the exception's name, or + a colon does follow but no text follows the colon). + - Issue #19834: Support unpickling of exceptions pickled by Python 2. - Issue #15798: Fixed subprocess.Popen() to no longer fail if file -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 04:33:43 2013 From: python-checkins at python.org (tim.peters) Date: Wed, 4 Dec 2013 04:33:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Null_merge_from_3=2E3_branch=2E?= Message-ID: <3dZ5J74yP8z7Ljw@mail.python.org> http://hg.python.org/cpython/rev/1a03f780f69a changeset: 87753:1a03f780f69a parent: 87750:c80083ad142d parent: 87752:017d7c27a4f6 user: Tim Peters date: Tue Dec 03 21:33:31 2013 -0600 summary: Null merge from 3.3 branch. files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 04:49:44 2013 From: python-checkins at python.org (tim.peters) Date: Wed, 4 Dec 2013 04:49:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Fudge_-_http?= =?utf-8?q?=2Eclient_doesn=27t_exist_in_2=2E7=2E__Use_Queue=2EEmpty_instea?= =?utf-8?q?d=2E?= Message-ID: <3dZ5fc1sFXz7LkH@mail.python.org> http://hg.python.org/cpython/rev/ae9fb85ab4e0 changeset: 87754:ae9fb85ab4e0 branch: 2.7 parent: 87751:5e14a3435f52 user: Tim Peters date: Tue Dec 03 21:49:30 2013 -0600 summary: Fudge - http.client doesn't exist in 2.7. Use Queue.Empty instead. files: Lib/test/test_doctest.py | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py --- a/Lib/test/test_doctest.py +++ b/Lib/test/test_doctest.py @@ -1024,10 +1024,10 @@ >>> def f(x): ... r''' - ... >>> from http.client import HTTPException - ... >>> raise HTTPException() #doctest: +IGNORE_EXCEPTION_DETAIL + ... >>> from Queue import Empty + ... >>> raise Empty() #doctest: +IGNORE_EXCEPTION_DETAIL ... Traceback (most recent call last): - ... foo.bar.HTTPException + ... foo.bar.Empty ... ''' >>> test = doctest.DocTestFinder().find(f)[0] >>> doctest.DocTestRunner(verbose=False).run(test) @@ -1037,10 +1037,10 @@ >>> def f(x): ... r''' - ... >>> from http.client import HTTPException - ... >>> raise HTTPException() #doctest: +IGNORE_EXCEPTION_DETAIL + ... >>> from Queue import Empty + ... >>> raise Empty() #doctest: +IGNORE_EXCEPTION_DETAIL ... Traceback (most recent call last): - ... foo.bar.HTTPException: + ... foo.bar.Empty: ... ''' >>> test = doctest.DocTestFinder().find(f)[0] >>> doctest.DocTestRunner(verbose=False).run(test) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 08:42:56 2013 From: python-checkins at python.org (christian.heimes) Date: Wed, 4 Dec 2013 08:42:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_upcast_int_to_size=5Ft_to_?= =?utf-8?q?silence_two_autological-constant-out-of-range-compare?= Message-ID: <3dZBqh0jq4z7LjY@mail.python.org> http://hg.python.org/cpython/rev/4994501035f2 changeset: 87755:4994501035f2 parent: 87753:1a03f780f69a user: Christian Heimes date: Wed Dec 04 08:42:46 2013 +0100 summary: upcast int to size_t to silence two autological-constant-out-of-range-compare warnings with clang. files: Parser/node.c | 2 +- Python/compile.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Parser/node.c b/Parser/node.c --- a/Parser/node.c +++ b/Parser/node.c @@ -91,7 +91,7 @@ if (current_capacity < 0 || required_capacity < 0) return E_OVERFLOW; if (current_capacity < required_capacity) { - if (required_capacity > PY_SIZE_MAX / sizeof(node)) { + if ((size_t)required_capacity > PY_SIZE_MAX / sizeof(node)) { return E_NOMEM; } n = n1->n_child; diff --git a/Python/compile.c b/Python/compile.c --- a/Python/compile.c +++ b/Python/compile.c @@ -3899,7 +3899,7 @@ a->a_lnotab = PyBytes_FromStringAndSize(NULL, DEFAULT_LNOTAB_SIZE); if (!a->a_lnotab) return 0; - if (nblocks > PY_SIZE_MAX / sizeof(basicblock *)) { + if ((size_t)nblocks > PY_SIZE_MAX / sizeof(basicblock *)) { PyErr_NoMemory(); return 0; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 08:50:43 2013 From: python-checkins at python.org (christian.heimes) Date: Wed, 4 Dec 2013 08:50:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogbmN1cnNlcycgd2lu?= =?utf-8?q?ch_and_mvwinch_return_an_unsigned_long?= Message-ID: <3dZC0g4KzDz7LjS@mail.python.org> http://hg.python.org/cpython/rev/f39a989f5b1a changeset: 87756:f39a989f5b1a branch: 3.3 parent: 87752:017d7c27a4f6 user: Christian Heimes date: Wed Dec 04 08:50:22 2013 +0100 summary: ncurses' winch and mvwinch return an unsigned long files: Modules/_cursesmodule.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c --- a/Modules/_cursesmodule.c +++ b/Modules/_cursesmodule.c @@ -1342,7 +1342,8 @@ static PyObject * PyCursesWindow_InCh(PyCursesWindowObject *self, PyObject *args) { - int x, y, rtn; + int x, y; + unsigned long rtn; switch (PyTuple_Size(args)) { case 0: @@ -1357,7 +1358,7 @@ PyErr_SetString(PyExc_TypeError, "inch requires 0 or 2 arguments"); return NULL; } - return PyLong_FromLong((long) rtn); + return PyLong_FromUnsignedLong(rtn); } static PyObject * -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 08:50:44 2013 From: python-checkins at python.org (christian.heimes) Date: Wed, 4 Dec 2013 08:50:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_ncurses=27_winch_and_mvwinch_return_an_unsigned_long?= Message-ID: <3dZC0h6GcXz7LjS@mail.python.org> http://hg.python.org/cpython/rev/75d1a9bccf99 changeset: 87757:75d1a9bccf99 parent: 87755:4994501035f2 parent: 87756:f39a989f5b1a user: Christian Heimes date: Wed Dec 04 08:50:32 2013 +0100 summary: ncurses' winch and mvwinch return an unsigned long files: Modules/_cursesmodule.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c --- a/Modules/_cursesmodule.c +++ b/Modules/_cursesmodule.c @@ -1414,7 +1414,8 @@ static PyObject * PyCursesWindow_InCh(PyCursesWindowObject *self, PyObject *args) { - int x, y, rtn; + int x, y; + unsigned long rtn; switch (PyTuple_Size(args)) { case 0: @@ -1429,7 +1430,7 @@ PyErr_SetString(PyExc_TypeError, "inch requires 0 or 2 arguments"); return NULL; } - return PyLong_FromLong((long) rtn); + return PyLong_FromUnsignedLong(rtn); } static PyObject * -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 09:28:32 2013 From: python-checkins at python.org (christian.heimes) Date: Wed, 4 Dec 2013 09:28:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Silence_expression_result_?= =?utf-8?q?unused_warnings_with_clang=2E?= Message-ID: <3dZCrJ1zRdz7LjR@mail.python.org> http://hg.python.org/cpython/rev/12e53abec5d0 changeset: 87758:12e53abec5d0 user: Christian Heimes date: Wed Dec 04 09:27:47 2013 +0100 summary: Silence expression result unused warnings with clang. The PyObject_INIT() macros returns obj: ../cpython/Objects/methodobject.c:32:23: warning: expression result unused [-Wunused-value] PyObject_INIT(op, &PyCFunction_Type); ^~ ../cpython/Include/objimpl.h:139:69: note: expanded from macro 'PyObject_INIT' ( Py_TYPE(op) = (typeobj), _Py_NewReference((PyObject *)(op)), (op) ) ^ 1 warning generated. files: Objects/bytesobject.c | 6 +++--- Objects/classobject.c | 2 +- Objects/complexobject.c | 2 +- Objects/floatobject.c | 2 +- Objects/longobject.c | 2 +- Objects/methodobject.c | 2 +- Objects/typeobject.c | 2 +- 7 files changed, 9 insertions(+), 9 deletions(-) diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -107,7 +107,7 @@ op = (PyBytesObject *)PyObject_MALLOC(PyBytesObject_SIZE + size); if (op == NULL) return PyErr_NoMemory(); - PyObject_INIT_VAR(op, &PyBytes_Type, size); + (void)PyObject_INIT_VAR(op, &PyBytes_Type, size); op->ob_shash = -1; if (str != NULL) Py_MEMCPY(op->ob_sval, str, size); @@ -155,7 +155,7 @@ op = (PyBytesObject *)PyObject_MALLOC(PyBytesObject_SIZE + size); if (op == NULL) return PyErr_NoMemory(); - PyObject_INIT_VAR(op, &PyBytes_Type, size); + (void)PyObject_INIT_VAR(op, &PyBytes_Type, size); op->ob_shash = -1; Py_MEMCPY(op->ob_sval, str, size+1); /* share short strings */ @@ -749,7 +749,7 @@ op = (PyBytesObject *)PyObject_MALLOC(PyBytesObject_SIZE + nbytes); if (op == NULL) return PyErr_NoMemory(); - PyObject_INIT_VAR(op, &PyBytes_Type, size); + (void)PyObject_INIT_VAR(op, &PyBytes_Type, size); op->ob_shash = -1; op->ob_sval[size] = '\0'; if (Py_SIZE(a) == 1 && n > 0) { diff --git a/Objects/classobject.c b/Objects/classobject.c --- a/Objects/classobject.c +++ b/Objects/classobject.c @@ -52,7 +52,7 @@ im = free_list; if (im != NULL) { free_list = (PyMethodObject *)(im->im_self); - PyObject_INIT(im, &PyMethod_Type); + (void)PyObject_INIT(im, &PyMethod_Type); numfree--; } else { diff --git a/Objects/complexobject.c b/Objects/complexobject.c --- a/Objects/complexobject.c +++ b/Objects/complexobject.c @@ -217,7 +217,7 @@ op = (PyComplexObject *) PyObject_MALLOC(sizeof(PyComplexObject)); if (op == NULL) return PyErr_NoMemory(); - PyObject_INIT(op, &PyComplex_Type); + (void)PyObject_INIT(op, &PyComplex_Type); op->cval = cval; return (PyObject *) op; } diff --git a/Objects/floatobject.c b/Objects/floatobject.c --- a/Objects/floatobject.c +++ b/Objects/floatobject.c @@ -119,7 +119,7 @@ return PyErr_NoMemory(); } /* Inline PyObject_New */ - PyObject_INIT(op, &PyFloat_Type); + (void)PyObject_INIT(op, &PyFloat_Type); op->ob_fval = fval; return (PyObject *) op; } diff --git a/Objects/longobject.c b/Objects/longobject.c --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -5089,7 +5089,7 @@ assert(v->ob_digit[0] == abs(ival)); } else { - PyObject_INIT(v, &PyLong_Type); + (void)PyObject_INIT(v, &PyLong_Type); } Py_SIZE(v) = size; v->ob_digit[0] = abs(ival); diff --git a/Objects/methodobject.c b/Objects/methodobject.c --- a/Objects/methodobject.c +++ b/Objects/methodobject.c @@ -29,7 +29,7 @@ op = free_list; if (op != NULL) { free_list = (PyCFunctionObject *)(op->m_self); - PyObject_INIT(op, &PyCFunction_Type); + (void)PyObject_INIT(op, &PyCFunction_Type); numfree--; } else { diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -787,7 +787,7 @@ Py_INCREF(type); if (type->tp_itemsize == 0) - PyObject_INIT(obj, type); + (void)PyObject_INIT(obj, type); else (void) PyObject_INIT_VAR((PyVarObject *)obj, type, nitems); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 09:31:58 2013 From: python-checkins at python.org (christian.heimes) Date: Wed, 4 Dec 2013 09:31:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_make_char*_const?= Message-ID: <3dZCwG4GYKz7LjQ@mail.python.org> http://hg.python.org/cpython/rev/0d23aa28a75e changeset: 87759:0d23aa28a75e user: Christian Heimes date: Wed Dec 04 09:31:47 2013 +0100 summary: make char* const readline() takes a const char* and the other readline_until_enter_or_signal() implementation already has const char*. files: Modules/readline.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/readline.c b/Modules/readline.c --- a/Modules/readline.c +++ b/Modules/readline.c @@ -1118,7 +1118,7 @@ static char * -readline_until_enter_or_signal(char *prompt, int *signal) +readline_until_enter_or_signal(const char *prompt, int *signal) { PyOS_sighandler_t old_inthandler; char *p; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 09:34:39 2013 From: python-checkins at python.org (christian.heimes) Date: Wed, 4 Dec 2013 09:34:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Silence_more_PyObject=5FIN?= =?utf-8?q?IT=28=29_unused_value_warnings=2E?= Message-ID: <3dZCzM61kVz7LjP@mail.python.org> http://hg.python.org/cpython/rev/0c7fff783b32 changeset: 87760:0c7fff783b32 user: Christian Heimes date: Wed Dec 04 09:34:29 2013 +0100 summary: Silence more PyObject_INIT() unused value warnings. files: Modules/_datetimemodule.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Modules/_datetimemodule.c b/Modules/_datetimemodule.c --- a/Modules/_datetimemodule.c +++ b/Modules/_datetimemodule.c @@ -613,7 +613,7 @@ sizeof(_PyDateTime_BaseTime)); if (self == NULL) return (PyObject *)PyErr_NoMemory(); - PyObject_INIT(self, type); + (void)PyObject_INIT(self, type); return self; } @@ -628,7 +628,7 @@ sizeof(_PyDateTime_BaseDateTime)); if (self == NULL) return (PyObject *)PyErr_NoMemory(); - PyObject_INIT(self, type); + (void)PyObject_INIT(self, type); return self; } -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Wed Dec 4 09:41:56 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 04 Dec 2013 09:41:56 +0100 Subject: [Python-checkins] Daily reference leaks (3c34ab550358): sum=0 Message-ID: results for 3c34ab550358 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog89Kgv8', '-x'] From python-checkins at python.org Wed Dec 4 09:43:30 2013 From: python-checkins at python.org (christian.heimes) Date: Wed, 4 Dec 2013 09:43:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_silence_unused-value_warni?= =?utf-8?q?ngs_in_libffi=27s_dlmalloc=2Ec?= Message-ID: <3dZD9Z4Q11z7LjP@mail.python.org> http://hg.python.org/cpython/rev/355af3657df3 changeset: 87761:355af3657df3 user: Christian Heimes date: Wed Dec 04 09:43:21 2013 +0100 summary: silence unused-value warnings in libffi's dlmalloc.c files: Modules/_ctypes/libffi.diff | 36 +++++++++++++++ Modules/_ctypes/libffi/src/dlmalloc.c | 8 +- 2 files changed, 40 insertions(+), 4 deletions(-) diff --git a/Modules/_ctypes/libffi.diff b/Modules/_ctypes/libffi.diff --- a/Modules/_ctypes/libffi.diff +++ b/Modules/_ctypes/libffi.diff @@ -167,3 +167,39 @@ #endif /* WIN32 */ #ifdef __OS2__ +@@ -3393,7 +3393,7 @@ + *ss = m->seg; /* Push current record */ + m->seg.base = tbase; + m->seg.size = tsize; +- set_segment_flags(&m->seg, mmapped); ++ (void)set_segment_flags(&m->seg, mmapped); + m->seg.next = ss; + + /* Insert trailing fenceposts */ +@@ -3553,7 +3553,7 @@ + if (!is_initialized(m)) { /* first-time initialization */ + m->seg.base = m->least_addr = tbase; + m->seg.size = tsize; +- set_segment_flags(&m->seg, mmap_flag); ++ (void)set_segment_flags(&m->seg, mmap_flag); + m->magic = mparams.magic; + init_bins(m); + if (is_global(m)) +@@ -4502,7 +4502,7 @@ + char* tbase = (char*)(CALL_MMAP(tsize)); + if (tbase != CMFAIL) { + m = init_user_mstate(tbase, tsize); +- set_segment_flags(&m->seg, IS_MMAPPED_BIT); ++ (void)set_segment_flags(&m->seg, IS_MMAPPED_BIT); + set_lock(m, locked); + } + } +@@ -4517,7 +4517,7 @@ + if (capacity > msize + TOP_FOOT_SIZE && + capacity < (size_t) -(msize + TOP_FOOT_SIZE + mparams.page_size)) { + m = init_user_mstate((char*)base, capacity); +- set_segment_flags(&m->seg, EXTERN_BIT); ++ (void)set_segment_flags(&m->seg, EXTERN_BIT); + set_lock(m, locked); + } + return (mspace)m; diff --git a/Modules/_ctypes/libffi/src/dlmalloc.c b/Modules/_ctypes/libffi/src/dlmalloc.c --- a/Modules/_ctypes/libffi/src/dlmalloc.c +++ b/Modules/_ctypes/libffi/src/dlmalloc.c @@ -3393,7 +3393,7 @@ *ss = m->seg; /* Push current record */ m->seg.base = tbase; m->seg.size = tsize; - set_segment_flags(&m->seg, mmapped); + (void)set_segment_flags(&m->seg, mmapped); m->seg.next = ss; /* Insert trailing fenceposts */ @@ -3553,7 +3553,7 @@ if (!is_initialized(m)) { /* first-time initialization */ m->seg.base = m->least_addr = tbase; m->seg.size = tsize; - set_segment_flags(&m->seg, mmap_flag); + (void)set_segment_flags(&m->seg, mmap_flag); m->magic = mparams.magic; init_bins(m); if (is_global(m)) @@ -4502,7 +4502,7 @@ char* tbase = (char*)(CALL_MMAP(tsize)); if (tbase != CMFAIL) { m = init_user_mstate(tbase, tsize); - set_segment_flags(&m->seg, IS_MMAPPED_BIT); + (void)set_segment_flags(&m->seg, IS_MMAPPED_BIT); set_lock(m, locked); } } @@ -4517,7 +4517,7 @@ if (capacity > msize + TOP_FOOT_SIZE && capacity < (size_t) -(msize + TOP_FOOT_SIZE + mparams.page_size)) { m = init_user_mstate((char*)base, capacity); - set_segment_flags(&m->seg, EXTERN_BIT); + (void)set_segment_flags(&m->seg, EXTERN_BIT); set_lock(m, locked); } return (mspace)m; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 11:31:22 2013 From: python-checkins at python.org (victor.stinner) Date: Wed, 4 Dec 2013 11:31:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_add_some_no?= =?utf-8?q?tes_on_the_ping_example?= Message-ID: <3dZGZ20dWWz7LnH@mail.python.org> http://hg.python.org/cpython/rev/0830670a9d9d changeset: 87762:0830670a9d9d user: Victor Stinner date: Wed Dec 04 11:16:17 2013 +0100 summary: asyncio doc: add some notes on the ping example files: Doc/library/asyncio-protocol.rst | 44 +++++++++++-------- 1 files changed, 26 insertions(+), 18 deletions(-) diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -579,43 +579,52 @@ This function returns a :ref:`coroutine `. -Protocol example: TCP echo client and server +Protocol example: TCP echo server and client ============================================ Echo server ----------- -TCP echo server example:: +TCP echo server example, send back received data and close the connection:: import asyncio class EchoServer(asyncio.Protocol): def connection_made(self, transport): - print('connection made') + peername = transport.get_extra_info('peername') + print('connection from {}'.format(peername)) self.transport = transport - def data_received(self, data): - print('data received:', data.decode()) + print('data received: {}'.format(data.decode())) self.transport.write(data) # close the socket self.transport.close() - def connection_lost(self, exc): - print('connection lost') + loop = asyncio.get_event_loop() + task = loop.create_server(EchoServer, '127.0.0.1', 8888) + server = loop.run_until_complete(task) + print('serving on {}'.format(server.sockets[0].getsockname())) - loop = asyncio.get_event_loop() - f = loop.create_server(EchoServer, '127.0.0.1', 8888) - s = loop.run_until_complete(f) - print('serving on', s.sockets[0].getsockname()) - loop.run_forever() + try: + loop.run_forever() + except KeyboardInterrupt: + print("exit") + finally: + server.close() + loop.close() + +:meth:`Transport.close` can be called immediatly after +:meth:`WriteTransport.write` even if data are not sent yet on the socket: both +methods are asynchronous. ``yield from`` is not needed because these transport +methods don't return coroutines. Echo client ----------- -TCP echo client example:: +TCP echo client example, send data and wait until the connection is closed:: import asyncio @@ -623,15 +632,14 @@ message = 'This is the message. It will be echoed.' def connection_made(self, transport): - self.transport = transport - self.transport.write(self.message.encode()) - print('data sent:', self.message) + transport.write(self.message.encode()) + print('data sent: {}'.format(self.message)) def data_received(self, data): - print('data received:', data.decode()) + print('data received: {}'.format(data.decode())) def connection_lost(self, exc): - print('connection lost') + print('server closed the connection') asyncio.get_event_loop().stop() loop = asyncio.get_event_loop() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 20:46:29 2013 From: python-checkins at python.org (christian.heimes) Date: Wed, 4 Dec 2013 20:46:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319509=3A_Don=27t_?= =?utf-8?q?close_the_socket_in_do=5Fhandshake=28=29_when_hostname?= Message-ID: <3dZVtY5lXzz7LkS@mail.python.org> http://hg.python.org/cpython/rev/b6fce698e467 changeset: 87763:b6fce698e467 user: Christian Heimes date: Wed Dec 04 20:46:20 2013 +0100 summary: Issue #19509: Don't close the socket in do_handshake() when hostname verification fails. files: Lib/ssl.py | 13 ++++--------- 1 files changed, 4 insertions(+), 9 deletions(-) diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -764,15 +764,10 @@ self.settimeout(timeout) if self.context.check_hostname: - try: - if not self.server_hostname: - raise ValueError("check_hostname needs server_hostname " - "argument") - match_hostname(self.getpeercert(), self.server_hostname) - except Exception: - self.shutdown(_SHUT_RDWR) - self.close() - raise + if not self.server_hostname: + raise ValueError("check_hostname needs server_hostname " + "argument") + match_hostname(self.getpeercert(), self.server_hostname) def _real_connect(self, addr, connect_ex): if self.server_side: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 20:50:15 2013 From: python-checkins at python.org (guido.van.rossum) Date: Wed, 4 Dec 2013 20:50:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_broken_docstring_conti?= =?utf-8?q?nuation_line_for_detach=28=29=2E?= Message-ID: <3dZVyv2ZJzz7LjN@mail.python.org> http://hg.python.org/cpython/rev/de531b2cfef0 changeset: 87764:de531b2cfef0 user: Guido van Rossum date: Wed Dec 04 11:50:09 2013 -0800 summary: Fix broken docstring continuation line for detach(). files: Modules/socketmodule.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -2322,8 +2322,8 @@ PyDoc_STRVAR(detach_doc, "detach()\n\ \n\ -Close the socket object without closing the underlying file descriptor.\ -The object cannot be used after this call, but the file descriptor\ +Close the socket object without closing the underlying file descriptor.\n\ +The object cannot be used after this call, but the file descriptor\n\ can be reused for other purposes. The file descriptor is returned."); static int -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 21:12:14 2013 From: python-checkins at python.org (guido.van.rossum) Date: Wed, 4 Dec 2013 21:12:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_Write_flow_cont?= =?utf-8?q?rol_for_proactor_event_loop=2E?= Message-ID: <3dZWSG6QV2z7LkR@mail.python.org> http://hg.python.org/cpython/rev/4fd8c173bcf0 changeset: 87765:4fd8c173bcf0 user: Guido van Rossum date: Wed Dec 04 12:12:07 2013 -0800 summary: asyncio: Write flow control for proactor event loop. files: Lib/asyncio/proactor_events.py | 116 ++++++++- Lib/test/test_asyncio/test_proactor_events.py | 14 +- 2 files changed, 106 insertions(+), 24 deletions(-) diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -24,12 +24,14 @@ self._sock = sock self._protocol = protocol self._server = server - self._buffer = [] + self._buffer = None # None or bytearray. self._read_fut = None self._write_fut = None self._conn_lost = 0 self._closing = False # Set when close() called. self._eof_written = False + self._protocol_paused = False + self.set_write_buffer_limits() if self._server is not None: self._server.attach(self) self._loop.call_soon(self._protocol.connection_made, self) @@ -63,7 +65,7 @@ if self._read_fut: self._read_fut.cancel() self._write_fut = self._read_fut = None - self._buffer = [] + self._buffer = None self._loop.call_soon(self._call_connection_lost, exc) def _call_connection_lost(self, exc): @@ -82,6 +84,53 @@ server.detach(self) self._server = None + # XXX The next four methods are nearly identical to corresponding + # ones in _SelectorTransport. Maybe refactor buffer management to + # share the implementations? (Also these are really only needed + # by _ProactorWritePipeTransport but since _buffer is defined on + # the base class I am putting it here for now.) + + def _maybe_pause_protocol(self): + size = self.get_write_buffer_size() + if size <= self._high_water: + return + if not self._protocol_paused: + self._protocol_paused = True + try: + self._protocol.pause_writing() + except Exception: + logger.exception('pause_writing() failed') + + def _maybe_resume_protocol(self): + if (self._protocol_paused and + self.get_write_buffer_size() <= self._low_water): + self._protocol_paused = False + try: + self._protocol.resume_writing() + except Exception: + logger.exception('resume_writing() failed') + + def set_write_buffer_limits(self, high=None, low=None): + if high is None: + if low is None: + high = 64*1024 + else: + high = 4*low + if low is None: + low = high // 4 + if not high >= low >= 0: + raise ValueError('high (%r) must be >= low (%r) must be >= 0' % + (high, low)) + self._high_water = high + self._low_water = low + + def get_write_buffer_size(self): + # NOTE: This doesn't take into account data already passed to + # send() even if send() hasn't finished yet. + if not self._buffer: + return 0 + return len(self._buffer) + class _ProactorReadPipeTransport(_ProactorBasePipeTransport, transports.ReadTransport): @@ -95,12 +144,15 @@ self._loop.call_soon(self._loop_reading) def pause_reading(self): - assert not self._closing, 'Cannot pause_reading() when closing' - assert not self._paused, 'Already paused' + if self._closing: + raise RuntimeError('Cannot pause_reading() when closing') + if self._paused: + raise RuntimeError('Already paused') self._paused = True def resume_reading(self): - assert self._paused, 'Not paused' + if not self._paused: + raise RuntimeError('Not paused') self._paused = False if self._closing: return @@ -155,9 +207,11 @@ """Transport for write pipes.""" def write(self, data): - assert isinstance(data, bytes), repr(data) + if not isinstance(data, (bytes, bytearray, memoryview)): + raise TypeError('data argument must be byte-ish (%r)', + type(data)) if self._eof_written: - raise IOError('write_eof() already called') + raise RuntimeError('write_eof() already called') if not data: return @@ -167,26 +221,53 @@ logger.warning('socket.send() raised exception.') self._conn_lost += 1 return - self._buffer.append(data) - if self._write_fut is None: - self._loop_writing() - def _loop_writing(self, f=None): + # Observable states: + # 1. IDLE: _write_fut and _buffer both None + # 2. WRITING: _write_fut set; _buffer None + # 3. BACKED UP: _write_fut set; _buffer a bytearray + # We always copy the data, so the caller can't modify it + # while we're still waiting for the I/O to happen. + if self._write_fut is None: # IDLE -> WRITING + assert self._buffer is None + # Pass a copy, except if it's already immutable. + self._loop_writing(data=bytes(data)) + # XXX Should we pause the protocol at this point + # if len(data) > self._high_water? (That would + # require keeping track of the number of bytes passed + # to a send() that hasn't finished yet.) + elif not self._buffer: # WRITING -> BACKED UP + # Make a mutable copy which we can extend. + self._buffer = bytearray(data) + self._maybe_pause_protocol() + else: # BACKED UP + # Append to buffer (also copies). + self._buffer.extend(data) + self._maybe_pause_protocol() + + def _loop_writing(self, f=None, data=None): try: assert f is self._write_fut self._write_fut = None if f: f.result() - data = b''.join(self._buffer) - self._buffer = [] + if data is None: + data = self._buffer + self._buffer = None if not data: if self._closing: self._loop.call_soon(self._call_connection_lost, None) if self._eof_written: self._sock.shutdown(socket.SHUT_WR) - return - self._write_fut = self._loop._proactor.send(self._sock, data) - self._write_fut.add_done_callback(self._loop_writing) + else: + self._write_fut = self._loop._proactor.send(self._sock, data) + self._write_fut.add_done_callback(self._loop_writing) + # Now that we've reduced the buffer size, tell the + # protocol to resume writing if it was paused. Note that + # we do this last since the callback is called immediately + # and it may add more data to the buffer (even causing the + # protocol to be paused again). + self._maybe_resume_protocol() except ConnectionResetError as exc: self._force_close(exc) except OSError as exc: @@ -330,7 +411,8 @@ self._csock.send(b'x') def _start_serving(self, protocol_factory, sock, ssl=None, server=None): - assert not ssl, 'IocpEventLoop is incompatible with SSL.' + if ssl: + raise ValueError('IocpEventLoop is incompatible with SSL.') def loop(f=None): try: diff --git a/Lib/test/test_asyncio/test_proactor_events.py b/Lib/test/test_asyncio/test_proactor_events.py --- a/Lib/test/test_asyncio/test_proactor_events.py +++ b/Lib/test/test_asyncio/test_proactor_events.py @@ -111,8 +111,8 @@ tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) tr._loop_writing = unittest.mock.Mock() tr.write(b'data') - self.assertEqual(tr._buffer, [b'data']) - self.assertTrue(tr._loop_writing.called) + self.assertEqual(tr._buffer, None) + tr._loop_writing.assert_called_with(data=b'data') def test_write_no_data(self): tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) @@ -124,12 +124,12 @@ tr._write_fut = unittest.mock.Mock() tr._loop_writing = unittest.mock.Mock() tr.write(b'data') - self.assertEqual(tr._buffer, [b'data']) + self.assertEqual(tr._buffer, b'data') self.assertFalse(tr._loop_writing.called) def test_loop_writing(self): tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) - tr._buffer = [b'da', b'ta'] + tr._buffer = bytearray(b'data') tr._loop_writing() self.loop._proactor.send.assert_called_with(self.sock, b'data') self.loop._proactor.send.return_value.add_done_callback.\ @@ -150,7 +150,7 @@ tr.write(b'data') tr.write(b'data') tr.write(b'data') - self.assertEqual(tr._buffer, []) + self.assertEqual(tr._buffer, None) m_log.warning.assert_called_with('socket.send() raised exception.') def test_loop_writing_stop(self): @@ -226,7 +226,7 @@ write_fut.cancel.assert_called_with() test_utils.run_briefly(self.loop) self.protocol.connection_lost.assert_called_with(None) - self.assertEqual([], tr._buffer) + self.assertEqual(None, tr._buffer) self.assertEqual(tr._conn_lost, 1) def test_force_close_idempotent(self): @@ -243,7 +243,7 @@ test_utils.run_briefly(self.loop) self.protocol.connection_lost.assert_called_with(None) - self.assertEqual([], tr._buffer) + self.assertEqual(None, tr._buffer) def test_call_connection_lost(self): tr = _ProactorSocketTransport(self.loop, self.sock, self.protocol) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 21:15:59 2013 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 4 Dec 2013 21:15:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5ODgy?= =?utf-8?q?=3A_tweak_docs_for_socket=2Eclose=28=29?= Message-ID: <3dZWXb4ghrz7LjR@mail.python.org> http://hg.python.org/cpython/rev/e10bb7c1b8f8 changeset: 87766:e10bb7c1b8f8 branch: 3.3 parent: 87756:f39a989f5b1a user: Antoine Pitrou date: Wed Dec 04 21:02:42 2013 +0100 summary: Issue #19882: tweak docs for socket.close() files: Doc/library/socket.rst | 28 +++++++++++++++++++--------- 1 files changed, 19 insertions(+), 9 deletions(-) diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -701,8 +701,9 @@ Socket Objects -------------- -Socket objects have the following methods. Except for :meth:`makefile` these -correspond to Unix system calls applicable to sockets. +Socket objects have the following methods. Except for +:meth:`~socket.makefile`, these correspond to Unix system calls applicable +to sockets. .. method:: socket.accept() @@ -721,9 +722,15 @@ .. method:: socket.close() - Close the socket. All future operations on the socket object will fail. The - remote end will receive no more data (after queued data is flushed). Sockets are - automatically closed when they are garbage-collected. + Mark the socket closed. The underlying system resource (e.g. a file + descriptor) is also closed when all file objects from :meth:`makefile()` + are closed. Once that happens, all future operations on the socket + object will fail. The remote end will receive no more data (after + queued data is flushed). + + Sockets are automatically closed when they are garbage-collected, but + it is recommended to :meth:`close` them explicitly, or to use a + :keyword:`with` statement around them. .. note:: :meth:`close()` releases the resource associated with a connection but @@ -829,10 +836,13 @@ type depends on the arguments given to :meth:`makefile`. These arguments are interpreted the same way as by the built-in :func:`open` function. - Closing the file object won't close the socket unless there are no remaining - references to the socket. The socket must be in blocking mode; it can have - a timeout, but the file object's internal buffer may end up in a inconsistent - state if a timeout occurs. + The socket must be in blocking mode; it can have a timeout, but the file + object's internal buffer may end up in a inconsistent state if a timeout + occurs. + + Closing the file object returned by :meth:`makefile` won't close the + original socket unless all other file objects have been closed and + :meth:`socket.close` has been called on the socket object. .. note:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 21:16:01 2013 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 4 Dec 2013 21:16:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Tweak_the_sock?= =?utf-8?q?et_module_doc_layout?= Message-ID: <3dZWXd0VvSz7Lkw@mail.python.org> http://hg.python.org/cpython/rev/489001c48bf1 changeset: 87767:489001c48bf1 branch: 3.3 user: Antoine Pitrou date: Wed Dec 04 21:11:03 2013 +0100 summary: Tweak the socket module doc layout files: Doc/library/socket.rst | 150 ++++++++++++++++------------ 1 files changed, 86 insertions(+), 64 deletions(-) diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -138,9 +138,12 @@ Module contents --------------- -The module :mod:`socket` exports the following constants and functions: +The module :mod:`socket` exports the following elements. +Exceptions +^^^^^^^^^^ + .. exception:: error A deprecated alias of :exc:`OSError`. @@ -186,6 +189,10 @@ .. versionchanged:: 3.3 This class was made a subclass of :exc:`OSError`. + +Constants +^^^^^^^^^ + .. data:: AF_UNIX AF_INET AF_INET6 @@ -290,6 +297,43 @@ this platform. +Functions +^^^^^^^^^ + +Creating sockets +'''''''''''''''' + +The following functions all create :ref:`socket objects `. + + +.. function:: socket([family[, type[, proto]]]) + + Create a new socket using the given address family, socket type and protocol + number. The address family should be :const:`AF_INET` (the default), + :const:`AF_INET6`, :const:`AF_UNIX`, :const:`AF_CAN` or :const:`AF_RDS`. The + socket type should be :const:`SOCK_STREAM` (the default), + :const:`SOCK_DGRAM`, :const:`SOCK_RAW` or perhaps one of the other ``SOCK_`` + constants. The protocol number is usually zero and may be omitted in that + case or :const:`CAN_RAW` in case the address family is :const:`AF_CAN`. + + .. versionchanged:: 3.3 + The AF_CAN family was added. + The AF_RDS family was added. + + +.. function:: socketpair([family[, type[, proto]]]) + + Build a pair of connected socket objects using the given address family, socket + type, and protocol number. Address family, socket type, and protocol number are + as for the :func:`.socket` function above. The default family is :const:`AF_UNIX` + if defined on the platform; otherwise, the default is :const:`AF_INET`. + Availability: Unix. + + .. versionchanged:: 3.2 + The returned socket objects now support the whole socket API, rather + than a subset. + + .. function:: create_connection(address[, timeout[, source_address]]) Connect to a TCP service listening on the Internet *address* (a 2-tuple @@ -316,6 +360,40 @@ support for the :keyword:`with` statement was added. +.. function:: fromfd(fd, family, type[, proto]) + + Duplicate the file descriptor *fd* (an integer as returned by a file object's + :meth:`fileno` method) and build a socket object from the result. Address + family, socket type and protocol number are as for the :func:`.socket` function + above. The file descriptor should refer to a socket, but this is not checked --- + subsequent operations on the object may fail if the file descriptor is invalid. + This function is rarely needed, but can be used to get or set socket options on + a socket passed to a program as standard input or output (such as a server + started by the Unix inet daemon). The socket is assumed to be in blocking mode. + + +.. function:: fromshare(data) + + Instantiate a socket from data obtained from the :meth:`socket.share` + method. The socket is assumed to be in blocking mode. + + Availability: Windows. + + .. versionadded:: 3.3 + + +.. data:: SocketType + + This is a Python type object that represents the socket object type. It is the + same as ``type(socket(...))``. + + +Other functions +''''''''''''''' + +The :mod:`socket` module also offers various network-related services: + + .. function:: getaddrinfo(host, port, family=0, type=0, proto=0, flags=0) Translate the *host*/*port* argument into a sequence of 5-tuples that contain @@ -445,46 +523,6 @@ ``'udp'``, otherwise any protocol will match. -.. function:: socket([family[, type[, proto]]]) - - Create a new socket using the given address family, socket type and protocol - number. The address family should be :const:`AF_INET` (the default), - :const:`AF_INET6`, :const:`AF_UNIX`, :const:`AF_CAN` or :const:`AF_RDS`. The - socket type should be :const:`SOCK_STREAM` (the default), - :const:`SOCK_DGRAM`, :const:`SOCK_RAW` or perhaps one of the other ``SOCK_`` - constants. The protocol number is usually zero and may be omitted in that - case or :const:`CAN_RAW` in case the address family is :const:`AF_CAN`. - - .. versionchanged:: 3.3 - The AF_CAN family was added. - The AF_RDS family was added. - - -.. function:: socketpair([family[, type[, proto]]]) - - Build a pair of connected socket objects using the given address family, socket - type, and protocol number. Address family, socket type, and protocol number are - as for the :func:`.socket` function above. The default family is :const:`AF_UNIX` - if defined on the platform; otherwise, the default is :const:`AF_INET`. - Availability: Unix. - - .. versionchanged:: 3.2 - The returned socket objects now support the whole socket API, rather - than a subset. - - -.. function:: fromfd(fd, family, type[, proto]) - - Duplicate the file descriptor *fd* (an integer as returned by a file object's - :meth:`fileno` method) and build a socket object from the result. Address - family, socket type and protocol number are as for the :func:`.socket` function - above. The file descriptor should refer to a socket, but this is not checked --- - subsequent operations on the object may fail if the file descriptor is invalid. - This function is rarely needed, but can be used to get or set socket options on - a socket passed to a program as standard input or output (such as a server - started by the Unix inet daemon). The socket is assumed to be in blocking mode. - - .. function:: ntohl(x) Convert 32-bit positive integers from network to host byte order. On machines @@ -680,22 +718,6 @@ .. versionadded:: 3.3 -.. function:: fromshare(data) - - Instantiate a socket from data obtained from :meth:`~socket.share`. - The socket is assumed to be in blocking mode. - - Availability: Windows. - - .. versionadded:: 3.3 - - -.. data:: SocketType - - This is a Python type object that represents the socket object type. It is the - same as ``type(socket(...))``. - - .. _socket-objects: Socket Objects @@ -1106,14 +1128,14 @@ .. method:: socket.share(process_id) - :platform: Windows + Duplicate a socket and prepare it for sharing with a target process. The + target process must be provided with *process_id*. The resulting bytes object + can then be passed to the target process using some form of interprocess + communication and the socket can be recreated there using :func:`fromshare`. + Once this method has been called, it is safe to close the socket since + the operating system has already duplicated it for the target process. - Duplacet a socket and prepare it for sharing with a target process. The - target process must be provided with *process_id*. The resulting bytes object - can then be passed to the target process using some form of interprocess - communication and the socket can be recreated there using :func:`fromshare`. - Once this method has been called, it is safe to close the socket since - the operating system has already duplicated it for the target process. + Availability: Windows. .. versionadded:: 3.3 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 21:16:02 2013 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 4 Dec 2013 21:16:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge_socket_doc_changes_from_3=2E3?= Message-ID: <3dZWXf3NTYz7LmP@mail.python.org> http://hg.python.org/cpython/rev/5c306845e67c changeset: 87768:5c306845e67c parent: 87765:4fd8c173bcf0 parent: 87767:489001c48bf1 user: Antoine Pitrou date: Wed Dec 04 21:15:24 2013 +0100 summary: Merge socket doc changes from 3.3 files: Doc/library/socket.rst | 216 ++++++++++++++++------------ 1 files changed, 124 insertions(+), 92 deletions(-) diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -138,9 +138,12 @@ Module contents --------------- -The module :mod:`socket` exports the following constants and functions: +The module :mod:`socket` exports the following elements. +Exceptions +^^^^^^^^^^ + .. exception:: error A deprecated alias of :exc:`OSError`. @@ -186,6 +189,10 @@ .. versionchanged:: 3.3 This class was made a subclass of :exc:`OSError`. + +Constants +^^^^^^^^^ + .. data:: AF_UNIX AF_INET AF_INET6 @@ -305,6 +312,57 @@ this platform. +Functions +^^^^^^^^^ + +Creating sockets +'''''''''''''''' + +The following functions all create :ref:`socket objects `. + + +.. function:: socket(family=AF_INET, type=SOCK_STREAM, proto=0, fileno=None) + + Create a new socket using the given address family, socket type and protocol + number. The address family should be :const:`AF_INET` (the default), + :const:`AF_INET6`, :const:`AF_UNIX`, :const:`AF_CAN` or :const:`AF_RDS`. The + socket type should be :const:`SOCK_STREAM` (the default), + :const:`SOCK_DGRAM`, :const:`SOCK_RAW` or perhaps one of the other ``SOCK_`` + constants. The protocol number is usually zero and may be omitted or in the + case where the address family is :const:`AF_CAN` the protocol should be one + of :const:`CAN_RAW` or :const:`CAN_BCM`. + + The newly created socket is :ref:`non-inheritable `. + + .. versionchanged:: 3.3 + The AF_CAN family was added. + The AF_RDS family was added. + + .. versionchanged:: 3.4 + The CAN_BCM protocol was added. + + .. versionchanged:: 3.4 + The returned socket is now non-inheritable. + + +.. function:: socketpair([family[, type[, proto]]]) + + Build a pair of connected socket objects using the given address family, socket + type, and protocol number. Address family, socket type, and protocol number are + as for the :func:`.socket` function above. The default family is :const:`AF_UNIX` + if defined on the platform; otherwise, the default is :const:`AF_INET`. + Availability: Unix. + + The newly created sockets are :ref:`non-inheritable `. + + .. versionchanged:: 3.2 + The returned socket objects now support the whole socket API, rather + than a subset. + + .. versionchanged:: 3.4 + The returned sockets are now non-inheritable. + + .. function:: create_connection(address[, timeout[, source_address]]) Connect to a TCP service listening on the Internet *address* (a 2-tuple @@ -331,6 +389,45 @@ support for the :keyword:`with` statement was added. +.. function:: fromfd(fd, family, type, proto=0) + + Duplicate the file descriptor *fd* (an integer as returned by a file object's + :meth:`fileno` method) and build a socket object from the result. Address + family, socket type and protocol number are as for the :func:`.socket` function + above. The file descriptor should refer to a socket, but this is not checked --- + subsequent operations on the object may fail if the file descriptor is invalid. + This function is rarely needed, but can be used to get or set socket options on + a socket passed to a program as standard input or output (such as a server + started by the Unix inet daemon). The socket is assumed to be in blocking mode. + + The newly created socket is :ref:`non-inheritable `. + + .. versionchanged:: 3.4 + The returned socket is now non-inheritable. + + +.. function:: fromshare(data) + + Instantiate a socket from data obtained from the :meth:`socket.share` + method. The socket is assumed to be in blocking mode. + + Availability: Windows. + + .. versionadded:: 3.3 + + +.. data:: SocketType + + This is a Python type object that represents the socket object type. It is the + same as ``type(socket(...))``. + + +Other functions +''''''''''''''' + +The :mod:`socket` module also offers various network-related services: + + .. function:: getaddrinfo(host, port, family=0, type=0, proto=0, flags=0) Translate the *host*/*port* argument into a sequence of 5-tuples that contain @@ -460,65 +557,6 @@ ``'udp'``, otherwise any protocol will match. -.. function:: socket(family=AF_INET, type=SOCK_STREAM, proto=0, fileno=None) - - Create a new socket using the given address family, socket type and protocol - number. The address family should be :const:`AF_INET` (the default), - :const:`AF_INET6`, :const:`AF_UNIX`, :const:`AF_CAN` or :const:`AF_RDS`. The - socket type should be :const:`SOCK_STREAM` (the default), - :const:`SOCK_DGRAM`, :const:`SOCK_RAW` or perhaps one of the other ``SOCK_`` - constants. The protocol number is usually zero and may be omitted or in the - case where the address family is :const:`AF_CAN` the protocol should be one - of :const:`CAN_RAW` or :const:`CAN_BCM`. - - The newly created socket is :ref:`non-inheritable `. - - .. versionchanged:: 3.3 - The AF_CAN family was added. - The AF_RDS family was added. - - .. versionchanged:: 3.4 - The CAN_BCM protocol was added. - - .. versionchanged:: 3.4 - The socket is now non-inheritable. - - -.. function:: socketpair([family[, type[, proto]]]) - - Build a pair of connected socket objects using the given address family, socket - type, and protocol number. Address family, socket type, and protocol number are - as for the :func:`.socket` function above. The default family is :const:`AF_UNIX` - if defined on the platform; otherwise, the default is :const:`AF_INET`. - Availability: Unix. - - The newly created sockets are :ref:`non-inheritable `. - - .. versionchanged:: 3.2 - The returned socket objects now support the whole socket API, rather - than a subset. - - .. versionchanged:: 3.4 - The sockets are now non-inheritable. - - -.. function:: fromfd(fd, family, type, proto=0) - - Duplicate the file descriptor *fd* (an integer as returned by a file object's - :meth:`fileno` method) and build a socket object from the result. Address - family, socket type and protocol number are as for the :func:`.socket` function - above. The file descriptor should refer to a socket, but this is not checked --- - subsequent operations on the object may fail if the file descriptor is invalid. - This function is rarely needed, but can be used to get or set socket options on - a socket passed to a program as standard input or output (such as a server - started by the Unix inet daemon). The socket is assumed to be in blocking mode. - - The newly created socket is :ref:`non-inheritable `. - - .. versionchanged:: 3.4 - The socket is now non-inheritable. - - .. function:: ntohl(x) Convert 32-bit positive integers from network to host byte order. On machines @@ -714,29 +752,14 @@ .. versionadded:: 3.3 -.. function:: fromshare(data) - - Instantiate a socket from data obtained from :meth:`~socket.share`. - The socket is assumed to be in blocking mode. - - Availability: Windows. - - .. versionadded:: 3.3 - - -.. data:: SocketType - - This is a Python type object that represents the socket object type. It is the - same as ``type(socket(...))``. - - .. _socket-objects: Socket Objects -------------- -Socket objects have the following methods. Except for :meth:`makefile` these -correspond to Unix system calls applicable to sockets. +Socket objects have the following methods. Except for +:meth:`~socket.makefile`, these correspond to Unix system calls applicable +to sockets. .. method:: socket.accept() @@ -760,9 +783,15 @@ .. method:: socket.close() - Close the socket. All future operations on the socket object will fail. The - remote end will receive no more data (after queued data is flushed). Sockets are - automatically closed when they are garbage-collected. + Mark the socket closed. The underlying system resource (e.g. a file + descriptor) is also closed when all file objects from :meth:`makefile()` + are closed. Once that happens, all future operations on the socket + object will fail. The remote end will receive no more data (after + queued data is flushed). + + Sockets are automatically closed when they are garbage-collected, but + it is recommended to :meth:`close` them explicitly, or to use a + :keyword:`with` statement around them. .. note:: :meth:`close()` releases the resource associated with a connection but @@ -887,10 +916,13 @@ type depends on the arguments given to :meth:`makefile`. These arguments are interpreted the same way as by the built-in :func:`open` function. - Closing the file object won't close the socket unless there are no remaining - references to the socket. The socket must be in blocking mode; it can have - a timeout, but the file object's internal buffer may end up in a inconsistent - state if a timeout occurs. + The socket must be in blocking mode; it can have a timeout, but the file + object's internal buffer may end up in a inconsistent state if a timeout + occurs. + + Closing the file object returned by :meth:`makefile` won't close the + original socket unless all other file objects have been closed and + :meth:`socket.close` has been called on the socket object. .. note:: @@ -1162,14 +1194,14 @@ .. method:: socket.share(process_id) - :platform: Windows + Duplicate a socket and prepare it for sharing with a target process. The + target process must be provided with *process_id*. The resulting bytes object + can then be passed to the target process using some form of interprocess + communication and the socket can be recreated there using :func:`fromshare`. + Once this method has been called, it is safe to close the socket since + the operating system has already duplicated it for the target process. - Duplacet a socket and prepare it for sharing with a target process. The - target process must be provided with *process_id*. The resulting bytes object - can then be passed to the target process using some form of interprocess - communication and the socket can be recreated there using :func:`fromshare`. - Once this method has been called, it is safe to close the socket since - the operating system has already duplicated it for the target process. + Availability: Windows. .. versionadded:: 3.3 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 21:59:10 2013 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 4 Dec 2013 21:59:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_Add_myself_as_expect_for_?= =?utf-8?q?pathlib?= Message-ID: <3dZXVQ1QmRzScj@mail.python.org> http://hg.python.org/devguide/rev/48469f022e72 changeset: 654:48469f022e72 user: Antoine Pitrou date: Wed Dec 04 21:59:06 2013 +0100 summary: Add myself as expect for pathlib files: experts.rst | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/experts.rst b/experts.rst --- a/experts.rst +++ b/experts.rst @@ -162,6 +162,7 @@ ossaudiodev packaging tarek*, eric.araujo*, alexis parser benjamin.peterson +pathlib pitrou* pdb georg.brandl* pickle alexandre.vassalotti, pitrou pickletools alexandre.vassalotti -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Wed Dec 4 23:30:13 2013 From: python-checkins at python.org (nadeem.vawda) Date: Wed, 4 Dec 2013 23:30:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzE5ODM5OiBGaXgg?= =?utf-8?q?regression_in_bz2_module=27s_handling_of_non-bzip2_data_at_EOF?= =?utf-8?q?=2E?= Message-ID: <3dZZWT4TQDz7LjS@mail.python.org> http://hg.python.org/cpython/rev/c5349a560703 changeset: 87769:c5349a560703 branch: 3.3 parent: 87767:489001c48bf1 user: Nadeem Vawda date: Wed Dec 04 23:01:15 2013 +0100 summary: #19839: Fix regression in bz2 module's handling of non-bzip2 data at EOF. files: Lib/bz2.py | 29 +++++++++++++++++++---------- Lib/test/test_bz2.py | 30 +++++++++++++++++++++++++++++- Misc/NEWS | 3 +++ 3 files changed, 51 insertions(+), 11 deletions(-) diff --git a/Lib/bz2.py b/Lib/bz2.py --- a/Lib/bz2.py +++ b/Lib/bz2.py @@ -199,8 +199,15 @@ # Continue to next stream. if self._decompressor.eof: self._decompressor = BZ2Decompressor() - - self._buffer = self._decompressor.decompress(rawblock) + try: + self._buffer = self._decompressor.decompress(rawblock) + except OSError: + # Trailing data isn't a valid bzip2 stream. We're done here. + self._mode = _MODE_READ_EOF + self._size = self._pos + return False + else: + self._buffer = self._decompressor.decompress(rawblock) self._buffer_offset = 0 return True @@ -488,17 +495,19 @@ For incremental decompression, use a BZ2Decompressor object instead. """ - if len(data) == 0: - return b"" - results = [] - while True: + while data: decomp = BZ2Decompressor() - results.append(decomp.decompress(data)) + try: + res = decomp.decompress(data) + except OSError: + if results: + break # Leftover data is not a valid bzip2 stream; ignore it. + else: + raise # Error on the first iteration; bail out. + results.append(res) if not decomp.eof: raise ValueError("Compressed data ended before the " "end-of-stream marker was reached") - if not decomp.unused_data: - return b"".join(results) - # There is unused data left over. Proceed to next stream. data = decomp.unused_data + return b"".join(results) diff --git a/Lib/test/test_bz2.py b/Lib/test/test_bz2.py --- a/Lib/test/test_bz2.py +++ b/Lib/test/test_bz2.py @@ -49,6 +49,7 @@ TEXT = b''.join(TEXT_LINES) DATA = b'BZh91AY&SY.\xc8N\x18\x00\x01>_\x80\x00\x10@\x02\xff\xf0\x01\x07n\x00?\xe7\xff\xe00\x01\x99\xaa\x00\xc0\x03F\x86\x8c#&\x83F\x9a\x03\x06\xa6\xd0\xa6\x93M\x0fQ\xa7\xa8\x06\x804hh\x12$\x11\xa4i4\xf14S\xd2\x88\xe5\xcd9gd6\x0b\n\xe9\x9b\xd5\x8a\x99\xf7\x08.K\x8ev\xfb\xf7xw\xbb\xdf\xa1\x92\xf1\xdd|/";\xa2\xba\x9f\xd5\xb1#A\xb6\xf6\xb3o\xc9\xc5y\\\xebO\xe7\x85\x9a\xbc\xb6f8\x952\xd5\xd7"%\x89>V,\xf7\xa6z\xe2\x9f\xa3\xdf\x11\x11"\xd6E)I\xa9\x13^\xca\xf3r\xd0\x03U\x922\xf26\xec\xb6\xed\x8b\xc3U\x13\x9d\xc5\x170\xa4\xfa^\x92\xacDF\x8a\x97\xd6\x19\xfe\xdd\xb8\xbd\x1a\x9a\x19\xa3\x80ankR\x8b\xe5\xd83]\xa9\xc6\x08\x82f\xf6\xb9"6l$\xb8j@\xc0\x8a\xb0l1..\xbak\x83ls\x15\xbc\xf4\xc1\x13\xbe\xf8E\xb8\x9d\r\xa8\x9dk\x84\xd3n\xfa\xacQ\x07\xb1%y\xaav\xb4\x08\xe0z\x1b\x16\xf5\x04\xe9\xcc\xb9\x08z\x1en7.G\xfc]\xc9\x14\xe1B@\xbb!8`' EMPTY_DATA = b'BZh9\x17rE8P\x90\x00\x00\x00\x00' + BAD_DATA = b'this is not a valid bzip2 file' def setUp(self): self.filename = TESTFN @@ -79,9 +80,10 @@ class BZ2FileTest(BaseTest): "Test BZ2File type miscellaneous methods." - def createTempFile(self, streams=1): + def createTempFile(self, streams=1, suffix=b""): with open(self.filename, "wb") as f: f.write(self.DATA * streams) + f.write(suffix) def testBadArgs(self): with self.assertRaises(TypeError): @@ -103,6 +105,11 @@ self.assertRaises(TypeError, bz2f.read, None) self.assertEqual(bz2f.read(), self.TEXT) + def testReadBadFile(self): + self.createTempFile(streams=0, suffix=self.BAD_DATA) + with BZ2File(self.filename) as bz2f: + self.assertRaises(OSError, bz2f.read) + def testReadMultiStream(self): self.createTempFile(streams=5) with BZ2File(self.filename) as bz2f: @@ -122,6 +129,16 @@ finally: bz2._BUFFER_SIZE = buffer_size + def testReadTrailingJunk(self): + self.createTempFile(suffix=self.BAD_DATA) + with BZ2File(self.filename) as bz2f: + self.assertEqual(bz2f.read(), self.TEXT) + + def testReadMultiStreamTrailingJunk(self): + self.createTempFile(streams=5, suffix=self.BAD_DATA) + with BZ2File(self.filename) as bz2f: + self.assertEqual(bz2f.read(), self.TEXT * 5) + def testRead0(self): self.createTempFile() with BZ2File(self.filename) as bz2f: @@ -707,10 +724,21 @@ def testDecompressIncomplete(self): self.assertRaises(ValueError, bz2.decompress, self.DATA[:-10]) + def testDecompressBadData(self): + self.assertRaises(OSError, bz2.decompress, self.BAD_DATA) + def testDecompressMultiStream(self): text = bz2.decompress(self.DATA * 5) self.assertEqual(text, self.TEXT * 5) + def testDecompressTrailingJunk(self): + text = bz2.decompress(self.DATA + self.BAD_DATA) + self.assertEqual(text, self.TEXT) + + def testDecompressMultiStreamTrailingJunk(self): + text = bz2.decompress(self.DATA * 5 + self.BAD_DATA) + self.assertEqual(text, self.TEXT * 5) + class OpenTest(BaseTest): def test_binary_modes(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,9 @@ Library ------- +- Issue #19839: Fix regression in bz2 module's handling of non-bzip2 data at + EOF. + - Issue #19138: doctest's IGNORE_EXCEPTION_DETAIL now allows a match when no exception detail exists (no colon following the exception's name, or a colon does follow but no text follows the colon). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 23:30:15 2013 From: python-checkins at python.org (nadeem.vawda) Date: Wed, 4 Dec 2013 23:30:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzE5ODM5OiBGaXgg?= =?utf-8?q?lzma_module=27s_handling_of_non-lzma_data_at_EOF=2E?= Message-ID: <3dZZWW0chfz7Lkt@mail.python.org> http://hg.python.org/cpython/rev/bec2033ee2ec changeset: 87770:bec2033ee2ec branch: 3.3 user: Nadeem Vawda date: Wed Dec 04 23:03:49 2013 +0100 summary: #19839: Fix lzma module's handling of non-lzma data at EOF. files: Lib/lzma.py | 28 +++++++++++++++++++++------- Lib/test/test_lzma.py | 26 ++++++++++++++++++++++++++ Misc/NEWS | 2 +- 3 files changed, 48 insertions(+), 8 deletions(-) diff --git a/Lib/lzma.py b/Lib/lzma.py --- a/Lib/lzma.py +++ b/Lib/lzma.py @@ -224,11 +224,18 @@ raise EOFError("Compressed file ended before the " "end-of-stream marker was reached") - # Continue to next stream. if self._decompressor.eof: + # Continue to next stream. self._decompressor = LZMADecompressor(**self._init_args) - - self._buffer = self._decompressor.decompress(rawblock) + try: + self._buffer = self._decompressor.decompress(rawblock) + except LZMAError: + # Trailing data isn't a valid compressed stream; ignore it. + self._mode = _MODE_READ_EOF + self._size = self._pos + return False + else: + self._buffer = self._decompressor.decompress(rawblock) # Read data until EOF. # If return_data is false, consume the data without returning it. @@ -444,11 +451,18 @@ results = [] while True: decomp = LZMADecompressor(format, memlimit, filters) - results.append(decomp.decompress(data)) + try: + res = decomp.decompress(data) + except LZMAError: + if results: + break # Leftover data is not a valid LZMA/XZ stream; ignore it. + else: + raise # Error on the first iteration; bail out. + results.append(res) if not decomp.eof: raise LZMAError("Compressed data ended before the " "end-of-stream marker was reached") - if not decomp.unused_data: - return b"".join(results) - # There is unused data left over. Proceed to next stream. data = decomp.unused_data + if not data: + break + return b"".join(results) diff --git a/Lib/test/test_lzma.py b/Lib/test/test_lzma.py --- a/Lib/test/test_lzma.py +++ b/Lib/test/test_lzma.py @@ -314,6 +314,8 @@ def test_decompress_bad_input(self): with self.assertRaises(LZMAError): + lzma.decompress(COMPRESSED_BOGUS) + with self.assertRaises(LZMAError): lzma.decompress(COMPRESSED_RAW_1) with self.assertRaises(LZMAError): lzma.decompress(COMPRESSED_ALONE, format=lzma.FORMAT_XZ) @@ -348,6 +350,16 @@ ddata = lzma.decompress(COMPRESSED_XZ + COMPRESSED_ALONE) self.assertEqual(ddata, INPUT * 2) + # Test robust handling of non-LZMA data following the compressed stream(s). + + def test_decompress_trailing_junk(self): + ddata = lzma.decompress(COMPRESSED_XZ + COMPRESSED_BOGUS) + self.assertEqual(ddata, INPUT) + + def test_decompress_multistream_trailing_junk(self): + ddata = lzma.decompress(COMPRESSED_XZ * 3 + COMPRESSED_BOGUS) + self.assertEqual(ddata, INPUT * 3) + class TempFile: """Context manager - creates a file, and deletes it on __exit__.""" @@ -658,6 +670,14 @@ finally: lzma._BUFFER_SIZE = saved_buffer_size + def test_read_trailing_junk(self): + with LZMAFile(BytesIO(COMPRESSED_XZ + COMPRESSED_BOGUS)) as f: + self.assertEqual(f.read(), INPUT) + + def test_read_multistream_trailing_junk(self): + with LZMAFile(BytesIO(COMPRESSED_XZ * 5 + COMPRESSED_BOGUS)) as f: + self.assertEqual(f.read(), INPUT * 5) + def test_read_from_file(self): with TempFile(TESTFN, COMPRESSED_XZ): with LZMAFile(TESTFN) as f: @@ -687,6 +707,10 @@ with LZMAFile(BytesIO(COMPRESSED_XZ)) as f: self.assertRaises(TypeError, f.read, None) + def test_read_bad_data(self): + with LZMAFile(BytesIO(COMPRESSED_BOGUS)) as f: + self.assertRaises(LZMAError, f.read) + def test_read1(self): with LZMAFile(BytesIO(COMPRESSED_XZ)) as f: blocks = [] @@ -1192,6 +1216,8 @@ Farewell. """ +COMPRESSED_BOGUS = b"this is not a valid lzma stream" + COMPRESSED_XZ = ( b"\xfd7zXZ\x00\x00\x04\xe6\xd6\xb4F\x02\x00!\x01\x16\x00\x00\x00t/\xe5\xa3" b"\xe0\x07\x80\x03\xdf]\x00\x05\x14\x07bX\x19\xcd\xddn\x98\x15\xe4\xb4\x9d" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -19,7 +19,7 @@ ------- - Issue #19839: Fix regression in bz2 module's handling of non-bzip2 data at - EOF. + EOF, and analogous bug in lzma module. - Issue #19138: doctest's IGNORE_EXCEPTION_DETAIL now allows a match when no exception detail exists (no colon following the exception's name, or -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 4 23:30:16 2013 From: python-checkins at python.org (nadeem.vawda) Date: Wed, 4 Dec 2013 23:30:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Closes_=2319839=3A_Fix_regression_in_bz2_module=27s_hand?= =?utf-8?q?ling_of_non-bzip2_data_at_EOF=2E?= Message-ID: <3dZZWX3l73z7Lnk@mail.python.org> http://hg.python.org/cpython/rev/1f1498fe50e5 changeset: 87771:1f1498fe50e5 parent: 87768:5c306845e67c parent: 87770:bec2033ee2ec user: Nadeem Vawda date: Wed Dec 04 23:29:51 2013 +0100 summary: Closes #19839: Fix regression in bz2 module's handling of non-bzip2 data at EOF. Also fix an analogous bug (not a regression) in the lzma module. files: Lib/bz2.py | 29 +++++++++++++++++++---------- Lib/lzma.py | 28 +++++++++++++++++++++------- Lib/test/test_bz2.py | 30 +++++++++++++++++++++++++++++- Lib/test/test_lzma.py | 26 ++++++++++++++++++++++++++ Misc/NEWS | 3 +++ 5 files changed, 98 insertions(+), 18 deletions(-) diff --git a/Lib/bz2.py b/Lib/bz2.py --- a/Lib/bz2.py +++ b/Lib/bz2.py @@ -207,8 +207,15 @@ if self._decompressor.eof: # Continue to next stream. self._decompressor = BZ2Decompressor() - - self._buffer = self._decompressor.decompress(rawblock) + try: + self._buffer = self._decompressor.decompress(rawblock) + except OSError: + # Trailing data isn't a valid bzip2 stream. We're done here. + self._mode = _MODE_READ_EOF + self._size = self._pos + return False + else: + self._buffer = self._decompressor.decompress(rawblock) self._buffer_offset = 0 return True @@ -496,17 +503,19 @@ For incremental decompression, use a BZ2Decompressor object instead. """ - if len(data) == 0: - return b"" - results = [] - while True: + while data: decomp = BZ2Decompressor() - results.append(decomp.decompress(data)) + try: + res = decomp.decompress(data) + except OSError: + if results: + break # Leftover data is not a valid bzip2 stream; ignore it. + else: + raise # Error on the first iteration; bail out. + results.append(res) if not decomp.eof: raise ValueError("Compressed data ended before the " "end-of-stream marker was reached") - if not decomp.unused_data: - return b"".join(results) - # There is unused data left over. Proceed to next stream. data = decomp.unused_data + return b"".join(results) diff --git a/Lib/lzma.py b/Lib/lzma.py --- a/Lib/lzma.py +++ b/Lib/lzma.py @@ -225,11 +225,18 @@ raise EOFError("Compressed file ended before the " "end-of-stream marker was reached") - # Continue to next stream. if self._decompressor.eof: + # Continue to next stream. self._decompressor = LZMADecompressor(**self._init_args) - - self._buffer = self._decompressor.decompress(rawblock) + try: + self._buffer = self._decompressor.decompress(rawblock) + except LZMAError: + # Trailing data isn't a valid compressed stream; ignore it. + self._mode = _MODE_READ_EOF + self._size = self._pos + return False + else: + self._buffer = self._decompressor.decompress(rawblock) self._buffer_offset = 0 return True @@ -487,11 +494,18 @@ results = [] while True: decomp = LZMADecompressor(format, memlimit, filters) - results.append(decomp.decompress(data)) + try: + res = decomp.decompress(data) + except LZMAError: + if results: + break # Leftover data is not a valid LZMA/XZ stream; ignore it. + else: + raise # Error on the first iteration; bail out. + results.append(res) if not decomp.eof: raise LZMAError("Compressed data ended before the " "end-of-stream marker was reached") - if not decomp.unused_data: - return b"".join(results) - # There is unused data left over. Proceed to next stream. data = decomp.unused_data + if not data: + break + return b"".join(results) diff --git a/Lib/test/test_bz2.py b/Lib/test/test_bz2.py --- a/Lib/test/test_bz2.py +++ b/Lib/test/test_bz2.py @@ -50,6 +50,7 @@ TEXT = b''.join(TEXT_LINES) DATA = b'BZh91AY&SY.\xc8N\x18\x00\x01>_\x80\x00\x10@\x02\xff\xf0\x01\x07n\x00?\xe7\xff\xe00\x01\x99\xaa\x00\xc0\x03F\x86\x8c#&\x83F\x9a\x03\x06\xa6\xd0\xa6\x93M\x0fQ\xa7\xa8\x06\x804hh\x12$\x11\xa4i4\xf14S\xd2\x88\xe5\xcd9gd6\x0b\n\xe9\x9b\xd5\x8a\x99\xf7\x08.K\x8ev\xfb\xf7xw\xbb\xdf\xa1\x92\xf1\xdd|/";\xa2\xba\x9f\xd5\xb1#A\xb6\xf6\xb3o\xc9\xc5y\\\xebO\xe7\x85\x9a\xbc\xb6f8\x952\xd5\xd7"%\x89>V,\xf7\xa6z\xe2\x9f\xa3\xdf\x11\x11"\xd6E)I\xa9\x13^\xca\xf3r\xd0\x03U\x922\xf26\xec\xb6\xed\x8b\xc3U\x13\x9d\xc5\x170\xa4\xfa^\x92\xacDF\x8a\x97\xd6\x19\xfe\xdd\xb8\xbd\x1a\x9a\x19\xa3\x80ankR\x8b\xe5\xd83]\xa9\xc6\x08\x82f\xf6\xb9"6l$\xb8j@\xc0\x8a\xb0l1..\xbak\x83ls\x15\xbc\xf4\xc1\x13\xbe\xf8E\xb8\x9d\r\xa8\x9dk\x84\xd3n\xfa\xacQ\x07\xb1%y\xaav\xb4\x08\xe0z\x1b\x16\xf5\x04\xe9\xcc\xb9\x08z\x1en7.G\xfc]\xc9\x14\xe1B@\xbb!8`' EMPTY_DATA = b'BZh9\x17rE8P\x90\x00\x00\x00\x00' + BAD_DATA = b'this is not a valid bzip2 file' def setUp(self): self.filename = support.TESTFN @@ -80,9 +81,10 @@ class BZ2FileTest(BaseTest): "Test the BZ2File class." - def createTempFile(self, streams=1): + def createTempFile(self, streams=1, suffix=b""): with open(self.filename, "wb") as f: f.write(self.DATA * streams) + f.write(suffix) def testBadArgs(self): self.assertRaises(TypeError, BZ2File, 123.456) @@ -98,6 +100,11 @@ self.assertRaises(TypeError, bz2f.read, None) self.assertEqual(bz2f.read(), self.TEXT) + def testReadBadFile(self): + self.createTempFile(streams=0, suffix=self.BAD_DATA) + with BZ2File(self.filename) as bz2f: + self.assertRaises(OSError, bz2f.read) + def testReadMultiStream(self): self.createTempFile(streams=5) with BZ2File(self.filename) as bz2f: @@ -117,6 +124,16 @@ finally: bz2._BUFFER_SIZE = buffer_size + def testReadTrailingJunk(self): + self.createTempFile(suffix=self.BAD_DATA) + with BZ2File(self.filename) as bz2f: + self.assertEqual(bz2f.read(), self.TEXT) + + def testReadMultiStreamTrailingJunk(self): + self.createTempFile(streams=5, suffix=self.BAD_DATA) + with BZ2File(self.filename) as bz2f: + self.assertEqual(bz2f.read(), self.TEXT * 5) + def testRead0(self): self.createTempFile() with BZ2File(self.filename) as bz2f: @@ -714,10 +731,21 @@ def testDecompressIncomplete(self): self.assertRaises(ValueError, bz2.decompress, self.DATA[:-10]) + def testDecompressBadData(self): + self.assertRaises(OSError, bz2.decompress, self.BAD_DATA) + def testDecompressMultiStream(self): text = bz2.decompress(self.DATA * 5) self.assertEqual(text, self.TEXT * 5) + def testDecompressTrailingJunk(self): + text = bz2.decompress(self.DATA + self.BAD_DATA) + self.assertEqual(text, self.TEXT) + + def testDecompressMultiStreamTrailingJunk(self): + text = bz2.decompress(self.DATA * 5 + self.BAD_DATA) + self.assertEqual(text, self.TEXT * 5) + class OpenTest(BaseTest): "Test the open function." diff --git a/Lib/test/test_lzma.py b/Lib/test/test_lzma.py --- a/Lib/test/test_lzma.py +++ b/Lib/test/test_lzma.py @@ -314,6 +314,8 @@ def test_decompress_bad_input(self): with self.assertRaises(LZMAError): + lzma.decompress(COMPRESSED_BOGUS) + with self.assertRaises(LZMAError): lzma.decompress(COMPRESSED_RAW_1) with self.assertRaises(LZMAError): lzma.decompress(COMPRESSED_ALONE, format=lzma.FORMAT_XZ) @@ -348,6 +350,16 @@ ddata = lzma.decompress(COMPRESSED_XZ + COMPRESSED_ALONE) self.assertEqual(ddata, INPUT * 2) + # Test robust handling of non-LZMA data following the compressed stream(s). + + def test_decompress_trailing_junk(self): + ddata = lzma.decompress(COMPRESSED_XZ + COMPRESSED_BOGUS) + self.assertEqual(ddata, INPUT) + + def test_decompress_multistream_trailing_junk(self): + ddata = lzma.decompress(COMPRESSED_XZ * 3 + COMPRESSED_BOGUS) + self.assertEqual(ddata, INPUT * 3) + class TempFile: """Context manager - creates a file, and deletes it on __exit__.""" @@ -676,6 +688,14 @@ finally: lzma._BUFFER_SIZE = saved_buffer_size + def test_read_trailing_junk(self): + with LZMAFile(BytesIO(COMPRESSED_XZ + COMPRESSED_BOGUS)) as f: + self.assertEqual(f.read(), INPUT) + + def test_read_multistream_trailing_junk(self): + with LZMAFile(BytesIO(COMPRESSED_XZ * 5 + COMPRESSED_BOGUS)) as f: + self.assertEqual(f.read(), INPUT * 5) + def test_read_from_file(self): with TempFile(TESTFN, COMPRESSED_XZ): with LZMAFile(TESTFN) as f: @@ -719,6 +739,10 @@ with LZMAFile(BytesIO(COMPRESSED_XZ)) as f: self.assertRaises(TypeError, f.read, None) + def test_read_bad_data(self): + with LZMAFile(BytesIO(COMPRESSED_BOGUS)) as f: + self.assertRaises(LZMAError, f.read) + def test_read1(self): with LZMAFile(BytesIO(COMPRESSED_XZ)) as f: blocks = [] @@ -1232,6 +1256,8 @@ Farewell. """ +COMPRESSED_BOGUS = b"this is not a valid lzma stream" + COMPRESSED_XZ = ( b"\xfd7zXZ\x00\x00\x04\xe6\xd6\xb4F\x02\x00!\x01\x16\x00\x00\x00t/\xe5\xa3" b"\xe0\x07\x80\x03\xdf]\x00\x05\x14\x07bX\x19\xcd\xddn\x98\x15\xe4\xb4\x9d" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,9 @@ Library ------- +- Issue #19839: Fix regression in bz2 module's handling of non-bzip2 data at + EOF, and analogous bug in lzma module. + - Issue #19138: doctest's IGNORE_EXCEPTION_DETAIL now allows a match when no exception detail exists (no colon following the exception's name, or a colon does follow but no text follows the colon). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 5 07:53:47 2013 From: python-checkins at python.org (christian.heimes) Date: Thu, 5 Dec 2013 07:53:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_exclude_=5Fhashopenssl=2Ec?= =?utf-8?q?=3A=5FsetException=28=29_from_LCOV_coverage?= Message-ID: <3dZnhW236kz7Ljw@mail.python.org> http://hg.python.org/cpython/rev/5435c14f690d changeset: 87772:5435c14f690d user: Christian Heimes date: Thu Dec 05 07:38:13 2013 +0100 summary: exclude _hashopenssl.c:_setException() from LCOV coverage files: Modules/_hashopenssl.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Modules/_hashopenssl.c b/Modules/_hashopenssl.c --- a/Modules/_hashopenssl.c +++ b/Modules/_hashopenssl.c @@ -557,6 +557,7 @@ return 1; } +/* LCOV_EXCL_START */ static PyObject * _setException(PyObject *exc) { @@ -585,6 +586,7 @@ } return NULL; } +/* LCOV_EXCL_STOP */ PyDoc_STRVAR(pbkdf2_hmac__doc__, "pbkdf2_hmac(hash_name, password, salt, iterations, dklen=None) -> key\n\ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 5 07:53:48 2013 From: python-checkins at python.org (christian.heimes) Date: Thu, 5 Dec 2013 07:53:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_touch_=5Flsprof=27s_clear?= =?utf-8?q?=28=29_method_for_C_code_coverage?= Message-ID: <3dZnhX3kQbz7LjZ@mail.python.org> http://hg.python.org/cpython/rev/0a4ae134d34e changeset: 87773:0a4ae134d34e user: Christian Heimes date: Thu Dec 05 07:40:29 2013 +0100 summary: touch _lsprof's clear() method for C code coverage files: Lib/test/test_cprofile.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_cprofile.py b/Lib/test/test_cprofile.py --- a/Lib/test/test_cprofile.py +++ b/Lib/test/test_cprofile.py @@ -29,6 +29,7 @@ obj.enable() obj = _lsprof.Profiler(1) obj.disable() + obj.clear() finally: sys.stderr = orig_stderr finally: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 5 07:53:49 2013 From: python-checkins at python.org (christian.heimes) Date: Thu, 5 Dec 2013 07:53:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Test_SSLSock=27s_context_g?= =?utf-8?q?etter_and_setter?= Message-ID: <3dZnhY5XGdz7LkC@mail.python.org> http://hg.python.org/cpython/rev/656e88290a09 changeset: 87774:656e88290a09 user: Christian Heimes date: Thu Dec 05 07:41:08 2013 +0100 summary: Test SSLSock's context getter and setter files: Lib/test/test_ssl.py | 14 ++++++++++++++ 1 files changed, 14 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1419,6 +1419,20 @@ s.close() self.assertEqual(len(ctx.get_ca_certs()), 1) + def test_context_setget(self): + # Check that the context of a connected socket can be replaced. + with support.transient_internet("svn.python.org"): + ctx1 = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + ctx2 = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + s = socket.socket(socket.AF_INET) + with ctx1.wrap_socket(s) as ss: + ss.connect(("svn.python.org", 443)) + self.assertIs(ss.context, ctx1) + self.assertIs(ss._sslobj.context, ctx1) + ss.context = ctx2 + self.assertIs(ss.context, ctx2) + self.assertIs(ss._sslobj.context, ctx2) + try: import threading except ImportError: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 5 07:53:50 2013 From: python-checkins at python.org (christian.heimes) Date: Thu, 5 Dec 2013 07:53:50 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_a_script_similar_to_xm?= =?utf-8?q?ltests=2Epy_to_run_all_SSL-related_unit_tests?= Message-ID: <3dZnhZ6v16z7Ll2@mail.python.org> http://hg.python.org/cpython/rev/567b9c18e8a8 changeset: 87775:567b9c18e8a8 user: Christian Heimes date: Thu Dec 05 07:45:36 2013 +0100 summary: Add a script similar to xmltests.py to run all SSL-related unit tests files: Lib/test/ssltests.py | 22 ++++++++++++++++++++++ 1 files changed, 22 insertions(+), 0 deletions(-) diff --git a/Lib/test/ssltests.py b/Lib/test/ssltests.py new file mode 100755 --- /dev/null +++ b/Lib/test/ssltests.py @@ -0,0 +1,22 @@ +# Convenience test module to run all of the SSL-related tests in the +# standard library. + +import sys +import subprocess + +TESTS = ['test_asyncio', 'test_ftplib', 'test_hashlib', 'test_httplib', + 'test_imaplib', 'test_nntplib', 'test_poplib', 'test_smtplib', + 'test_smtpnet', 'test_urllib2_localnet', 'test_venv'] + +def run_regrtests(*extra_args): + args = [sys.executable, "-m", "test"] + if not extra_args: + args.append("-unetwork") + else: + args.extend(extra_args) + args.extend(TESTS) + result = subprocess.call(args) + sys.exit(result) + +if __name__ == '__main__': + run_regrtests(*sys.argv[1:]) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 5 07:53:52 2013 From: python-checkins at python.org (christian.heimes) Date: Thu, 5 Dec 2013 07:53:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_19509=3A_Don=27t_cal?= =?utf-8?q?l_match=5Fhostname=28=29_twice_in_http=2Eclient=2E?= Message-ID: <3dZnhc18V7z7LjZ@mail.python.org> http://hg.python.org/cpython/rev/1a945fb875bf changeset: 87776:1a945fb875bf user: Christian Heimes date: Thu Dec 05 07:51:17 2013 +0100 summary: Issue 19509: Don't call match_hostname() twice in http.client. files: Lib/http/client.py | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -1204,13 +1204,13 @@ server_hostname = self.host if ssl.HAS_SNI else None self.sock = self._context.wrap_socket(sock, server_hostname=server_hostname) - try: - if self._check_hostname: + if not self._context.check_hostname and self._check_hostname: + try: ssl.match_hostname(self.sock.getpeercert(), self.host) - except Exception: - self.sock.shutdown(socket.SHUT_RDWR) - self.sock.close() - raise + except Exception: + self.sock.shutdown(socket.SHUT_RDWR) + self.sock.close() + raise __all__.append("HTTPSConnection") -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 5 07:53:53 2013 From: python-checkins at python.org (christian.heimes) Date: Thu, 5 Dec 2013 07:53:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_normalize_white_space?= Message-ID: <3dZnhd2Rpdz7Lks@mail.python.org> http://hg.python.org/cpython/rev/14ff7b1383a7 changeset: 87777:14ff7b1383a7 user: Christian Heimes date: Thu Dec 05 07:53:38 2013 +0100 summary: normalize white space files: Lib/test/ssltests.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/ssltests.py b/Lib/test/ssltests.py --- a/Lib/test/ssltests.py +++ b/Lib/test/ssltests.py @@ -5,7 +5,7 @@ import subprocess TESTS = ['test_asyncio', 'test_ftplib', 'test_hashlib', 'test_httplib', - 'test_imaplib', 'test_nntplib', 'test_poplib', 'test_smtplib', + 'test_imaplib', 'test_nntplib', 'test_poplib', 'test_smtplib', 'test_smtpnet', 'test_urllib2_localnet', 'test_venv'] def run_regrtests(*extra_args): -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Thu Dec 5 09:40:58 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 05 Dec 2013 09:40:58 +0100 Subject: [Python-checkins] Daily reference leaks (1f1498fe50e5): sum=0 Message-ID: results for 1f1498fe50e5 on branch "default" -------------------------------------------- test_site leaked [0, 2, -2] references, sum=0 test_site leaked [0, 2, -2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog0Ppyjd', '-x'] From python-checkins at python.org Thu Dec 5 11:05:12 2013 From: python-checkins at python.org (kristjan.jonsson) Date: Thu, 5 Dec 2013 11:05:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzcxMDU6?= =?utf-8?q?__weak_dict_iterators_are_fragile_because_of_unpredictable_GC_r?= =?utf-8?q?uns?= Message-ID: <3dZsxN1KKpz7LjX@mail.python.org> http://hg.python.org/cpython/rev/03fcc12282fc changeset: 87778:03fcc12282fc branch: 2.7 parent: 87754:ae9fb85ab4e0 user: Kristj?n Valur J?nsson date: Thu Dec 05 10:03:45 2013 +0000 summary: Issue #7105: weak dict iterators are fragile because of unpredictable GC runs Backport the fix from pyton 3.x for this issue. files: Doc/library/weakref.rst | 4 +- Lib/test/test_weakref.py | 97 ++++++++++++++++++++- Lib/test/test_weakset.py | 49 +++++++++++ Lib/weakref.py | 120 ++++++++++++++++++++------ 4 files changed, 235 insertions(+), 35 deletions(-) diff --git a/Doc/library/weakref.rst b/Doc/library/weakref.rst --- a/Doc/library/weakref.rst +++ b/Doc/library/weakref.rst @@ -163,7 +163,7 @@ .. method:: WeakKeyDictionary.iterkeyrefs() - Return an :term:`iterator` that yields the weak references to the keys. + Return an iterable of the weak references to the keys. .. versionadded:: 2.5 @@ -195,7 +195,7 @@ .. method:: WeakValueDictionary.itervaluerefs() - Return an :term:`iterator` that yields the weak references to the values. + Return an iterable of the weak references to the values. .. versionadded:: 2.5 diff --git a/Lib/test/test_weakref.py b/Lib/test/test_weakref.py --- a/Lib/test/test_weakref.py +++ b/Lib/test/test_weakref.py @@ -4,6 +4,8 @@ import UserList import weakref import operator +import contextlib +import copy from test import test_support @@ -903,7 +905,7 @@ def check_len_cycles(self, dict_type, cons): N = 20 items = [RefCycle() for i in range(N)] - dct = dict_type(cons(o) for o in items) + dct = dict_type(cons(i, o) for i, o in enumerate(items)) # Keep an iterator alive it = dct.iteritems() try: @@ -913,18 +915,23 @@ del items gc.collect() n1 = len(dct) + list(it) del it gc.collect() n2 = len(dct) - # one item may be kept alive inside the iterator - self.assertIn(n1, (0, 1)) + # iteration should prevent garbage collection here + # Note that this is a test on an implementation detail. The requirement + # is only to provide stable iteration, not that the size of the container + # stay fixed. + self.assertEqual(n1, 20) + #self.assertIn(n1, (0, 1)) self.assertEqual(n2, 0) def test_weak_keyed_len_cycles(self): - self.check_len_cycles(weakref.WeakKeyDictionary, lambda k: (k, 1)) + self.check_len_cycles(weakref.WeakKeyDictionary, lambda n, k: (k, n)) def test_weak_valued_len_cycles(self): - self.check_len_cycles(weakref.WeakValueDictionary, lambda k: (1, k)) + self.check_len_cycles(weakref.WeakValueDictionary, lambda n, k: (n, k)) def check_len_race(self, dict_type, cons): # Extended sanity checks for len() in the face of cyclic collection @@ -1090,6 +1097,86 @@ self.assertEqual(len(values), 0, "itervalues() did not touch all values") + def check_weak_destroy_while_iterating(self, dict, objects, iter_name): + n = len(dict) + it = iter(getattr(dict, iter_name)()) + next(it) # Trigger internal iteration + # Destroy an object + del objects[-1] + gc.collect() # just in case + # We have removed either the first consumed object, or another one + self.assertIn(len(list(it)), [len(objects), len(objects) - 1]) + del it + # The removal has been committed + self.assertEqual(len(dict), n - 1) + + def check_weak_destroy_and_mutate_while_iterating(self, dict, testcontext): + # Check that we can explicitly mutate the weak dict without + # interfering with delayed removal. + # `testcontext` should create an iterator, destroy one of the + # weakref'ed objects and then return a new key/value pair corresponding + # to the destroyed object. + with testcontext() as (k, v): + self.assertFalse(k in dict) + with testcontext() as (k, v): + self.assertRaises(KeyError, dict.__delitem__, k) + self.assertFalse(k in dict) + with testcontext() as (k, v): + self.assertRaises(KeyError, dict.pop, k) + self.assertFalse(k in dict) + with testcontext() as (k, v): + dict[k] = v + self.assertEqual(dict[k], v) + ddict = copy.copy(dict) + with testcontext() as (k, v): + dict.update(ddict) + self.assertEqual(dict, ddict) + with testcontext() as (k, v): + dict.clear() + self.assertEqual(len(dict), 0) + + def test_weak_keys_destroy_while_iterating(self): + # Issue #7105: iterators shouldn't crash when a key is implicitly removed + dict, objects = self.make_weak_keyed_dict() + self.check_weak_destroy_while_iterating(dict, objects, 'iterkeys') + self.check_weak_destroy_while_iterating(dict, objects, 'iteritems') + self.check_weak_destroy_while_iterating(dict, objects, 'itervalues') + self.check_weak_destroy_while_iterating(dict, objects, 'iterkeyrefs') + dict, objects = self.make_weak_keyed_dict() + @contextlib.contextmanager + def testcontext(): + try: + it = iter(dict.iteritems()) + next(it) + # Schedule a key/value for removal and recreate it + v = objects.pop().arg + gc.collect() # just in case + yield Object(v), v + finally: + it = None # should commit all removals + self.check_weak_destroy_and_mutate_while_iterating(dict, testcontext) + + def test_weak_values_destroy_while_iterating(self): + # Issue #7105: iterators shouldn't crash when a key is implicitly removed + dict, objects = self.make_weak_valued_dict() + self.check_weak_destroy_while_iterating(dict, objects, 'iterkeys') + self.check_weak_destroy_while_iterating(dict, objects, 'iteritems') + self.check_weak_destroy_while_iterating(dict, objects, 'itervalues') + self.check_weak_destroy_while_iterating(dict, objects, 'itervaluerefs') + dict, objects = self.make_weak_valued_dict() + @contextlib.contextmanager + def testcontext(): + try: + it = iter(dict.iteritems()) + next(it) + # Schedule a key/value for removal and recreate it + k = objects.pop().arg + gc.collect() # just in case + yield k, Object(k) + finally: + it = None # should commit all removals + self.check_weak_destroy_and_mutate_while_iterating(dict, testcontext) + def test_make_weak_keyed_dict_from_dict(self): o = Object(3) dict = weakref.WeakKeyDictionary({o:364}) diff --git a/Lib/test/test_weakset.py b/Lib/test/test_weakset.py --- a/Lib/test/test_weakset.py +++ b/Lib/test/test_weakset.py @@ -11,6 +11,7 @@ import collections import gc import contextlib +from UserString import UserString as ustr class Foo: @@ -448,6 +449,54 @@ self.assertGreaterEqual(n2, 0) self.assertLessEqual(n2, n1) + def test_weak_destroy_while_iterating(self): + # Issue #7105: iterators shouldn't crash when a key is implicitly removed + # Create new items to be sure no-one else holds a reference + items = [ustr(c) for c in ('a', 'b', 'c')] + s = WeakSet(items) + it = iter(s) + next(it) # Trigger internal iteration + # Destroy an item + del items[-1] + gc.collect() # just in case + # We have removed either the first consumed items, or another one + self.assertIn(len(list(it)), [len(items), len(items) - 1]) + del it + # The removal has been committed + self.assertEqual(len(s), len(items)) + + def test_weak_destroy_and_mutate_while_iterating(self): + # Issue #7105: iterators shouldn't crash when a key is implicitly removed + items = [ustr(c) for c in string.ascii_letters] + s = WeakSet(items) + @contextlib.contextmanager + def testcontext(): + try: + it = iter(s) + next(it) + # Schedule an item for removal and recreate it + u = ustr(str(items.pop())) + gc.collect() # just in case + yield u + finally: + it = None # should commit all removals + + with testcontext() as u: + self.assertFalse(u in s) + with testcontext() as u: + self.assertRaises(KeyError, s.remove, u) + self.assertFalse(u in s) + with testcontext() as u: + s.add(u) + self.assertTrue(u in s) + t = s.copy() + with testcontext() as u: + s.update(t) + self.assertEqual(len(s), len(t)) + with testcontext() as u: + s.clear() + self.assertEqual(len(s), 0) + def test_main(verbose=None): test_support.run_unittest(TestWeakSet) diff --git a/Lib/weakref.py b/Lib/weakref.py --- a/Lib/weakref.py +++ b/Lib/weakref.py @@ -20,7 +20,7 @@ ProxyType, ReferenceType) -from _weakrefset import WeakSet +from _weakrefset import WeakSet, _IterationGuard from exceptions import ReferenceError @@ -48,10 +48,24 @@ def remove(wr, selfref=ref(self)): self = selfref() if self is not None: - del self.data[wr.key] + if self._iterating: + self._pending_removals.append(wr.key) + else: + del self.data[wr.key] self._remove = remove + # A list of keys to be removed + self._pending_removals = [] + self._iterating = set() UserDict.UserDict.__init__(self, *args, **kw) + def _commit_removals(self): + l = self._pending_removals + d = self.data + # We shouldn't encounter any KeyError, because this method should + # always be called *before* mutating the dict. + while l: + del d[l.pop()] + def __getitem__(self, key): o = self.data[key]() if o is None: @@ -59,6 +73,11 @@ else: return o + def __delitem__(self, key): + if self._pending_removals: + self._commit_removals() + del self.data[key] + def __contains__(self, key): try: o = self.data[key]() @@ -77,8 +96,15 @@ return "" % id(self) def __setitem__(self, key, value): + if self._pending_removals: + self._commit_removals() self.data[key] = KeyedRef(value, self._remove, key) + def clear(self): + if self._pending_removals: + self._commit_removals() + self.data.clear() + def copy(self): new = WeakValueDictionary() for key, wr in self.data.items(): @@ -120,16 +146,18 @@ return L def iteritems(self): - for wr in self.data.itervalues(): - value = wr() - if value is not None: - yield wr.key, value + with _IterationGuard(self): + for wr in self.data.itervalues(): + value = wr() + if value is not None: + yield wr.key, value def iterkeys(self): - return self.data.iterkeys() + with _IterationGuard(self): + for k in self.data.iterkeys(): + yield k - def __iter__(self): - return self.data.iterkeys() + __iter__ = iterkeys def itervaluerefs(self): """Return an iterator that yields the weak references to the values. @@ -141,15 +169,20 @@ keep the values around longer than needed. """ - return self.data.itervalues() + with _IterationGuard(self): + for wr in self.data.itervalues(): + yield wr def itervalues(self): - for wr in self.data.itervalues(): - obj = wr() - if obj is not None: - yield obj + with _IterationGuard(self): + for wr in self.data.itervalues(): + obj = wr() + if obj is not None: + yield obj def popitem(self): + if self._pending_removals: + self._commit_removals() while 1: key, wr = self.data.popitem() o = wr() @@ -157,6 +190,8 @@ return key, o def pop(self, key, *args): + if self._pending_removals: + self._commit_removals() try: o = self.data.pop(key)() except KeyError: @@ -172,12 +207,16 @@ try: wr = self.data[key] except KeyError: + if self._pending_removals: + self._commit_removals() self.data[key] = KeyedRef(default, self._remove, key) return default else: return wr() def update(self, dict=None, **kwargs): + if self._pending_removals: + self._commit_removals() d = self.data if dict is not None: if not hasattr(dict, "items"): @@ -245,9 +284,29 @@ def remove(k, selfref=ref(self)): self = selfref() if self is not None: - del self.data[k] + if self._iterating: + self._pending_removals.append(k) + else: + del self.data[k] self._remove = remove - if dict is not None: self.update(dict) + # A list of dead weakrefs (keys to be removed) + self._pending_removals = [] + self._iterating = set() + if dict is not None: + self.update(dict) + + def _commit_removals(self): + # NOTE: We don't need to call this method before mutating the dict, + # because a dead weakref never compares equal to a live weakref, + # even if they happened to refer to equal objects. + # However, it means keys may already have been removed. + l = self._pending_removals + d = self.data + while l: + try: + del d[l.pop()] + except KeyError: + pass def __delitem__(self, key): del self.data[ref(key)] @@ -306,10 +365,11 @@ return L def iteritems(self): - for wr, value in self.data.iteritems(): - key = wr() - if key is not None: - yield key, value + with _IterationGuard(self): + for wr, value in self.data.iteritems(): + key = wr() + if key is not None: + yield key, value def iterkeyrefs(self): """Return an iterator that yields the weak references to the keys. @@ -321,19 +381,23 @@ keep the keys around longer than needed. """ - return self.data.iterkeys() + with _IterationGuard(self): + for wr in self.data.iterkeys(): + yield wr def iterkeys(self): - for wr in self.data.iterkeys(): - obj = wr() - if obj is not None: - yield obj + with _IterationGuard(self): + for wr in self.data.iterkeys(): + obj = wr() + if obj is not None: + yield obj - def __iter__(self): - return self.iterkeys() + __iter__ = iterkeys def itervalues(self): - return self.data.itervalues() + with _IterationGuard(self): + for value in self.data.itervalues(): + yield value def keyrefs(self): """Return a list of weak references to the keys. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 5 16:13:37 2013 From: python-checkins at python.org (christian.heimes) Date: Thu, 5 Dec 2013 16:13:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Test_syslog=2Eopenlog=28?= =?utf-8?q?=29_without_args_to_test_syslog=5Fget=5Fargv=28=29?= Message-ID: <3db0nF4hkmz7LjN@mail.python.org> http://hg.python.org/cpython/rev/69681e4615ee changeset: 87779:69681e4615ee parent: 87777:14ff7b1383a7 user: Christian Heimes date: Thu Dec 05 13:56:56 2013 +0100 summary: Test syslog.openlog() without args to test syslog_get_argv() files: Lib/test/test_syslog.py | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_syslog.py b/Lib/test/test_syslog.py --- a/Lib/test/test_syslog.py +++ b/Lib/test/test_syslog.py @@ -32,6 +32,10 @@ def test_log_upto(self): syslog.LOG_UPTO(syslog.LOG_INFO) + def test_openlog_noargs(self): + syslog.openlog() + syslog.syslog('test message from python test_syslog') + def test_main(): support.run_unittest(__name__) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 5 16:13:38 2013 From: python-checkins at python.org (christian.heimes) Date: Thu, 5 Dec 2013 16:13:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_simple_test_for_fcntl?= =?utf-8?b?LmZsb2NrKCk=?= Message-ID: <3db0nG6dDJz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/6f8a5458108d changeset: 87780:6f8a5458108d user: Christian Heimes date: Thu Dec 05 16:13:03 2013 +0100 summary: Add simple test for fcntl.flock() files: Lib/test/test_fcntl.py | 15 +++++++++++++++ 1 files changed, 15 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_fcntl.py b/Lib/test/test_fcntl.py --- a/Lib/test/test_fcntl.py +++ b/Lib/test/test_fcntl.py @@ -115,6 +115,21 @@ finally: os.close(fd) + def test_flock(self): + self.f = open(TESTFN, 'wb') + fileno = self.f.fileno() + fcntl.flock(fileno, fcntl.LOCK_SH) + fcntl.flock(fileno, fcntl.LOCK_UN) + fcntl.flock(self.f, fcntl.LOCK_SH | fcntl.LOCK_NB) + fcntl.flock(self.f, fcntl.LOCK_UN) + fcntl.flock(fileno, fcntl.LOCK_EX) + fcntl.flock(fileno, fcntl.LOCK_UN) + + self.assertRaises(ValueError, fcntl.flock, -1, fcntl.LOCK_SH) + self.assertRaises(TypeError, fcntl.flock, 'spam', fcntl.LOCK_SH) + self.assertRaises(OverflowError, fcntl.flock, _testcapi.INT_MAX+1, + fcntl.LOCK_SH) + def test_main(): run_unittest(TestFcntl) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 5 22:54:05 2013 From: python-checkins at python.org (charles-francois.natali) Date: Thu, 5 Dec 2013 22:54:05 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319850=3A_asyncio?= =?utf-8?q?=3A_Set_SA=5FRESTART_when_registering_a_signal_handler_to?= Message-ID: <3db9gK63sFz7LjR@mail.python.org> http://hg.python.org/cpython/rev/546cad3627e2 changeset: 87781:546cad3627e2 user: Charles-Fran?ois Natali date: Thu Dec 05 22:47:19 2013 +0100 summary: Issue #19850: asyncio: Set SA_RESTART when registering a signal handler to limit EINTR occurrences. files: Lib/asyncio/unix_events.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -74,6 +74,8 @@ try: signal.signal(sig, self._handle_signal) + # Set SA_RESTART to limit EINTR occurrences. + signal.siginterrupt(sig, False) except OSError as exc: del self._signal_handlers[sig] if not self._signal_handlers: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 5 23:48:19 2013 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 5 Dec 2013 23:48:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE4ODQw?= =?utf-8?q?=3A_Introduce_the_json_module_in_the_tutorial=2C_and_deemphasiz?= =?utf-8?q?e_the?= Message-ID: <3dbBsv41qfz7LjX@mail.python.org> http://hg.python.org/cpython/rev/90cf299dcf9b changeset: 87782:90cf299dcf9b branch: 3.3 parent: 87770:bec2033ee2ec user: Antoine Pitrou date: Thu Dec 05 23:46:32 2013 +0100 summary: Issue #18840: Introduce the json module in the tutorial, and deemphasize the pickle module. files: Doc/glossary.rst | 24 ++++++- Doc/tutorial/inputoutput.rst | 75 ++++++++++++++--------- Misc/NEWS | 3 + 3 files changed, 69 insertions(+), 33 deletions(-) diff --git a/Doc/glossary.rst b/Doc/glossary.rst --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -78,6 +78,13 @@ Benevolent Dictator For Life, a.k.a. `Guido van Rossum `_, Python's creator. + binary file + A :term:`file object` able to read and write + :term:`bytes-like objects `. + + .. seealso:: + A :term:`text file` reads and writes :class:`str` objects. + bytes-like object An object that supports the :ref:`bufferobjects`, like :class:`bytes`, :class:`bytearray` or :class:`memoryview`. Bytes-like objects can @@ -225,10 +232,11 @@ etc.). File objects are also called :dfn:`file-like objects` or :dfn:`streams`. - There are actually three categories of file objects: raw binary files, - buffered binary files and text files. Their interfaces are defined in the - :mod:`io` module. The canonical way to create a file object is by using - the :func:`open` function. + There are actually three categories of file objects: raw + :term:`binary files `, buffered + :term:`binary files ` and :term:`text files `. + Their interfaces are defined in the :mod:`io` module. The canonical + way to create a file object is by using the :func:`open` function. file-like object A synonym for :term:`file object`. @@ -780,6 +788,14 @@ :meth:`~collections.somenamedtuple._asdict`. Examples of struct sequences include :data:`sys.float_info` and the return value of :func:`os.stat`. + text file + A :term:`file object` able to read and write :class:`str` objects. + Often, a text file actually accesses a byte-oriented datastream + and handles the text encoding automatically. + + .. seealso:: + A :term:`binary file` reads and write :class:`bytes` objects. + triple-quoted string A string which is bound by three instances of either a quotation mark (") or an apostrophe ('). While they don't provide any functionality diff --git a/Doc/tutorial/inputoutput.rst b/Doc/tutorial/inputoutput.rst --- a/Doc/tutorial/inputoutput.rst +++ b/Doc/tutorial/inputoutput.rst @@ -377,47 +377,64 @@ Reference for a complete guide to file objects. -.. _tut-pickle: +.. _tut-json: -The :mod:`pickle` Module ------------------------- +Saving structured data with :mod:`json` +--------------------------------------- -.. index:: module: pickle +.. index:: module: json -Strings can easily be written to and read from a file. Numbers take a bit more +Strings can easily be written to and read from a file. Numbers take a bit more effort, since the :meth:`read` method only returns strings, which will have to be passed to a function like :func:`int`, which takes a string like ``'123'`` -and returns its numeric value 123. However, when you want to save more complex -data types like lists, dictionaries, or class instances, things get a lot more -complicated. +and returns its numeric value 123. When you want to save more complex data +types like nested lists and dictionaries, parsing and serializing by hand +becomes complicated. -Rather than have users be constantly writing and debugging code to save -complicated data types, Python provides a standard module called :mod:`pickle`. -This is an amazing module that can take almost any Python object (even some -forms of Python code!), and convert it to a string representation; this process -is called :dfn:`pickling`. Reconstructing the object from the string -representation is called :dfn:`unpickling`. Between pickling and unpickling, -the string representing the object may have been stored in a file or data, or +Rather than having users constantly writing and debugging code to save +complicated data types to files, Python allows you to use the popular data +interchange format called `JSON (JavaScript Object Notation) +`_. The standard module called :mod:`json` can take Python +data hierarchies, and convert them to string representations; this process is +called :dfn:`serializing`. Reconstructing the data from the string representation +is called :dfn:`deserializing`. Between serializing and deserializing, the +string representing the object may have been stored in a file or data, or sent over a network connection to some distant machine. -If you have an object ``x``, and a file object ``f`` that's been opened for -writing, the simplest way to pickle the object takes only one line of code:: +.. note:: + The JSON format is commonly used by modern applications to allow for data + exchange. Many programmers are already familiar with it, which makes + it a good choice for interoperability. - pickle.dump(x, f) +If you have an object ``x``, you can view its JSON string representation with a +simple line of code:: -To unpickle the object again, if ``f`` is a file object which has been opened -for reading:: + >>> json.dumps([1, 'simple', 'list']) + '[1, "simple", "list"]' - x = pickle.load(f) +Another variant of the :func:`~json.dumps` function, called :func:`~json.dump`, +simply serializes the object to a :term:`text file`. So if ``f`` is a +:term:`text file` object opened for writing, we can do this:: -(There are other variants of this, used when pickling many objects or when you -don't want to write the pickled data to a file; consult the complete -documentation for :mod:`pickle` in the Python Library Reference.) + json.dump(x, f) -:mod:`pickle` is the standard way to make Python objects which can be stored and -reused by other programs or by a future invocation of the same program; the -technical term for this is a :dfn:`persistent` object. Because :mod:`pickle` is -so widely used, many authors who write Python extensions take care to ensure -that new data types such as matrices can be properly pickled and unpickled. +To decode the object again, if ``f`` is a :term:`text file` object which has +been opened for reading:: + x = json.load(f) +This simple serialization technique can handle lists and dictionaries, but +serializing arbitrary class instances in JSON requires a bit of extra effort. +The reference for the :mod:`json` module contains an explanation of this. + +.. seealso:: + + :mod:`pickle` - the pickle module + + Contrary to :ref:`JSON `, *pickle* is a protocol which allows + the serialization of arbitrarily complex Python objects. As such, it is + specific to Python and cannot be used to communicate with applications + written in other languages. It is also insecure by default: + deserializing pickle data coming from an untrusted source can execute + arbitrary code, if the data was crafted by a skilled attacker. + diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -113,6 +113,9 @@ Documentation ------------- +- Issue #18840: Introduce the json module in the tutorial, and deemphasize + the pickle module. + - Issue #19845: Updated the Compiling Python on Windows section. - Issue #19795: Improved markup of True/False constants. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 5 23:48:21 2013 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 5 Dec 2013 23:48:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2318840=3A_Introduce_the_json_module_in_the_tutor?= =?utf-8?q?ial=2C_and_deemphasize_the?= Message-ID: <3dbBsx04knz7Lkl@mail.python.org> http://hg.python.org/cpython/rev/1009b77f59fd changeset: 87783:1009b77f59fd parent: 87781:546cad3627e2 parent: 87782:90cf299dcf9b user: Antoine Pitrou date: Thu Dec 05 23:48:10 2013 +0100 summary: Issue #18840: Introduce the json module in the tutorial, and deemphasize the pickle module. files: Doc/glossary.rst | 24 ++++++- Doc/tutorial/inputoutput.rst | 75 ++++++++++++++--------- Misc/NEWS | 3 + 3 files changed, 69 insertions(+), 33 deletions(-) diff --git a/Doc/glossary.rst b/Doc/glossary.rst --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -78,6 +78,13 @@ Benevolent Dictator For Life, a.k.a. `Guido van Rossum `_, Python's creator. + binary file + A :term:`file object` able to read and write + :term:`bytes-like objects `. + + .. seealso:: + A :term:`text file` reads and writes :class:`str` objects. + bytes-like object An object that supports the :ref:`bufferobjects`, like :class:`bytes`, :class:`bytearray` or :class:`memoryview`. Bytes-like objects can @@ -225,10 +232,11 @@ etc.). File objects are also called :dfn:`file-like objects` or :dfn:`streams`. - There are actually three categories of file objects: raw binary files, - buffered binary files and text files. Their interfaces are defined in the - :mod:`io` module. The canonical way to create a file object is by using - the :func:`open` function. + There are actually three categories of file objects: raw + :term:`binary files `, buffered + :term:`binary files ` and :term:`text files `. + Their interfaces are defined in the :mod:`io` module. The canonical + way to create a file object is by using the :func:`open` function. file-like object A synonym for :term:`file object`. @@ -800,6 +808,14 @@ :meth:`~collections.somenamedtuple._asdict`. Examples of struct sequences include :data:`sys.float_info` and the return value of :func:`os.stat`. + text file + A :term:`file object` able to read and write :class:`str` objects. + Often, a text file actually accesses a byte-oriented datastream + and handles the text encoding automatically. + + .. seealso:: + A :term:`binary file` reads and write :class:`bytes` objects. + triple-quoted string A string which is bound by three instances of either a quotation mark (") or an apostrophe ('). While they don't provide any functionality diff --git a/Doc/tutorial/inputoutput.rst b/Doc/tutorial/inputoutput.rst --- a/Doc/tutorial/inputoutput.rst +++ b/Doc/tutorial/inputoutput.rst @@ -377,47 +377,64 @@ Reference for a complete guide to file objects. -.. _tut-pickle: +.. _tut-json: -The :mod:`pickle` Module ------------------------- +Saving structured data with :mod:`json` +--------------------------------------- -.. index:: module: pickle +.. index:: module: json -Strings can easily be written to and read from a file. Numbers take a bit more +Strings can easily be written to and read from a file. Numbers take a bit more effort, since the :meth:`read` method only returns strings, which will have to be passed to a function like :func:`int`, which takes a string like ``'123'`` -and returns its numeric value 123. However, when you want to save more complex -data types like lists, dictionaries, or class instances, things get a lot more -complicated. +and returns its numeric value 123. When you want to save more complex data +types like nested lists and dictionaries, parsing and serializing by hand +becomes complicated. -Rather than have users be constantly writing and debugging code to save -complicated data types, Python provides a standard module called :mod:`pickle`. -This is an amazing module that can take almost any Python object (even some -forms of Python code!), and convert it to a string representation; this process -is called :dfn:`pickling`. Reconstructing the object from the string -representation is called :dfn:`unpickling`. Between pickling and unpickling, -the string representing the object may have been stored in a file or data, or +Rather than having users constantly writing and debugging code to save +complicated data types to files, Python allows you to use the popular data +interchange format called `JSON (JavaScript Object Notation) +`_. The standard module called :mod:`json` can take Python +data hierarchies, and convert them to string representations; this process is +called :dfn:`serializing`. Reconstructing the data from the string representation +is called :dfn:`deserializing`. Between serializing and deserializing, the +string representing the object may have been stored in a file or data, or sent over a network connection to some distant machine. -If you have an object ``x``, and a file object ``f`` that's been opened for -writing, the simplest way to pickle the object takes only one line of code:: +.. note:: + The JSON format is commonly used by modern applications to allow for data + exchange. Many programmers are already familiar with it, which makes + it a good choice for interoperability. - pickle.dump(x, f) +If you have an object ``x``, you can view its JSON string representation with a +simple line of code:: -To unpickle the object again, if ``f`` is a file object which has been opened -for reading:: + >>> json.dumps([1, 'simple', 'list']) + '[1, "simple", "list"]' - x = pickle.load(f) +Another variant of the :func:`~json.dumps` function, called :func:`~json.dump`, +simply serializes the object to a :term:`text file`. So if ``f`` is a +:term:`text file` object opened for writing, we can do this:: -(There are other variants of this, used when pickling many objects or when you -don't want to write the pickled data to a file; consult the complete -documentation for :mod:`pickle` in the Python Library Reference.) + json.dump(x, f) -:mod:`pickle` is the standard way to make Python objects which can be stored and -reused by other programs or by a future invocation of the same program; the -technical term for this is a :dfn:`persistent` object. Because :mod:`pickle` is -so widely used, many authors who write Python extensions take care to ensure -that new data types such as matrices can be properly pickled and unpickled. +To decode the object again, if ``f`` is a :term:`text file` object which has +been opened for reading:: + x = json.load(f) +This simple serialization technique can handle lists and dictionaries, but +serializing arbitrary class instances in JSON requires a bit of extra effort. +The reference for the :mod:`json` module contains an explanation of this. + +.. seealso:: + + :mod:`pickle` - the pickle module + + Contrary to :ref:`JSON `, *pickle* is a protocol which allows + the serialization of arbitrarily complex Python objects. As such, it is + specific to Python and cannot be used to communicate with applications + written in other languages. It is also insecure by default: + deserializing pickle data coming from an untrusted source can execute + arbitrary code, if the data was crafted by a skilled attacker. + diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -103,6 +103,9 @@ Documentation ------------- +- Issue #18840: Introduce the json module in the tutorial, and deemphasize + the pickle module. + - Issue #19845: Updated the Compiling Python on Windows section. - Issue #19795: Improved markup of True/False constants. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 5 23:52:08 2013 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 5 Dec 2013 23:52:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE4ODQw?= =?utf-8?q?=3A_Introduce_the_json_module_in_the_tutorial=2C_and_deemphasiz?= =?utf-8?q?e_the?= Message-ID: <3dbByJ62jhz7LjX@mail.python.org> http://hg.python.org/cpython/rev/481b30bfe496 changeset: 87784:481b30bfe496 branch: 2.7 parent: 87778:03fcc12282fc user: Antoine Pitrou date: Thu Dec 05 23:46:32 2013 +0100 summary: Issue #18840: Introduce the json module in the tutorial, and deemphasize the pickle module. files: Doc/tutorial/inputoutput.rst | 75 ++++++++++++++--------- Misc/NEWS | 3 + 2 files changed, 49 insertions(+), 29 deletions(-) diff --git a/Doc/tutorial/inputoutput.rst b/Doc/tutorial/inputoutput.rst --- a/Doc/tutorial/inputoutput.rst +++ b/Doc/tutorial/inputoutput.rst @@ -358,47 +358,64 @@ Reference for a complete guide to file objects. -.. _tut-pickle: +.. _tut-json: -The :mod:`pickle` Module ------------------------- +Saving structured data with :mod:`json` +--------------------------------------- -.. index:: module: pickle +.. index:: module: json -Strings can easily be written to and read from a file. Numbers take a bit more +Strings can easily be written to and read from a file. Numbers take a bit more effort, since the :meth:`read` method only returns strings, which will have to be passed to a function like :func:`int`, which takes a string like ``'123'`` -and returns its numeric value 123. However, when you want to save more complex -data types like lists, dictionaries, or class instances, things get a lot more -complicated. +and returns its numeric value 123. When you want to save more complex data +types like nested lists and dictionaries, parsing and serializing by hand +becomes complicated. -Rather than have users be constantly writing and debugging code to save -complicated data types, Python provides a standard module called :mod:`pickle`. -This is an amazing module that can take almost any Python object (even some -forms of Python code!), and convert it to a string representation; this process -is called :dfn:`pickling`. Reconstructing the object from the string -representation is called :dfn:`unpickling`. Between pickling and unpickling, -the string representing the object may have been stored in a file or data, or +Rather than having users constantly writing and debugging code to save +complicated data types to files, Python allows you to use the popular data +interchange format called `JSON (JavaScript Object Notation) +`_. The standard module called :mod:`json` can take Python +data hierarchies, and convert them to string representations; this process is +called :dfn:`serializing`. Reconstructing the data from the string representation +is called :dfn:`deserializing`. Between serializing and deserializing, the +string representing the object may have been stored in a file or data, or sent over a network connection to some distant machine. -If you have an object ``x``, and a file object ``f`` that's been opened for -writing, the simplest way to pickle the object takes only one line of code:: +.. note:: + The JSON format is commonly used by modern applications to allow for data + exchange. Many programmers are already familiar with it, which makes + it a good choice for interoperability. - pickle.dump(x, f) +If you have an object ``x``, you can view its JSON string representation with a +simple line of code:: -To unpickle the object again, if ``f`` is a file object which has been opened -for reading:: + >>> json.dumps([1, 'simple', 'list']) + '[1, "simple", "list"]' - x = pickle.load(f) +Another variant of the :func:`~json.dumps` function, called :func:`~json.dump`, +simply serializes the object to a file. So if ``f`` is a :term:`file object` +opened for writing, we can do this:: -(There are other variants of this, used when pickling many objects or when you -don't want to write the pickled data to a file; consult the complete -documentation for :mod:`pickle` in the Python Library Reference.) + json.dump(x, f) -:mod:`pickle` is the standard way to make Python objects which can be stored and -reused by other programs or by a future invocation of the same program; the -technical term for this is a :dfn:`persistent` object. Because :mod:`pickle` is -so widely used, many authors who write Python extensions take care to ensure -that new data types such as matrices can be properly pickled and unpickled. +To decode the object again, if ``f`` is a :term:`file object` which has +been opened for reading:: + x = json.load(f) +This simple serialization technique can handle lists and dictionaries, but +serializing arbitrary class instances in JSON requires a bit of extra effort. +The reference for the :mod:`json` module contains an explanation of this. + +.. seealso:: + + :mod:`pickle` - the pickle module + + Contrary to :ref:`JSON `, *pickle* is a protocol which allows + the serialization of arbitrarily complex Python objects. As such, it is + specific to Python and cannot be used to communicate with applications + written in other languages. It is also insecure by default: + deserializing pickle data coming from an untrusted source can execute + arbitrary code, if the data was crafted by a skilled attacker. + diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -81,6 +81,9 @@ Documentation ------------- +- Issue #18840: Introduce the json module in the tutorial, and deemphasize + the pickle module. + - Issue #19795: Improved markup of True/False constants. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 6 00:29:10 2013 From: python-checkins at python.org (christian.heimes) Date: Fri, 6 Dec 2013 00:29:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319296=3A_Silence_?= =?utf-8?q?compiler_warning_in_dbm=5Fopen=2E?= Message-ID: <3dbCn21d37z7Lk1@mail.python.org> http://hg.python.org/cpython/rev/3068ff3d9d1e changeset: 87785:3068ff3d9d1e parent: 87783:1009b77f59fd user: Christian Heimes date: Fri Dec 06 00:20:00 2013 +0100 summary: Issue #19296: Silence compiler warning in dbm_open. Some dbm header files declare the first argument as char * instead of a const char *. files: Misc/NEWS | 2 ++ Modules/_dbmmodule.c | 3 ++- 2 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,8 @@ Library ------- +- Issue #19296: Silence compiler warning in dbm_open + - Issue #19839: Fix regression in bz2 module's handling of non-bzip2 data at EOF, and analogous bug in lzma module. diff --git a/Modules/_dbmmodule.c b/Modules/_dbmmodule.c --- a/Modules/_dbmmodule.c +++ b/Modules/_dbmmodule.c @@ -66,7 +66,8 @@ if (dp == NULL) return NULL; dp->di_size = -1; - if ( (dp->di_dbm = dbm_open(file, flags, mode)) == 0 ) { + /* See issue #19296 */ + if ( (dp->di_dbm = dbm_open((char *)file, flags, mode)) == 0 ) { PyErr_SetFromErrno(DbmError); Py_DECREF(dp); return NULL; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 6 00:29:11 2013 From: python-checkins at python.org (christian.heimes) Date: Fri, 6 Dec 2013 00:29:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319509=3A_Finish_i?= =?utf-8?q?mplementation_of_check=5Fhostname?= Message-ID: <3dbCn34Sv0z7Lkr@mail.python.org> http://hg.python.org/cpython/rev/1605eda93392 changeset: 87786:1605eda93392 user: Christian Heimes date: Fri Dec 06 00:23:13 2013 +0100 summary: Issue #19509: Finish implementation of check_hostname The new asyncio package now supports the new feature and comes with additional tests for SSL. files: Lib/asyncio/selector_events.py | 25 +- Lib/test/keycert3.pem | 0 Lib/test/pycacert.pem | 0 Lib/test/test_asyncio/sample.crt | 14 - Lib/test/test_asyncio/sample.key | 15 - Lib/test/ssl_cert.pem | 0 Lib/test/ssl_key.pem | 0 Lib/test/test_asyncio/test_events.py | 140 +++++++++++++- 8 files changed, 137 insertions(+), 57 deletions(-) diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -583,7 +583,8 @@ # cadefault=True. if hasattr(ssl, '_create_stdlib_context'): sslcontext = ssl._create_stdlib_context( - cert_reqs=ssl.CERT_REQUIRED) + cert_reqs=ssl.CERT_REQUIRED, + check_hostname=bool(server_hostname)) else: # Fallback for Python 3.3. sslcontext = ssl.SSLContext(ssl.PROTOCOL_SSLv23) @@ -639,17 +640,19 @@ self._loop.remove_reader(self._sock_fd) self._loop.remove_writer(self._sock_fd) - # Verify hostname if requested. peercert = self._sock.getpeercert() - if (self._server_hostname and - self._sslcontext.verify_mode != ssl.CERT_NONE): - try: - ssl.match_hostname(peercert, self._server_hostname) - except Exception as exc: - self._sock.close() - if self._waiter is not None: - self._waiter.set_exception(exc) - return + if not hasattr(self._sslcontext, 'check_hostname'): + # Verify hostname if requested, Python 3.4+ uses check_hostname + # and checks the hostname in do_handshake() + if (self._server_hostname and + self._sslcontext.verify_mode != ssl.CERT_NONE): + try: + ssl.match_hostname(peercert, self._server_hostname) + except Exception as exc: + self._sock.close() + if self._waiter is not None: + self._waiter.set_exception(exc) + return # Add extra info that becomes available after handshake. self._extra.update(peercert=peercert, diff --git a/Lib/test/keycert3.pem b/Lib/test/test_asyncio/keycert3.pem copy from Lib/test/keycert3.pem copy to Lib/test/test_asyncio/keycert3.pem diff --git a/Lib/test/pycacert.pem b/Lib/test/test_asyncio/pycacert.pem copy from Lib/test/pycacert.pem copy to Lib/test/test_asyncio/pycacert.pem diff --git a/Lib/test/test_asyncio/sample.crt b/Lib/test/test_asyncio/sample.crt deleted file mode 100644 --- a/Lib/test/test_asyncio/sample.crt +++ /dev/null @@ -1,14 +0,0 @@ ------BEGIN CERTIFICATE----- -MIICMzCCAZwCCQDFl4ys0fU7iTANBgkqhkiG9w0BAQUFADBeMQswCQYDVQQGEwJV -UzETMBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuLUZyYW5jaXNjbzEi -MCAGA1UECgwZUHl0aG9uIFNvZnR3YXJlIEZvbmRhdGlvbjAeFw0xMzAzMTgyMDA3 -MjhaFw0yMzAzMTYyMDA3MjhaMF4xCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxp -Zm9ybmlhMRYwFAYDVQQHDA1TYW4tRnJhbmNpc2NvMSIwIAYDVQQKDBlQeXRob24g -U29mdHdhcmUgRm9uZGF0aW9uMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCn -t3s+J7L0xP/YdAQOacpPi9phlrzKZhcXL3XMu2LCUg2fNJpx/47Vc5TZSaO11uO7 -gdwVz3Z7Q2epAgwo59JLffLt5fia8+a/SlPweI/j4+wcIIIiqusnLfpqR8cIAavg -Z06cLYCDvb9wMlheIvSJY12skc1nnphWS2YJ0Xm6uQIDAQABMA0GCSqGSIb3DQEB -BQUAA4GBAE9PknG6pv72+5z/gsDGYy8sK5UNkbWSNr4i4e5lxVsF03+/M71H+3AB -MxVX4+A+Vlk2fmU+BrdHIIUE0r1dDcO3josQ9hc9OJpp5VLSQFP8VeuJCmzYPp9I -I8WbW93cnXnChTrYQVdgVoFdv7GE9YgU7NYkrGIM0nZl1/f/bHPB ------END CERTIFICATE----- diff --git a/Lib/test/test_asyncio/sample.key b/Lib/test/test_asyncio/sample.key deleted file mode 100644 --- a/Lib/test/test_asyncio/sample.key +++ /dev/null @@ -1,15 +0,0 @@ ------BEGIN RSA PRIVATE KEY----- -MIICXQIBAAKBgQCnt3s+J7L0xP/YdAQOacpPi9phlrzKZhcXL3XMu2LCUg2fNJpx -/47Vc5TZSaO11uO7gdwVz3Z7Q2epAgwo59JLffLt5fia8+a/SlPweI/j4+wcIIIi -qusnLfpqR8cIAavgZ06cLYCDvb9wMlheIvSJY12skc1nnphWS2YJ0Xm6uQIDAQAB -AoGABfm8k19Yue3W68BecKEGS0VBV57GRTPT+MiBGvVGNIQ15gk6w3sGfMZsdD1y -bsUkQgcDb2d/4i5poBTpl/+Cd41V+c20IC/sSl5X1IEreHMKSLhy/uyjyiyfXlP1 -iXhToFCgLWwENWc8LzfUV8vuAV5WG6oL9bnudWzZxeqx8V0CQQDR7xwVj6LN70Eb -DUhSKLkusmFw5Gk9NJ/7wZ4eHg4B8c9KNVvSlLCLhcsVTQXuqYeFpOqytI45SneP -lr0vrvsDAkEAzITYiXu6ox5huDCG7imX2W9CAYuX638urLxBqBXMS7GqBzojD6RL -21Q8oPwJWJquERa3HDScq1deiQbM9uKIkwJBAIa1PLslGN216Xv3UPHPScyKD/aF -ynXIv+OnANPoiyp6RH4ksQ/18zcEGiVH8EeNpvV9tlAHhb+DZibQHgNr74sCQQC0 -zhToplu/bVKSlUQUNO0rqrI9z30FErDewKeCw5KSsIRSU1E/uM3fHr9iyq4wiL6u -GNjUtKZ0y46lsT9uW6LFAkB5eqeEQnshAdr3X5GykWHJ8DDGBXPPn6Rce1NX4RSq -V9khG2z1bFyfo+hMqpYnF2k32hVq3E54RS8YYnwBsVof ------END RSA PRIVATE KEY----- diff --git a/Lib/test/ssl_cert.pem b/Lib/test/test_asyncio/ssl_cert.pem copy from Lib/test/ssl_cert.pem copy to Lib/test/test_asyncio/ssl_cert.pem diff --git a/Lib/test/ssl_key.pem b/Lib/test/test_asyncio/ssl_key.pem copy from Lib/test/ssl_key.pem copy to Lib/test/test_asyncio/ssl_key.pem diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -17,7 +17,7 @@ import errno import unittest import unittest.mock -from test.support import find_unused_port, IPV6_ENABLED +from test import support # find_unused_port, IPV6_ENABLED, TEST_HOME_DIR from asyncio import futures @@ -30,10 +30,27 @@ from asyncio import locks +def data_file(filename): + if hasattr(support, 'TEST_HOME_DIR'): + fullname = os.path.join(support.TEST_HOME_DIR, filename) + if os.path.isfile(fullname): + return fullname + fullname = os.path.join(os.path.dirname(__file__), filename) + if os.path.isfile(fullname): + return fullname + raise FileNotFoundError(filename) + +ONLYCERT = data_file('ssl_cert.pem') +ONLYKEY = data_file('ssl_key.pem') +SIGNED_CERTFILE = data_file('keycert3.pem') +SIGNING_CA = data_file('pycacert.pem') + + class MyProto(protocols.Protocol): done = None def __init__(self, loop=None): + self.transport = None self.state = 'INITIAL' self.nbytes = 0 if loop is not None: @@ -523,7 +540,7 @@ def test_create_connection_local_addr(self): with test_utils.run_test_server() as httpd: - port = find_unused_port() + port = support.find_unused_port() f = self.loop.create_connection( lambda: MyProto(loop=self.loop), *httpd.address, local_addr=(httpd.address[0], port)) @@ -587,6 +604,20 @@ # close server server.close() + def _make_ssl_server(self, factory, certfile, keyfile=None): + sslcontext = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + sslcontext.options |= ssl.OP_NO_SSLv2 + sslcontext.load_cert_chain(certfile, keyfile) + + f = self.loop.create_server( + factory, '127.0.0.1', 0, ssl=sslcontext) + + server = self.loop.run_until_complete(f) + sock = server.sockets[0] + host, port = sock.getsockname() + self.assertEqual(host, '127.0.0.1') + return server, host, port + @unittest.skipIf(ssl is None, 'No ssl module') def test_create_server_ssl(self): proto = None @@ -602,19 +633,7 @@ proto = MyProto(loop=self.loop) return proto - here = os.path.dirname(__file__) - sslcontext = ssl.SSLContext(ssl.PROTOCOL_SSLv23) - sslcontext.load_cert_chain( - certfile=os.path.join(here, 'sample.crt'), - keyfile=os.path.join(here, 'sample.key')) - - f = self.loop.create_server( - factory, '127.0.0.1', 0, ssl=sslcontext) - - server = self.loop.run_until_complete(f) - sock = server.sockets[0] - host, port = sock.getsockname() - self.assertEqual(host, '127.0.0.1') + server, host, port = self._make_ssl_server(factory, ONLYCERT, ONLYKEY) f_c = self.loop.create_connection(ClientMyProto, host, port, ssl=test_utils.dummy_ssl_context()) @@ -646,6 +665,93 @@ # stop serving server.close() + @unittest.skipIf(ssl is None, 'No ssl module') + def test_create_server_ssl_verify_failed(self): + proto = None + + def factory(): + nonlocal proto + proto = MyProto(loop=self.loop) + return proto + + server, host, port = self._make_ssl_server(factory, SIGNED_CERTFILE) + + sslcontext_client = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + sslcontext_client.options |= ssl.OP_NO_SSLv2 + sslcontext_client.verify_mode = ssl.CERT_REQUIRED + if hasattr(sslcontext_client, 'check_hostname'): + sslcontext_client.check_hostname = True + + # no CA loaded + f_c = self.loop.create_connection(MyProto, host, port, + ssl=sslcontext_client) + with self.assertRaisesRegex(ssl.SSLError, + 'certificate verify failed '): + self.loop.run_until_complete(f_c) + + # close connection + self.assertIsNone(proto.transport) + server.close() + + @unittest.skipIf(ssl is None, 'No ssl module') + def test_create_server_ssl_match_failed(self): + proto = None + + def factory(): + nonlocal proto + proto = MyProto(loop=self.loop) + return proto + + server, host, port = self._make_ssl_server(factory, SIGNED_CERTFILE) + + sslcontext_client = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + sslcontext_client.options |= ssl.OP_NO_SSLv2 + sslcontext_client.verify_mode = ssl.CERT_REQUIRED + sslcontext_client.load_verify_locations( + cafile=SIGNING_CA) + if hasattr(sslcontext_client, 'check_hostname'): + sslcontext_client.check_hostname = True + + # incorrect server_hostname + f_c = self.loop.create_connection(MyProto, host, port, + ssl=sslcontext_client) + with self.assertRaisesRegex(ssl.CertificateError, + "hostname '127.0.0.1' doesn't match 'localhost'"): + self.loop.run_until_complete(f_c) + + # close connection + proto.transport.close() + server.close() + + @unittest.skipIf(ssl is None, 'No ssl module') + def test_create_server_ssl_verified(self): + proto = None + + def factory(): + nonlocal proto + proto = MyProto(loop=self.loop) + return proto + + server, host, port = self._make_ssl_server(factory, SIGNED_CERTFILE) + + sslcontext_client = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + sslcontext_client.options |= ssl.OP_NO_SSLv2 + sslcontext_client.verify_mode = ssl.CERT_REQUIRED + sslcontext_client.load_verify_locations(cafile=SIGNING_CA) + if hasattr(sslcontext_client, 'check_hostname'): + sslcontext_client.check_hostname = True + + # Connection succeeds with correct CA and server hostname. + f_c = self.loop.create_connection(MyProto, host, port, + ssl=sslcontext_client, + server_hostname='localhost') + client, pr = self.loop.run_until_complete(f_c) + + # close connection + proto.transport.close() + client.close() + server.close() + def test_create_server_sock(self): proto = futures.Future(loop=self.loop) @@ -688,7 +794,7 @@ server.close() - @unittest.skipUnless(IPV6_ENABLED, 'IPv6 not supported or enabled') + @unittest.skipUnless(support.IPV6_ENABLED, 'IPv6 not supported or enabled') def test_create_server_dual_stack(self): f_proto = futures.Future(loop=self.loop) @@ -700,7 +806,7 @@ try_count = 0 while True: try: - port = find_unused_port() + port = support.find_unused_port() f = self.loop.create_server(TestMyProto, host=None, port=port) server = self.loop.run_until_complete(f) except OSError as ex: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 6 02:58:32 2013 From: python-checkins at python.org (christian.heimes) Date: Fri, 6 Dec 2013 02:58:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_fix_cert_names_for_asyncio?= =?utf-8?q?_test?= Message-ID: <3dbH5N2qRbz7LjX@mail.python.org> http://hg.python.org/cpython/rev/ec1e7fc9b5a4 changeset: 87787:ec1e7fc9b5a4 user: Christian Heimes date: Fri Dec 06 02:58:23 2013 +0100 summary: fix cert names for asyncio test files: Lib/asyncio/test_utils.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py --- a/Lib/asyncio/test_utils.py +++ b/Lib/asyncio/test_utils.py @@ -95,8 +95,8 @@ if not os.path.isdir(here): here = os.path.join(os.path.dirname(os.__file__), 'test', 'test_asyncio') - keyfile = os.path.join(here, 'sample.key') - certfile = os.path.join(here, 'sample.crt') + keyfile = os.path.join(here, 'ssl_key.pem') + certfile = os.path.join(here, 'ssl_cert.pem') ssock = ssl.wrap_socket(request, keyfile=keyfile, certfile=certfile, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 6 04:29:44 2013 From: python-checkins at python.org (alexandre.vassalotti) Date: Fri, 6 Dec 2013 04:29:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319881=3A_Fix_bad_?= =?utf-8?q?pickling_of_large_bytes_in_cpickle=2E?= Message-ID: <3dbK6c0TNFz7LkK@mail.python.org> http://hg.python.org/cpython/rev/2612ea573ff7 changeset: 87788:2612ea573ff7 user: Alexandre Vassalotti date: Thu Dec 05 19:29:32 2013 -0800 summary: Issue #19881: Fix bad pickling of large bytes in cpickle. files: Lib/test/pickletester.py | 75 ++++++++++++++++++++------- Misc/NEWS | 4 + Modules/_pickle.c | 2 +- 3 files changed, 59 insertions(+), 22 deletions(-) diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -3,6 +3,7 @@ import pickle import pickletools import random +import struct import sys import unittest import weakref @@ -1611,9 +1612,9 @@ data = 1 << (8 * size) try: for proto in protocols: + if proto < 2: + continue with self.subTest(proto=proto): - if proto < 2: - continue with self.assertRaises((ValueError, OverflowError)): self.dumps(data, protocol=proto) finally: @@ -1628,13 +1629,17 @@ data = b"abcd" * (size // 4) try: for proto in protocols: + if proto < 3: + continue with self.subTest(proto=proto): - if proto < 3: - continue try: pickled = self.dumps(data, protocol=proto) - self.assertTrue(b"abcd" in pickled[:19]) - self.assertTrue(b"abcd" in pickled[-18:]) + header = (pickle.BINBYTES + + struct.pack("proto >= 4) { header[0] = BINBYTES8; _write_size64(header + 1, size); - len = 8; + len = 9; } else { PyErr_SetString(PyExc_OverflowError, -- Repository URL: http://hg.python.org/cpython From victor.stinner at gmail.com Mon Dec 2 13:25:00 2013 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 2 Dec 2013 13:25:00 +0100 Subject: [Python-checkins] cpython (3.3): Issue #19728: Fix sys.getfilesystemencoding() documentation In-Reply-To: References: <3dY3hr6BG1z7LrQ@mail.python.org> Message-ID: Oops, I happens to me sometimes when I open too many tabs in Firefox. The correct issue number is #19847: "Setting the default filesystem-encoding". Victor 2013/12/2 Nick Coghlan : > On 2 Dec 2013 21:18, "victor.stinner" wrote: >> >> http://hg.python.org/cpython/rev/b231e0c3fd26 >> changeset: 87692:b231e0c3fd26 >> branch: 3.3 >> parent: 87690:7d3297f127ae >> user: Victor Stinner >> date: Mon Dec 02 12:16:46 2013 +0100 >> summary: >> Issue #19728: Fix sys.getfilesystemencoding() documentation > > This doesn't appear to be the right issue number (ironically, this number > refers to the PEP 453 Windows installer one I was trying to reference when I > got a commit message wrong the other day and ended up referencing an asyncio > bug instead). > > Cheers, > Nick. From solipsis at pitrou.net Fri Dec 6 10:10:33 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 06 Dec 2013 10:10:33 +0100 Subject: [Python-checkins] Daily reference leaks (ec1e7fc9b5a4): sum=-4 Message-ID: results for ec1e7fc9b5a4 on branch "default" -------------------------------------------- test_site leaked [-2, 0, 0] references, sum=-2 test_site leaked [-2, 0, 0] memory blocks, sum=-2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogRv843s', '-x'] From python-checkins at python.org Fri Dec 6 12:23:21 2013 From: python-checkins at python.org (vinay.sajip) Date: Fri, 6 Dec 2013 12:23:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Added_minor_cl?= =?utf-8?q?arification_in_logging_HOWTO=2E?= Message-ID: <3dbWd55Hw8z7LjZ@mail.python.org> http://hg.python.org/cpython/rev/80142c15a920 changeset: 87789:80142c15a920 branch: 2.7 parent: 87784:481b30bfe496 user: Vinay Sajip date: Fri Dec 06 11:21:15 2013 +0000 summary: Added minor clarification in logging HOWTO. files: Doc/howto/logging.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst --- a/Doc/howto/logging.rst +++ b/Doc/howto/logging.rst @@ -122,7 +122,8 @@ ^^^^^^^^^^^^^^^^^ A very common situation is that of recording logging events in a file, so let's -look at that next:: +look at that next. Be sure to try the following in a newly-started Python +interpreter, and don't just continue from the session described above:: import logging logging.basicConfig(filename='example.log',level=logging.DEBUG) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 6 12:23:22 2013 From: python-checkins at python.org (vinay.sajip) Date: Fri, 6 Dec 2013 12:23:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Added_minor_cl?= =?utf-8?q?arification_in_logging_HOWTO=2E?= Message-ID: <3dbWd675jdz7LjZ@mail.python.org> http://hg.python.org/cpython/rev/9a3a4d32f91e changeset: 87790:9a3a4d32f91e branch: 3.3 parent: 87782:90cf299dcf9b user: Vinay Sajip date: Fri Dec 06 11:22:24 2013 +0000 summary: Added minor clarification in logging HOWTO. files: Doc/howto/logging.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst --- a/Doc/howto/logging.rst +++ b/Doc/howto/logging.rst @@ -122,7 +122,8 @@ ^^^^^^^^^^^^^^^^^ A very common situation is that of recording logging events in a file, so let's -look at that next:: +look at that next. Be sure to try the following in a newly-started Python +interpreter, and don't just continue from the session described above:: import logging logging.basicConfig(filename='example.log',level=logging.DEBUG) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 6 12:23:24 2013 From: python-checkins at python.org (vinay.sajip) Date: Fri, 6 Dec 2013 12:23:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merged_minor_documentation_update_from_3=2E3=2E?= Message-ID: <3dbWd81kndz7Ljp@mail.python.org> http://hg.python.org/cpython/rev/7a668179d691 changeset: 87791:7a668179d691 parent: 87788:2612ea573ff7 parent: 87790:9a3a4d32f91e user: Vinay Sajip date: Fri Dec 06 11:23:08 2013 +0000 summary: Merged minor documentation update from 3.3. files: Doc/howto/logging.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst --- a/Doc/howto/logging.rst +++ b/Doc/howto/logging.rst @@ -122,7 +122,8 @@ ^^^^^^^^^^^^^^^^^ A very common situation is that of recording logging events in a file, so let's -look at that next:: +look at that next. Be sure to try the following in a newly-started Python +interpreter, and don't just continue from the session described above:: import logging logging.basicConfig(filename='example.log',level=logging.DEBUG) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 6 16:14:49 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 6 Dec 2013 16:14:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319908=3A_pathlib_?= =?utf-8?q?now_joins_relative_Windows_paths_correctly_when_a_drive?= Message-ID: <3dbcm93H9dzLr1@mail.python.org> http://hg.python.org/cpython/rev/1ba15c3f45fa changeset: 87792:1ba15c3f45fa user: Serhiy Storchaka date: Fri Dec 06 17:14:12 2013 +0200 summary: Issue #19908: pathlib now joins relative Windows paths correctly when a drive is present. Original patch by Antoine Pitrou. files: Lib/pathlib.py | 14 ++++++--- Lib/test/test_pathlib.py | 42 ++++++++++++++++++++++++++++ Misc/NEWS | 2 + 3 files changed, 53 insertions(+), 5 deletions(-) diff --git a/Lib/pathlib.py b/Lib/pathlib.py --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -89,12 +89,16 @@ (drive, root, parts) tuples. Return a new (drive, root, parts) tuple. """ if root2: - parts = parts2 - root = root2 + if not drv2 and drv: + return drv, root2, [drv + root2] + parts2[1:] + elif drv2: + if drv2 == drv or self.casefold(drv2) == self.casefold(drv): + # Same drive => second path is relative to the first + return drv, root, parts + parts2[1:] else: - parts = parts + parts2 - # XXX raise error if drv and drv2 are different? - return drv2 or drv, root, parts + # Second path is non-anchored (common case) + return drv, root, parts + parts2 + return drv2, root2, parts2 class _WindowsFlavour(_Flavour): diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -975,6 +975,48 @@ self.assertTrue(P('//a/b/c').is_absolute()) self.assertTrue(P('//a/b/c/d').is_absolute()) + def test_join(self): + P = self.cls + p = P('C:/a/b') + pp = p.joinpath('x/y') + self.assertEqual(pp, P('C:/a/b/x/y')) + pp = p.joinpath('/x/y') + self.assertEqual(pp, P('C:/x/y')) + # Joining with a different drive => the first path is ignored, even + # if the second path is relative. + pp = p.joinpath('D:x/y') + self.assertEqual(pp, P('D:x/y')) + pp = p.joinpath('D:/x/y') + self.assertEqual(pp, P('D:/x/y')) + pp = p.joinpath('//host/share/x/y') + self.assertEqual(pp, P('//host/share/x/y')) + # Joining with the same drive => the first path is appended to if + # the second path is relative. + pp = p.joinpath('c:x/y') + self.assertEqual(pp, P('C:/a/b/x/y')) + pp = p.joinpath('c:/x/y') + self.assertEqual(pp, P('C:/x/y')) + + def test_div(self): + # Basically the same as joinpath() + P = self.cls + p = P('C:/a/b') + self.assertEqual(p / 'x/y', P('C:/a/b/x/y')) + self.assertEqual(p / 'x' / 'y', P('C:/a/b/x/y')) + self.assertEqual(p / '/x/y', P('C:/x/y')) + self.assertEqual(p / '/x' / 'y', P('C:/x/y')) + # Joining with a different drive => the first path is ignored, even + # if the second path is relative. + self.assertEqual(p / 'D:x/y', P('D:x/y')) + self.assertEqual(p / 'D:' / 'x/y', P('D:x/y')) + self.assertEqual(p / 'D:/x/y', P('D:/x/y')) + self.assertEqual(p / 'D:' / '/x/y', P('D:/x/y')) + self.assertEqual(p / '//host/share/x/y', P('//host/share/x/y')) + # Joining with the same drive => the first path is appended to if + # the second path is relative. + self.assertEqual(p / 'C:x/y', P('C:/a/b/x/y')) + self.assertEqual(p / 'C:/x/y', P('C:/x/y')) + def test_is_reserved(self): P = self.cls self.assertIs(False, P('').is_reserved()) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,8 @@ Library ------- +- Issue #19908: pathlib now joins relative Windows paths correctly when a drive + is present. Original patch by Antoine Pitrou. - Issue #19296: Silence compiler warning in dbm_open -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 6 16:26:17 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 6 Dec 2013 16:26:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Test_same_drive_in_differe?= =?utf-8?q?nt_cases_=28issue_=2319908=29=2E?= Message-ID: <3dbd1P5v6tz7LjX@mail.python.org> http://hg.python.org/cpython/rev/0c508d87f80b changeset: 87793:0c508d87f80b user: Serhiy Storchaka date: Fri Dec 06 17:25:51 2013 +0200 summary: Test same drive in different cases (issue #19908). files: Lib/test/test_pathlib.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -1014,8 +1014,8 @@ self.assertEqual(p / '//host/share/x/y', P('//host/share/x/y')) # Joining with the same drive => the first path is appended to if # the second path is relative. - self.assertEqual(p / 'C:x/y', P('C:/a/b/x/y')) - self.assertEqual(p / 'C:/x/y', P('C:/x/y')) + self.assertEqual(p / 'c:x/y', P('C:/a/b/x/y')) + self.assertEqual(p / 'c:/x/y', P('C:/x/y')) def test_is_reserved(self): P = self.cls -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 6 18:07:34 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 6 Dec 2013 18:07:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319712=3A_Update_t?= =?utf-8?q?est=2Etest=5Fimportlib=2Eimport=5F_to_test/use_PEP_451?= Message-ID: <3dbgGG5D65z7LjZ@mail.python.org> http://hg.python.org/cpython/rev/543c76769c14 changeset: 87794:543c76769c14 user: Brett Cannon date: Fri Dec 06 12:07:25 2013 -0500 summary: Issue #19712: Update test.test_importlib.import_ to test/use PEP 451 where appropriate. files: Lib/test/test_importlib/import_/test___package__.py | 31 ++++- Lib/test/test_importlib/import_/test_caching.py | 18 ++- Lib/test/test_importlib/import_/test_fromlist.py | 2 +- Lib/test/test_importlib/import_/test_meta_path.py | 50 +++++---- Lib/test/test_importlib/import_/test_packages.py | 14 +- Lib/test/test_importlib/import_/test_path.py | 10 +- Lib/test/test_importlib/import_/test_relative_imports.py | 2 +- Lib/test/test_importlib/util.py | 48 ++++++++- 8 files changed, 114 insertions(+), 61 deletions(-) diff --git a/Lib/test/test_importlib/import_/test___package__.py b/Lib/test/test_importlib/import_/test___package__.py --- a/Lib/test/test_importlib/import_/test___package__.py +++ b/Lib/test/test_importlib/import_/test___package__.py @@ -36,7 +36,7 @@ def test_using___package__(self): # [__package__] - with util.mock_modules('pkg.__init__', 'pkg.fake') as importer: + with self.mock_modules('pkg.__init__', 'pkg.fake') as importer: with util.import_state(meta_path=[importer]): self.__import__('pkg.fake') module = self.__import__('', @@ -49,7 +49,7 @@ globals_ = {'__name__': 'pkg.fake', '__path__': []} if package_as_None: globals_['__package__'] = None - with util.mock_modules('pkg.__init__', 'pkg.fake') as importer: + with self.mock_modules('pkg.__init__', 'pkg.fake') as importer: with util.import_state(meta_path=[importer]): self.__import__('pkg.fake') module = self.__import__('', globals= globals_, @@ -70,11 +70,20 @@ with self.assertRaises(TypeError): self.__import__('', globals, {}, ['relimport'], 1) -Frozen_UsingPackage, Source_UsingPackage = util.test_both( - Using__package__, __import__=import_util.__import__) +class Using__package__PEP302(Using__package__): + mock_modules = util.mock_modules +Frozen_UsingPackagePEP302, Source_UsingPackagePEP302 = util.test_both( + Using__package__PEP302, __import__=import_util.__import__) -class Setting__package__(unittest.TestCase): +class Using__package__PEP302(Using__package__): + mock_modules = util.mock_spec + +Frozen_UsingPackagePEP451, Source_UsingPackagePEP451 = util.test_both( + Using__package__PEP302, __import__=import_util.__import__) + + +class Setting__package__: """Because __package__ is a new feature, it is not always set by a loader. Import will set it as needed to help with the transition to relying on @@ -90,7 +99,7 @@ # [top-level] def test_top_level(self): - with util.mock_modules('top_level') as mock: + with self.mock_modules('top_level') as mock: with util.import_state(meta_path=[mock]): del mock['top_level'].__package__ module = self.__import__('top_level') @@ -98,7 +107,7 @@ # [package] def test_package(self): - with util.mock_modules('pkg.__init__') as mock: + with self.mock_modules('pkg.__init__') as mock: with util.import_state(meta_path=[mock]): del mock['pkg'].__package__ module = self.__import__('pkg') @@ -106,13 +115,19 @@ # [submodule] def test_submodule(self): - with util.mock_modules('pkg.__init__', 'pkg.mod') as mock: + with self.mock_modules('pkg.__init__', 'pkg.mod') as mock: with util.import_state(meta_path=[mock]): del mock['pkg.mod'].__package__ pkg = self.__import__('pkg.mod') module = getattr(pkg, 'mod') self.assertEqual(module.__package__, 'pkg') +class Setting__package__PEP302(Setting__package__, unittest.TestCase): + mock_modules = util.mock_modules + +class Setting__package__PEP451(Setting__package__, unittest.TestCase): + mock_modules = util.mock_spec + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_importlib/import_/test_caching.py b/Lib/test/test_importlib/import_/test_caching.py --- a/Lib/test/test_importlib/import_/test_caching.py +++ b/Lib/test/test_importlib/import_/test_caching.py @@ -39,6 +39,16 @@ self.__import__(name) self.assertEqual(cm.exception.name, name) +Frozen_UseCache, Source_UseCache = util.test_both( + UseCache, __import__=import_util.__import__) + + +class ImportlibUseCache(UseCache, unittest.TestCase): + + # Pertinent only to PEP 302; exec_module() doesn't return a module. + + __import__ = import_util.__import__[1] + def create_mock(self, *names, return_=None): mock = util.mock_modules(*names) original_load = mock.load_module @@ -48,14 +58,6 @@ mock.load_module = MethodType(load_module, mock) return mock -Frozen_UseCache, Source_UseCache = util.test_both( - UseCache, __import__=import_util.__import__) - - -class ImportlibUseCache(UseCache, unittest.TestCase): - - __import__ = import_util.__import__[1] - # __import__ inconsistent between loaders and built-in import when it comes # to when to use the module in sys.modules and when not to. def test_using_cache_after_loader(self): diff --git a/Lib/test/test_importlib/import_/test_fromlist.py b/Lib/test/test_importlib/import_/test_fromlist.py --- a/Lib/test/test_importlib/import_/test_fromlist.py +++ b/Lib/test/test_importlib/import_/test_fromlist.py @@ -17,7 +17,7 @@ def test_return_from_import(self): # [import return] - with util.mock_modules('pkg.__init__', 'pkg.module') as importer: + with util.mock_spec('pkg.__init__', 'pkg.module') as importer: with util.import_state(meta_path=[importer]): module = self.__import__('pkg.module') self.assertEqual(module.__name__, 'pkg') diff --git a/Lib/test/test_importlib/import_/test_meta_path.py b/Lib/test/test_importlib/import_/test_meta_path.py --- a/Lib/test/test_importlib/import_/test_meta_path.py +++ b/Lib/test/test_importlib/import_/test_meta_path.py @@ -18,23 +18,18 @@ def test_first_called(self): # [first called] mod = 'top_level' - first = util.mock_modules(mod) - second = util.mock_modules(mod) - with util.mock_modules(mod) as first, util.mock_modules(mod) as second: - first.modules[mod] = 42 - second.modules[mod] = -13 + with util.mock_spec(mod) as first, util.mock_spec(mod) as second: with util.import_state(meta_path=[first, second]): - self.assertEqual(self.__import__(mod), 42) + self.assertIs(self.__import__(mod), first.modules[mod]) def test_continuing(self): # [continuing] mod_name = 'for_real' - with util.mock_modules('nonexistent') as first, \ - util.mock_modules(mod_name) as second: - first.find_module = lambda self, fullname, path=None: None - second.modules[mod_name] = 42 + with util.mock_spec('nonexistent') as first, \ + util.mock_spec(mod_name) as second: + first.find_spec = lambda self, fullname, path=None, parent=None: None with util.import_state(meta_path=[first, second]): - self.assertEqual(self.__import__(mod_name), 42) + self.assertIs(self.__import__(mod_name), second.modules[mod_name]) def test_empty(self): # Raise an ImportWarning if sys.meta_path is empty. @@ -61,29 +56,27 @@ [no path]. Otherwise, the value for __path__ is passed in for the 'path' argument [path set].""" - def log(self, fxn): + def log_finder(self, importer): + fxn = getattr(importer, self.finder_name) log = [] def wrapper(self, *args, **kwargs): log.append([args, kwargs]) return fxn(*args, **kwargs) return log, wrapper - def test_no_path(self): # [no path] mod_name = 'top_level' assert '.' not in mod_name - with util.mock_modules(mod_name) as importer: - log, wrapped_call = self.log(importer.find_module) - importer.find_module = MethodType(wrapped_call, importer) + with self.mock_modules(mod_name) as importer: + log, wrapped_call = self.log_finder(importer) + setattr(importer, self.finder_name, MethodType(wrapped_call, importer)) with util.import_state(meta_path=[importer]): self.__import__(mod_name) assert len(log) == 1 args = log[0][0] kwargs = log[0][1] # Assuming all arguments are positional. - self.assertEqual(len(args), 2) - self.assertEqual(len(kwargs), 0) self.assertEqual(args[0], mod_name) self.assertIsNone(args[1]) @@ -93,10 +86,10 @@ mod_name = pkg_name + '.module' path = [42] assert '.' in mod_name - with util.mock_modules(pkg_name+'.__init__', mod_name) as importer: + with self.mock_modules(pkg_name+'.__init__', mod_name) as importer: importer.modules[pkg_name].__path__ = path - log, wrapped_call = self.log(importer.find_module) - importer.find_module = MethodType(wrapped_call, importer) + log, wrapped_call = self.log_finder(importer) + setattr(importer, self.finder_name, MethodType(wrapped_call, importer)) with util.import_state(meta_path=[importer]): self.__import__(mod_name) assert len(log) == 2 @@ -107,8 +100,19 @@ self.assertEqual(args[0], mod_name) self.assertIs(args[1], path) -Frozen_CallSignature, Source_CallSignature = util.test_both( - CallSignature, __import__=import_util.__import__) +class CallSignaturePEP302(CallSignature): + mock_modules = util.mock_modules + finder_name = 'find_module' + +Frozen_CallSignaturePEP302, Source_CallSignaturePEP302 = util.test_both( + CallSignaturePEP302, __import__=import_util.__import__) + +class CallSignaturePEP451(CallSignature): + mock_modules = util.mock_spec + finder_name = 'find_spec' + +Frozen_CallSignaturePEP451, Source_CallSignaturePEP451 = util.test_both( + CallSignaturePEP451, __import__=import_util.__import__) if __name__ == '__main__': diff --git a/Lib/test/test_importlib/import_/test_packages.py b/Lib/test/test_importlib/import_/test_packages.py --- a/Lib/test/test_importlib/import_/test_packages.py +++ b/Lib/test/test_importlib/import_/test_packages.py @@ -11,13 +11,13 @@ """Importing a submodule should import the parent modules.""" def test_import_parent(self): - with util.mock_modules('pkg.__init__', 'pkg.module') as mock: + with util.mock_spec('pkg.__init__', 'pkg.module') as mock: with util.import_state(meta_path=[mock]): module = self.__import__('pkg.module') self.assertIn('pkg', sys.modules) def test_bad_parent(self): - with util.mock_modules('pkg.module') as mock: + with util.mock_spec('pkg.module') as mock: with util.import_state(meta_path=[mock]): with self.assertRaises(ImportError) as cm: self.__import__('pkg.module') @@ -27,7 +27,7 @@ def __init__(): import pkg.module 1/0 - mock = util.mock_modules('pkg.__init__', 'pkg.module', + mock = util.mock_spec('pkg.__init__', 'pkg.module', module_code={'pkg': __init__}) with mock: with util.import_state(meta_path=[mock]): @@ -44,7 +44,7 @@ def __init__(): from . import module 1/0 - mock = util.mock_modules('pkg.__init__', 'pkg.module', + mock = util.mock_spec('pkg.__init__', 'pkg.module', module_code={'pkg': __init__}) with mock: with util.import_state(meta_path=[mock]): @@ -63,7 +63,7 @@ def __init__(): from ..subpkg import module 1/0 - mock = util.mock_modules('pkg.__init__', 'pkg.subpkg.__init__', + mock = util.mock_spec('pkg.__init__', 'pkg.subpkg.__init__', 'pkg.subpkg.module', module_code={'pkg.subpkg': __init__}) with mock: @@ -93,9 +93,9 @@ subname = name + '.b' def module_injection(): sys.modules[subname] = 'total bunk' - mock_modules = util.mock_modules('mod', + mock_spec = util.mock_spec('mod', module_code={'mod': module_injection}) - with mock_modules as mock: + with mock_spec as mock: with util.import_state(meta_path=[mock]): try: submodule = self.__import__(subname) diff --git a/Lib/test/test_importlib/import_/test_path.py b/Lib/test/test_importlib/import_/test_path.py --- a/Lib/test/test_importlib/import_/test_path.py +++ b/Lib/test/test_importlib/import_/test_path.py @@ -27,7 +27,7 @@ # Implicitly tests that sys.path_importer_cache is used. module = '' path = '' - importer = util.mock_modules(module) + importer = util.mock_spec(module) with util.import_state(path_importer_cache={path: importer}, path=[path]): loader = self.machinery.PathFinder.find_module(module) @@ -38,7 +38,7 @@ # Implicitly tests that sys.path_importer_cache is used. module = '' path = '' - importer = util.mock_modules(module) + importer = util.mock_spec(module) with util.import_state(path_importer_cache={path: importer}): loader = self.machinery.PathFinder.find_module(module, [path]) self.assertIs(loader, importer) @@ -47,7 +47,7 @@ # An empty list should not count as asking for sys.path. module = 'module' path = '' - importer = util.mock_modules(module) + importer = util.mock_spec(module) with util.import_state(path_importer_cache={path: importer}, path=[path]): self.assertIsNone(self.machinery.PathFinder.find_module('module', [])) @@ -57,7 +57,7 @@ # Test that sys.path_importer_cache is set. module = '' path = '' - importer = util.mock_modules(module) + importer = util.mock_spec(module) hook = import_util.mock_path_hook(path, importer=importer) with util.import_state(path_hooks=[hook]): loader = self.machinery.PathFinder.find_module(module, [path]) @@ -82,7 +82,7 @@ # The empty string should create a finder using the cwd. path = '' module = '' - importer = util.mock_modules(module) + importer = util.mock_spec(module) hook = import_util.mock_path_hook(os.getcwd(), importer=importer) with util.import_state(path=[path], path_hooks=[hook]): loader = self.machinery.PathFinder.find_module(module) diff --git a/Lib/test/test_importlib/import_/test_relative_imports.py b/Lib/test/test_importlib/import_/test_relative_imports.py --- a/Lib/test/test_importlib/import_/test_relative_imports.py +++ b/Lib/test/test_importlib/import_/test_relative_imports.py @@ -64,7 +64,7 @@ uncache_names.append(name) else: uncache_names.append(name[:-len('.__init__')]) - with util.mock_modules(*create) as importer: + with util.mock_spec(*create) as importer: with util.import_state(meta_path=[importer]): for global_ in globals_: with util.uncache(*uncache_names): diff --git a/Lib/test/test_importlib/util.py b/Lib/test/test_importlib/util.py --- a/Lib/test/test_importlib/util.py +++ b/Lib/test/test_importlib/util.py @@ -1,4 +1,5 @@ from contextlib import contextmanager +from importlib import util import os.path from test import support import unittest @@ -101,9 +102,9 @@ setattr(sys, attr, value) -class mock_modules: +class _ImporterMock: - """A mock importer/loader.""" + """Base class to help with creating importer mocks.""" def __init__(self, *names, module_code={}): self.modules = {} @@ -133,6 +134,19 @@ def __getitem__(self, name): return self.modules[name] + def __enter__(self): + self._uncache = uncache(*self.modules.keys()) + self._uncache.__enter__() + return self + + def __exit__(self, *exc_info): + self._uncache.__exit__(None, None, None) + + +class mock_modules(_ImporterMock): + + """Importer mock using PEP 302 APIs.""" + def find_module(self, fullname, path=None): if fullname not in self.modules: return None @@ -152,10 +166,28 @@ raise return self.modules[fullname] - def __enter__(self): - self._uncache = uncache(*self.modules.keys()) - self._uncache.__enter__() - return self +class mock_spec(_ImporterMock): - def __exit__(self, *exc_info): - self._uncache.__exit__(None, None, None) + """Importer mock using PEP 451 APIs.""" + + def find_spec(self, fullname, path=None, parent=None): + try: + module = self.modules[fullname] + except KeyError: + return None + is_package = hasattr(module, '__path__') + spec = util.spec_from_file_location( + fullname, module.__file__, loader=self, + submodule_search_locations=getattr(module, '__path__', None)) + return spec + + def create_module(self, spec): + if spec.name not in self.modules: + raise ImportError + return self.modules[spec.name] + + def exec_module(self, module): + try: + self.module_code[module.__spec__.name]() + except KeyError: + pass -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 6 20:25:10 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 6 Dec 2013 20:25:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319712=3A_Update_t?= =?utf-8?q?est=2Etest=5Fimportlib=2Esource_for_PEP_451?= Message-ID: <3dbkK20j97z7Ljb@mail.python.org> http://hg.python.org/cpython/rev/07ef52e751f3 changeset: 87795:07ef52e751f3 user: Brett Cannon date: Fri Dec 06 14:25:01 2013 -0500 summary: Issue #19712: Update test.test_importlib.source for PEP 451 files: Lib/test/test_importlib/source/test_case_sensitivity.py | 31 +++- Lib/test/test_importlib/source/test_file_loader.py | 71 ++++++++- Lib/test/test_importlib/source/test_finder.py | 21 ++- Lib/test/test_importlib/source/test_source_encoding.py | 41 +++++- 4 files changed, 140 insertions(+), 24 deletions(-) diff --git a/Lib/test/test_importlib/source/test_case_sensitivity.py b/Lib/test/test_importlib/source/test_case_sensitivity.py --- a/Lib/test/test_importlib/source/test_case_sensitivity.py +++ b/Lib/test/test_importlib/source/test_case_sensitivity.py @@ -21,13 +21,12 @@ name = 'MoDuLe' assert name != name.lower() - def find(self, path): - finder = self.machinery.FileFinder(path, + def finder(self, path): + return self.machinery.FileFinder(path, (self.machinery.SourceFileLoader, self.machinery.SOURCE_SUFFIXES), (self.machinery.SourcelessFileLoader, self.machinery.BYTECODE_SUFFIXES)) - return finder.find_module(self.name) def sensitivity_test(self): """Look for a module with matching and non-matching sensitivity.""" @@ -37,7 +36,9 @@ with context as mapping: sensitive_path = os.path.join(mapping['.root'], 'sensitive') insensitive_path = os.path.join(mapping['.root'], 'insensitive') - return self.find(sensitive_path), self.find(insensitive_path) + sensitive_finder = self.finder(sensitive_path) + insensitive_finder = self.finder(insensitive_path) + return self.find(sensitive_finder), self.find(insensitive_finder) def test_sensitive(self): with test_support.EnvironmentVarGuard() as env: @@ -46,7 +47,7 @@ self.skipTest('os.environ changes not reflected in ' '_os.environ') sensitive, insensitive = self.sensitivity_test() - self.assertTrue(hasattr(sensitive, 'load_module')) + self.assertIsNotNone(sensitive) self.assertIn(self.name, sensitive.get_filename(self.name)) self.assertIsNone(insensitive) @@ -57,13 +58,25 @@ self.skipTest('os.environ changes not reflected in ' '_os.environ') sensitive, insensitive = self.sensitivity_test() - self.assertTrue(hasattr(sensitive, 'load_module')) + self.assertIsNotNone(sensitive) self.assertIn(self.name, sensitive.get_filename(self.name)) - self.assertTrue(hasattr(insensitive, 'load_module')) + self.assertIsNotNone(insensitive) self.assertIn(self.name, insensitive.get_filename(self.name)) -Frozen_CaseSensitivityTest, Source_CaseSensitivityTest = util.test_both( - CaseSensitivityTest, importlib=importlib, machinery=machinery) +class CaseSensitivityTestPEP302(CaseSensitivityTest): + def find(self, finder): + return finder.find_module(self.name) + +Frozen_CaseSensitivityTestPEP302, Source_CaseSensitivityTestPEP302 = util.test_both( + CaseSensitivityTestPEP302, importlib=importlib, machinery=machinery) + +class CaseSensitivityTestPEP451(CaseSensitivityTest): + def find(self, finder): + found = finder.find_spec(self.name) + return found.loader if found is not None else found + +Frozen_CaseSensitivityTestPEP451, Source_CaseSensitivityTestPEP451 = util.test_both( + CaseSensitivityTestPEP451, importlib=importlib, machinery=machinery) if __name__ == '__main__': diff --git a/Lib/test/test_importlib/source/test_file_loader.py b/Lib/test/test_importlib/source/test_file_loader.py --- a/Lib/test/test_importlib/source/test_file_loader.py +++ b/Lib/test/test_importlib/source/test_file_loader.py @@ -121,6 +121,10 @@ file.write('+++ bad syntax +++') loader = self.machinery.SourceFileLoader('_temp', mapping['_temp']) with self.assertRaises(SyntaxError): + loader.exec_module(orig_module) + for attr in attributes: + self.assertEqual(getattr(orig_module, attr), value) + with self.assertRaises(SyntaxError): loader.load_module(name) for attr in attributes: self.assertEqual(getattr(orig_module, attr), value) @@ -171,15 +175,27 @@ raise self.skipTest("cannot set modification time to large integer ({})".format(e)) loader = self.machinery.SourceFileLoader('_temp', mapping['_temp']) - mod = loader.load_module('_temp') + # PEP 451 + module = types.ModuleType('_temp') + module.__spec__ = self.util.spec_from_loader('_temp', loader) + loader.exec_module(module) + self.assertEqual(module.x, 5) + self.assertTrue(os.path.exists(compiled)) + os.unlink(compiled) + # PEP 302 + mod = loader.load_module('_temp') # XXX # Sanity checks. self.assertEqual(mod.__cached__, compiled) self.assertEqual(mod.x, 5) # The pyc file was created. - os.stat(compiled) + self.assertTrue(os.path.exists(compiled)) def test_unloadable(self): loader = self.machinery.SourceFileLoader('good name', {}) + module = types.ModuleType('bad name') + module.__spec__ = self.machinery.ModuleSpec('bad name', loader) + with self.assertRaises(ImportError): + loader.exec_module(module) with self.assertRaises(ImportError): loader.load_module('bad name') @@ -291,8 +307,23 @@ lambda bc: b'\x00\x00\x00\x00' + bc[4:]) test('_temp', mapping, bc_path) +class BadBytecodeTestPEP451(BadBytecodeTest): -class SourceLoaderBadBytecodeTest(BadBytecodeTest): + def import_(self, file, module_name): + loader = self.loader(module_name, file) + module = types.ModuleType(module_name) + module.__spec__ = self.util.spec_from_loader(module_name, loader) + loader.exec_module(module) + +class BadBytecodeTestPEP302(BadBytecodeTest): + + def import_(self, file, module_name): + loader = self.loader(module_name, file) + module = loader.load_module(module_name) + self.assertIn(module_name, sys.modules) + + +class SourceLoaderBadBytecodeTest: @classmethod def setUpClass(cls): @@ -418,12 +449,24 @@ # Make writable for eventual clean-up. os.chmod(bytecode_path, stat.S_IWUSR) -Frozen_SourceBadBytecode, Source_SourceBadBytecode = util.test_both( - SourceLoaderBadBytecodeTest, importlib=importlib, machinery=machinery, +class SourceLoaderBadBytecodeTestPEP451( + SourceLoaderBadBytecodeTest, BadBytecodeTestPEP451): + pass + +Frozen_SourceBadBytecodePEP451, Source_SourceBadBytecodePEP451 = util.test_both( + SourceLoaderBadBytecodeTestPEP451, importlib=importlib, machinery=machinery, abc=importlib_abc, util=importlib_util) +class SourceLoaderBadBytecodeTestPEP302( + SourceLoaderBadBytecodeTest, BadBytecodeTestPEP302): + pass -class SourcelessLoaderBadBytecodeTest(BadBytecodeTest): +Frozen_SourceBadBytecodePEP302, Source_SourceBadBytecodePEP302 = util.test_both( + SourceLoaderBadBytecodeTestPEP302, importlib=importlib, machinery=machinery, + abc=importlib_abc, util=importlib_util) + + +class SourcelessLoaderBadBytecodeTest: @classmethod def setUpClass(cls): @@ -482,8 +525,20 @@ def test_non_code_marshal(self): self._test_non_code_marshal(del_source=True) -Frozen_SourcelessBadBytecode, Source_SourcelessBadBytecode = util.test_both( - SourcelessLoaderBadBytecodeTest, importlib=importlib, +class SourcelessLoaderBadBytecodeTestPEP451(SourcelessLoaderBadBytecodeTest, + BadBytecodeTestPEP451): + pass + +Frozen_SourcelessBadBytecodePEP451, Source_SourcelessBadBytecodePEP451 = util.test_both( + SourcelessLoaderBadBytecodeTestPEP451, importlib=importlib, + machinery=machinery, abc=importlib_abc, util=importlib_util) + +class SourcelessLoaderBadBytecodeTestPEP302(SourcelessLoaderBadBytecodeTest, + BadBytecodeTestPEP302): + pass + +Frozen_SourcelessBadBytecodePEP302, Source_SourcelessBadBytecodePEP302 = util.test_both( + SourcelessLoaderBadBytecodeTestPEP302, importlib=importlib, machinery=machinery, abc=importlib_abc, util=importlib_util) diff --git a/Lib/test/test_importlib/source/test_finder.py b/Lib/test/test_importlib/source/test_finder.py --- a/Lib/test/test_importlib/source/test_finder.py +++ b/Lib/test/test_importlib/source/test_finder.py @@ -46,9 +46,6 @@ self.machinery.BYTECODE_SUFFIXES)] return self.machinery.FileFinder(root, *loader_details) - def import_(self, root, module): - return self.get_finder(root).find_module(module) - def run_test(self, test, create=None, *, compile_=None, unlink=None): """Test the finding of 'test' with the creation of modules listed in 'create'. @@ -182,7 +179,23 @@ finder = self.get_finder(file_obj.name) self.assertEqual((None, []), finder.find_loader('doesnotexist')) -Frozen_FinderTests, Source_FinderTests = util.test_both(FinderTests, machinery=machinery) +class FinderTestsPEP451(FinderTests): + + def import_(self, root, module): + found = self.get_finder(root).find_spec(module) + return found.loader if found is not None else found + +Frozen_FinderTestsPEP451, Source_FinderTestsPEP451 = util.test_both( + FinderTestsPEP451, machinery=machinery) + + +class FinderTestsPEP302(FinderTests): + + def import_(self, root, module): + return self.get_finder(root).find_module(module) + +Frozen_FinderTestsPEP302, Source_FinderTestsPEP302 = util.test_both( + FinderTestsPEP302, machinery=machinery) diff --git a/Lib/test/test_importlib/source/test_source_encoding.py b/Lib/test/test_importlib/source/test_source_encoding.py --- a/Lib/test/test_importlib/source/test_source_encoding.py +++ b/Lib/test/test_importlib/source/test_source_encoding.py @@ -4,8 +4,10 @@ machinery = util.import_importlib('importlib.machinery') import codecs +import importlib.util import re import sys +import types # Because sys.path gets essentially blanked, need to have unicodedata already # imported for the parser to use. import unicodedata @@ -39,7 +41,7 @@ file.write(source) loader = self.machinery.SourceFileLoader(self.module_name, mapping[self.module_name]) - return loader.load_module(self.module_name) + return self.load(loader) def create_source(self, encoding): encoding_line = "# coding={0}".format(encoding) @@ -86,7 +88,24 @@ with self.assertRaises(SyntaxError): self.run_test(source) -Frozen_EncodingTest, Source_EncodingTest = util.test_both(EncodingTest, machinery=machinery) +class EncodingTestPEP451(EncodingTest): + + def load(self, loader): + module = types.ModuleType(self.module_name) + module.__spec__ = importlib.util.spec_from_loader(self.module_name, loader) + loader.exec_module(module) + return module + +Frozen_EncodingTestPEP451, Source_EncodingTestPEP451 = util.test_both( + EncodingTestPEP451, machinery=machinery) + +class EncodingTestPEP302(EncodingTest): + + def load(self, loader): + return loader.load_module(self.module_name) + +Frozen_EncodingTestPEP302, Source_EncodingTestPEP302 = util.test_both( + EncodingTestPEP302, machinery=machinery) class LineEndingTest: @@ -117,8 +136,24 @@ def test_lf(self): self.run_test(b'\n') -Frozen_LineEndings, Source_LineEndings = util.test_both(LineEndingTest, machinery=machinery) +class LineEndingTestPEP451(LineEndingTest): + def load(self, loader): + module = types.ModuleType(self.module_name) + module.__spec__ = importlib.util.spec_from_loader(self.module_name, loader) + loader.exec_module(module) + return module + +Frozen_LineEndingTestPEP451, Source_LineEndingTestPEP451 = util.test_both( + LineEndingTestPEP451, machinery=machinery) + +class LineEndingTestPEP302(LineEndingTest): + + def load(self, loader): + return loader.load_module(self.module_name) + +Frozen_LineEndingTestPEP302, Source_LineEndingTestPEP302 = util.test_both( + LineEndingTestPEP302, machinery=machinery) if __name__ == '__main__': -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 6 21:57:48 2013 From: python-checkins at python.org (guido.van.rossum) Date: Fri, 6 Dec 2013 21:57:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_Add_Task=2Ecurr?= =?utf-8?q?ent=5Ftask=28=29_class_method=2E?= Message-ID: <3dbmMw5Zfyz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/07cfe9f5561b changeset: 87796:07cfe9f5561b user: Guido van Rossum date: Fri Dec 06 12:57:40 2013 -0800 summary: asyncio: Add Task.current_task() class method. files: Lib/asyncio/tasks.py | 20 +++++++++ Lib/asyncio/test_utils.py | 2 +- Lib/test/test_asyncio/test_tasks.py | 36 +++++++++++++++++ 3 files changed, 57 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -122,6 +122,22 @@ # Weak set containing all tasks alive. _all_tasks = weakref.WeakSet() + # Dictionary containing tasks that are currently active in + # all running event loops. {EventLoop: Task} + _current_tasks = {} + + @classmethod + def current_task(cls, loop=None): + """Return the currently running task in an event loop or None. + + By default the current task for the current event loop is returned. + + None is returned when called not in the context of a Task. + """ + if loop is None: + loop = events.get_event_loop() + return cls._current_tasks.get(loop) + @classmethod def all_tasks(cls, loop=None): """Return a set of all tasks for an event loop. @@ -252,6 +268,8 @@ self._must_cancel = False coro = self._coro self._fut_waiter = None + + self.__class__._current_tasks[self._loop] = self # Call either coro.throw(exc) or coro.send(value). try: if exc is not None: @@ -302,6 +320,8 @@ self._step, None, RuntimeError( 'Task got bad yield: {!r}'.format(result))) + finally: + self.__class__._current_tasks.pop(self._loop) self = None def _wakeup(self, future): diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py --- a/Lib/asyncio/test_utils.py +++ b/Lib/asyncio/test_utils.py @@ -88,7 +88,7 @@ class SSLWSGIServer(SilentWSGIServer): def finish_request(self, request, client_address): # The relative location of our test directory (which - # contains the sample key and certificate files) differs + # contains the ssl key and certificate files) differs # between the stdlib and stand-alone Tulip/asyncio. # Prefer our own if we can find it. here = os.path.join(os.path.dirname(__file__), '..', 'tests') diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -1113,6 +1113,42 @@ self.assertEqual(res, 'test') self.assertIsNone(t2.result()) + def test_current_task(self): + self.assertIsNone(tasks.Task.current_task(loop=self.loop)) + @tasks.coroutine + def coro(loop): + self.assertTrue(tasks.Task.current_task(loop=loop) is task) + + task = tasks.Task(coro(self.loop), loop=self.loop) + self.loop.run_until_complete(task) + self.assertIsNone(tasks.Task.current_task(loop=self.loop)) + + def test_current_task_with_interleaving_tasks(self): + self.assertIsNone(tasks.Task.current_task(loop=self.loop)) + + fut1 = futures.Future(loop=self.loop) + fut2 = futures.Future(loop=self.loop) + + @tasks.coroutine + def coro1(loop): + self.assertTrue(tasks.Task.current_task(loop=loop) is task1) + yield from fut1 + self.assertTrue(tasks.Task.current_task(loop=loop) is task1) + fut2.set_result(True) + + @tasks.coroutine + def coro2(loop): + self.assertTrue(tasks.Task.current_task(loop=loop) is task2) + fut1.set_result(True) + yield from fut2 + self.assertTrue(tasks.Task.current_task(loop=loop) is task2) + + task1 = tasks.Task(coro1(self.loop), loop=self.loop) + task2 = tasks.Task(coro2(self.loop), loop=self.loop) + + self.loop.run_until_complete(tasks.wait((task1, task2), loop=self.loop)) + self.assertIsNone(tasks.Task.current_task(loop=self.loop)) + # Some thorough tests for cancellation propagation through # coroutines, tasks and wait(). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 6 22:24:28 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 6 Dec 2013 22:24:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE2Mzcz?= =?utf-8?q?=3A_Prevent_infinite_recursion_for_ABC_Set_class_comparisons=2E?= Message-ID: <3dbmyh2rlCz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/9bf55766d935 changeset: 87797:9bf55766d935 branch: 2.7 parent: 87789:80142c15a920 user: Serhiy Storchaka date: Fri Dec 06 23:23:15 2013 +0200 summary: Issue #16373: Prevent infinite recursion for ABC Set class comparisons. files: Lib/_abcoll.py | 4 +- Lib/test/test_collections.py | 29 ++++++++++++++++++++++++ Misc/NEWS | 2 + 3 files changed, 33 insertions(+), 2 deletions(-) diff --git a/Lib/_abcoll.py b/Lib/_abcoll.py --- a/Lib/_abcoll.py +++ b/Lib/_abcoll.py @@ -165,12 +165,12 @@ def __gt__(self, other): if not isinstance(other, Set): return NotImplemented - return other < self + return other.__lt__(self) def __ge__(self, other): if not isinstance(other, Set): return NotImplemented - return other <= self + return other.__le__(self) def __eq__(self, other): if not isinstance(other, Set): diff --git a/Lib/test/test_collections.py b/Lib/test/test_collections.py --- a/Lib/test/test_collections.py +++ b/Lib/test/test_collections.py @@ -594,6 +594,35 @@ s |= s self.assertEqual(s, full) + def test_issue16373(self): + # Recursion error comparing comparable and noncomparable + # Set instances + class MyComparableSet(Set): + def __contains__(self, x): + return False + def __len__(self): + return 0 + def __iter__(self): + return iter([]) + class MyNonComparableSet(Set): + def __contains__(self, x): + return False + def __len__(self): + return 0 + def __iter__(self): + return iter([]) + def __le__(self, x): + return NotImplemented + def __lt__(self, x): + return NotImplemented + + cs = MyComparableSet() + ncs = MyNonComparableSet() + self.assertFalse(ncs < cs) + self.assertFalse(ncs <= cs) + self.assertFalse(cs > ncs) + self.assertFalse(cs >= ncs) + def test_Mapping(self): for sample in [dict]: self.assertIsInstance(sample(), Mapping) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,8 @@ Library ------- +- Issue #16373: Prevent infinite recursion for ABC Set class comparisons. + - Issue #19138: doctest's IGNORE_EXCEPTION_DETAIL now allows a match when no exception detail exists (no colon following the exception's name, or a colon does follow but no text follows the colon). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 6 23:44:44 2013 From: python-checkins at python.org (christian.heimes) Date: Fri, 6 Dec 2013 23:44:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Load_expat=5Fconfig=2Eh_an?= =?utf-8?q?d_therefore_pyconfig=2Eh_before_C_stdlib_headers_are_loaded=2E?= Message-ID: <3dbplJ5kBnz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/b04557ae099a changeset: 87798:b04557ae099a parent: 87796:07cfe9f5561b user: Christian Heimes date: Fri Dec 06 23:43:50 2013 +0100 summary: Load expat_config.h and therefore pyconfig.h before C stdlib headers are loaded. This silences the pre-processor warning '_POSIX_C_SOURCE redefined'. files: Modules/expat/xmlparse.c | 12 ++++++------ Modules/expat/xmlrole.c | 4 ++-- Modules/expat/xmltok.c | 4 ++-- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/Modules/expat/xmlparse.c b/Modules/expat/xmlparse.c --- a/Modules/expat/xmlparse.c +++ b/Modules/expat/xmlparse.c @@ -2,12 +2,6 @@ See the file COPYING for copying permission. */ -#include -#include /* memset(), memcpy() */ -#include -#include /* UINT_MAX */ -#include /* time() */ - #define XML_BUILDING_EXPAT 1 #ifdef COMPILED_FROM_DSP @@ -22,6 +16,12 @@ #include #endif /* ndef COMPILED_FROM_DSP */ +#include +#include /* memset(), memcpy() */ +#include +#include /* UINT_MAX */ +#include /* time() */ + #include "ascii.h" #include "expat.h" diff --git a/Modules/expat/xmlrole.c b/Modules/expat/xmlrole.c --- a/Modules/expat/xmlrole.c +++ b/Modules/expat/xmlrole.c @@ -2,8 +2,6 @@ See the file COPYING for copying permission. */ -#include - #ifdef COMPILED_FROM_DSP #include "winconfig.h" #elif defined(MACOS_CLASSIC) @@ -18,6 +16,8 @@ #endif #endif /* ndef COMPILED_FROM_DSP */ +#include + #include "expat_external.h" #include "internal.h" #include "xmlrole.h" diff --git a/Modules/expat/xmltok.c b/Modules/expat/xmltok.c --- a/Modules/expat/xmltok.c +++ b/Modules/expat/xmltok.c @@ -2,8 +2,6 @@ See the file COPYING for copying permission. */ -#include - #ifdef COMPILED_FROM_DSP #include "winconfig.h" #elif defined(MACOS_CLASSIC) @@ -18,6 +16,8 @@ #endif #endif /* ndef COMPILED_FROM_DSP */ +#include + #include "expat_external.h" #include "internal.h" #include "xmltok.h" -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 7 00:09:52 2013 From: python-checkins at python.org (christian.heimes) Date: Sat, 7 Dec 2013 00:09:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Skip_new_SSL_tests_with_Io?= =?utf-8?q?cpEventLoop?= Message-ID: <3dbqJJ4smfz7Ljd@mail.python.org> http://hg.python.org/cpython/rev/2a9683c45f6e changeset: 87799:2a9683c45f6e user: Christian Heimes date: Sat Dec 07 00:09:45 2013 +0100 summary: Skip new SSL tests with IocpEventLoop files: Lib/test/test_asyncio/test_events.py | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -1398,6 +1398,15 @@ def test_create_server_ssl(self): raise unittest.SkipTest("IocpEventLoop imcompatible with SSL") + def test_create_server_ssl_verify_failed(self): + raise unittest.SkipTest("IocpEventLoop imcompatible with SSL") + + def test_create_server_ssl_match_failed(self): + raise unittest.SkipTest("IocpEventLoop imcompatible with SSL") + + def test_create_server_ssl_verified(self): + raise unittest.SkipTest("IocpEventLoop imcompatible with SSL") + def test_reader_callback(self): raise unittest.SkipTest("IocpEventLoop does not have add_reader()") -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 7 00:15:04 2013 From: python-checkins at python.org (christian.heimes) Date: Sat, 7 Dec 2013 00:15:04 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_copy_=27n_paste_typo_=28th?= =?utf-8?q?x_Antoine=29?= Message-ID: <3dbqQJ17wWz7Ljd@mail.python.org> http://hg.python.org/cpython/rev/7ce3cd4dd561 changeset: 87800:7ce3cd4dd561 user: Christian Heimes date: Sat Dec 07 00:14:55 2013 +0100 summary: copy 'n paste typo (thx Antoine) files: Lib/test/test_asyncio/test_events.py | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -1393,19 +1393,19 @@ return windows_events.ProactorEventLoop() def test_create_ssl_connection(self): - raise unittest.SkipTest("IocpEventLoop imcompatible with SSL") + raise unittest.SkipTest("IocpEventLoop incompatible with SSL") def test_create_server_ssl(self): - raise unittest.SkipTest("IocpEventLoop imcompatible with SSL") + raise unittest.SkipTest("IocpEventLoop incompatible with SSL") def test_create_server_ssl_verify_failed(self): - raise unittest.SkipTest("IocpEventLoop imcompatible with SSL") + raise unittest.SkipTest("IocpEventLoop incompatible with SSL") def test_create_server_ssl_match_failed(self): - raise unittest.SkipTest("IocpEventLoop imcompatible with SSL") + raise unittest.SkipTest("IocpEventLoop incompatible with SSL") def test_create_server_ssl_verified(self): - raise unittest.SkipTest("IocpEventLoop imcompatible with SSL") + raise unittest.SkipTest("IocpEventLoop incompatible with SSL") def test_reader_callback(self): raise unittest.SkipTest("IocpEventLoop does not have add_reader()") -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 7 00:57:52 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 7 Dec 2013 00:57:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5OTAw?= =?utf-8?q?=3A_improve_generalities_at_the_start_of_the_pickle_module_doc?= Message-ID: <3dbrMh5RgCz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/609325d187bf changeset: 87801:609325d187bf branch: 3.3 parent: 87790:9a3a4d32f91e user: Antoine Pitrou date: Sat Dec 07 00:56:59 2013 +0100 summary: Issue #19900: improve generalities at the start of the pickle module doc files: Doc/library/pickle.rst | 73 +++++++++++++++++++++-------- 1 files changed, 53 insertions(+), 20 deletions(-) diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -15,13 +15,14 @@ .. sectionauthor:: Barry Warsaw -The :mod:`pickle` module implements a fundamental, but powerful algorithm for -serializing and de-serializing a Python object structure. "Pickling" is the -process whereby a Python object hierarchy is converted into a byte stream, and -"unpickling" is the inverse operation, whereby a byte stream is converted back -into an object hierarchy. Pickling (and unpickling) is alternatively known as -"serialization", "marshalling," [#]_ or "flattening", however, to avoid -confusion, the terms used here are "pickling" and "unpickling".. +The :mod:`pickle` module implements binary protocols for serializing and +de-serializing a Python object structure. *"Pickling"* is the process +whereby a Python object hierarchy is converted into a byte stream, and +*"unpickling"* is the inverse operation, whereby a byte stream +(from a :term:`binary file` or :term:`bytes-like object`) is converted +back into an object hierarchy. Pickling (and unpickling) is alternatively +known as "serialization", "marshalling," [#]_ or "flattening"; however, to +avoid confusion, the terms used here are "pickling" and "unpickling". .. warning:: @@ -33,9 +34,8 @@ Relationship to other Python modules ------------------------------------ -The :mod:`pickle` module has an transparent optimizer (:mod:`_pickle`) written -in C. It is used whenever available. Otherwise the pure Python implementation is -used. +Comparison with ``marshal`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^ Python has a more primitive serialization module called :mod:`marshal`, but in general :mod:`pickle` should always be the preferred way to serialize Python @@ -69,17 +69,30 @@ The :mod:`pickle` serialization format is guaranteed to be backwards compatible across Python releases. -Note that serialization is a more primitive notion than persistence; although -:mod:`pickle` reads and writes file objects, it does not handle the issue of -naming persistent objects, nor the (even more complicated) issue of concurrent -access to persistent objects. The :mod:`pickle` module can transform a complex -object into a byte stream and it can transform the byte stream into an object -with the same internal structure. Perhaps the most obvious thing to do with -these byte streams is to write them onto a file, but it is also conceivable to -send them across a network or store them in a database. The module -:mod:`shelve` provides a simple interface to pickle and unpickle objects on -DBM-style database files. +Comparison with ``json`` +^^^^^^^^^^^^^^^^^^^^^^^^ +There are fundamental differences between the pickle protocols and +`JSON (JavaScript Object Notation) `_: + +* JSON is a text serialization format (it outputs unicode text, although + most of the time it is then encoded to ``utf-8``), while pickle is + a binary serialization format; + +* JSON is human-readable, while pickle is not; + +* JSON is interoperable and widely used outside of the Python ecosystem, + while pickle is Python-specific; + +* JSON, by default, can only represent a subset of the Python built-in + types, and no custom classes; pickle can represent an extremely large + number of Python types (many of them automatically, by clever usage + of Python's introspection facilities; complex cases can be tackled by + implementing :ref:`specific object APIs `). + +.. seealso:: + The :mod:`json` module: a standard library module allowing JSON + serialization and deserialization. Data stream format ------------------ @@ -117,6 +130,18 @@ the default as well as the current recommended protocol; use it whenever possible. +.. note:: + Serialization is a more primitive notion than persistence; although + :mod:`pickle` reads and writes file objects, it does not handle the issue of + naming persistent objects, nor the (even more complicated) issue of concurrent + access to persistent objects. The :mod:`pickle` module can transform a complex + object into a byte stream and it can transform the byte stream into an object + with the same internal structure. Perhaps the most obvious thing to do with + these byte streams is to write them onto a file, but it is also conceivable to + send them across a network or store them in a database. The :mod:`shelve` + module provides a simple interface to pickle and unpickle objects on + DBM-style database files. + Module Interface ---------------- @@ -793,6 +818,14 @@ third-party solutions. +Performance +----------- + +Recent versions of the pickle protocol (from protocol 2 and upwards) feature +efficient binary encodings for several common features and built-in types. +Also, the :mod:`pickle` module has a transparent optimizer written in C. + + .. _pickle-example: Examples -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 7 00:57:54 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 7 Dec 2013 00:57:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319900=3A_improve_generalities_at_the_start_of_t?= =?utf-8?q?he_pickle_module_doc?= Message-ID: <3dbrMk1CqYz7LkF@mail.python.org> http://hg.python.org/cpython/rev/595b8f82569c changeset: 87802:595b8f82569c parent: 87800:7ce3cd4dd561 parent: 87801:609325d187bf user: Antoine Pitrou date: Sat Dec 07 00:57:44 2013 +0100 summary: Issue #19900: improve generalities at the start of the pickle module doc files: Doc/library/pickle.rst | 73 +++++++++++++++++++++-------- 1 files changed, 53 insertions(+), 20 deletions(-) diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -15,13 +15,14 @@ .. sectionauthor:: Barry Warsaw -The :mod:`pickle` module implements a fundamental, but powerful algorithm for -serializing and de-serializing a Python object structure. "Pickling" is the -process whereby a Python object hierarchy is converted into a byte stream, and -"unpickling" is the inverse operation, whereby a byte stream is converted back -into an object hierarchy. Pickling (and unpickling) is alternatively known as -"serialization", "marshalling," [#]_ or "flattening", however, to avoid -confusion, the terms used here are "pickling" and "unpickling".. +The :mod:`pickle` module implements binary protocols for serializing and +de-serializing a Python object structure. *"Pickling"* is the process +whereby a Python object hierarchy is converted into a byte stream, and +*"unpickling"* is the inverse operation, whereby a byte stream +(from a :term:`binary file` or :term:`bytes-like object`) is converted +back into an object hierarchy. Pickling (and unpickling) is alternatively +known as "serialization", "marshalling," [#]_ or "flattening"; however, to +avoid confusion, the terms used here are "pickling" and "unpickling". .. warning:: @@ -33,9 +34,8 @@ Relationship to other Python modules ------------------------------------ -The :mod:`pickle` module has an transparent optimizer (:mod:`_pickle`) written -in C. It is used whenever available. Otherwise the pure Python implementation is -used. +Comparison with ``marshal`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^ Python has a more primitive serialization module called :mod:`marshal`, but in general :mod:`pickle` should always be the preferred way to serialize Python @@ -69,17 +69,30 @@ The :mod:`pickle` serialization format is guaranteed to be backwards compatible across Python releases. -Note that serialization is a more primitive notion than persistence; although -:mod:`pickle` reads and writes file objects, it does not handle the issue of -naming persistent objects, nor the (even more complicated) issue of concurrent -access to persistent objects. The :mod:`pickle` module can transform a complex -object into a byte stream and it can transform the byte stream into an object -with the same internal structure. Perhaps the most obvious thing to do with -these byte streams is to write them onto a file, but it is also conceivable to -send them across a network or store them in a database. The module -:mod:`shelve` provides a simple interface to pickle and unpickle objects on -DBM-style database files. +Comparison with ``json`` +^^^^^^^^^^^^^^^^^^^^^^^^ +There are fundamental differences between the pickle protocols and +`JSON (JavaScript Object Notation) `_: + +* JSON is a text serialization format (it outputs unicode text, although + most of the time it is then encoded to ``utf-8``), while pickle is + a binary serialization format; + +* JSON is human-readable, while pickle is not; + +* JSON is interoperable and widely used outside of the Python ecosystem, + while pickle is Python-specific; + +* JSON, by default, can only represent a subset of the Python built-in + types, and no custom classes; pickle can represent an extremely large + number of Python types (many of them automatically, by clever usage + of Python's introspection facilities; complex cases can be tackled by + implementing :ref:`specific object APIs `). + +.. seealso:: + The :mod:`json` module: a standard library module allowing JSON + serialization and deserialization. Data stream format ------------------ @@ -117,6 +130,18 @@ the default as well as the current recommended protocol; use it whenever possible. +.. note:: + Serialization is a more primitive notion than persistence; although + :mod:`pickle` reads and writes file objects, it does not handle the issue of + naming persistent objects, nor the (even more complicated) issue of concurrent + access to persistent objects. The :mod:`pickle` module can transform a complex + object into a byte stream and it can transform the byte stream into an object + with the same internal structure. Perhaps the most obvious thing to do with + these byte streams is to write them onto a file, but it is also conceivable to + send them across a network or store them in a database. The :mod:`shelve` + module provides a simple interface to pickle and unpickle objects on + DBM-style database files. + Module Interface ---------------- @@ -811,6 +836,14 @@ third-party solutions. +Performance +----------- + +Recent versions of the pickle protocol (from protocol 2 and upwards) feature +efficient binary encodings for several common features and built-in types. +Also, the :mod:`pickle` module has a transparent optimizer written in C. + + .. _pickle-example: Examples -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 7 01:06:03 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 7 Dec 2013 01:06:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Mention_pickle_protocol_4?= =?utf-8?q?=2C_and_some_tweaks=2E?= Message-ID: <3dbrY74575z7Lk0@mail.python.org> http://hg.python.org/cpython/rev/8001f0dfe84f changeset: 87803:8001f0dfe84f user: Antoine Pitrou date: Sat Dec 07 01:05:57 2013 +0100 summary: Mention pickle protocol 4, and some tweaks. files: Doc/library/pickle.rst | 25 ++++++++++++++++++------- 1 files changed, 18 insertions(+), 7 deletions(-) diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -94,6 +94,9 @@ The :mod:`json` module: a standard library module allowing JSON serialization and deserialization. + +.. _pickle-protocols: + Data stream format ------------------ @@ -125,10 +128,15 @@ efficient pickling of :term:`new-style class`\es. Refer to :pep:`307` for information about improvements brought by protocol 2. -* Protocol version 3 was added in Python 3. It has explicit support for +* Protocol version 3 was added in Python 3.0. It has explicit support for :class:`bytes` objects and cannot be unpickled by Python 2.x. This is - the default as well as the current recommended protocol; use it whenever - possible. + the default protocol, and the recommended protocol when compatibility with + other Python 3 versions is required. + +* Protocol version 4 was added in Python 3.4. It adds support for very large + objects, pickling more kinds of objects, and some data format + optimizations. Refer to :pep:`3154` for information about improvements + brought by protocol 4. .. note:: Serialization is a more primitive notion than persistence; although @@ -156,13 +164,16 @@ .. data:: HIGHEST_PROTOCOL - The highest protocol version available. This value can be passed as a - *protocol* value. + An integer, the highest :ref:`protocol version ` + available. This value can be passed as a *protocol* value to functions + :func:`dump` and :func:`dumps` as well as the :class:`Pickler` + constructor. .. data:: DEFAULT_PROTOCOL - The default protocol used for pickling. May be less than HIGHEST_PROTOCOL. - Currently the default protocol is 3, a new protocol designed for Python 3.0. + An integer, the default :ref:`protocol version ` used + for pickling. May be less than :data:`HIGHEST_PROTOCOL`. Currently the + default protocol is 3, a new protocol designed for Python 3.0. The :mod:`pickle` module provides the following functions to make the pickling -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 7 02:14:57 2013 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 7 Dec 2013 02:14:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_document_that_?= =?utf-8?q?compile=28=29_can_take_bytes_=28closes_=2319910=29?= Message-ID: <3dbt4d5JX0z7LjW@mail.python.org> http://hg.python.org/cpython/rev/500cc1acc42f changeset: 87804:500cc1acc42f branch: 3.3 parent: 87801:609325d187bf user: Benjamin Peterson date: Fri Dec 06 20:12:39 2013 -0500 summary: document that compile() can take bytes (closes #19910) files: Doc/library/functions.rst | 6 +++--- Python/bltinmodule.c | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -193,9 +193,9 @@ .. function:: compile(source, filename, mode, flags=0, dont_inherit=False, optimize=-1) Compile the *source* into a code or AST object. Code objects can be executed - by :func:`exec` or :func:`eval`. *source* can either be a string or an AST - object. Refer to the :mod:`ast` module documentation for information on how - to work with AST objects. + by :func:`exec` or :func:`eval`. *source* can either be a normal string, a + byte string, or an AST object. Refer to the :mod:`ast` module documentation + for information on how to work with AST objects. The *filename* argument should give the file from which the code was read; pass some recognizable value if it wasn't read from a file (``''`` is diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c --- a/Python/bltinmodule.c +++ b/Python/bltinmodule.c @@ -676,7 +676,7 @@ PyDoc_STRVAR(compile_doc, "compile(source, filename, mode[, flags[, dont_inherit]]) -> code object\n\ \n\ -Compile the source string (a Python module, statement or expression)\n\ +Compile the source (a Python module, statement or expression)\n\ into a code object that can be executed by exec() or eval().\n\ The filename will be used for run-time error messages.\n\ The mode must be 'exec' to compile a module, 'single' to compile a\n\ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 7 02:14:59 2013 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 7 Dec 2013 02:14:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4zICgjMTk5MTAp?= Message-ID: <3dbt4g06fBz7LjW@mail.python.org> http://hg.python.org/cpython/rev/44dacafdd48a changeset: 87805:44dacafdd48a parent: 87803:8001f0dfe84f parent: 87804:500cc1acc42f user: Benjamin Peterson date: Fri Dec 06 20:12:51 2013 -0500 summary: merge 3.3 (#19910) files: Doc/library/functions.rst | 6 +++--- Python/bltinmodule.c | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -193,9 +193,9 @@ .. function:: compile(source, filename, mode, flags=0, dont_inherit=False, optimize=-1) Compile the *source* into a code or AST object. Code objects can be executed - by :func:`exec` or :func:`eval`. *source* can either be a string or an AST - object. Refer to the :mod:`ast` module documentation for information on how - to work with AST objects. + by :func:`exec` or :func:`eval`. *source* can either be a normal string, a + byte string, or an AST object. Refer to the :mod:`ast` module documentation + for information on how to work with AST objects. The *filename* argument should give the file from which the code was read; pass some recognizable value if it wasn't read from a file (``''`` is diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c --- a/Python/bltinmodule.c +++ b/Python/bltinmodule.c @@ -695,7 +695,7 @@ PyDoc_STRVAR(compile_doc, "compile(source, filename, mode[, flags[, dont_inherit]]) -> code object\n\ \n\ -Compile the source string (a Python module, statement or expression)\n\ +Compile the source (a Python module, statement or expression)\n\ into a code object that can be executed by exec() or eval().\n\ The filename will be used for run-time error messages.\n\ The mode must be 'exec' to compile a module, 'single' to compile a\n\ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 7 02:46:52 2013 From: python-checkins at python.org (guido.van.rossum) Date: Sat, 7 Dec 2013 02:46:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_indentation_of_switch_?= =?utf-8?q?cases=2E?= Message-ID: <3dbtnS6V85z7LlW@mail.python.org> http://hg.python.org/cpython/rev/e63a9695a0d4 changeset: 87806:e63a9695a0d4 user: Guido van Rossum date: Fri Dec 06 17:46:22 2013 -0800 summary: Fix indentation of switch cases. files: Modules/selectmodule.c | 22 +++++++++++----------- 1 files changed, 11 insertions(+), 11 deletions(-) diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -1317,16 +1317,16 @@ return NULL; } - switch(op) { - case EPOLL_CTL_ADD: - case EPOLL_CTL_MOD: + switch (op) { + case EPOLL_CTL_ADD: + case EPOLL_CTL_MOD: ev.events = events; ev.data.fd = fd; Py_BEGIN_ALLOW_THREADS result = epoll_ctl(epfd, op, fd, &ev); Py_END_ALLOW_THREADS break; - case EPOLL_CTL_DEL: + case EPOLL_CTL_DEL: /* In kernel versions before 2.6.9, the EPOLL_CTL_DEL * operation required a non-NULL pointer in event, even * though this argument is ignored. */ @@ -1339,7 +1339,7 @@ } Py_END_ALLOW_THREADS break; - default: + default: result = -1; errno = EINVAL; } @@ -1795,22 +1795,22 @@ } switch (op) { - case Py_EQ: + case Py_EQ: result = (result == 0); break; - case Py_NE: + case Py_NE: result = (result != 0); break; - case Py_LE: + case Py_LE: result = (result <= 0); break; - case Py_GE: + case Py_GE: result = (result >= 0); break; - case Py_LT: + case Py_LT: result = (result < 0); break; - case Py_GT: + case Py_GT: result = (result > 0); break; } -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sat Dec 7 09:41:49 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 07 Dec 2013 09:41:49 +0100 Subject: [Python-checkins] Daily reference leaks (e63a9695a0d4): sum=0 Message-ID: results for e63a9695a0d4 on branch "default" -------------------------------------------- test_site leaked [2, -2, 0] references, sum=0 test_site leaked [2, -2, 0] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/refloglxu6y8', '-x'] From python-checkins at python.org Sat Dec 7 10:09:40 2013 From: python-checkins at python.org (alexandre.vassalotti) Date: Sat, 7 Dec 2013 10:09:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=236784=3A_Strings_f?= =?utf-8?q?rom_Python_2_can_now_be_unpickled_as_bytes_objects=2E?= Message-ID: <3dc4cN0XBKz7Ljp@mail.python.org> http://hg.python.org/cpython/rev/bd71352e950f changeset: 87807:bd71352e950f user: Alexandre Vassalotti date: Sat Dec 07 01:09:27 2013 -0800 summary: Issue #6784: Strings from Python 2 can now be unpickled as bytes objects. Initial patch by Merlijn van Deen. I've added a few unrelated docstring fixes in the patch while I was at it, which makes the documentation for pickle a bit more consistent. files: Doc/library/pickle.rst | 88 ++-- Lib/pickle.py | 71 ++- Lib/pickletools.py | 193 +++++----- Lib/test/pickletester.py | 30 +- Lib/test/test_pickle.py | 4 + Misc/ACKS | 1 + Misc/NEWS | 4 + Modules/_pickle.c | 480 ++++++++++++++------------ 8 files changed, 476 insertions(+), 395 deletions(-) diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -173,7 +173,7 @@ An integer, the default :ref:`protocol version ` used for pickling. May be less than :data:`HIGHEST_PROTOCOL`. Currently the - default protocol is 3, a new protocol designed for Python 3.0. + default protocol is 3, a new protocol designed for Python 3. The :mod:`pickle` module provides the following functions to make the pickling @@ -184,9 +184,9 @@ Write a pickled representation of *obj* to the open :term:`file object` *file*. This is equivalent to ``Pickler(file, protocol).dump(obj)``. - The optional *protocol* argument tells the pickler to use the given protocol; - supported protocols are 0, 1, 2, 3. The default protocol is 3; a - backward-incompatible protocol designed for Python 3.0. + The optional *protocol* argument tells the pickler to use the given + protocol; supported protocols are 0, 1, 2, 3. The default protocol is 3; a + backward-incompatible protocol designed for Python 3. Specifying a negative protocol version selects the highest protocol version supported. The higher the protocol used, the more recent the version of @@ -198,64 +198,66 @@ interface. If *fix_imports* is true and *protocol* is less than 3, pickle will try to - map the new Python 3.x names to the old module names used in Python 2.x, - so that the pickle data stream is readable with Python 2.x. + map the new Python 3 names to the old module names used in Python 2, so + that the pickle data stream is readable with Python 2. .. function:: dumps(obj, protocol=None, \*, fix_imports=True) - Return the pickled representation of the object as a :class:`bytes` - object, instead of writing it to a file. + Return the pickled representation of the object as a :class:`bytes` object, + instead of writing it to a file. - The optional *protocol* argument tells the pickler to use the given protocol; - supported protocols are 0, 1, 2, 3. The default protocol is 3; a - backward-incompatible protocol designed for Python 3.0. + The optional *protocol* argument tells the pickler to use the given + protocol; supported protocols are 0, 1, 2, 3 and 4. The default protocol + is 3; a backward-incompatible protocol designed for Python 3. Specifying a negative protocol version selects the highest protocol version supported. The higher the protocol used, the more recent the version of Python needed to read the pickle produced. If *fix_imports* is true and *protocol* is less than 3, pickle will try to - map the new Python 3.x names to the old module names used in Python 2.x, - so that the pickle data stream is readable with Python 2.x. + map the new Python 3 names to the old module names used in Python 2, so + that the pickle data stream is readable with Python 2. .. function:: load(file, \*, fix_imports=True, encoding="ASCII", errors="strict") - Read a pickled object representation from the open :term:`file object` *file* - and return the reconstituted object hierarchy specified therein. This is - equivalent to ``Unpickler(file).load()``. + Read a pickled object representation from the open :term:`file object` + *file* and return the reconstituted object hierarchy specified therein. + This is equivalent to ``Unpickler(file).load()``. - The protocol version of the pickle is detected automatically, so no protocol - argument is needed. Bytes past the pickled object's representation are - ignored. + The protocol version of the pickle is detected automatically, so no + protocol argument is needed. Bytes past the pickled object's + representation are ignored. The argument *file* must have two methods, a read() method that takes an integer argument, and a readline() method that requires no arguments. Both - methods should return bytes. Thus *file* can be an on-disk file opened - for binary reading, a :class:`io.BytesIO` object, or any other custom object + methods should return bytes. Thus *file* can be an on-disk file opened for + binary reading, a :class:`io.BytesIO` object, or any other custom object that meets this interface. Optional keyword arguments are *fix_imports*, *encoding* and *errors*, which are used to control compatibility support for pickle stream generated - by Python 2.x. If *fix_imports* is true, pickle will try to map the old - Python 2.x names to the new names used in Python 3.x. The *encoding* and + by Python 2. If *fix_imports* is true, pickle will try to map the old + Python 2 names to the new names used in Python 3. The *encoding* and *errors* tell pickle how to decode 8-bit string instances pickled by Python - 2.x; these default to 'ASCII' and 'strict', respectively. + 2; these default to 'ASCII' and 'strict', respectively. The *encoding* can + be 'bytes' to read these 8-bit string instances as bytes objects. .. function:: loads(bytes_object, \*, fix_imports=True, encoding="ASCII", errors="strict") Read a pickled object hierarchy from a :class:`bytes` object and return the reconstituted object hierarchy specified therein - The protocol version of the pickle is detected automatically, so no protocol - argument is needed. Bytes past the pickled object's representation are - ignored. + The protocol version of the pickle is detected automatically, so no + protocol argument is needed. Bytes past the pickled object's + representation are ignored. Optional keyword arguments are *fix_imports*, *encoding* and *errors*, which are used to control compatibility support for pickle stream generated - by Python 2.x. If *fix_imports* is true, pickle will try to map the old - Python 2.x names to the new names used in Python 3.x. The *encoding* and + by Python 2. If *fix_imports* is true, pickle will try to map the old + Python 2 names to the new names used in Python 3. The *encoding* and *errors* tell pickle how to decode 8-bit string instances pickled by Python - 2.x; these default to 'ASCII' and 'strict', respectively. + 2; these default to 'ASCII' and 'strict', respectively. The *encoding* can + be 'bytes' to read these 8-bit string instances as bytes objects. The :mod:`pickle` module defines three exceptions: @@ -290,9 +292,9 @@ This takes a binary file for writing a pickle data stream. - The optional *protocol* argument tells the pickler to use the given protocol; - supported protocols are 0, 1, 2, 3. The default protocol is 3; a - backward-incompatible protocol designed for Python 3.0. + The optional *protocol* argument tells the pickler to use the given + protocol; supported protocols are 0, 1, 2, 3 and 4. The default protocol + is 3; a backward-incompatible protocol designed for Python 3. Specifying a negative protocol version selects the highest protocol version supported. The higher the protocol used, the more recent the version of @@ -300,11 +302,12 @@ The *file* argument must have a write() method that accepts a single bytes argument. It can thus be an on-disk file opened for binary writing, a - :class:`io.BytesIO` instance, or any other custom object that meets this interface. + :class:`io.BytesIO` instance, or any other custom object that meets this + interface. If *fix_imports* is true and *protocol* is less than 3, pickle will try to - map the new Python 3.x names to the old module names used in Python 2.x, - so that the pickle data stream is readable with Python 2.x. + map the new Python 3 names to the old module names used in Python 2, so + that the pickle data stream is readable with Python 2. .. method:: dump(obj) @@ -366,16 +369,17 @@ The argument *file* must have two methods, a read() method that takes an integer argument, and a readline() method that requires no arguments. Both - methods should return bytes. Thus *file* can be an on-disk file object opened - for binary reading, a :class:`io.BytesIO` object, or any other custom object - that meets this interface. + methods should return bytes. Thus *file* can be an on-disk file object + opened for binary reading, a :class:`io.BytesIO` object, or any other + custom object that meets this interface. Optional keyword arguments are *fix_imports*, *encoding* and *errors*, which are used to control compatibility support for pickle stream generated - by Python 2.x. If *fix_imports* is true, pickle will try to map the old - Python 2.x names to the new names used in Python 3.x. The *encoding* and + by Python 2. If *fix_imports* is true, pickle will try to map the old + Python 2 names to the new names used in Python 3. The *encoding* and *errors* tell pickle how to decode 8-bit string instances pickled by Python - 2.x; these default to 'ASCII' and 'strict', respectively. + 2; these default to 'ASCII' and 'strict', respectively. The *encoding* can + be 'bytes' to read these ?8-bit string instances as bytes objects. .. method:: load() diff --git a/Lib/pickle.py b/Lib/pickle.py --- a/Lib/pickle.py +++ b/Lib/pickle.py @@ -348,24 +348,25 @@ def __init__(self, file, protocol=None, *, fix_imports=True): """This takes a binary file for writing a pickle data stream. - The optional protocol argument tells the pickler to use the + The optional *protocol* argument tells the pickler to use the given protocol; supported protocols are 0, 1, 2, 3 and 4. The - default protocol is 3; a backward-incompatible protocol designed for - Python 3. + default protocol is 3; a backward-incompatible protocol designed + for Python 3. Specifying a negative protocol version selects the highest protocol version supported. The higher the protocol used, the more recent the version of Python needed to read the pickle produced. - The file argument must have a write() method that accepts a single - bytes argument. It can thus be a file object opened for binary - writing, a io.BytesIO instance, or any other custom object that - meets this interface. + The *file* argument must have a write() method that accepts a + single bytes argument. It can thus be a file object opened for + binary writing, a io.BytesIO instance, or any other custom + object that meets this interface. - If fix_imports is True and protocol is less than 3, pickle will try to - map the new Python 3 names to the old module names used in Python 2, - so that the pickle data stream is readable with Python 2. + If *fix_imports* is True and *protocol* is less than 3, pickle + will try to map the new Python 3 names to the old module names + used in Python 2, so that the pickle data stream is readable + with Python 2. """ if protocol is None: protocol = DEFAULT_PROTOCOL @@ -389,10 +390,9 @@ """Clears the pickler's "memo". The memo is the data structure that remembers which objects the - pickler has already seen, so that shared or recursive objects are - pickled by reference and not by value. This method is useful when - re-using picklers. - + pickler has already seen, so that shared or recursive objects + are pickled by reference and not by value. This method is + useful when re-using picklers. """ self.memo.clear() @@ -975,8 +975,14 @@ encoding="ASCII", errors="strict"): """This takes a binary file for reading a pickle data stream. - The protocol version of the pickle is detected automatically, so no - proto argument is needed. + The protocol version of the pickle is detected automatically, so + no proto argument is needed. + + The argument *file* must have two methods, a read() method that + takes an integer argument, and a readline() method that requires + no arguments. Both methods should return bytes. Thus *file* + can be a binary file object opened for reading, a io.BytesIO + object, or any other custom object that meets this interface. The file-like object must have two methods, a read() method that takes an integer argument, and a readline() method that @@ -985,13 +991,14 @@ reading, a BytesIO object, or any other custom object that meets this interface. - Optional keyword arguments are *fix_imports*, *encoding* and *errors*, - which are used to control compatiblity support for pickle stream - generated by Python 2.x. If *fix_imports* is True, pickle will try to - map the old Python 2.x names to the new names used in Python 3.x. The - *encoding* and *errors* tell pickle how to decode 8-bit string - instances pickled by Python 2.x; these default to 'ASCII' and - 'strict', respectively. + Optional keyword arguments are *fix_imports*, *encoding* and + *errors*, which are used to control compatiblity support for + pickle stream generated by Python 2. If *fix_imports* is True, + pickle will try to map the old Python 2 names to the new names + used in Python 3. The *encoding* and *errors* tell pickle how + to decode 8-bit string instances pickled by Python 2; these + default to 'ASCII' and 'strict', respectively. *encoding* can be + 'bytes' to read theses 8-bit string instances as bytes objects. """ self._file_readline = file.readline self._file_read = file.read @@ -1139,6 +1146,15 @@ self.append(unpack('>d', self.read(8))[0]) dispatch[BINFLOAT[0]] = load_binfloat + def _decode_string(self, value): + # Used to allow strings from Python 2 to be decoded either as + # bytes or Unicode strings. This should be used only with the + # STRING, BINSTRING and SHORT_BINSTRING opcodes. + if self.encoding == "bytes": + return value + else: + return value.decode(self.encoding, self.errors) + def load_string(self): data = self.readline()[:-1] # Strip outermost quotes @@ -1146,8 +1162,7 @@ data = data[1:-1] else: raise UnpicklingError("the STRING opcode argument must be quoted") - self.append(codecs.escape_decode(data)[0] - .decode(self.encoding, self.errors)) + self.append(self._decode_string(codecs.escape_decode(data)[0])) dispatch[STRING[0]] = load_string def load_binstring(self): @@ -1156,8 +1171,7 @@ if len < 0: raise UnpicklingError("BINSTRING pickle has negative byte count") data = self.read(len) - value = str(data, self.encoding, self.errors) - self.append(value) + self.append(self._decode_string(data)) dispatch[BINSTRING[0]] = load_binstring def load_binbytes(self): @@ -1191,8 +1205,7 @@ def load_short_binstring(self): len = self.read(1)[0] data = self.read(len) - value = str(data, self.encoding, self.errors) - self.append(value) + self.append(self._decode_string(data)) dispatch[SHORT_BINSTRING[0]] = load_short_binstring def load_short_binbytes(self): diff --git a/Lib/pickletools.py b/Lib/pickletools.py --- a/Lib/pickletools.py +++ b/Lib/pickletools.py @@ -969,113 +969,107 @@ return self.name -pyint = StackObject( - name='int', - obtype=int, - doc="A short (as opposed to long) Python integer object.") - -pylong = StackObject( - name='long', - obtype=int, - doc="A long (as opposed to short) Python integer object.") +pyint = pylong = StackObject( + name='int', + obtype=int, + doc="A Python integer object.") pyinteger_or_bool = StackObject( - name='int_or_bool', - obtype=(int, bool), - doc="A Python integer object (short or long), or " - "a Python bool.") + name='int_or_bool', + obtype=(int, bool), + doc="A Python integer or boolean object.") pybool = StackObject( - name='bool', - obtype=(bool,), - doc="A Python bool object.") + name='bool', + obtype=bool, + doc="A Python boolean object.") pyfloat = StackObject( - name='float', - obtype=float, - doc="A Python float object.") - -pystring = StackObject( - name='string', - obtype=bytes, - doc="A Python (8-bit) string object.") + name='float', + obtype=float, + doc="A Python float object.") + +pybytes_or_str = pystring = StackObject( + name='bytes_or_str', + obtype=(bytes, str), + doc="A Python bytes or (Unicode) string object.") pybytes = StackObject( - name='bytes', - obtype=bytes, - doc="A Python bytes object.") + name='bytes', + obtype=bytes, + doc="A Python bytes object.") pyunicode = StackObject( - name='str', - obtype=str, - doc="A Python (Unicode) string object.") + name='str', + obtype=str, + doc="A Python (Unicode) string object.") pynone = StackObject( - name="None", - obtype=type(None), - doc="The Python None object.") + name="None", + obtype=type(None), + doc="The Python None object.") pytuple = StackObject( - name="tuple", - obtype=tuple, - doc="A Python tuple object.") + name="tuple", + obtype=tuple, + doc="A Python tuple object.") pylist = StackObject( - name="list", - obtype=list, - doc="A Python list object.") + name="list", + obtype=list, + doc="A Python list object.") pydict = StackObject( - name="dict", - obtype=dict, - doc="A Python dict object.") + name="dict", + obtype=dict, + doc="A Python dict object.") pyset = StackObject( - name="set", - obtype=set, - doc="A Python set object.") + name="set", + obtype=set, + doc="A Python set object.") pyfrozenset = StackObject( - name="frozenset", - obtype=set, - doc="A Python frozenset object.") + name="frozenset", + obtype=set, + doc="A Python frozenset object.") anyobject = StackObject( - name='any', - obtype=object, - doc="Any kind of object whatsoever.") + name='any', + obtype=object, + doc="Any kind of object whatsoever.") markobject = StackObject( - name="mark", - obtype=StackObject, - doc="""'The mark' is a unique object. - - Opcodes that operate on a variable number of objects - generally don't embed the count of objects in the opcode, - or pull it off the stack. Instead the MARK opcode is used - to push a special marker object on the stack, and then - some other opcodes grab all the objects from the top of - the stack down to (but not including) the topmost marker - object. - """) + name="mark", + obtype=StackObject, + doc="""'The mark' is a unique object. + +Opcodes that operate on a variable number of objects +generally don't embed the count of objects in the opcode, +or pull it off the stack. Instead the MARK opcode is used +to push a special marker object on the stack, and then +some other opcodes grab all the objects from the top of +the stack down to (but not including) the topmost marker +object. +""") stackslice = StackObject( - name="stackslice", - obtype=StackObject, - doc="""An object representing a contiguous slice of the stack. - - This is used in conjunction with markobject, to represent all - of the stack following the topmost markobject. For example, - the POP_MARK opcode changes the stack from - - [..., markobject, stackslice] - to - [...] - - No matter how many object are on the stack after the topmost - markobject, POP_MARK gets rid of all of them (including the - topmost markobject too). - """) + name="stackslice", + obtype=StackObject, + doc="""An object representing a contiguous slice of the stack. + +This is used in conjunction with markobject, to represent all +of the stack following the topmost markobject. For example, +the POP_MARK opcode changes the stack from + + [..., markobject, stackslice] +to + [...] + +No matter how many object are on the stack after the topmost +markobject, POP_MARK gets rid of all of them (including the +topmost markobject too). +""") ############################################################################## # Descriptors for pickle opcodes. @@ -1212,7 +1206,7 @@ code='L', arg=decimalnl_long, stack_before=[], - stack_after=[pylong], + stack_after=[pyint], proto=0, doc="""Push a long integer. @@ -1230,7 +1224,7 @@ code='\x8a', arg=long1, stack_before=[], - stack_after=[pylong], + stack_after=[pyint], proto=2, doc="""Long integer using one-byte length. @@ -1241,7 +1235,7 @@ code='\x8b', arg=long4, stack_before=[], - stack_after=[pylong], + stack_after=[pyint], proto=2, doc="""Long integer using found-byte length. @@ -1254,45 +1248,50 @@ code='S', arg=stringnl, stack_before=[], - stack_after=[pystring], + stack_after=[pybytes_or_str], proto=0, doc="""Push a Python string object. The argument is a repr-style string, with bracketing quote characters, and perhaps embedded escapes. The argument extends until the next - newline character. (Actually, they are decoded into a str instance + newline character. These are usually decoded into a str instance using the encoding given to the Unpickler constructor. or the default, - 'ASCII'.) + 'ASCII'. If the encoding given was 'bytes' however, they will be + decoded as bytes object instead. """), I(name='BINSTRING', code='T', arg=string4, stack_before=[], - stack_after=[pystring], + stack_after=[pybytes_or_str], proto=1, doc="""Push a Python string object. - There are two arguments: the first is a 4-byte little-endian signed int - giving the number of bytes in the string, and the second is that many - bytes, which are taken literally as the string content. (Actually, - they are decoded into a str instance using the encoding given to the - Unpickler constructor. or the default, 'ASCII'.) + There are two arguments: the first is a 4-byte little-endian + signed int giving the number of bytes in the string, and the + second is that many bytes, which are taken literally as the string + content. These are usually decoded into a str instance using the + encoding given to the Unpickler constructor. or the default, + 'ASCII'. If the encoding given was 'bytes' however, they will be + decoded as bytes object instead. """), I(name='SHORT_BINSTRING', code='U', arg=string1, stack_before=[], - stack_after=[pystring], + stack_after=[pybytes_or_str], proto=1, doc="""Push a Python string object. - There are two arguments: the first is a 1-byte unsigned int giving - the number of bytes in the string, and the second is that many bytes, - which are taken literally as the string content. (Actually, they - are decoded into a str instance using the encoding given to the - Unpickler constructor. or the default, 'ASCII'.) + There are two arguments: the first is a 1-byte unsigned int giving + the number of bytes in the string, and the second is that many + bytes, which are taken literally as the string content. These are + usually decoded into a str instance using the encoding given to + the Unpickler constructor. or the default, 'ASCII'. If the + encoding given was 'bytes' however, they will be decoded as bytes + object instead. """), # Bytes (protocol 3 only; older protocols don't support bytes at all) diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -1305,6 +1305,35 @@ dumped = self.dumps(set([3]), 2) self.assertEqual(dumped, DATA6) + def test_load_python2_str_as_bytes(self): + # From Python 2: pickle.dumps('a\x00\xa0', protocol=0) + self.assertEqual(self.loads(b"S'a\\x00\\xa0'\n.", + encoding="bytes"), b'a\x00\xa0') + # From Python 2: pickle.dumps('a\x00\xa0', protocol=1) + self.assertEqual(self.loads(b'U\x03a\x00\xa0.', + encoding="bytes"), b'a\x00\xa0') + # From Python 2: pickle.dumps('a\x00\xa0', protocol=2) + self.assertEqual(self.loads(b'\x80\x02U\x03a\x00\xa0.', + encoding="bytes"), b'a\x00\xa0') + + def test_load_python2_unicode_as_str(self): + # From Python 2: pickle.dumps(u'?', protocol=0) + self.assertEqual(self.loads(b'V\\u03c0\n.', + encoding='bytes'), '?') + # From Python 2: pickle.dumps(u'?', protocol=1) + self.assertEqual(self.loads(b'X\x02\x00\x00\x00\xcf\x80.', + encoding="bytes"), '?') + # From Python 2: pickle.dumps(u'?', protocol=2) + self.assertEqual(self.loads(b'\x80\x02X\x02\x00\x00\x00\xcf\x80.', + encoding="bytes"), '?') + + def test_load_long_python2_str_as_bytes(self): + # From Python 2: pickle.dumps('x' * 300, protocol=1) + self.assertEqual(self.loads(pickle.BINSTRING + + struct.pack("encoding, self->errors); - Py_DECREF(bytes); - if (str == NULL) - return -1; - - PDATA_PUSH(self->stack, str, -1); - return 0; -} - -static int -load_counted_binbytes(UnpicklerObject *self, int nbytes) -{ - PyObject *bytes; - Py_ssize_t size; - char *s; - - if (_Unpickler_Read(self, &s, nbytes) < 0) - return -1; - - size = calc_binsize(s, nbytes); - if (size < 0) { - PyErr_Format(PyExc_OverflowError, - "BINBYTES exceeds system's maximum size of %zd bytes", - PY_SSIZE_T_MAX); - return -1; - } - - if (_Unpickler_Read(self, &s, size) < 0) - return -1; - - bytes = PyBytes_FromStringAndSize(s, size); - if (bytes == NULL) - return -1; - - PDATA_PUSH(self->stack, bytes, -1); + + /* Leave the Python 2.x strings as bytes if the *encoding* given to the + Unpickler was 'bytes'. Otherwise, convert them to unicode. */ + if (strcmp(self->encoding, "bytes") == 0) { + obj = bytes; + } + else { + obj = PyUnicode_FromEncodedObject(bytes, self->encoding, self->errors); + Py_DECREF(bytes); + if (obj == NULL) { + return -1; + } + } + + PDATA_PUSH(self->stack, obj, -1); return 0; } static int load_counted_binstring(UnpicklerObject *self, int nbytes) { - PyObject *str; + PyObject *obj; Py_ssize_t size; char *s; @@ -4916,12 +4892,49 @@ if (_Unpickler_Read(self, &s, size) < 0) return -1; - /* Convert Python 2.x strings to unicode. */ - str = PyUnicode_Decode(s, size, self->encoding, self->errors); - if (str == NULL) - return -1; - - PDATA_PUSH(self->stack, str, -1); + + /* Convert Python 2.x strings to bytes if the *encoding* given to the + Unpickler was 'bytes'. Otherwise, convert them to unicode. */ + if (strcmp(self->encoding, "bytes") == 0) { + obj = PyBytes_FromStringAndSize(s, size); + } + else { + obj = PyUnicode_Decode(s, size, self->encoding, self->errors); + } + if (obj == NULL) { + return -1; + } + + PDATA_PUSH(self->stack, obj, -1); + return 0; +} + +static int +load_counted_binbytes(UnpicklerObject *self, int nbytes) +{ + PyObject *bytes; + Py_ssize_t size; + char *s; + + if (_Unpickler_Read(self, &s, nbytes) < 0) + return -1; + + size = calc_binsize(s, nbytes); + if (size < 0) { + PyErr_Format(PyExc_OverflowError, + "BINBYTES exceeds system's maximum size of %zd bytes", + PY_SSIZE_T_MAX); + return -1; + } + + if (_Unpickler_Read(self, &s, size) < 0) + return -1; + + bytes = PyBytes_FromStringAndSize(s, size); + if (bytes == NULL) + return -1; + + PDATA_PUSH(self->stack, bytes, -1); return 0; } @@ -6258,25 +6271,25 @@ Load a pickle. -Read a pickled object representation from the open file object given in -the constructor, and return the reconstituted object hierarchy specified -therein. +Read a pickled object representation from the open file object given +in the constructor, and return the reconstituted object hierarchy +specified therein. [clinic]*/ PyDoc_STRVAR(_pickle_Unpickler_load__doc__, "load()\n" "Load a pickle.\n" "\n" -"Read a pickled object representation from the open file object given in\n" -"the constructor, and return the reconstituted object hierarchy specified\n" -"therein."); +"Read a pickled object representation from the open file object given\n" +"in the constructor, and return the reconstituted object hierarchy\n" +"specified therein."); #define _PICKLE_UNPICKLER_LOAD_METHODDEF \ {"load", (PyCFunction)_pickle_Unpickler_load, METH_NOARGS, _pickle_Unpickler_load__doc__}, static PyObject * _pickle_Unpickler_load(PyObject *self) -/*[clinic checksum: 9a30ba4e4d9221d4dcd705e1471ab11b2c9e3ac6]*/ +/*[clinic checksum: c2ae1263f0dd000f34ccf0fe59d7c544464babc4]*/ { UnpicklerObject *unpickler = (UnpicklerObject*)self; @@ -6310,8 +6323,9 @@ Return an object from a specified module. -If necessary, the module will be imported. Subclasses may override this -method (e.g. to restrict unpickling of arbitrary classes and functions). +If necessary, the module will be imported. Subclasses may override +this method (e.g. to restrict unpickling of arbitrary classes and +functions). This method is called whenever a class or a function object is needed. Both arguments passed are str objects. @@ -6321,8 +6335,9 @@ "find_class(module_name, global_name)\n" "Return an object from a specified module.\n" "\n" -"If necessary, the module will be imported. Subclasses may override this\n" -"method (e.g. to restrict unpickling of arbitrary classes and functions).\n" +"If necessary, the module will be imported. Subclasses may override\n" +"this method (e.g. to restrict unpickling of arbitrary classes and\n" +"functions).\n" "\n" "This method is called whenever a class or a function object is\n" "needed. Both arguments passed are str objects."); @@ -6352,7 +6367,7 @@ static PyObject * _pickle_Unpickler_find_class_impl(UnpicklerObject *self, PyObject *module_name, PyObject *global_name) -/*[clinic checksum: b7d05d4dd8adc698e5780c1ac2be0f5062d33915]*/ +/*[clinic checksum: 1f353d13a32c9d94feb1466b3c2d0529a7e5650e]*/ { PyObject *global; PyObject *modules_dict; @@ -6515,23 +6530,23 @@ This takes a binary file for reading a pickle data stream. The protocol version of the pickle is detected automatically, so no -proto argument is needed. - -The file-like object must have two methods, a read() method -that takes an integer argument, and a readline() method that -requires no arguments. Both methods should return bytes. -Thus file-like object can be a binary file object opened for -reading, a BytesIO object, or any other custom object that -meets this interface. +protocol argument is needed. Bytes past the pickled object's +representation are ignored. + +The argument *file* must have two methods, a read() method that takes +an integer argument, and a readline() method that requires no +arguments. Both methods should return bytes. Thus *file* can be a +binary file object opened for reading, a io.BytesIO object, or any +other custom object that meets this interface. Optional keyword arguments are *fix_imports*, *encoding* and *errors*, which are used to control compatiblity support for pickle stream -generated by Python 2.x. If *fix_imports* is True, pickle will try to -map the old Python 2.x names to the new names used in Python 3.x. The +generated by Python 2. If *fix_imports* is True, pickle will try to +map the old Python 2 names to the new names used in Python 3. The *encoding* and *errors* tell pickle how to decode 8-bit string -instances pickled by Python 2.x; these default to 'ASCII' and -'strict', respectively. - +instances pickled by Python 2; these default to 'ASCII' and 'strict', +respectively. The *encoding* can be 'bytes' to read these 8-bit +string instances as bytes objects. [clinic]*/ PyDoc_STRVAR(_pickle_Unpickler___init____doc__, @@ -6539,22 +6554,23 @@ "This takes a binary file for reading a pickle data stream.\n" "\n" "The protocol version of the pickle is detected automatically, so no\n" -"proto argument is needed.\n" +"protocol argument is needed. Bytes past the pickled object\'s\n" +"representation are ignored.\n" "\n" -"The file-like object must have two methods, a read() method\n" -"that takes an integer argument, and a readline() method that\n" -"requires no arguments. Both methods should return bytes.\n" -"Thus file-like object can be a binary file object opened for\n" -"reading, a BytesIO object, or any other custom object that\n" -"meets this interface.\n" +"The argument *file* must have two methods, a read() method that takes\n" +"an integer argument, and a readline() method that requires no\n" +"arguments. Both methods should return bytes. Thus *file* can be a\n" +"binary file object opened for reading, a io.BytesIO object, or any\n" +"other custom object that meets this interface.\n" "\n" "Optional keyword arguments are *fix_imports*, *encoding* and *errors*,\n" "which are used to control compatiblity support for pickle stream\n" -"generated by Python 2.x. If *fix_imports* is True, pickle will try to\n" -"map the old Python 2.x names to the new names used in Python 3.x. The\n" +"generated by Python 2. If *fix_imports* is True, pickle will try to\n" +"map the old Python 2 names to the new names used in Python 3. The\n" "*encoding* and *errors* tell pickle how to decode 8-bit string\n" -"instances pickled by Python 2.x; these default to \'ASCII\' and\n" -"\'strict\', respectively."); +"instances pickled by Python 2; these default to \'ASCII\' and \'strict\',\n" +"respectively. The *encoding* can be \'bytes\' to read these 8-bit\n" +"string instances as bytes objects."); #define _PICKLE_UNPICKLER___INIT___METHODDEF \ {"__init__", (PyCFunction)_pickle_Unpickler___init__, METH_VARARGS|METH_KEYWORDS, _pickle_Unpickler___init____doc__}, @@ -6584,7 +6600,7 @@ static PyObject * _pickle_Unpickler___init___impl(UnpicklerObject *self, PyObject *file, int fix_imports, const char *encoding, const char *errors) -/*[clinic checksum: bed0d8bbe1c647960ccc6f997b33bf33935fa56f]*/ +/*[clinic checksum: 9ce6783224e220573d42a94fe1bb7199d6f1c5a6]*/ { _Py_IDENTIFIER(persistent_load); @@ -7033,48 +7049,50 @@ Write a pickled representation of obj to the open file object file. -This is equivalent to ``Pickler(file, protocol).dump(obj)``, but may be more -efficient. - -The optional protocol argument tells the pickler to use the given protocol -supported protocols are 0, 1, 2, 3. The default protocol is 3; a -backward-incompatible protocol designed for Python 3.0. - -Specifying a negative protocol version selects the highest protocol version -supported. The higher the protocol used, the more recent the version of -Python needed to read the pickle produced. - -The file argument must have a write() method that accepts a single bytes -argument. It can thus be a file object opened for binary writing, a -io.BytesIO instance, or any other custom object that meets this interface. - -If fix_imports is True and protocol is less than 3, pickle will try to -map the new Python 3.x names to the old module names used in Python 2.x, -so that the pickle data stream is readable with Python 2.x. +This is equivalent to ``Pickler(file, protocol).dump(obj)``, but may +be more efficient. + +The optional *protocol* argument tells the pickler to use the given +protocol supported protocols are 0, 1, 2, 3 and 4. The default +protocol is 3; a backward-incompatible protocol designed for Python 3. + +Specifying a negative protocol version selects the highest protocol +version supported. The higher the protocol used, the more recent the +version of Python needed to read the pickle produced. + +The *file* argument must have a write() method that accepts a single +bytes argument. It can thus be a file object opened for binary +writing, a io.BytesIO instance, or any other custom object that meets +this interface. + +If *fix_imports* is True and protocol is less than 3, pickle will try +to map the new Python 3 names to the old module names used in Python +2, so that the pickle data stream is readable with Python 2. [clinic]*/ PyDoc_STRVAR(_pickle_dump__doc__, "dump(obj, file, protocol=None, *, fix_imports=True)\n" "Write a pickled representation of obj to the open file object file.\n" "\n" -"This is equivalent to ``Pickler(file, protocol).dump(obj)``, but may be more\n" -"efficient.\n" +"This is equivalent to ``Pickler(file, protocol).dump(obj)``, but may\n" +"be more efficient.\n" "\n" -"The optional protocol argument tells the pickler to use the given protocol\n" -"supported protocols are 0, 1, 2, 3. The default protocol is 3; a\n" -"backward-incompatible protocol designed for Python 3.0.\n" +"The optional *protocol* argument tells the pickler to use the given\n" +"protocol supported protocols are 0, 1, 2, 3 and 4. The default\n" +"protocol is 3; a backward-incompatible protocol designed for Python 3.\n" "\n" -"Specifying a negative protocol version selects the highest protocol version\n" -"supported. The higher the protocol used, the more recent the version of\n" -"Python needed to read the pickle produced.\n" +"Specifying a negative protocol version selects the highest protocol\n" +"version supported. The higher the protocol used, the more recent the\n" +"version of Python needed to read the pickle produced.\n" "\n" -"The file argument must have a write() method that accepts a single bytes\n" -"argument. It can thus be a file object opened for binary writing, a\n" -"io.BytesIO instance, or any other custom object that meets this interface.\n" +"The *file* argument must have a write() method that accepts a single\n" +"bytes argument. It can thus be a file object opened for binary\n" +"writing, a io.BytesIO instance, or any other custom object that meets\n" +"this interface.\n" "\n" -"If fix_imports is True and protocol is less than 3, pickle will try to\n" -"map the new Python 3.x names to the old module names used in Python 2.x,\n" -"so that the pickle data stream is readable with Python 2.x."); +"If *fix_imports* is True and protocol is less than 3, pickle will try\n" +"to map the new Python 3 names to the old module names used in Python\n" +"2, so that the pickle data stream is readable with Python 2."); #define _PICKLE_DUMP_METHODDEF \ {"dump", (PyCFunction)_pickle_dump, METH_VARARGS|METH_KEYWORDS, _pickle_dump__doc__}, @@ -7104,7 +7122,7 @@ static PyObject * _pickle_dump_impl(PyModuleDef *module, PyObject *obj, PyObject *file, PyObject *protocol, int fix_imports) -/*[clinic checksum: e442721b16052d921b5e3fbd146d0a62e94a459e]*/ +/*[clinic checksum: eb5c23e64da34477178230b704d2cc9c6b6650ea]*/ { PicklerObject *pickler = _Pickler_New(); @@ -7142,34 +7160,34 @@ Return the pickled representation of the object as a bytes object. -The optional protocol argument tells the pickler to use the given protocol; -supported protocols are 0, 1, 2, 3. The default protocol is 3; a -backward-incompatible protocol designed for Python 3.0. - -Specifying a negative protocol version selects the highest protocol version -supported. The higher the protocol used, the more recent the version of -Python needed to read the pickle produced. - -If fix_imports is True and *protocol* is less than 3, pickle will try to -map the new Python 3.x names to the old module names used in Python 2.x, -so that the pickle data stream is readable with Python 2.x. +The optional *protocol* argument tells the pickler to use the given +protocol; supported protocols are 0, 1, 2, 3 and 4. The default +protocol is 3; a backward-incompatible protocol designed for Python 3. + +Specifying a negative protocol version selects the highest protocol +version supported. The higher the protocol used, the more recent the +version of Python needed to read the pickle produced. + +If *fix_imports* is True and *protocol* is less than 3, pickle will +try to map the new Python 3 names to the old module names used in +Python 2, so that the pickle data stream is readable with Python 2. [clinic]*/ PyDoc_STRVAR(_pickle_dumps__doc__, "dumps(obj, protocol=None, *, fix_imports=True)\n" "Return the pickled representation of the object as a bytes object.\n" "\n" -"The optional protocol argument tells the pickler to use the given protocol;\n" -"supported protocols are 0, 1, 2, 3. The default protocol is 3; a\n" -"backward-incompatible protocol designed for Python 3.0.\n" +"The optional *protocol* argument tells the pickler to use the given\n" +"protocol; supported protocols are 0, 1, 2, 3 and 4. The default\n" +"protocol is 3; a backward-incompatible protocol designed for Python 3.\n" "\n" -"Specifying a negative protocol version selects the highest protocol version\n" -"supported. The higher the protocol used, the more recent the version of\n" -"Python needed to read the pickle produced.\n" +"Specifying a negative protocol version selects the highest protocol\n" +"version supported. The higher the protocol used, the more recent the\n" +"version of Python needed to read the pickle produced.\n" "\n" -"If fix_imports is True and *protocol* is less than 3, pickle will try to\n" -"map the new Python 3.x names to the old module names used in Python 2.x,\n" -"so that the pickle data stream is readable with Python 2.x."); +"If *fix_imports* is True and *protocol* is less than 3, pickle will\n" +"try to map the new Python 3 names to the old module names used in\n" +"Python 2, so that the pickle data stream is readable with Python 2."); #define _PICKLE_DUMPS_METHODDEF \ {"dumps", (PyCFunction)_pickle_dumps, METH_VARARGS|METH_KEYWORDS, _pickle_dumps__doc__}, @@ -7198,7 +7216,7 @@ static PyObject * _pickle_dumps_impl(PyModuleDef *module, PyObject *obj, PyObject *protocol, int fix_imports) -/*[clinic checksum: df6262c4c487f537f47aec8a1709318204c1e174]*/ +/*[clinic checksum: e9b915d61202a9692cb6c6718db74fe54fc9c4d1]*/ { PyObject *result; PicklerObject *pickler = _Pickler_New(); @@ -7231,50 +7249,56 @@ encoding: str = 'ASCII' errors: str = 'strict' -Return a reconstituted object from the pickle data stored in a file. - -This is equivalent to ``Unpickler(file).load()``, but may be more efficient. - -The protocol version of the pickle is detected automatically, so no protocol -argument is needed. Bytes past the pickled object's representation are -ignored. - -The argument file must have two methods, a read() method that takes an -integer argument, and a readline() method that requires no arguments. Both -methods should return bytes. Thus *file* can be a binary file object opened -for reading, a BytesIO object, or any other custom object that meets this -interface. - -Optional keyword arguments are fix_imports, encoding and errors, -which are used to control compatiblity support for pickle stream generated -by Python 2.x. If fix_imports is True, pickle will try to map the old -Python 2.x names to the new names used in Python 3.x. The encoding and -errors tell pickle how to decode 8-bit string instances pickled by Python -2.x; these default to 'ASCII' and 'strict', respectively. +Read and return an object from the pickle data stored in a file. + +This is equivalent to ``Unpickler(file).load()``, but may be more +efficient. + +The protocol version of the pickle is detected automatically, so no +protocol argument is needed. Bytes past the pickled object's +representation are ignored. + +The argument *file* must have two methods, a read() method that takes +an integer argument, and a readline() method that requires no +arguments. Both methods should return bytes. Thus *file* can be a +binary file object opened for reading, a io.BytesIO object, or any +other custom object that meets this interface. + +Optional keyword arguments are *fix_imports*, *encoding* and *errors*, +which are used to control compatiblity support for pickle stream +generated by Python 2. If *fix_imports* is True, pickle will try to +map the old Python 2 names to the new names used in Python 3. The +*encoding* and *errors* tell pickle how to decode 8-bit string +instances pickled by Python 2; these default to 'ASCII' and 'strict', +respectively. The *encoding* can be 'bytes' to read these 8-bit +string instances as bytes objects. [clinic]*/ PyDoc_STRVAR(_pickle_load__doc__, "load(file, *, fix_imports=True, encoding=\'ASCII\', errors=\'strict\')\n" -"Return a reconstituted object from the pickle data stored in a file.\n" +"Read and return an object from the pickle data stored in a file.\n" "\n" -"This is equivalent to ``Unpickler(file).load()``, but may be more efficient.\n" +"This is equivalent to ``Unpickler(file).load()``, but may be more\n" +"efficient.\n" "\n" -"The protocol version of the pickle is detected automatically, so no protocol\n" -"argument is needed. Bytes past the pickled object\'s representation are\n" -"ignored.\n" +"The protocol version of the pickle is detected automatically, so no\n" +"protocol argument is needed. Bytes past the pickled object\'s\n" +"representation are ignored.\n" "\n" -"The argument file must have two methods, a read() method that takes an\n" -"integer argument, and a readline() method that requires no arguments. Both\n" -"methods should return bytes. Thus *file* can be a binary file object opened\n" -"for reading, a BytesIO object, or any other custom object that meets this\n" -"interface.\n" +"The argument *file* must have two methods, a read() method that takes\n" +"an integer argument, and a readline() method that requires no\n" +"arguments. Both methods should return bytes. Thus *file* can be a\n" +"binary file object opened for reading, a io.BytesIO object, or any\n" +"other custom object that meets this interface.\n" "\n" -"Optional keyword arguments are fix_imports, encoding and errors,\n" -"which are used to control compatiblity support for pickle stream generated\n" -"by Python 2.x. If fix_imports is True, pickle will try to map the old\n" -"Python 2.x names to the new names used in Python 3.x. The encoding and\n" -"errors tell pickle how to decode 8-bit string instances pickled by Python\n" -"2.x; these default to \'ASCII\' and \'strict\', respectively."); +"Optional keyword arguments are *fix_imports*, *encoding* and *errors*,\n" +"which are used to control compatiblity support for pickle stream\n" +"generated by Python 2. If *fix_imports* is True, pickle will try to\n" +"map the old Python 2 names to the new names used in Python 3. The\n" +"*encoding* and *errors* tell pickle how to decode 8-bit string\n" +"instances pickled by Python 2; these default to \'ASCII\' and \'strict\',\n" +"respectively. The *encoding* can be \'bytes\' to read these 8-bit\n" +"string instances as bytes objects."); #define _PICKLE_LOAD_METHODDEF \ {"load", (PyCFunction)_pickle_load, METH_VARARGS|METH_KEYWORDS, _pickle_load__doc__}, @@ -7304,7 +7328,7 @@ static PyObject * _pickle_load_impl(PyModuleDef *module, PyObject *file, int fix_imports, const char *encoding, const char *errors) -/*[clinic checksum: e10796f6765b22ce48dca6940f11b3933853ca35]*/ +/*[clinic checksum: b41f06970e57acf2fd602e4b7f88e3f3e1e53087]*/ { PyObject *result; UnpicklerObject *unpickler = _Unpickler_New(); @@ -7339,34 +7363,38 @@ encoding: str = 'ASCII' errors: str = 'strict' -Return a reconstituted object from the given pickle data. - -The protocol version of the pickle is detected automatically, so no protocol -argument is needed. Bytes past the pickled object's representation are -ignored. - -Optional keyword arguments are fix_imports, encoding and errors, which -are used to control compatiblity support for pickle stream generated -by Python 2.x. If fix_imports is True, pickle will try to map the old -Python 2.x names to the new names used in Python 3.x. The encoding and -errors tell pickle how to decode 8-bit string instances pickled by Python -2.x; these default to 'ASCII' and 'strict', respectively. +Read and return an object from the given pickle data. + +The protocol version of the pickle is detected automatically, so no +protocol argument is needed. Bytes past the pickled object's +representation are ignored. + +Optional keyword arguments are *fix_imports*, *encoding* and *errors*, +which are used to control compatiblity support for pickle stream +generated by Python 2. If *fix_imports* is True, pickle will try to +map the old Python 2 names to the new names used in Python 3. The +*encoding* and *errors* tell pickle how to decode 8-bit string +instances pickled by Python 2; these default to 'ASCII' and 'strict', +respectively. The *encoding* can be 'bytes' to read these 8-bit +string instances as bytes objects. [clinic]*/ PyDoc_STRVAR(_pickle_loads__doc__, "loads(data, *, fix_imports=True, encoding=\'ASCII\', errors=\'strict\')\n" -"Return a reconstituted object from the given pickle data.\n" +"Read and return an object from the given pickle data.\n" "\n" -"The protocol version of the pickle is detected automatically, so no protocol\n" -"argument is needed. Bytes past the pickled object\'s representation are\n" -"ignored.\n" +"The protocol version of the pickle is detected automatically, so no\n" +"protocol argument is needed. Bytes past the pickled object\'s\n" +"representation are ignored.\n" "\n" -"Optional keyword arguments are fix_imports, encoding and errors, which\n" -"are used to control compatiblity support for pickle stream generated\n" -"by Python 2.x. If fix_imports is True, pickle will try to map the old\n" -"Python 2.x names to the new names used in Python 3.x. The encoding and\n" -"errors tell pickle how to decode 8-bit string instances pickled by Python\n" -"2.x; these default to \'ASCII\' and \'strict\', respectively."); +"Optional keyword arguments are *fix_imports*, *encoding* and *errors*,\n" +"which are used to control compatiblity support for pickle stream\n" +"generated by Python 2. If *fix_imports* is True, pickle will try to\n" +"map the old Python 2 names to the new names used in Python 3. The\n" +"*encoding* and *errors* tell pickle how to decode 8-bit string\n" +"instances pickled by Python 2; these default to \'ASCII\' and \'strict\',\n" +"respectively. The *encoding* can be \'bytes\' to read these 8-bit\n" +"string instances as bytes objects."); #define _PICKLE_LOADS_METHODDEF \ {"loads", (PyCFunction)_pickle_loads, METH_VARARGS|METH_KEYWORDS, _pickle_loads__doc__}, @@ -7396,7 +7424,7 @@ static PyObject * _pickle_loads_impl(PyModuleDef *module, PyObject *data, int fix_imports, const char *encoding, const char *errors) -/*[clinic checksum: 29ee725efcbf51a3533c19cb8261a8e267b7080a]*/ +/*[clinic checksum: 0663de43aca6c21508a777e29d98c9c3a6e7f72d]*/ { PyObject *result; UnpicklerObject *unpickler = _Unpickler_New(); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 7 15:03:23 2013 From: python-checkins at python.org (victor.stinner) Date: Sat, 7 Dec 2013 15:03:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Remove_a_duplicated_import?= Message-ID: <3dcC7H6wmBz7LjQ@mail.python.org> http://hg.python.org/cpython/rev/3fe2fd4ffa32 changeset: 87808:3fe2fd4ffa32 user: Victor Stinner date: Sat Dec 07 15:02:09 2013 +0100 summary: Remove a duplicated import files: Lib/asyncio/test_utils.py | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py --- a/Lib/asyncio/test_utils.py +++ b/Lib/asyncio/test_utils.py @@ -3,7 +3,6 @@ import collections import contextlib import io -import unittest.mock import os import sys import threading -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 7 15:44:34 2013 From: python-checkins at python.org (christian.heimes) Date: Sat, 7 Dec 2013 15:44:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_Explain_how_to_use_the_bu?= =?utf-8?q?ildbot_scripts_to_provide_dependencies_on_Windows=2E?= Message-ID: <3dcD2p1GP0z7LkL@mail.python.org> http://hg.python.org/devguide/rev/6aa0dc8c0bc6 changeset: 655:6aa0dc8c0bc6 user: Christian Heimes date: Sat Dec 07 15:44:27 2013 +0100 summary: Explain how to use the buildbot scripts to provide dependencies on Windows. files: index.rst | 6 +++++- setup.rst | 5 ++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/index.rst b/index.rst --- a/index.rst +++ b/index.rst @@ -24,7 +24,11 @@ On :ref:`Windows `, open the solution file :file:`PCbuild\\pcbuild.sln` in Visual Studio, select :menuselection:`Debug`, - and :menuselection:`Build --> Build Solution`. + and :menuselection:`Build --> Build Solution`. Run + :file:`Tools\buildbot\external.bat` or + :file:`Tools\buildbot\external-amd64.bat` to download and compile 3rd + party libraries. + 3. :doc:`Run the tests `:: ./python -m test -j3 diff --git a/setup.rst b/setup.rst --- a/setup.rst +++ b/setup.rst @@ -220,7 +220,10 @@ files or directories. These do not necessarily mean that Python failed to build. If you prefer, you can exclude the offending projects from the build process by unchecking them inside the -:menuselection:`Build --> Configuration Manager...` settings. +:menuselection:`Build --> Configuration Manager...` settings. You can +also use the script :file:`Tools\buildbot\external.bat` or +:file:`Tools\buildbot\external-amd64.bat` (as applicable) to download and +compile missing dependencies. Once built you might want to set Python as a startup project. Pressing F5 in Visual Studio, or choosing Start Debugging from the Debug menu, will launch -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Sat Dec 7 16:02:38 2013 From: python-checkins at python.org (christian.heimes) Date: Sat, 7 Dec 2013 16:02:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_quote_backslashes?= Message-ID: <3dcDRf5nmpz7Ljb@mail.python.org> http://hg.python.org/devguide/rev/99419f310cf1 changeset: 656:99419f310cf1 user: Christian Heimes date: Sat Dec 07 16:02:31 2013 +0100 summary: quote backslashes files: index.rst | 4 ++-- setup.rst | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/index.rst b/index.rst --- a/index.rst +++ b/index.rst @@ -25,8 +25,8 @@ On :ref:`Windows `, open the solution file :file:`PCbuild\\pcbuild.sln` in Visual Studio, select :menuselection:`Debug`, and :menuselection:`Build --> Build Solution`. Run - :file:`Tools\buildbot\external.bat` or - :file:`Tools\buildbot\external-amd64.bat` to download and compile 3rd + :file:`Tools\\buildbot\\external.bat` or + :file:`Tools\\buildbot\\external-amd64.bat` to download and compile 3rd party libraries. 3. :doc:`Run the tests `:: diff --git a/setup.rst b/setup.rst --- a/setup.rst +++ b/setup.rst @@ -221,8 +221,8 @@ to build. If you prefer, you can exclude the offending projects from the build process by unchecking them inside the :menuselection:`Build --> Configuration Manager...` settings. You can -also use the script :file:`Tools\buildbot\external.bat` or -:file:`Tools\buildbot\external-amd64.bat` (as applicable) to download and +also use the script :file:`Tools\\buildbot\external.bat` or +:file:`Tools\\buildbot\external-amd64.bat` (as applicable) to download and compile missing dependencies. Once built you might want to set Python as a startup project. Pressing F5 in -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Sat Dec 7 18:19:29 2013 From: python-checkins at python.org (christian.heimes) Date: Sat, 7 Dec 2013 18:19:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Solaris_needs_a_readable_f?= =?utf-8?q?ile_for_shared_lock?= Message-ID: <3dcHTY1HtZz7Ljb@mail.python.org> http://hg.python.org/cpython/rev/4e9fb4ce7cf9 changeset: 87809:4e9fb4ce7cf9 user: Christian Heimes date: Sat Dec 07 18:19:21 2013 +0100 summary: Solaris needs a readable file for shared lock files: Lib/test/test_fcntl.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_fcntl.py b/Lib/test/test_fcntl.py --- a/Lib/test/test_fcntl.py +++ b/Lib/test/test_fcntl.py @@ -116,7 +116,8 @@ os.close(fd) def test_flock(self): - self.f = open(TESTFN, 'wb') + # Solaris needs readable file for shared lock + self.f = open(TESTFN, 'wb+') fileno = self.f.fileno() fcntl.flock(fileno, fcntl.LOCK_SH) fcntl.flock(fileno, fcntl.LOCK_UN) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 7 20:53:38 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sat, 7 Dec 2013 20:53:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319857=3A_Make_sur?= =?utf-8?q?e_that_test=5Fimaplib_reaps_server_threads_even_in_face_of?= Message-ID: <3dcLvQ1cC8z7LjQ@mail.python.org> http://hg.python.org/cpython/rev/fbbb591dea09 changeset: 87810:fbbb591dea09 parent: 85678:bda5a87df1c8 user: Charles-Fran?ois Natali date: Sat Dec 07 20:03:12 2013 +0100 summary: Issue #19857: Make sure that test_imaplib reaps server threads even in face of error upon client disconnection. files: Lib/test/test_imaplib.py | 13 ++++++------- 1 files changed, 6 insertions(+), 7 deletions(-) diff --git a/Lib/test/test_imaplib.py b/Lib/test/test_imaplib.py --- a/Lib/test/test_imaplib.py +++ b/Lib/test/test_imaplib.py @@ -210,13 +210,12 @@ @contextmanager def reaped_pair(self, hdlr): - server, thread = self.make_server((support.HOST, 0), hdlr) - client = self.imap_class(*server.server_address) - try: - yield server, client - finally: - client.logout() - self.reap_server(server, thread) + with self.reaped_server(hdlr) as server: + client = self.imap_class(*server.server_address) + try: + yield server, client + finally: + client.logout() @reap_threads def test_connect(self): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 7 20:53:39 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sat, 7 Dec 2013 20:53:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319857=3A_Make_sur?= =?utf-8?q?e_that_test=5Fimaplib_reaps_server_threads_even_in_face_of?= Message-ID: <3dcLvR3RVwz7LkB@mail.python.org> http://hg.python.org/cpython/rev/6c81df506739 changeset: 87811:6c81df506739 parent: 87809:4e9fb4ce7cf9 user: Charles-Fran?ois Natali date: Sat Dec 07 20:27:41 2013 +0100 summary: Issue #19857: Make sure that test_imaplib reaps server threads even in face of error upon client disconnection. files: Lib/test/test_imaplib.py | 13 ++++++------- 1 files changed, 6 insertions(+), 7 deletions(-) diff --git a/Lib/test/test_imaplib.py b/Lib/test/test_imaplib.py --- a/Lib/test/test_imaplib.py +++ b/Lib/test/test_imaplib.py @@ -211,13 +211,12 @@ @contextmanager def reaped_pair(self, hdlr): - server, thread = self.make_server((support.HOST, 0), hdlr) - client = self.imap_class(*server.server_address) - try: - yield server, client - finally: - client.logout() - self.reap_server(server, thread) + with self.reaped_server(hdlr) as server: + client = self.imap_class(*server.server_address) + try: + yield server, client + finally: + client.logout() @reap_threads def test_connect(self): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 7 20:53:40 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sat, 7 Dec 2013 20:53:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?b?KTogTWVyZ2Uu?= Message-ID: <3dcLvS5PsMz7Lk8@mail.python.org> http://hg.python.org/cpython/rev/8f43ddadbf9f changeset: 87812:8f43ddadbf9f parent: 87811:6c81df506739 parent: 87810:fbbb591dea09 user: Charles-Fran?ois Natali date: Sat Dec 07 20:28:46 2013 +0100 summary: Merge. files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 7 20:53:42 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sat, 7 Dec 2013 20:53:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5ODU3?= =?utf-8?q?=3A_Make_sure_that_test=5Fimaplib_reaps_server_threads_even_in_?= =?utf-8?q?face_of?= Message-ID: <3dcLvV079Vz7Lkp@mail.python.org> http://hg.python.org/cpython/rev/78efa2c06447 changeset: 87813:78efa2c06447 branch: 3.3 parent: 87804:500cc1acc42f user: Charles-Fran?ois Natali date: Sat Dec 07 20:30:17 2013 +0100 summary: Issue #19857: Make sure that test_imaplib reaps server threads even in face of error upon client disconnection. files: Lib/test/test_imaplib.py | 13 ++++++------- 1 files changed, 6 insertions(+), 7 deletions(-) diff --git a/Lib/test/test_imaplib.py b/Lib/test/test_imaplib.py --- a/Lib/test/test_imaplib.py +++ b/Lib/test/test_imaplib.py @@ -210,13 +210,12 @@ @contextmanager def reaped_pair(self, hdlr): - server, thread = self.make_server((support.HOST, 0), hdlr) - client = self.imap_class(*server.server_address) - try: - yield server, client - finally: - client.logout() - self.reap_server(server, thread) + with self.reaped_server(hdlr) as server: + client = self.imap_class(*server.server_address) + try: + yield server, client + finally: + client.logout() @reap_threads def test_connect(self): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 7 20:53:43 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sat, 7 Dec 2013 20:53:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogTWVyZ2Uu?= Message-ID: <3dcLvW1rlhz7LlM@mail.python.org> http://hg.python.org/cpython/rev/412b94cd882a changeset: 87814:412b94cd882a parent: 87812:8f43ddadbf9f parent: 87813:78efa2c06447 user: Charles-Fran?ois Natali date: Sat Dec 07 20:30:41 2013 +0100 summary: Merge. files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 7 23:39:42 2013 From: python-checkins at python.org (christian.heimes) Date: Sat, 7 Dec 2013 23:39:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319922=3A_define_?= =?utf-8?q?=5FINCLUDE=5F=5FSTDC=5FA1=5FSOURCE_in_HP-UX_to_include_mbstate?= =?utf-8?q?=5Ft?= Message-ID: <3dcQb23ktbz7Ljq@mail.python.org> http://hg.python.org/cpython/rev/d8cfc7106f41 changeset: 87815:d8cfc7106f41 user: Christian Heimes date: Sat Dec 07 23:39:33 2013 +0100 summary: Issue #19922: define _INCLUDE__STDC_A1_SOURCE in HP-UX to include mbstate_t for mbrtowc(). files: Misc/NEWS | 3 +++ configure | 9 +++++++++ configure.ac | 7 +++++++ pyconfig.h.in | 3 +++ 4 files changed, 22 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -108,6 +108,9 @@ Build ----- +- Issue #19922: define _INCLUDE__STDC_A1_SOURCE in HP-UX to include mbstate_t + for mbrtowc(). + - Issue #19788: kill_python(_d).exe is now run as a PreBuildEvent on the pythoncore sub-project. This should prevent build errors due a previous build's python(_d).exe still running. diff --git a/configure b/configure --- a/configure +++ b/configure @@ -3445,6 +3445,15 @@ fi +# On HP-UX mbstate_t requires _INCLUDE__STDC_A1_SOURCE +case $ac_sys_system in +hp*|HP*) + +$as_echo "#define _INCLUDE__STDC_A1_SOURCE 1" >>confdefs.h + + ;; +esac + # # SGI compilers allow the specification of the both the ABI and the # ISA on the command line. Depending on the values of these switches, diff --git a/configure.ac b/configure.ac --- a/configure.ac +++ b/configure.ac @@ -522,6 +522,13 @@ AC_DEFINE(_POSIX_C_SOURCE, 200809L, Define to activate features from IEEE Stds 1003.1-2008) fi +# On HP-UX mbstate_t requires _INCLUDE__STDC_A1_SOURCE +case $ac_sys_system in +hp*|HP*) + AC_DEFINE(_INCLUDE__STDC_A1_SOURCE, 1, Define to include mbstate_t for mbrtowc) + ;; +esac + # # SGI compilers allow the specification of the both the ABI and the # ISA on the command line. Depending on the values of these switches, diff --git a/pyconfig.h.in b/pyconfig.h.in --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -1382,6 +1382,9 @@ /* Define on Linux to activate all library features */ #undef _GNU_SOURCE +/* Define to include mbstate_t for mbrtowc */ +#undef _INCLUDE__STDC_A1_SOURCE + /* This must be defined on some systems to enable large file support. */ #undef _LARGEFILE_SOURCE -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 00:57:15 2013 From: python-checkins at python.org (guido.van.rossum) Date: Sun, 8 Dec 2013 00:57:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Silently_ignore_unregister?= =?utf-8?q?ing_closed_files=2E_Fixes_issue_19876=2E_With_docs_and?= Message-ID: <3dcSJW5MvXz7Ljq@mail.python.org> http://hg.python.org/cpython/rev/39e7995f9ad1 changeset: 87816:39e7995f9ad1 user: Guido van Rossum date: Sat Dec 07 15:57:01 2013 -0800 summary: Silently ignore unregistering closed files. Fixes issue 19876. With docs and slight test refactor. files: Doc/library/selectors.rst | 7 +- Lib/selectors.py | 74 +++++++++++++++++-- Lib/test/test_selectors.py | 96 +++++++++++++++---------- 3 files changed, 129 insertions(+), 48 deletions(-) diff --git a/Doc/library/selectors.rst b/Doc/library/selectors.rst --- a/Doc/library/selectors.rst +++ b/Doc/library/selectors.rst @@ -102,7 +102,8 @@ Register a file object for selection, monitoring it for I/O events. - *fileobj* is the file object to monitor. + *fileobj* is the file object to monitor. It may either be an integer + file descriptor or an object with a ``fileno()`` method. *events* is a bitwise mask of events to monitor. *data* is an opaque object. @@ -118,7 +119,9 @@ *fileobj* must be a file object previously registered. This returns the associated :class:`SelectorKey` instance, or raises a - :exc:`KeyError` if the file object is not registered. + :exc:`KeyError` if *fileobj* is not registered. It will raise + :exc:`ValueError` if *fileobj* is invalid (e.g. it has no ``fileno()`` + method or its ``fileno()`` method has an invalid return value). .. method:: modify(fileobj, events, data=None) diff --git a/Lib/selectors.py b/Lib/selectors.py --- a/Lib/selectors.py +++ b/Lib/selectors.py @@ -25,6 +25,9 @@ Returns: corresponding file descriptor + + Raises: + ValueError if the object is invalid """ if isinstance(fileobj, int): fd = fileobj @@ -55,7 +58,8 @@ def __getitem__(self, fileobj): try: - return self._selector._fd_to_key[_fileobj_to_fd(fileobj)] + fd = self._selector._fileobj_lookup(fileobj) + return self._selector._fd_to_key[fd] except KeyError: raise KeyError("{!r} is not registered".format(fileobj)) from None @@ -89,6 +93,15 @@ Returns: SelectorKey instance + + Raises: + ValueError if events is invalid + KeyError if fileobj is already registered + OSError if fileobj is closed or otherwise is unacceptable to + the underlying system call (if a system call is made) + + Note: + OSError may or may not be raised """ raise NotImplementedError @@ -101,6 +114,13 @@ Returns: SelectorKey instance + + Raises: + KeyError if fileobj is not registered + + Note: + If fileobj is registered but has since been closed this does + *not* raise OSError (even if the wrapped syscall does) """ raise NotImplementedError @@ -114,6 +134,9 @@ Returns: SelectorKey instance + + Raises: + Anything that unregister() or register() raises """ self.unregister(fileobj) return self.register(fileobj, events, data) @@ -177,22 +200,41 @@ # read-only mapping returned by get_map() self._map = _SelectorMapping(self) + def _fileobj_lookup(self, fileobj): + """Return a file descriptor from a file object. + + This wraps _fileobj_to_fd() to do an exhaustive search in case + the object is invalid but we still have it in our map. This + is used by unregister() so we can unregister an object that + was previously registered even if it is closed. It is also + used by _SelectorMapping. + """ + try: + return _fileobj_to_fd(fileobj) + except ValueError: + # Do an exhaustive search. + for key in self._fd_to_key.values(): + if key.fileobj is fileobj: + return key.fd + # Raise ValueError after all. + raise + def register(self, fileobj, events, data=None): if (not events) or (events & ~(EVENT_READ | EVENT_WRITE)): raise ValueError("Invalid events: {!r}".format(events)) - key = SelectorKey(fileobj, _fileobj_to_fd(fileobj), events, data) + key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data) if key.fd in self._fd_to_key: - raise KeyError("{!r} (FD {}) is already " - "registered".format(fileobj, key.fd)) + raise KeyError("{!r} (FD {}) is already registered" + .format(fileobj, key.fd)) self._fd_to_key[key.fd] = key return key def unregister(self, fileobj): try: - key = self._fd_to_key.pop(_fileobj_to_fd(fileobj)) + key = self._fd_to_key.pop(self._fileobj_lookup(fileobj)) except KeyError: raise KeyError("{!r} is not registered".format(fileobj)) from None return key @@ -200,7 +242,7 @@ def modify(self, fileobj, events, data=None): # TODO: Subclasses can probably optimize this even further. try: - key = self._fd_to_key[_fileobj_to_fd(fileobj)] + key = self._fd_to_key[self._fileobj_lookup(fileobj)] except KeyError: raise KeyError("{!r} is not registered".format(fileobj)) from None if events != key.events: @@ -352,7 +394,12 @@ def unregister(self, fileobj): key = super().unregister(fileobj) - self._epoll.unregister(key.fd) + try: + self._epoll.unregister(key.fd) + except OSError: + # This can happen if the FD was closed since it + # was registered. + pass return key def select(self, timeout=None): @@ -409,11 +456,20 @@ if key.events & EVENT_READ: kev = select.kevent(key.fd, select.KQ_FILTER_READ, select.KQ_EV_DELETE) - self._kqueue.control([kev], 0, 0) + try: + self._kqueue.control([kev], 0, 0) + except OSError: + # This can happen if the FD was closed since it + # was registered. + pass if key.events & EVENT_WRITE: kev = select.kevent(key.fd, select.KQ_FILTER_WRITE, select.KQ_EV_DELETE) - self._kqueue.control([kev], 0, 0) + try: + self._kqueue.control([kev], 0, 0) + except OSError: + # See comment above. + pass return key def select(self, timeout=None): diff --git a/Lib/test/test_selectors.py b/Lib/test/test_selectors.py --- a/Lib/test/test_selectors.py +++ b/Lib/test/test_selectors.py @@ -1,4 +1,5 @@ import errno +import os import random import selectors import signal @@ -49,13 +50,17 @@ class BaseSelectorTestCase(unittest.TestCase): + def make_socketpair(self): + rd, wr = socketpair() + self.addCleanup(rd.close) + self.addCleanup(wr.close) + return rd, wr + def test_register(self): s = self.SELECTOR() self.addCleanup(s.close) - rd, wr = socketpair() - self.addCleanup(rd.close) - self.addCleanup(wr.close) + rd, wr = self.make_socketpair() key = s.register(rd, selectors.EVENT_READ, "data") self.assertIsInstance(key, selectors.SelectorKey) @@ -81,9 +86,7 @@ s = self.SELECTOR() self.addCleanup(s.close) - rd, wr = socketpair() - self.addCleanup(rd.close) - self.addCleanup(wr.close) + rd, wr = self.make_socketpair() s.register(rd, selectors.EVENT_READ) s.unregister(rd) @@ -94,13 +97,51 @@ # unregister twice self.assertRaises(KeyError, s.unregister, rd) + def test_unregister_after_fd_close(self): + s = self.SELECTOR() + self.addCleanup(s.close) + rd, wr = self.make_socketpair() + r, w = rd.fileno(), wr.fileno() + s.register(r, selectors.EVENT_READ) + s.register(w, selectors.EVENT_WRITE) + rd.close() + wr.close() + s.unregister(r) + s.unregister(w) + + def test_unregister_after_fd_close_and_reuse(self): + s = self.SELECTOR() + self.addCleanup(s.close) + rd, wr = self.make_socketpair() + r, w = rd.fileno(), wr.fileno() + s.register(r, selectors.EVENT_READ) + s.register(w, selectors.EVENT_WRITE) + rd2, wr2 = self.make_socketpair() + rd.close() + wr.close() + os.dup2(rd2.fileno(), r) + os.dup2(wr2.fileno(), w) + self.addCleanup(os.close, r) + self.addCleanup(os.close, w) + s.unregister(r) + s.unregister(w) + + def test_unregister_after_socket_close(self): + s = self.SELECTOR() + self.addCleanup(s.close) + rd, wr = self.make_socketpair() + s.register(rd, selectors.EVENT_READ) + s.register(wr, selectors.EVENT_WRITE) + rd.close() + wr.close() + s.unregister(rd) + s.unregister(wr) + def test_modify(self): s = self.SELECTOR() self.addCleanup(s.close) - rd, wr = socketpair() - self.addCleanup(rd.close) - self.addCleanup(wr.close) + rd, wr = self.make_socketpair() key = s.register(rd, selectors.EVENT_READ) @@ -138,9 +179,7 @@ s = self.SELECTOR() self.addCleanup(s.close) - rd, wr = socketpair() - self.addCleanup(rd.close) - self.addCleanup(wr.close) + rd, wr = self.make_socketpair() s.register(rd, selectors.EVENT_READ) s.register(wr, selectors.EVENT_WRITE) @@ -153,9 +192,7 @@ s = self.SELECTOR() self.addCleanup(s.close) - rd, wr = socketpair() - self.addCleanup(rd.close) - self.addCleanup(wr.close) + rd, wr = self.make_socketpair() key = s.register(rd, selectors.EVENT_READ, "data") self.assertEqual(key, s.get_key(rd)) @@ -167,9 +204,7 @@ s = self.SELECTOR() self.addCleanup(s.close) - rd, wr = socketpair() - self.addCleanup(rd.close) - self.addCleanup(wr.close) + rd, wr = self.make_socketpair() keys = s.get_map() self.assertFalse(keys) @@ -194,9 +229,7 @@ s = self.SELECTOR() self.addCleanup(s.close) - rd, wr = socketpair() - self.addCleanup(rd.close) - self.addCleanup(wr.close) + rd, wr = self.make_socketpair() s.register(rd, selectors.EVENT_READ) wr_key = s.register(wr, selectors.EVENT_WRITE) @@ -214,9 +247,7 @@ s = self.SELECTOR() self.addCleanup(s.close) - rd, wr = socketpair() - self.addCleanup(rd.close) - self.addCleanup(wr.close) + rd, wr = self.make_socketpair() with s as sel: sel.register(rd, selectors.EVENT_READ) @@ -247,9 +278,7 @@ w2r = {} for i in range(NUM_SOCKETS): - rd, wr = socketpair() - self.addCleanup(rd.close) - self.addCleanup(wr.close) + rd, wr = self.make_socketpair() s.register(rd, selectors.EVENT_READ) s.register(wr, selectors.EVENT_WRITE) readers.append(rd) @@ -293,9 +322,7 @@ s = self.SELECTOR() self.addCleanup(s.close) - rd, wr = socketpair() - self.addCleanup(rd.close) - self.addCleanup(wr.close) + rd, wr = self.make_socketpair() s.register(wr, selectors.EVENT_WRITE) t = time() @@ -322,9 +349,7 @@ s = self.SELECTOR() self.addCleanup(s.close) - rd, wr = socketpair() - self.addCleanup(rd.close) - self.addCleanup(wr.close) + rd, wr = self.make_socketpair() orig_alrm_handler = signal.signal(signal.SIGALRM, lambda *args: None) self.addCleanup(signal.signal, signal.SIGALRM, orig_alrm_handler) @@ -364,16 +389,13 @@ for i in range(NUM_FDS // 2): try: - rd, wr = socketpair() + rd, wr = self.make_socketpair() except OSError: # too many FDs, skip - note that we should only catch EMFILE # here, but apparently *BSD and Solaris can fail upon connect() # or bind() with EADDRNOTAVAIL, so let's be safe self.skipTest("FD limit reached") - self.addCleanup(rd.close) - self.addCleanup(wr.close) - try: s.register(rd, selectors.EVENT_READ) s.register(wr, selectors.EVENT_WRITE) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 01:03:47 2013 From: python-checkins at python.org (guido.van.rossum) Date: Sun, 8 Dec 2013 01:03:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_News_item_for_issue_19876?= =?utf-8?q?=2E?= Message-ID: <3dcSS300rZz7Ljq@mail.python.org> http://hg.python.org/cpython/rev/f334dd2471e7 changeset: 87817:f334dd2471e7 user: Guido van Rossum date: Sat Dec 07 16:03:36 2013 -0800 summary: News item for issue 19876. files: Misc/NEWS | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,9 @@ Library ------- +- Issue #19876: selectors unregister() no longer raises ValueError or OSError + if the FD is closed (as long as it was registered). + - Issue #19908: pathlib now joins relative Windows paths correctly when a drive is present. Original patch by Antoine Pitrou. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 03:37:56 2013 From: python-checkins at python.org (eric.snow) Date: Sun, 8 Dec 2013 03:37:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319758=3A_silence_?= =?utf-8?q?PendingDeprecationWarnings_in_test=5Fimportlib=2E?= Message-ID: <3dcWsw5GQYz7LjQ@mail.python.org> http://hg.python.org/cpython/rev/94593dcb195a changeset: 87818:94593dcb195a user: Eric Snow date: Sat Dec 07 19:37:31 2013 -0700 summary: Issue #19758: silence PendingDeprecationWarnings in test_importlib. files: Lib/test/test_importlib/test_spec.py | 12 ++++++++---- 1 files changed, 8 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_importlib/test_spec.py b/Lib/test/test_importlib/test_spec.py --- a/Lib/test/test_importlib/test_spec.py +++ b/Lib/test/test_importlib/test_spec.py @@ -10,6 +10,7 @@ from test.support import CleanImport import unittest import sys +import warnings @@ -55,10 +56,13 @@ HAM = -1 - @frozen_util.module_for_loader - def load_module(self, module): - module.ham = self.HAM - return module + with warnings.catch_warnings(): + warnings.simplefilter("ignore", PendingDeprecationWarning) + + @frozen_util.module_for_loader + def load_module(self, module): + module.ham = self.HAM + return module class ModuleSpecTests: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 04:15:15 2013 From: python-checkins at python.org (gregory.p.smith) Date: Sun, 8 Dec 2013 04:15:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Fixes_issue_?= =?utf-8?q?=2319506=3A_Use_a_memoryview_to_avoid_a_data_copy_when_piping_d?= =?utf-8?q?ata?= Message-ID: <3dcXhz5zbnz7Lln@mail.python.org> http://hg.python.org/cpython/rev/44948f5bdc12 changeset: 87819:44948f5bdc12 branch: 3.3 parent: 87813:78efa2c06447 user: Gregory P. Smith date: Sat Dec 07 19:12:46 2013 -0800 summary: Fixes issue #19506: Use a memoryview to avoid a data copy when piping data to stdin within subprocess.Popen.communicate. 5-10% less cpu usage. files: Lib/subprocess.py | 7 +++++-- Misc/NEWS | 3 +++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -1621,6 +1621,9 @@ self._save_input(input) + if self._input: + input_view = memoryview(self._input) + while self._fd2file: timeout = self._remaining_time(endtime) if timeout is not None and timeout < 0: @@ -1638,8 +1641,8 @@ for fd, mode in ready: if mode & select.POLLOUT: - chunk = self._input[self._input_offset : - self._input_offset + _PIPE_BUF] + chunk = input_view[self._input_offset : + self._input_offset + _PIPE_BUF] try: self._input_offset += os.write(fd, chunk) except OSError as e: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,9 @@ Library ------- +- Issue #19506: Use a memoryview to avoid a data copy when piping data + to stdin within subprocess.Popen.communicate. 5-10% less cpu usage. + - Issue #19839: Fix regression in bz2 module's handling of non-bzip2 data at EOF, and analogous bug in lzma module. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 04:15:17 2013 From: python-checkins at python.org (gregory.p.smith) Date: Sun, 8 Dec 2013 04:15:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Fixes_issue_=2319506=3A_Use_a_memoryview_to_avoid_a_data?= =?utf-8?q?_copy_when_piping_data?= Message-ID: <3dcXj10pqwz7LlX@mail.python.org> http://hg.python.org/cpython/rev/5379bba2fb21 changeset: 87820:5379bba2fb21 parent: 87818:94593dcb195a parent: 87819:44948f5bdc12 user: Gregory P. Smith date: Sat Dec 07 19:14:59 2013 -0800 summary: Fixes issue #19506: Use a memoryview to avoid a data copy when piping data to stdin within subprocess.Popen.communicate. 5-10% less cpu usage. files: Lib/subprocess.py | 7 +++++-- Misc/NEWS | 3 +++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -1562,6 +1562,9 @@ self._save_input(input) + if self._input: + input_view = memoryview(self._input) + with _PopenSelector() as selector: if self.stdin and input: selector.register(self.stdin, selectors.EVENT_WRITE) @@ -1583,8 +1586,8 @@ for key, events in ready: if key.fileobj is self.stdin: - chunk = self._input[self._input_offset : - self._input_offset + _PIPE_BUF] + chunk = input_view[self._input_offset : + self._input_offset + _PIPE_BUF] try: self._input_offset += os.write(key.fd, chunk) except OSError as e: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,9 @@ Library ------- +- Issue #19506: Use a memoryview to avoid a data copy when piping data + to stdin within subprocess.Popen.communicate. 5-10% less cpu usage. + - Issue #19876: selectors unregister() no longer raises ValueError or OSError if the FD is closed (as long as it was registered). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 07:45:26 2013 From: python-checkins at python.org (zach.ware) Date: Sun, 8 Dec 2013 07:45:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgMTk1NzI6?= =?utf-8?q?_More_silently_skipped_tests_explicitly_skipped=2E?= Message-ID: <3dcdMV4TG3z7LkB@mail.python.org> http://hg.python.org/cpython/rev/3283fb24106d changeset: 87821:3283fb24106d branch: 3.3 parent: 87819:44948f5bdc12 user: Zachary Ware date: Sun Dec 08 00:20:35 2013 -0600 summary: Issue 19572: More silently skipped tests explicitly skipped. files: Lib/test/datetimetester.py | 1 + Lib/test/multibytecodec_support.py | 4 +- Lib/test/string_tests.py | 4 +- Lib/test/test_array.py | 2 +- Lib/test/test_codecencodings_iso2022.py | 1 + Lib/test/test_configparser.py | 3 +- Lib/test/test_decimal.py | 19 ++--- Lib/test/test_dis.py | 14 ++- Lib/test/test_fileio.py | 3 +- Lib/test/test_float.py | 2 +- Lib/test/test_functools.py | 10 +- Lib/test/test_grp.py | 4 +- Lib/test/test_imp.py | 2 +- Lib/test/test_io.py | 12 +-- Lib/test/test_memoryview.py | 14 ++-- Lib/test/test_multiprocessing.py | 22 +++--- Lib/test/test_nis.py | 6 +- Lib/test/test_ntpath.py | 2 +- Lib/test/test_os.py | 4 +- Lib/test/test_pwd.py | 17 +++- Lib/test/test_reprlib.py | 3 + Lib/test/test_shutil.py | 4 +- Lib/test/test_site.py | 1 + Lib/test/test_socket.py | 14 ++-- Lib/test/test_strptime.py | 6 +- Lib/test/test_tempfile.py | 12 +-- Lib/test/test_thread.py | 42 +++++------- Lib/test/test_time.py | 3 +- Lib/test/test_unicode.py | 10 +- Lib/test/test_urllibnet.py | 7 +- Lib/test/test_warnings.py | 2 +- Lib/test/test_xmlrpc_net.py | 2 - Lib/test/test_zipimport.py | 2 +- Misc/NEWS | 2 +- 34 files changed, 122 insertions(+), 134 deletions(-) diff --git a/Lib/test/datetimetester.py b/Lib/test/datetimetester.py --- a/Lib/test/datetimetester.py +++ b/Lib/test/datetimetester.py @@ -2025,6 +2025,7 @@ class TestSubclassDateTime(TestDateTime): theclass = SubclassDatetime # Override tests not designed for subclass + @unittest.skip('not appropriate for subclasses') def test_roundtrip(self): pass diff --git a/Lib/test/multibytecodec_support.py b/Lib/test/multibytecodec_support.py --- a/Lib/test/multibytecodec_support.py +++ b/Lib/test/multibytecodec_support.py @@ -73,7 +73,7 @@ def test_xmlcharrefreplace(self): if self.has_iso10646: - return + self.skipTest('encoding contains full ISO 10646 map') s = "\u0b13\u0b23\u0b60 nd eggs" self.assertEqual( @@ -83,7 +83,7 @@ def test_customreplace_encode(self): if self.has_iso10646: - return + self.skipTest('encoding contains full ISO 10646 map') from html.entities import codepoint2name diff --git a/Lib/test/string_tests.py b/Lib/test/string_tests.py --- a/Lib/test/string_tests.py +++ b/Lib/test/string_tests.py @@ -663,10 +663,10 @@ self.checkraises(TypeError, 'hello', 'replace', 42, 'h') self.checkraises(TypeError, 'hello', 'replace', 'h', 42) + @unittest.skipIf(sys.maxsize > (1 << 32) or struct.calcsize('P') != 4, + 'only applies to 32-bit platforms') def test_replace_overflow(self): # Check for overflow checking on 32 bit machines - if sys.maxsize != 2147483647 or struct.calcsize("P") > 4: - return A2_16 = "A" * (2**16) self.checkraises(OverflowError, A2_16, "replace", "", A2_16) self.checkraises(OverflowError, A2_16, "replace", "A", A2_16) diff --git a/Lib/test/test_array.py b/Lib/test/test_array.py --- a/Lib/test/test_array.py +++ b/Lib/test/test_array.py @@ -946,7 +946,7 @@ try: import gc except ImportError: - return + self.skipTest('gc module not available') a = array.array(self.typecode) l = [iter(a)] l.append(l) diff --git a/Lib/test/test_codecencodings_iso2022.py b/Lib/test/test_codecencodings_iso2022.py --- a/Lib/test/test_codecencodings_iso2022.py +++ b/Lib/test/test_codecencodings_iso2022.py @@ -36,6 +36,7 @@ # iso2022_kr.txt cannot be used to test "chunk coding": the escape # sequence is only written on the first line + @unittest.skip('iso2022_kr.txt cannot be used to test "chunk coding"') def test_chunkcoding(self): pass diff --git a/Lib/test/test_configparser.py b/Lib/test/test_configparser.py --- a/Lib/test/test_configparser.py +++ b/Lib/test/test_configparser.py @@ -707,8 +707,7 @@ def test_read_returns_file_list(self): if self.delimiters[0] != '=': - # skip reading the file if we're using an incompatible format - return + self.skipTest('incompatible format') file1 = support.findfile("cfgparser.1") # check when we pass a mix of readable and non-readable files: cf = self.newconfig() diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -290,7 +290,6 @@ global skip_expected if skip_expected: raise unittest.SkipTest - return with open(file) as f: for line in f: line = line.replace('\r\n', '').replace('\n', '') @@ -301,7 +300,6 @@ #Exception raised where there shouldn't have been one. self.fail('Exception "'+exception.__class__.__name__ + '" raised on line '+line) - return def eval_line(self, s): if s.find(' -> ') >= 0 and s[:2] != '--' and not s.startswith(' --'): @@ -461,7 +459,6 @@ self.assertEqual(myexceptions, theirexceptions, 'Incorrect flags set in ' + s + ' -- got ' + str(myexceptions)) - return def getexceptions(self): return [e for e in Signals[self.decimal] if self.context.flags[e]] @@ -1073,7 +1070,7 @@ try: from locale import CHAR_MAX except ImportError: - return + self.skipTest('locale.CHAR_MAX not available') def make_grouping(lst): return ''.join([chr(x) for x in lst]) if self.decimal == C else lst @@ -1164,8 +1161,12 @@ decimal_point = locale.localeconv()['decimal_point'] thousands_sep = locale.localeconv()['thousands_sep'] - if decimal_point != '\u066b' or thousands_sep != '\u066c': - return + if decimal_point != '\u066b': + self.skipTest('inappropriate decimal point separator' + '({!r} not {!r})'.format(decimal_point, '\u066b')) + if thousands_sep != '\u066c': + self.skipTest('inappropriate thousands separator' + '({!r} not {!r})'.format(thousands_sep, '\u066c')) self.assertEqual(format(Decimal('100000000.123'), 'n'), '100\u066c000\u066c000\u066b123') @@ -1515,7 +1516,6 @@ cls.assertTrue(c1.flags[Inexact]) for sig in Overflow, Underflow, DivisionByZero, InvalidOperation: cls.assertFalse(c1.flags[sig]) - return def thfunc2(cls): Decimal = cls.decimal.Decimal @@ -1560,7 +1560,6 @@ cls.assertTrue(thiscontext.flags[Inexact]) for sig in Overflow, Underflow, DivisionByZero, InvalidOperation: cls.assertFalse(thiscontext.flags[sig]) - return class ThreadingTest(unittest.TestCase): '''Unit tests for thread local contexts in Decimal.''' @@ -1602,7 +1601,6 @@ DefaultContext.prec = save_prec DefaultContext.Emax = save_emax DefaultContext.Emin = save_emin - return @unittest.skipUnless(threading, 'threading required') class CThreadingTest(ThreadingTest): @@ -4525,7 +4523,6 @@ self.assertEqual(d1._sign, b1._sign) self.assertEqual(d1._int, b1._int) self.assertEqual(d1._exp, b1._exp) - return Decimal(d1) self.assertEqual(d1._sign, b1._sign) @@ -5271,7 +5268,7 @@ try: from locale import CHAR_MAX except ImportError: - return + self.skipTest('locale.CHAR_MAX not available') def make_grouping(lst): return ''.join([chr(x) for x in lst]) diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py --- a/Lib/test/test_dis.py +++ b/Lib/test/test_dis.py @@ -217,16 +217,18 @@ def test_bug_708901(self): self.do_disassembly_test(bug708901, dis_bug708901) + # Test has been disabled due to change in the way + # list comps are handled. The byte code now includes + # a memory address and a file location, so they change from + # run to run. + @unittest.skip('disabled due to a change in the way list comps are handled') def test_bug_1333982(self): # XXX: re-enable this test! # This one is checking bytecodes generated for an `assert` statement, # so fails if the tests are run with -O. Skip this test then. - pass # Test has been disabled due to change in the way - # list comps are handled. The byte code now includes - # a memory address and a file location, so they change from - # run to run. - # if __debug__: - # self.do_disassembly_test(bug1333982, dis_bug1333982) + + if __debug__: + self.do_disassembly_test(bug1333982, dis_bug1333982) def test_big_linenos(self): def func(count): diff --git a/Lib/test/test_fileio.py b/Lib/test/test_fileio.py --- a/Lib/test/test_fileio.py +++ b/Lib/test/test_fileio.py @@ -341,8 +341,7 @@ try: fn = TESTFN.encode("ascii") except UnicodeEncodeError: - # Skip test - return + self.skipTest('could not encode %r to ascii' % TESTFN) f = _FileIO(fn, "w") try: f.write(b"abc") diff --git a/Lib/test/test_float.py b/Lib/test/test_float.py --- a/Lib/test/test_float.py +++ b/Lib/test/test_float.py @@ -70,7 +70,7 @@ # it still has to accept the normal python syntax import locale if not locale.localeconv()['decimal_point'] == ',': - return + self.skipTest('decimal_point is not ","') self.assertEqual(float(" 3.14 "), 3.14) self.assertEqual(float("+3.14 "), 3.14) diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -45,8 +45,6 @@ self.assertEqual(p.args, (1, 2)) self.assertEqual(p.keywords, dict(a=10, b=20)) # attributes should not be writable - if not isinstance(self.thetype, type): - return self.assertRaises(AttributeError, setattr, p, 'func', map) self.assertRaises(AttributeError, setattr, p, 'args', (1, 2)) self.assertRaises(AttributeError, setattr, p, 'keywords', dict(a=1, b=2)) @@ -210,11 +208,13 @@ thetype = PythonPartial # the python version hasn't a nice repr - def test_repr(self): pass + test_repr = None # the python version isn't picklable - def test_pickle(self): pass - def test_setstate_refcount(self): pass + test_pickle = test_setstate_refcount = None + + # the python version isn't a type + test_attributes = None class TestUpdateWrapper(unittest.TestCase): diff --git a/Lib/test/test_grp.py b/Lib/test/test_grp.py --- a/Lib/test/test_grp.py +++ b/Lib/test/test_grp.py @@ -26,8 +26,10 @@ for e in entries: self.check_value(e) + def test_values_extended(self): + entries = grp.getgrall() if len(entries) > 1000: # Huge group file (NIS?) -- skip the rest - return + self.skipTest('huge group file, extended test skipped') for e in entries: e2 = grp.getgrgid(e.gr_gid) diff --git a/Lib/test/test_imp.py b/Lib/test/test_imp.py --- a/Lib/test/test_imp.py +++ b/Lib/test/test_imp.py @@ -245,7 +245,7 @@ if found[0] is not None: found[0].close() if found[2][2] != imp.C_EXTENSION: - return + self.skipTest("found module doesn't appear to be a C extension") imp.load_module(name, None, *found[1:]) def test_multiple_calls_to_get_data(self): diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -421,14 +421,9 @@ # a long time to build the >2GB file and takes >2GB of disk space # therefore the resource must be enabled to run this test. if sys.platform[:3] == 'win' or sys.platform == 'darwin': - if not support.is_resource_enabled("largefile"): - print("\nTesting large file ops skipped on %s." % sys.platform, - file=sys.stderr) - print("It requires %d bytes and a long time." % self.LARGE, - file=sys.stderr) - print("Use 'regrtest.py -u largefile test_io' to run it.", - file=sys.stderr) - return + support.requires( + 'largefile', + 'test requires %s bytes and a long time to run' % self.LARGE) with self.open(support.TESTFN, "w+b", 0) as f: self.large_file_ops(f) with self.open(support.TESTFN, "w+b") as f: @@ -698,6 +693,7 @@ self.assertEqual(42, bufio.fileno()) + @unittest.skip('test having existential crisis') def test_no_fileno(self): # XXX will we always have fileno() function? If so, kill # this test. Else, write it. diff --git a/Lib/test/test_memoryview.py b/Lib/test/test_memoryview.py --- a/Lib/test/test_memoryview.py +++ b/Lib/test/test_memoryview.py @@ -57,7 +57,7 @@ def test_setitem_readonly(self): if not self.ro_type: - return + self.skipTest("no read-only type to test") b = self.ro_type(self._source) oldrefcount = sys.getrefcount(b) m = self._view(b) @@ -71,7 +71,7 @@ def test_setitem_writable(self): if not self.rw_type: - return + self.skipTest("no writable type to test") tp = self.rw_type b = self.rw_type(self._source) oldrefcount = sys.getrefcount(b) @@ -189,13 +189,13 @@ def test_attributes_readonly(self): if not self.ro_type: - return + self.skipTest("no read-only type to test") m = self.check_attributes_with_type(self.ro_type) self.assertEqual(m.readonly, True) def test_attributes_writable(self): if not self.rw_type: - return + self.skipTest("no writable type to test") m = self.check_attributes_with_type(self.rw_type) self.assertEqual(m.readonly, False) @@ -301,7 +301,7 @@ # buffer as writable causing a segfault if using mmap tp = self.ro_type if tp is None: - return + self.skipTest("no read-only type to test") b = tp(self._source) m = self._view(b) i = io.BytesIO(b'ZZZZ') @@ -370,12 +370,12 @@ itemsize = array.array('i').itemsize format = 'i' + @unittest.skip('XXX test should be adapted for non-byte buffers') def test_getbuffer(self): - # XXX Test should be adapted for non-byte buffers pass + @unittest.skip('XXX NotImplementedError: tolist() only supports byte views') def test_tolist(self): - # XXX NotImplementedError: tolist() only supports byte views pass diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -195,7 +195,7 @@ def test_current(self): if self.TYPE == 'threads': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) current = self.current_process() authkey = current.authkey @@ -209,7 +209,7 @@ def test_daemon_argument(self): if self.TYPE == "threads": - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) # By default uses the current process's daemon flag. proc0 = self.Process(target=self._test) @@ -274,7 +274,7 @@ def test_terminate(self): if self.TYPE == 'threads': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) p = self.Process(target=self._test_terminate) p.daemon = True @@ -378,7 +378,7 @@ def test_sentinel(self): if self.TYPE == "threads": - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) event = self.Event() p = self.Process(target=self._test_sentinel, args=(event,)) with self.assertRaises(ValueError): @@ -434,7 +434,7 @@ def test_stderr_flush(self): # sys.stderr is flushed at process shutdown (issue #13812) if self.TYPE == "threads": - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) testfn = test.support.TESTFN self.addCleanup(test.support.unlink, testfn) @@ -462,7 +462,7 @@ def test_sys_exit(self): # See Issue 13854 if self.TYPE == 'threads': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) testfn = test.support.TESTFN self.addCleanup(test.support.unlink, testfn) @@ -671,7 +671,7 @@ try: self.assertEqual(q.qsize(), 0) except NotImplementedError: - return + self.skipTest('qsize method not implemented') q.put(1) self.assertEqual(q.qsize(), 1) q.put(5) @@ -779,7 +779,7 @@ def test_timeout(self): if self.TYPE != 'processes': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) sem = self.Semaphore(0) acquire = TimingWrapper(sem.acquire) @@ -1399,7 +1399,7 @@ def test_thousand(self): if self.TYPE == 'manager': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) passes = 1000 lock = self.Lock() conn, child_conn = self.Pipe(False) @@ -1694,7 +1694,7 @@ def test_map_unplicklable(self): # Issue #19425 -- failure to pickle should not cause a hang if self.TYPE == 'threads': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) class A(object): def __reduce__(self): raise RuntimeError('cannot pickle') @@ -2188,7 +2188,7 @@ def test_sendbytes(self): if self.TYPE != 'processes': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) msg = latin('abcdefghijklmnopqrstuvwxyz') a, b = self.Pipe() diff --git a/Lib/test/test_nis.py b/Lib/test/test_nis.py --- a/Lib/test/test_nis.py +++ b/Lib/test/test_nis.py @@ -12,11 +12,7 @@ maps = nis.maps() except nis.error as msg: # NIS is probably not active, so this test isn't useful - if support.verbose: - print("Test Skipped:", msg) - # Can't raise SkipTest as regrtest only recognizes the exception - # import time. - return + self.skipTest(str(msg)) try: # On some systems, this map is only accessible to the # super user diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py --- a/Lib/test/test_ntpath.py +++ b/Lib/test/test_ntpath.py @@ -218,7 +218,7 @@ import nt tester('ntpath.abspath("C:\\")', "C:\\") except ImportError: - pass + self.skipTest('nt module not available') def test_relpath(self): currentdir = os.path.split(os.getcwd())[-1] diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -263,7 +263,7 @@ except OSError as e: # On AtheOS, glibc always returns ENOSYS if e.errno == errno.ENOSYS: - return + self.skipTest('glibc always returns ENOSYS on AtheOS') # Make sure direct access works self.assertEqual(result.f_bfree, result[3]) @@ -480,7 +480,7 @@ os.stat(r"c:\pagefile.sys") except WindowsError as e: if e.errno == 2: # file does not exist; cannot run test - return + self.skipTest(r'c:\pagefile.sys does not exist') self.fail("Could not stat pagefile.sys") @unittest.skipUnless(sys.platform == "win32", "Win32 specific tests") diff --git a/Lib/test/test_pwd.py b/Lib/test/test_pwd.py --- a/Lib/test/test_pwd.py +++ b/Lib/test/test_pwd.py @@ -8,8 +8,6 @@ def test_values(self): entries = pwd.getpwall() - entriesbyname = {} - entriesbyuid = {} for e in entries: self.assertEqual(len(e), 7) @@ -32,13 +30,20 @@ # for one uid # self.assertEqual(pwd.getpwuid(e.pw_uid), e) # instead of this collect all entries for one uid - # and check afterwards + # and check afterwards (done in test_values_extended) + + def test_values_extended(self): + entries = pwd.getpwall() + entriesbyname = {} + entriesbyuid = {} + + if len(entries) > 1000: # Huge passwd file (NIS?) -- skip this test + self.skipTest('passwd file is huge; extended test skipped') + + for e in entries: entriesbyname.setdefault(e.pw_name, []).append(e) entriesbyuid.setdefault(e.pw_uid, []).append(e) - if len(entries) > 1000: # Huge passwd file (NIS?) -- skip the rest - return - # check whether the entry returned by getpwuid() # for each uid is among those from getpwall() for this uid for e in entries: diff --git a/Lib/test/test_reprlib.py b/Lib/test/test_reprlib.py --- a/Lib/test/test_reprlib.py +++ b/Lib/test/test_reprlib.py @@ -166,6 +166,7 @@ eq(r([[[[[[{}]]]]]]), "[[[[[[{}]]]]]]") eq(r([[[[[[[{}]]]]]]]), "[[[[[[[...]]]]]]]") + @unittest.skip('hard to catch a cell object') def test_cell(self): # XXX Hmm? How to get at a cell object? pass @@ -272,6 +273,7 @@ eq(repr(foo.foo), "" % foo.__name__) + @unittest.skip('need a suitable object') def test_object(self): # XXX Test the repr of a type with a really long tp_name but with no # tp_repr. WIBNI we had ::Inline? :) @@ -319,6 +321,7 @@ '= 0, "Error resolving host to ip.") try: hname, aliases, ipaddrs = socket.gethostbyaddr(ip) except socket.error: # Probably a similar problem as above; skip this test - return + self.skipTest('name lookup failure') all_host_names = [hostname, hname] + aliases fqhn = socket.getfqdn(ip) if not fqhn in all_host_names: @@ -932,9 +932,9 @@ try: from socket import inet_pton, AF_INET6, has_ipv6 if not has_ipv6: - return + self.skipTest('IPv6 not available') except ImportError: - return + self.skipTest('could not import needed symbols from socket') f = lambda a: inet_pton(AF_INET6, a) assertInvalid = lambda a: self.assertRaises( (socket.error, ValueError), f, a @@ -1010,9 +1010,9 @@ try: from socket import inet_ntop, AF_INET6, has_ipv6 if not has_ipv6: - return + self.skipTest('IPv6 not available') except ImportError: - return + self.skipTest('could not import needed symbols from socket') f = lambda a: inet_ntop(AF_INET6, a) assertInvalid = lambda a: self.assertRaises( (socket.error, ValueError), f, a @@ -1045,7 +1045,7 @@ my_ip_addr = socket.gethostbyname(socket.gethostname()) except socket.error: # Probably name lookup wasn't set up right; skip this test - return + self.skipTest('name lookup failure') self.assertIn(name[0], ("0.0.0.0", my_ip_addr), '%s invalid' % name[0]) self.assertEqual(name[1], port) diff --git a/Lib/test/test_strptime.py b/Lib/test/test_strptime.py --- a/Lib/test/test_strptime.py +++ b/Lib/test/test_strptime.py @@ -323,7 +323,7 @@ # when time.tzname[0] == time.tzname[1] and time.daylight tz_name = time.tzname[0] if tz_name.upper() in ("UTC", "GMT"): - return + self.skipTest('need non-UTC/GMT timezone') try: original_tzname = time.tzname original_daylight = time.daylight @@ -536,7 +536,7 @@ try: locale.setlocale(locale.LC_TIME, ('en_US', 'UTF8')) except locale.Error: - return + self.skipTest('test needs en_US.UTF8 locale') try: _strptime._strptime_time('10', '%d') # Get id of current cache object. @@ -553,7 +553,7 @@ # If this is the case just suppress the exception and fall-through # to the resetting to the original locale. except locale.Error: - pass + self.skipTest('test needs de_DE.UTF8 locale') # Make sure we don't trample on the locale setting once we leave the # test. finally: diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -324,10 +324,9 @@ finally: os.rmdir(dir) + @unittest.skipUnless(has_stat, 'os.stat not available') def test_file_mode(self): # _mkstemp_inner creates files with the proper mode - if not has_stat: - return # ugh, can't use SkipTest. file = self.do_create() mode = stat.S_IMODE(os.stat(file.name).st_mode) @@ -339,10 +338,9 @@ expected = user * (1 + 8 + 64) self.assertEqual(mode, expected) + @unittest.skipUnless(has_spawnl, 'os.spawnl not available') def test_noinherit(self): # _mkstemp_inner file handles are not inherited by child processes - if not has_spawnl: - return # ugh, can't use SkipTest. if support.verbose: v="v" @@ -377,10 +375,9 @@ "child process caught fatal signal %d" % -retval) self.assertFalse(retval > 0, "child process reports failure %d"%retval) + @unittest.skipUnless(has_textmode, "text mode not available") def test_textmode(self): # _mkstemp_inner can create files in text mode - if not has_textmode: - return # ugh, can't use SkipTest. # A text file is truncated at the first Ctrl+Z byte f = self.do_create(bin=0) @@ -556,10 +553,9 @@ finally: os.rmdir(dir) + @unittest.skipUnless(has_stat, 'os.stat not available') def test_mode(self): # mkdtemp creates directories with the proper mode - if not has_stat: - return # ugh, can't use SkipTest. dir = self.do_create() try: diff --git a/Lib/test/test_thread.py b/Lib/test/test_thread.py --- a/Lib/test/test_thread.py +++ b/Lib/test/test_thread.py @@ -68,39 +68,35 @@ thread.stack_size(0) self.assertEqual(thread.stack_size(), 0, "stack_size not reset to default") - if os.name not in ("nt", "os2", "posix"): - return - - tss_supported = True + @unittest.skipIf(os.name not in ("nt", "os2", "posix"), 'test meant for nt, os2, and posix') + def test_nt_and_posix_stack_size(self): try: thread.stack_size(4096) except ValueError: verbose_print("caught expected ValueError setting " "stack_size(4096)") except thread.error: - tss_supported = False - verbose_print("platform does not support changing thread stack " - "size") + self.skipTest("platform does not support changing thread stack " + "size") - if tss_supported: - fail_msg = "stack_size(%d) failed - should succeed" - for tss in (262144, 0x100000, 0): - thread.stack_size(tss) - self.assertEqual(thread.stack_size(), tss, fail_msg % tss) - verbose_print("successfully set stack_size(%d)" % tss) + fail_msg = "stack_size(%d) failed - should succeed" + for tss in (262144, 0x100000, 0): + thread.stack_size(tss) + self.assertEqual(thread.stack_size(), tss, fail_msg % tss) + verbose_print("successfully set stack_size(%d)" % tss) - for tss in (262144, 0x100000): - verbose_print("trying stack_size = (%d)" % tss) - self.next_ident = 0 - self.created = 0 - for i in range(NUMTASKS): - self.newtask() + for tss in (262144, 0x100000): + verbose_print("trying stack_size = (%d)" % tss) + self.next_ident = 0 + self.created = 0 + for i in range(NUMTASKS): + self.newtask() - verbose_print("waiting for all tasks to complete") - self.done_mutex.acquire() - verbose_print("all tasks done") + verbose_print("waiting for all tasks to complete") + self.done_mutex.acquire() + verbose_print("all tasks done") - thread.stack_size(0) + thread.stack_size(0) def test__count(self): # Test the _count() function. diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py --- a/Lib/test/test_time.py +++ b/Lib/test/test_time.py @@ -463,8 +463,7 @@ try: tmp = locale.setlocale(locale.LC_ALL, "fr_FR") except locale.Error: - # skip this test - return + self.skipTest('could not set locale.LC_ALL to fr_FR') # This should not cause an exception time.strftime("%B", (2009,2,1,0,0,0,0,0,0)) diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -1952,12 +1952,12 @@ self.assertEqual(repr('\U00010000'), "'%c'" % (0x10000,)) # printable self.assertEqual(repr('\U00014000'), "'\\U00014000'") # nonprintable + # This test only affects 32-bit platforms because expandtabs can only take + # an int as the max value, not a 64-bit C long. If expandtabs is changed + # to take a 64-bit long, this test should apply to all platforms. + @unittest.skipIf(sys.maxsize > (1 << 32) or struct.calcsize('P') != 4, + 'only applies to 32-bit platforms') def test_expandtabs_overflows_gracefully(self): - # This test only affects 32-bit platforms because expandtabs can only take - # an int as the max value, not a 64-bit C long. If expandtabs is changed - # to take a 64-bit long, this test should apply to all platforms. - if sys.maxsize > (1 << 32) or struct.calcsize('P') != 4: - return self.assertRaises(OverflowError, 't\tt\t'.expandtabs, sys.maxsize) @support.cpython_only diff --git a/Lib/test/test_urllibnet.py b/Lib/test/test_urllibnet.py --- a/Lib/test/test_urllibnet.py +++ b/Lib/test/test_urllibnet.py @@ -98,11 +98,10 @@ open_url.close() self.assertEqual(code, 404) + # On Windows, socket handles are not file descriptors; this + # test can't pass on Windows. + @unittest.skipIf(sys.platform in ('win32',), 'not appropriate for Windows') def test_fileno(self): - if sys.platform in ('win32',): - # On Windows, socket handles are not file descriptors; this - # test can't pass on Windows. - return # Make sure fd returned by fileno is valid. with self.urlopen("http://www.python.org/", timeout=None) as open_url: fd = open_url.fileno() diff --git a/Lib/test/test_warnings.py b/Lib/test/test_warnings.py --- a/Lib/test/test_warnings.py +++ b/Lib/test/test_warnings.py @@ -682,7 +682,7 @@ # Explicit tests for the test.support convenience wrapper wmod = self.module if wmod is not sys.modules['warnings']: - return + self.skipTest('module to test is not loaded warnings module') with support.check_warnings(quiet=False) as w: self.assertEqual(w.warnings, []) wmod.simplefilter("always") diff --git a/Lib/test/test_xmlrpc_net.py b/Lib/test/test_xmlrpc_net.py --- a/Lib/test/test_xmlrpc_net.py +++ b/Lib/test/test_xmlrpc_net.py @@ -20,7 +20,6 @@ t0 = server.currentTime.getCurrentTime() except socket.error as e: self.skipTest("network error: %s" % e) - return # Perform a minimal sanity check on the result, just to be sure # the request means what we think it means. @@ -44,7 +43,6 @@ builders = server.getAllBuilders() except socket.error as e: self.skipTest("network error: %s" % e) - return self.addCleanup(lambda: server('close')()) # Perform a minimal sanity check on the result, just to be sure diff --git a/Lib/test/test_zipimport.py b/Lib/test/test_zipimport.py --- a/Lib/test/test_zipimport.py +++ b/Lib/test/test_zipimport.py @@ -111,7 +111,7 @@ # so we'll simply skip it then. Bug #765456. # if "zlib" in sys.builtin_module_names: - return + self.skipTest('zlib is a builtin module') if "zlib" in sys.modules: del sys.modules["zlib"] files = {"zlib.py": (NOW, test_src)} diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -109,7 +109,7 @@ import, converting from test_main to unittest.main, and running the _testcapi module tests within a unittest TestCase. -- Issue #18702: All skipped tests now reported as skipped. +- Issue #18702, 19572: All skipped tests now reported as skipped. - Issue #19085: Added basic tests for all tkinter widget options. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 07:45:27 2013 From: python-checkins at python.org (zach.ware) Date: Sun, 8 Dec 2013 07:45:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Normalize_whit?= =?utf-8?q?espace?= Message-ID: <3dcdMW6DyKz7Llr@mail.python.org> http://hg.python.org/cpython/rev/9826bfb802ca changeset: 87822:9826bfb802ca branch: 3.3 user: Zachary Ware date: Sun Dec 08 00:38:54 2013 -0600 summary: Normalize whitespace files: Lib/test/test_dis.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py --- a/Lib/test/test_dis.py +++ b/Lib/test/test_dis.py @@ -228,7 +228,7 @@ # so fails if the tests are run with -O. Skip this test then. if __debug__: - self.do_disassembly_test(bug1333982, dis_bug1333982) + self.do_disassembly_test(bug1333982, dis_bug1333982) def test_big_linenos(self): def func(count): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 07:45:29 2013 From: python-checkins at python.org (zach.ware) Date: Sun, 8 Dec 2013 07:45:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_19572=3A_More_silently_skipped_tests_explicitly_sk?= =?utf-8?q?ipped=2E?= Message-ID: <3dcdMY5K3vz7Llr@mail.python.org> http://hg.python.org/cpython/rev/03afd2d7d395 changeset: 87823:03afd2d7d395 parent: 87820:5379bba2fb21 parent: 87822:9826bfb802ca user: Zachary Ware date: Sun Dec 08 00:44:27 2013 -0600 summary: Issue 19572: More silently skipped tests explicitly skipped. files: Lib/test/_test_multiprocessing.py | 22 +++--- Lib/test/datetimetester.py | 1 + Lib/test/multibytecodec_support.py | 4 +- Lib/test/string_tests.py | 4 +- Lib/test/test_array.py | 2 +- Lib/test/test_asyncio/test_events.py | 2 +- Lib/test/test_codecencodings_iso2022.py | 1 + Lib/test/test_configparser.py | 3 +- Lib/test/test_decimal.py | 18 ++--- Lib/test/test_dis.py | 14 ++- Lib/test/test_fileio.py | 3 +- Lib/test/test_float.py | 2 +- Lib/test/test_functools.py | 29 ++++---- Lib/test/test_grp.py | 4 +- Lib/test/test_imp.py | 2 +- Lib/test/test_io.py | 12 +-- Lib/test/test_memoryview.py | 14 ++-- Lib/test/test_nis.py | 6 +- Lib/test/test_ntpath.py | 2 +- Lib/test/test_os.py | 2 +- Lib/test/test_pwd.py | 17 +++- Lib/test/test_reprlib.py | 3 + Lib/test/test_shutil.py | 4 +- Lib/test/test_site.py | 1 + Lib/test/test_socket.py | 18 ++-- Lib/test/test_strptime.py | 6 +- Lib/test/test_tempfile.py | 12 +-- Lib/test/test_thread.py | 42 +++++------- Lib/test/test_time.py | 3 +- Lib/test/test_unicode.py | 10 +- Lib/test/test_urllibnet.py | 7 +- Lib/test/test_warnings.py | 2 +- Lib/test/test_xmlrpc_net.py | 1 - Lib/test/test_zipimport.py | 2 +- Misc/NEWS | 2 + 35 files changed, 135 insertions(+), 142 deletions(-) diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -198,7 +198,7 @@ def test_current(self): if self.TYPE == 'threads': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) current = self.current_process() authkey = current.authkey @@ -212,7 +212,7 @@ def test_daemon_argument(self): if self.TYPE == "threads": - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) # By default uses the current process's daemon flag. proc0 = self.Process(target=self._test) @@ -277,7 +277,7 @@ def test_terminate(self): if self.TYPE == 'threads': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) p = self.Process(target=self._test_terminate) p.daemon = True @@ -385,7 +385,7 @@ def test_sentinel(self): if self.TYPE == "threads": - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) event = self.Event() p = self.Process(target=self._test_sentinel, args=(event,)) with self.assertRaises(ValueError): @@ -441,7 +441,7 @@ def test_stderr_flush(self): # sys.stderr is flushed at process shutdown (issue #13812) if self.TYPE == "threads": - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) testfn = test.support.TESTFN self.addCleanup(test.support.unlink, testfn) @@ -469,7 +469,7 @@ def test_sys_exit(self): # See Issue 13854 if self.TYPE == 'threads': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) testfn = test.support.TESTFN self.addCleanup(test.support.unlink, testfn) @@ -678,7 +678,7 @@ try: self.assertEqual(q.qsize(), 0) except NotImplementedError: - return + self.skipTest('qsize method not implemented') q.put(1) self.assertEqual(q.qsize(), 1) q.put(5) @@ -786,7 +786,7 @@ def test_timeout(self): if self.TYPE != 'processes': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) sem = self.Semaphore(0) acquire = TimingWrapper(sem.acquire) @@ -1406,7 +1406,7 @@ def test_thousand(self): if self.TYPE == 'manager': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) passes = 1000 lock = self.Lock() conn, child_conn = self.Pipe(False) @@ -1701,7 +1701,7 @@ def test_map_unplicklable(self): # Issue #19425 -- failure to pickle should not cause a hang if self.TYPE == 'threads': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) class A(object): def __reduce__(self): raise RuntimeError('cannot pickle') @@ -2224,7 +2224,7 @@ def test_sendbytes(self): if self.TYPE != 'processes': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) msg = latin('abcdefghijklmnopqrstuvwxyz') a, b = self.Pipe() diff --git a/Lib/test/datetimetester.py b/Lib/test/datetimetester.py --- a/Lib/test/datetimetester.py +++ b/Lib/test/datetimetester.py @@ -2029,6 +2029,7 @@ class TestSubclassDateTime(TestDateTime): theclass = SubclassDatetime # Override tests not designed for subclass + @unittest.skip('not appropriate for subclasses') def test_roundtrip(self): pass diff --git a/Lib/test/multibytecodec_support.py b/Lib/test/multibytecodec_support.py --- a/Lib/test/multibytecodec_support.py +++ b/Lib/test/multibytecodec_support.py @@ -73,7 +73,7 @@ def test_xmlcharrefreplace(self): if self.has_iso10646: - return + self.skipTest('encoding contains full ISO 10646 map') s = "\u0b13\u0b23\u0b60 nd eggs" self.assertEqual( @@ -83,7 +83,7 @@ def test_customreplace_encode(self): if self.has_iso10646: - return + self.skipTest('encoding contains full ISO 10646 map') from html.entities import codepoint2name diff --git a/Lib/test/string_tests.py b/Lib/test/string_tests.py --- a/Lib/test/string_tests.py +++ b/Lib/test/string_tests.py @@ -676,10 +676,10 @@ self.checkraises(TypeError, 'hello', 'replace', 42, 'h') self.checkraises(TypeError, 'hello', 'replace', 'h', 42) + @unittest.skipIf(sys.maxsize > (1 << 32) or struct.calcsize('P') != 4, + 'only applies to 32-bit platforms') def test_replace_overflow(self): # Check for overflow checking on 32 bit machines - if sys.maxsize != 2147483647 or struct.calcsize("P") > 4: - return A2_16 = "A" * (2**16) self.checkraises(OverflowError, A2_16, "replace", "", A2_16) self.checkraises(OverflowError, A2_16, "replace", "A", A2_16) diff --git a/Lib/test/test_array.py b/Lib/test/test_array.py --- a/Lib/test/test_array.py +++ b/Lib/test/test_array.py @@ -946,7 +946,7 @@ try: import gc except ImportError: - return + self.skipTest('gc module not available') a = array.array(self.typecode) l = [iter(a)] l.append(l) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -899,7 +899,7 @@ def test_internal_fds(self): loop = self.create_event_loop() if not isinstance(loop, selector_events.BaseSelectorEventLoop): - return + self.skipTest('loop is not a BaseSelectorEventLoop') self.assertEqual(1, loop._internal_fds) loop.close() diff --git a/Lib/test/test_codecencodings_iso2022.py b/Lib/test/test_codecencodings_iso2022.py --- a/Lib/test/test_codecencodings_iso2022.py +++ b/Lib/test/test_codecencodings_iso2022.py @@ -36,6 +36,7 @@ # iso2022_kr.txt cannot be used to test "chunk coding": the escape # sequence is only written on the first line + @unittest.skip('iso2022_kr.txt cannot be used to test "chunk coding"') def test_chunkcoding(self): pass diff --git a/Lib/test/test_configparser.py b/Lib/test/test_configparser.py --- a/Lib/test/test_configparser.py +++ b/Lib/test/test_configparser.py @@ -707,8 +707,7 @@ def test_read_returns_file_list(self): if self.delimiters[0] != '=': - # skip reading the file if we're using an incompatible format - return + self.skipTest('incompatible format') file1 = support.findfile("cfgparser.1") # check when we pass a mix of readable and non-readable files: cf = self.newconfig() diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -300,7 +300,6 @@ #Exception raised where there shouldn't have been one. self.fail('Exception "'+exception.__class__.__name__ + '" raised on line '+line) - return def eval_line(self, s): if s.find(' -> ') >= 0 and s[:2] != '--' and not s.startswith(' --'): @@ -460,7 +459,6 @@ self.assertEqual(myexceptions, theirexceptions, 'Incorrect flags set in ' + s + ' -- got ' + str(myexceptions)) - return def getexceptions(self): return [e for e in Signals[self.decimal] if self.context.flags[e]] @@ -1072,7 +1070,7 @@ try: from locale import CHAR_MAX except ImportError: - return + self.skipTest('locale.CHAR_MAX not available') def make_grouping(lst): return ''.join([chr(x) for x in lst]) if self.decimal == C else lst @@ -1163,8 +1161,12 @@ decimal_point = locale.localeconv()['decimal_point'] thousands_sep = locale.localeconv()['thousands_sep'] - if decimal_point != '\u066b' or thousands_sep != '\u066c': - return + if decimal_point != '\u066b': + self.skipTest('inappropriate decimal point separator' + '({!r} not {!r})'.format(decimal_point, '\u066b')) + if thousands_sep != '\u066c': + self.skipTest('inappropriate thousands separator' + '({!r} not {!r})'.format(thousands_sep, '\u066c')) self.assertEqual(format(Decimal('100000000.123'), 'n'), '100\u066c000\u066c000\u066b123') @@ -1514,7 +1516,6 @@ cls.assertTrue(c1.flags[Inexact]) for sig in Overflow, Underflow, DivisionByZero, InvalidOperation: cls.assertFalse(c1.flags[sig]) - return def thfunc2(cls): Decimal = cls.decimal.Decimal @@ -1559,7 +1560,6 @@ cls.assertTrue(thiscontext.flags[Inexact]) for sig in Overflow, Underflow, DivisionByZero, InvalidOperation: cls.assertFalse(thiscontext.flags[sig]) - return class ThreadingTest(unittest.TestCase): '''Unit tests for thread local contexts in Decimal.''' @@ -1601,7 +1601,6 @@ DefaultContext.prec = save_prec DefaultContext.Emax = save_emax DefaultContext.Emin = save_emin - return @unittest.skipUnless(threading, 'threading required') class CThreadingTest(ThreadingTest): @@ -4524,7 +4523,6 @@ self.assertEqual(d1._sign, b1._sign) self.assertEqual(d1._int, b1._int) self.assertEqual(d1._exp, b1._exp) - return Decimal(d1) self.assertEqual(d1._sign, b1._sign) @@ -5270,7 +5268,7 @@ try: from locale import CHAR_MAX except ImportError: - return + self.skipTest('locale.CHAR_MAX not available') def make_grouping(lst): return ''.join([chr(x) for x in lst]) diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py --- a/Lib/test/test_dis.py +++ b/Lib/test/test_dis.py @@ -265,16 +265,18 @@ def test_bug_708901(self): self.do_disassembly_test(bug708901, dis_bug708901) + # Test has been disabled due to change in the way + # list comps are handled. The byte code now includes + # a memory address and a file location, so they change from + # run to run. + @unittest.skip('disabled due to a change in the way list comps are handled') def test_bug_1333982(self): # XXX: re-enable this test! # This one is checking bytecodes generated for an `assert` statement, # so fails if the tests are run with -O. Skip this test then. - pass # Test has been disabled due to change in the way - # list comps are handled. The byte code now includes - # a memory address and a file location, so they change from - # run to run. - # if __debug__: - # self.do_disassembly_test(bug1333982, dis_bug1333982) + + if __debug__: + self.do_disassembly_test(bug1333982, dis_bug1333982) def test_big_linenos(self): def func(count): diff --git a/Lib/test/test_fileio.py b/Lib/test/test_fileio.py --- a/Lib/test/test_fileio.py +++ b/Lib/test/test_fileio.py @@ -341,8 +341,7 @@ try: fn = TESTFN.encode("ascii") except UnicodeEncodeError: - # Skip test - return + self.skipTest('could not encode %r to ascii' % TESTFN) f = _FileIO(fn, "w") try: f.write(b"abc") diff --git a/Lib/test/test_float.py b/Lib/test/test_float.py --- a/Lib/test/test_float.py +++ b/Lib/test/test_float.py @@ -71,7 +71,7 @@ # it still has to accept the normal python syntax import locale if not locale.localeconv()['decimal_point'] == ',': - return + self.skipTest('decimal_point is not ","') self.assertEqual(float(" 3.14 "), 3.14) self.assertEqual(float("+3.14 "), 3.14) diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -42,20 +42,6 @@ self.assertEqual(p.func, capture) self.assertEqual(p.args, (1, 2)) self.assertEqual(p.keywords, dict(a=10, b=20)) - # attributes should not be writable - if not isinstance(self.partial, type): - return - self.assertRaises(AttributeError, setattr, p, 'func', map) - self.assertRaises(AttributeError, setattr, p, 'args', (1, 2)) - self.assertRaises(AttributeError, setattr, p, 'keywords', dict(a=1, b=2)) - - p = self.partial(hex) - try: - del p.__dict__ - except TypeError: - pass - else: - self.fail('partial object allowed __dict__ to be deleted') def test_argument_checking(self): self.assertRaises(TypeError, self.partial) # need at least a func arg @@ -151,6 +137,21 @@ if c_functools: partial = c_functools.partial + def test_attributes_unwritable(self): + # attributes should not be writable + p = self.partial(capture, 1, 2, a=10, b=20) + self.assertRaises(AttributeError, setattr, p, 'func', map) + self.assertRaises(AttributeError, setattr, p, 'args', (1, 2)) + self.assertRaises(AttributeError, setattr, p, 'keywords', dict(a=1, b=2)) + + p = self.partial(hex) + try: + del p.__dict__ + except TypeError: + pass + else: + self.fail('partial object allowed __dict__ to be deleted') + def test_repr(self): args = (object(), object()) args_repr = ', '.join(repr(a) for a in args) diff --git a/Lib/test/test_grp.py b/Lib/test/test_grp.py --- a/Lib/test/test_grp.py +++ b/Lib/test/test_grp.py @@ -26,8 +26,10 @@ for e in entries: self.check_value(e) + def test_values_extended(self): + entries = grp.getgrall() if len(entries) > 1000: # Huge group file (NIS?) -- skip the rest - return + self.skipTest('huge group file, extended test skipped') for e in entries: e2 = grp.getgrgid(e.gr_gid) diff --git a/Lib/test/test_imp.py b/Lib/test/test_imp.py --- a/Lib/test/test_imp.py +++ b/Lib/test/test_imp.py @@ -272,7 +272,7 @@ if found[0] is not None: found[0].close() if found[2][2] != imp.C_EXTENSION: - return + self.skipTest("found module doesn't appear to be a C extension") imp.load_module(name, None, *found[1:]) @unittest.skipIf(sys.dont_write_bytecode, diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -421,14 +421,9 @@ # a long time to build the >2GB file and takes >2GB of disk space # therefore the resource must be enabled to run this test. if sys.platform[:3] == 'win' or sys.platform == 'darwin': - if not support.is_resource_enabled("largefile"): - print("\nTesting large file ops skipped on %s." % sys.platform, - file=sys.stderr) - print("It requires %d bytes and a long time." % self.LARGE, - file=sys.stderr) - print("Use 'regrtest.py -u largefile test_io' to run it.", - file=sys.stderr) - return + support.requires( + 'largefile', + 'test requires %s bytes and a long time to run' % self.LARGE) with self.open(support.TESTFN, "w+b", 0) as f: self.large_file_ops(f) with self.open(support.TESTFN, "w+b") as f: @@ -698,6 +693,7 @@ self.assertEqual(42, bufio.fileno()) + @unittest.skip('test having existential crisis') def test_no_fileno(self): # XXX will we always have fileno() function? If so, kill # this test. Else, write it. diff --git a/Lib/test/test_memoryview.py b/Lib/test/test_memoryview.py --- a/Lib/test/test_memoryview.py +++ b/Lib/test/test_memoryview.py @@ -57,7 +57,7 @@ def test_setitem_readonly(self): if not self.ro_type: - return + self.skipTest("no read-only type to test") b = self.ro_type(self._source) oldrefcount = sys.getrefcount(b) m = self._view(b) @@ -71,7 +71,7 @@ def test_setitem_writable(self): if not self.rw_type: - return + self.skipTest("no writable type to test") tp = self.rw_type b = self.rw_type(self._source) oldrefcount = sys.getrefcount(b) @@ -189,13 +189,13 @@ def test_attributes_readonly(self): if not self.ro_type: - return + self.skipTest("no read-only type to test") m = self.check_attributes_with_type(self.ro_type) self.assertEqual(m.readonly, True) def test_attributes_writable(self): if not self.rw_type: - return + self.skipTest("no writable type to test") m = self.check_attributes_with_type(self.rw_type) self.assertEqual(m.readonly, False) @@ -301,7 +301,7 @@ # buffer as writable causing a segfault if using mmap tp = self.ro_type if tp is None: - return + self.skipTest("no read-only type to test") b = tp(self._source) m = self._view(b) i = io.BytesIO(b'ZZZZ') @@ -379,12 +379,12 @@ itemsize = array.array('i').itemsize format = 'i' + @unittest.skip('XXX test should be adapted for non-byte buffers') def test_getbuffer(self): - # XXX Test should be adapted for non-byte buffers pass + @unittest.skip('XXX NotImplementedError: tolist() only supports byte views') def test_tolist(self): - # XXX NotImplementedError: tolist() only supports byte views pass diff --git a/Lib/test/test_nis.py b/Lib/test/test_nis.py --- a/Lib/test/test_nis.py +++ b/Lib/test/test_nis.py @@ -12,11 +12,7 @@ maps = nis.maps() except nis.error as msg: # NIS is probably not active, so this test isn't useful - if support.verbose: - print("Test Skipped:", msg) - # Can't raise SkipTest as regrtest only recognizes the exception - # import time. - return + self.skipTest(str(msg)) try: # On some systems, this map is only accessible to the # super user diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py --- a/Lib/test/test_ntpath.py +++ b/Lib/test/test_ntpath.py @@ -218,7 +218,7 @@ import nt tester('ntpath.abspath("C:\\")', "C:\\") except ImportError: - pass + self.skipTest('nt module not available') def test_relpath(self): currentdir = os.path.split(os.getcwd())[-1] diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -508,7 +508,7 @@ try: os.stat(r"c:\pagefile.sys") except FileNotFoundError: - pass # file does not exist; cannot run test + self.skipTest(r'c:\pagefile.sys does not exist') except OSError as e: self.fail("Could not stat pagefile.sys") diff --git a/Lib/test/test_pwd.py b/Lib/test/test_pwd.py --- a/Lib/test/test_pwd.py +++ b/Lib/test/test_pwd.py @@ -8,8 +8,6 @@ def test_values(self): entries = pwd.getpwall() - entriesbyname = {} - entriesbyuid = {} for e in entries: self.assertEqual(len(e), 7) @@ -32,13 +30,20 @@ # for one uid # self.assertEqual(pwd.getpwuid(e.pw_uid), e) # instead of this collect all entries for one uid - # and check afterwards + # and check afterwards (done in test_values_extended) + + def test_values_extended(self): + entries = pwd.getpwall() + entriesbyname = {} + entriesbyuid = {} + + if len(entries) > 1000: # Huge passwd file (NIS?) -- skip this test + self.skipTest('passwd file is huge; extended test skipped') + + for e in entries: entriesbyname.setdefault(e.pw_name, []).append(e) entriesbyuid.setdefault(e.pw_uid, []).append(e) - if len(entries) > 1000: # Huge passwd file (NIS?) -- skip the rest - return - # check whether the entry returned by getpwuid() # for each uid is among those from getpwall() for this uid for e in entries: diff --git a/Lib/test/test_reprlib.py b/Lib/test/test_reprlib.py --- a/Lib/test/test_reprlib.py +++ b/Lib/test/test_reprlib.py @@ -166,6 +166,7 @@ eq(r([[[[[[{}]]]]]]), "[[[[[[{}]]]]]]") eq(r([[[[[[[{}]]]]]]]), "[[[[[[[...]]]]]]]") + @unittest.skip('hard to catch a cell object') def test_cell(self): # XXX Hmm? How to get at a cell object? pass @@ -274,6 +275,7 @@ eq(repr(foo.foo), "" % foo.__name__) + @unittest.skip('need a suitable object') def test_object(self): # XXX Test the repr of a type with a really long tp_name but with no # tp_repr. WIBNI we had ::Inline? :) @@ -321,6 +323,7 @@ '= 0, "Error resolving host to ip.") try: hname, aliases, ipaddrs = socket.gethostbyaddr(ip) except OSError: # Probably a similar problem as above; skip this test - return + self.skipTest('name lookup failure') all_host_names = [hostname, hname] + aliases fqhn = socket.getfqdn(ip) if not fqhn in all_host_names: @@ -977,16 +977,16 @@ try: from socket import inet_pton, AF_INET6, has_ipv6 if not has_ipv6: - return + self.skipTest('IPv6 not available') except ImportError: - return + self.skipTest('could not import needed symbols from socket') if sys.platform == "win32": try: inet_pton(AF_INET6, '::') except OSError as e: if e.winerror == 10022: - return # IPv6 might not be installed on this PC + self.skipTest('IPv6 might not be supported') f = lambda a: inet_pton(AF_INET6, a) assertInvalid = lambda a: self.assertRaises( @@ -1063,16 +1063,16 @@ try: from socket import inet_ntop, AF_INET6, has_ipv6 if not has_ipv6: - return + self.skipTest('IPv6 not available') except ImportError: - return + self.skipTest('could not import needed symbols from socket') if sys.platform == "win32": try: inet_ntop(AF_INET6, b'\x00' * 16) except OSError as e: if e.winerror == 10022: - return # IPv6 might not be installed on this PC + self.skipTest('IPv6 might not be supported') f = lambda a: inet_ntop(AF_INET6, a) assertInvalid = lambda a: self.assertRaises( @@ -1106,7 +1106,7 @@ my_ip_addr = socket.gethostbyname(socket.gethostname()) except OSError: # Probably name lookup wasn't set up right; skip this test - return + self.skipTest('name lookup failure') self.assertIn(name[0], ("0.0.0.0", my_ip_addr), '%s invalid' % name[0]) self.assertEqual(name[1], port) diff --git a/Lib/test/test_strptime.py b/Lib/test/test_strptime.py --- a/Lib/test/test_strptime.py +++ b/Lib/test/test_strptime.py @@ -323,7 +323,7 @@ # when time.tzname[0] == time.tzname[1] and time.daylight tz_name = time.tzname[0] if tz_name.upper() in ("UTC", "GMT"): - return + self.skipTest('need non-UTC/GMT timezone') try: original_tzname = time.tzname original_daylight = time.daylight @@ -536,7 +536,7 @@ try: locale.setlocale(locale.LC_TIME, ('en_US', 'UTF8')) except locale.Error: - return + self.skipTest('test needs en_US.UTF8 locale') try: _strptime._strptime_time('10', '%d') # Get id of current cache object. @@ -553,7 +553,7 @@ # If this is the case just suppress the exception and fall-through # to the resetting to the original locale. except locale.Error: - pass + self.skipTest('test needs de_DE.UTF8 locale') # Make sure we don't trample on the locale setting once we leave the # test. finally: diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -324,10 +324,9 @@ finally: os.rmdir(dir) + @unittest.skipUnless(has_stat, 'os.stat not available') def test_file_mode(self): # _mkstemp_inner creates files with the proper mode - if not has_stat: - return # ugh, can't use SkipTest. file = self.do_create() mode = stat.S_IMODE(os.stat(file.name).st_mode) @@ -339,10 +338,9 @@ expected = user * (1 + 8 + 64) self.assertEqual(mode, expected) + @unittest.skipUnless(has_spawnl, 'os.spawnl not available') def test_noinherit(self): # _mkstemp_inner file handles are not inherited by child processes - if not has_spawnl: - return # ugh, can't use SkipTest. if support.verbose: v="v" @@ -378,10 +376,9 @@ "child process caught fatal signal %d" % -retval) self.assertFalse(retval > 0, "child process reports failure %d"%retval) + @unittest.skipUnless(has_textmode, "text mode not available") def test_textmode(self): # _mkstemp_inner can create files in text mode - if not has_textmode: - return # ugh, can't use SkipTest. # A text file is truncated at the first Ctrl+Z byte f = self.do_create(bin=0) @@ -571,10 +568,9 @@ finally: os.rmdir(dir) + @unittest.skipUnless(has_stat, 'os.stat not available') def test_mode(self): # mkdtemp creates directories with the proper mode - if not has_stat: - return # ugh, can't use SkipTest. dir = self.do_create() try: diff --git a/Lib/test/test_thread.py b/Lib/test/test_thread.py --- a/Lib/test/test_thread.py +++ b/Lib/test/test_thread.py @@ -68,39 +68,35 @@ thread.stack_size(0) self.assertEqual(thread.stack_size(), 0, "stack_size not reset to default") - if os.name not in ("nt", "posix"): - return - - tss_supported = True + @unittest.skipIf(os.name not in ("nt", "posix"), 'test meant for nt and posix') + def test_nt_and_posix_stack_size(self): try: thread.stack_size(4096) except ValueError: verbose_print("caught expected ValueError setting " "stack_size(4096)") except thread.error: - tss_supported = False - verbose_print("platform does not support changing thread stack " - "size") + self.skipTest("platform does not support changing thread stack " + "size") - if tss_supported: - fail_msg = "stack_size(%d) failed - should succeed" - for tss in (262144, 0x100000, 0): - thread.stack_size(tss) - self.assertEqual(thread.stack_size(), tss, fail_msg % tss) - verbose_print("successfully set stack_size(%d)" % tss) + fail_msg = "stack_size(%d) failed - should succeed" + for tss in (262144, 0x100000, 0): + thread.stack_size(tss) + self.assertEqual(thread.stack_size(), tss, fail_msg % tss) + verbose_print("successfully set stack_size(%d)" % tss) - for tss in (262144, 0x100000): - verbose_print("trying stack_size = (%d)" % tss) - self.next_ident = 0 - self.created = 0 - for i in range(NUMTASKS): - self.newtask() + for tss in (262144, 0x100000): + verbose_print("trying stack_size = (%d)" % tss) + self.next_ident = 0 + self.created = 0 + for i in range(NUMTASKS): + self.newtask() - verbose_print("waiting for all tasks to complete") - self.done_mutex.acquire() - verbose_print("all tasks done") + verbose_print("waiting for all tasks to complete") + self.done_mutex.acquire() + verbose_print("all tasks done") - thread.stack_size(0) + thread.stack_size(0) def test__count(self): # Test the _count() function. diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py --- a/Lib/test/test_time.py +++ b/Lib/test/test_time.py @@ -472,8 +472,7 @@ try: tmp = locale.setlocale(locale.LC_ALL, "fr_FR") except locale.Error: - # skip this test - return + self.skipTest('could not set locale.LC_ALL to fr_FR') # This should not cause an exception time.strftime("%B", (2009,2,1,0,0,0,0,0,0)) diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -2002,12 +2002,12 @@ self.assertEqual(repr('\U00010000'), "'%c'" % (0x10000,)) # printable self.assertEqual(repr('\U00014000'), "'\\U00014000'") # nonprintable + # This test only affects 32-bit platforms because expandtabs can only take + # an int as the max value, not a 64-bit C long. If expandtabs is changed + # to take a 64-bit long, this test should apply to all platforms. + @unittest.skipIf(sys.maxsize > (1 << 32) or struct.calcsize('P') != 4, + 'only applies to 32-bit platforms') def test_expandtabs_overflows_gracefully(self): - # This test only affects 32-bit platforms because expandtabs can only take - # an int as the max value, not a 64-bit C long. If expandtabs is changed - # to take a 64-bit long, this test should apply to all platforms. - if sys.maxsize > (1 << 32) or struct.calcsize('P') != 4: - return self.assertRaises(OverflowError, 't\tt\t'.expandtabs, sys.maxsize) @support.cpython_only diff --git a/Lib/test/test_urllibnet.py b/Lib/test/test_urllibnet.py --- a/Lib/test/test_urllibnet.py +++ b/Lib/test/test_urllibnet.py @@ -98,11 +98,10 @@ open_url.close() self.assertEqual(code, 404) + # On Windows, socket handles are not file descriptors; this + # test can't pass on Windows. + @unittest.skipIf(sys.platform in ('win32',), 'not appropriate for Windows') def test_fileno(self): - if sys.platform in ('win32',): - # On Windows, socket handles are not file descriptors; this - # test can't pass on Windows. - return # Make sure fd returned by fileno is valid. with self.urlopen("http://www.python.org/", timeout=None) as open_url: fd = open_url.fileno() diff --git a/Lib/test/test_warnings.py b/Lib/test/test_warnings.py --- a/Lib/test/test_warnings.py +++ b/Lib/test/test_warnings.py @@ -695,7 +695,7 @@ # Explicit tests for the test.support convenience wrapper wmod = self.module if wmod is not sys.modules['warnings']: - return + self.skipTest('module to test is not loaded warnings module') with support.check_warnings(quiet=False) as w: self.assertEqual(w.warnings, []) wmod.simplefilter("always") diff --git a/Lib/test/test_xmlrpc_net.py b/Lib/test/test_xmlrpc_net.py --- a/Lib/test/test_xmlrpc_net.py +++ b/Lib/test/test_xmlrpc_net.py @@ -19,7 +19,6 @@ builders = server.getAllBuilders() except OSError as e: self.skipTest("network error: %s" % e) - return self.addCleanup(lambda: server('close')()) # Perform a minimal sanity check on the result, just to be sure diff --git a/Lib/test/test_zipimport.py b/Lib/test/test_zipimport.py --- a/Lib/test/test_zipimport.py +++ b/Lib/test/test_zipimport.py @@ -136,7 +136,7 @@ # so we'll simply skip it then. Bug #765456. # if "zlib" in sys.builtin_module_names: - return + self.skipTest('zlib is a builtin module') if "zlib" in sys.modules: del sys.modules["zlib"] files = {"zlib.py": (NOW, test_src)} diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -106,6 +106,8 @@ Tests ----- +- Issue #19572: More skipped tests explicitly marked as skipped. + - Issue #19595: Re-enabled a long-disabled test in test_winsound. - Issue #19588: Fixed tests in test_random that were silently skipped most -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 08:02:38 2013 From: python-checkins at python.org (zach.ware) Date: Sun, 8 Dec 2013 08:02:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5OTI2?= =?utf-8?q?=3A_Removed_unneeded_test=5Fmain_from_test=5Fabstract=5Fnumbers?= =?utf-8?q?=2E?= Message-ID: <3dcdlL0tqXz7LmL@mail.python.org> http://hg.python.org/cpython/rev/22c865babf0a changeset: 87824:22c865babf0a branch: 3.3 parent: 87822:9826bfb802ca user: Zachary Ware date: Sun Dec 08 01:00:14 2013 -0600 summary: Issue #19926: Removed unneeded test_main from test_abstract_numbers. Patch by Vajrasky Kok. files: Lib/test/test_abstract_numbers.py | 4 ---- Misc/NEWS | 3 +++ 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_abstract_numbers.py b/Lib/test/test_abstract_numbers.py --- a/Lib/test/test_abstract_numbers.py +++ b/Lib/test/test_abstract_numbers.py @@ -4,7 +4,6 @@ import operator import unittest from numbers import Complex, Real, Rational, Integral -from test import support class TestNumbers(unittest.TestCase): def test_int(self): @@ -40,9 +39,6 @@ self.assertRaises(TypeError, float, c1) self.assertRaises(TypeError, int, c1) -def test_main(): - support.run_unittest(TestNumbers) - if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -97,6 +97,9 @@ Tests ----- +- Issue #19926: Removed unneeded test_main from test_abstract_numbers. + Patch by Vajrasky Kok. + - Issue #19595: Re-enabled a long-disabled test in test_winsound. - Issue #19588: Fixed tests in test_random that were silently skipped most -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 08:02:39 2013 From: python-checkins at python.org (zach.ware) Date: Sun, 8 Dec 2013 08:02:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319926=3A_Removed_unneeded_test=5Fmain_from_test?= =?utf-8?q?=5Fabstract=5Fnumbers=2E?= Message-ID: <3dcdlM2kyLz7Lmg@mail.python.org> http://hg.python.org/cpython/rev/7ff101a449b4 changeset: 87825:7ff101a449b4 parent: 87823:03afd2d7d395 parent: 87824:22c865babf0a user: Zachary Ware date: Sun Dec 08 01:01:42 2013 -0600 summary: Issue #19926: Removed unneeded test_main from test_abstract_numbers. Patch by Vajrasky Kok. files: Lib/test/test_abstract_numbers.py | 4 ---- Misc/NEWS | 3 +++ 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_abstract_numbers.py b/Lib/test/test_abstract_numbers.py --- a/Lib/test/test_abstract_numbers.py +++ b/Lib/test/test_abstract_numbers.py @@ -4,7 +4,6 @@ import operator import unittest from numbers import Complex, Real, Rational, Integral -from test import support class TestNumbers(unittest.TestCase): def test_int(self): @@ -40,9 +39,6 @@ self.assertRaises(TypeError, float, c1) self.assertRaises(TypeError, int, c1) -def test_main(): - support.run_unittest(TestNumbers) - if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -106,6 +106,9 @@ Tests ----- +- Issue #19926: Removed unneeded test_main from test_abstract_numbers. + Patch by Vajrasky Kok. + - Issue #19572: More skipped tests explicitly marked as skipped. - Issue #19595: Re-enabled a long-disabled test in test_winsound. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 09:39:45 2013 From: python-checkins at python.org (gregory.p.smith) Date: Sun, 8 Dec 2013 09:39:45 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Remove_mention?= =?utf-8?q?s_of_Python_2=2Ex_and_being_externally_maintained_from?= Message-ID: <3dcgvP2W9Qz7LjP@mail.python.org> http://hg.python.org/cpython/rev/3580c718add3 changeset: 87826:3580c718add3 branch: 3.3 parent: 87824:22c865babf0a user: Gregory P. Smith date: Sun Dec 08 00:39:07 2013 -0800 summary: Remove mentions of Python 2.x and being externally maintained from the bundled json module. Replace that with a mention of it being a version of the externally maintained simplejson module. files: Lib/json/__init__.py | 7 ++----- 1 files changed, 2 insertions(+), 5 deletions(-) diff --git a/Lib/json/__init__.py b/Lib/json/__init__.py --- a/Lib/json/__init__.py +++ b/Lib/json/__init__.py @@ -3,11 +3,8 @@ interchange format. :mod:`json` exposes an API familiar to users of the standard library -:mod:`marshal` and :mod:`pickle` modules. It is the externally maintained -version of the :mod:`json` library contained in Python 2.6, but maintains -compatibility with Python 2.4 and Python 2.5 and (currently) has -significant performance advantages, even without using the optional C -extension for speedups. +:mod:`marshal` and :mod:`pickle` modules. It is derived from a +version of the externally maintained simplejson library. Encoding basic Python object hierarchies:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 09:39:46 2013 From: python-checkins at python.org (gregory.p.smith) Date: Sun, 8 Dec 2013 09:39:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Remove_mentions_of_Python_2=2Ex_and_being_externally_mai?= =?utf-8?q?ntained_from?= Message-ID: <3dcgvQ4GWnz7LjP@mail.python.org> http://hg.python.org/cpython/rev/8f789599b26a changeset: 87827:8f789599b26a parent: 87825:7ff101a449b4 parent: 87826:3580c718add3 user: Gregory P. Smith date: Sun Dec 08 00:39:36 2013 -0800 summary: Remove mentions of Python 2.x and being externally maintained from the bundled json module. Replace that with a mention of it being a version of the externally maintained simplejson module. files: Lib/json/__init__.py | 7 ++----- 1 files changed, 2 insertions(+), 5 deletions(-) diff --git a/Lib/json/__init__.py b/Lib/json/__init__.py --- a/Lib/json/__init__.py +++ b/Lib/json/__init__.py @@ -3,11 +3,8 @@ interchange format. :mod:`json` exposes an API familiar to users of the standard library -:mod:`marshal` and :mod:`pickle` modules. It is the externally maintained -version of the :mod:`json` library contained in Python 2.6, but maintains -compatibility with Python 2.4 and Python 2.5 and (currently) has -significant performance advantages, even without using the optional C -extension for speedups. +:mod:`marshal` and :mod:`pickle` modules. It is derived from a +version of the externally maintained simplejson library. Encoding basic Python object hierarchies:: -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sun Dec 8 09:43:12 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 08 Dec 2013 09:43:12 +0100 Subject: [Python-checkins] Daily reference leaks (f334dd2471e7): sum=0 Message-ID: results for f334dd2471e7 on branch "default" -------------------------------------------- test_site leaked [0, -2, 2] references, sum=0 test_site leaked [0, -2, 2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogKKRGUp', '-x'] From python-checkins at python.org Sun Dec 8 10:11:54 2013 From: python-checkins at python.org (charles-francois.natali) Date: Sun, 8 Dec 2013 10:11:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_test=5Fselectors_failu?= =?utf-8?q?re_introduced_by_39e7995f9ad1=2E?= Message-ID: <3dchcV1XZcz7LjN@mail.python.org> http://hg.python.org/cpython/rev/01676a4c16ff changeset: 87828:01676a4c16ff user: Charles-Fran?ois Natali date: Sun Dec 08 10:06:04 2013 +0100 summary: Fix test_selectors failure introduced by 39e7995f9ad1. files: Lib/test/test_selectors.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_selectors.py b/Lib/test/test_selectors.py --- a/Lib/test/test_selectors.py +++ b/Lib/test/test_selectors.py @@ -109,6 +109,7 @@ s.unregister(r) s.unregister(w) + @unittest.skipUnless(os.name == 'posix', "requires posix") def test_unregister_after_fd_close_and_reuse(self): s = self.SELECTOR() self.addCleanup(s.close) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 14:36:15 2013 From: python-checkins at python.org (christian.heimes) Date: Sun, 8 Dec 2013 14:36:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319343=3A_Expose_F?= =?utf-8?q?reeBSD-specific_APIs_in_resource_module=2E__Original_patch?= Message-ID: <3dcpTW2Fjyz7LjS@mail.python.org> http://hg.python.org/cpython/rev/ad2cd599f1cf changeset: 87829:ad2cd599f1cf user: Christian Heimes date: Sun Dec 08 14:35:55 2013 +0100 summary: Issue #19343: Expose FreeBSD-specific APIs in resource module. Original patch by Koobs. files: Doc/library/resource.rst | 28 +++++++++++++++++++++++++++ Lib/test/test_resource.py | 6 +++++ Misc/NEWS | 3 ++ Modules/resource.c | 14 +++++++++++++ 4 files changed, 51 insertions(+), 0 deletions(-) diff --git a/Doc/library/resource.rst b/Doc/library/resource.rst --- a/Doc/library/resource.rst +++ b/Doc/library/resource.rst @@ -217,6 +217,34 @@ .. versionadded:: 3.4 +.. data:: RLIMIT_SBSIZE + + The maximum size (in bytes) of socket buffer usage for this user. + This limits the amount of network memory, and hence the amount of mbufs, + that this user may hold at any time. + + Availability: FreeBSD 9 or later. + + .. versionadded:: 3.4 + +.. data:: RLIMIT_SWAP + + The maximum size (in bytes) of the swap space that may be reserved or + used by all of this user id's processes. + This limit is enforced only if bit 1 of the vm.overcommit sysctl is set. + Please see :manpage:`tuning(7)` for a complete description of this sysctl. + + Availability: FreeBSD 9 or later. + + .. versionadded:: 3.4 + +.. data:: RLIMIT_NPTS + + The maximum number of pseudo-terminals created by this user id. + + Availability: FreeBSD 9 or later. + + .. versionadded:: 3.4 Resource Usage -------------- diff --git a/Lib/test/test_resource.py b/Lib/test/test_resource.py --- a/Lib/test/test_resource.py +++ b/Lib/test/test_resource.py @@ -138,6 +138,12 @@ with contextlib.suppress(AttributeError): self.assertIsInstance(getattr(resource, 'RLIMIT_' + attr), int) + @support.requires_freebsd_version(9) + def test_freebsd_contants(self): + for attr in ['SWAP', 'SBSIZE', 'NPTS']: + with contextlib.suppress(AttributeError): + self.assertIsInstance(getattr(resource, 'RLIMIT_' + attr), int) + @unittest.skipUnless(hasattr(resource, 'prlimit'), 'no prlimit') @support.requires_linux_version(2, 6, 36) def test_prlimit(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,9 @@ Library ------- +- Issue #19343: Expose FreeBSD-specific APIs in resource module. Original + patch by Koobs. + - Issue #19506: Use a memoryview to avoid a data copy when piping data to stdin within subprocess.Popen.communicate. 5-10% less cpu usage. diff --git a/Modules/resource.c b/Modules/resource.c --- a/Modules/resource.c +++ b/Modules/resource.c @@ -424,6 +424,20 @@ PyModule_AddIntMacro(m, RUSAGE_THREAD); #endif +/* FreeBSD specific */ + +#ifdef RLIMIT_SWAP + PyModule_AddIntMacro(m, RLIMIT_SWAP); +#endif + +#ifdef RLIMIT_SBSIZE + PyModule_AddIntMacro(m, RLIMIT_SBSIZE); +#endif + +#ifdef RLIMIT_NPTS + PyModule_AddIntMacro(m, RLIMIT_NPTS); +#endif + #if defined(HAVE_LONG_LONG) if (sizeof(RLIM_INFINITY) > sizeof(long)) { v = PyLong_FromLongLong((PY_LONG_LONG) RLIM_INFINITY); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 15:21:17 2013 From: python-checkins at python.org (christian.heimes) Date: Sun, 8 Dec 2013 15:21:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Attempt_to_fix_OpenIndiana?= =?utf-8?q?_build_issue_introduced_by_=2319922?= Message-ID: <3dcqTT4zthz7Ll8@mail.python.org> http://hg.python.org/cpython/rev/4221d5d9ac84 changeset: 87830:4221d5d9ac84 user: Christian Heimes date: Sun Dec 08 15:21:08 2013 +0100 summary: Attempt to fix OpenIndiana build issue introduced by #19922 files: configure | 12 +++++++++--- configure.ac | 12 +++++++++--- 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/configure b/configure --- a/configure +++ b/configure @@ -3447,12 +3447,18 @@ # On HP-UX mbstate_t requires _INCLUDE__STDC_A1_SOURCE case $ac_sys_system in -hp*|HP*) + hp*|HP*) + define_stdc_a1=yes;; + *) + define_stdc_a1=no;; +esac + +if test $define_stdc_a1 = yes +then $as_echo "#define _INCLUDE__STDC_A1_SOURCE 1" >>confdefs.h - ;; -esac +fi # # SGI compilers allow the specification of the both the ABI and the diff --git a/configure.ac b/configure.ac --- a/configure.ac +++ b/configure.ac @@ -524,10 +524,16 @@ # On HP-UX mbstate_t requires _INCLUDE__STDC_A1_SOURCE case $ac_sys_system in -hp*|HP*) + hp*|HP*) + define_stdc_a1=yes;; + *) + define_stdc_a1=no;; +esac + +if test $define_stdc_a1 = yes +then AC_DEFINE(_INCLUDE__STDC_A1_SOURCE, 1, Define to include mbstate_t for mbrtowc) - ;; -esac +fi # # SGI compilers allow the specification of the both the ABI and the -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 15:23:34 2013 From: python-checkins at python.org (matthias.klose) Date: Sun, 8 Dec 2013 15:23:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_-_Issue_=2319736=3A_Add_mo?= =?utf-8?q?dule-level_statvfs_constants_defined_for_GNU/glibc?= Message-ID: <3dcqX665Xdz7LjN@mail.python.org> http://hg.python.org/cpython/rev/1d0b7e90da4d changeset: 87831:1d0b7e90da4d user: doko at ubuntu.com date: Sun Dec 08 15:23:07 2013 +0100 summary: - Issue #19736: Add module-level statvfs constants defined for GNU/glibc based systems. files: Doc/library/os.rst | 15 +++++++++++++++ Misc/NEWS | 3 +++ Modules/posixmodule.c | 29 +++++++++++++++++++++++++++++ 3 files changed, 47 insertions(+), 0 deletions(-) diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -1980,11 +1980,26 @@ read-only, and if :const:`ST_NOSUID` is set, the semantics of setuid/setgid bits are disabled or not supported. + Additional module-level constants are defined for GNU/glibc based systems. + These are :const:`ST_NODEV` (disallow access to device special files), + :const:`ST_NOEXEC` (disallow program execution), :const:`ST_SYNCHRONOUS` + (writes are synced at once), :const:`ST_MANDLOCK` (allow mandatory locks on an FS), + :const:`ST_WRITE` (write on file/directory/symlink), :const:`ST_APPEND` + (append-only file), :const:`ST_IMMUTABLE` (immutable file), :const:`ST_NOATIME` + (do not update access times), :const:`ST_NODIRATIME` (do not update directory access + times), :const:`ST_RELATIME` (update atime relative to mtime/ctime). + This function can support :ref:`specifying a file descriptor `. .. versionchanged:: 3.2 The :const:`ST_RDONLY` and :const:`ST_NOSUID` constants were added. + .. versionchanged:: 3.4 + The :const:`ST_NODEV`, :const:`ST_NOEXEC`, :const:`ST_SYNCHRONOUS`, + :const:`ST_MANDLOCK`, :const:`ST_WRITE`, :const:`ST_APPEND`, + :const:`ST_IMMUTABLE`, :const:`ST_NOATIME`, :const:`ST_NODIRATIME`, + and :const:`ST_RELATIME` constants were added. + Availability: Unix. .. versionadded:: 3.3 diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #19736: Add module-level statvfs constants defined for GNU/glibc + based systems. + - Issue #19729: In str.format(), fix recursive expansion in format spec. - Issue #19638: Fix possible crash / undefined behaviour from huge (more than 2 diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -11858,6 +11858,35 @@ if (PyModule_AddIntMacro(m, ST_NOSUID)) return -1; #endif /* ST_NOSUID */ + /* GNU extensions */ +#ifdef ST_NODEV + if (PyModule_AddIntMacro(m, ST_NODEV)) return -1; +#endif /* ST_NODEV */ +#ifdef ST_NOEXEC + if (PyModule_AddIntMacro(m, ST_NOEXEC)) return -1; +#endif /* ST_NOEXEC */ +#ifdef ST_SYNCHRONOUS + if (PyModule_AddIntMacro(m, ST_SYNCHRONOUS)) return -1; +#endif /* ST_SYNCHRONOUS */ +#ifdef ST_MANDLOCK + if (PyModule_AddIntMacro(m, ST_MANDLOCK)) return -1; +#endif /* ST_MANDLOCK */ +#ifdef ST_WRITE + if (PyModule_AddIntMacro(m, ST_WRITE)) return -1; +#endif /* ST_WRITE */ +#ifdef ST_APPEND + if (PyModule_AddIntMacro(m, ST_APPEND)) return -1; +#endif /* ST_APPEND */ +#ifdef ST_NOATIME + if (PyModule_AddIntMacro(m, ST_NOATIME)) return -1; +#endif /* ST_NOATIME */ +#ifdef ST_NODIRATIME + if (PyModule_AddIntMacro(m, ST_NODIRATIME)) return -1; +#endif /* ST_NODIRATIME */ +#ifdef ST_RELATIME + if (PyModule_AddIntMacro(m, ST_RELATIME)) return -1; +#endif /* ST_RELATIME */ + /* FreeBSD sendfile() constants */ #ifdef SF_NODISKIO if (PyModule_AddIntMacro(m, SF_NODISKIO)) return -1; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 15:49:32 2013 From: python-checkins at python.org (nadeem.vawda) Date: Sun, 8 Dec 2013 15:49:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogQ2xvc2VzICMxOTg3?= =?utf-8?q?8=3A_Fix_segfault_in_bz2_module=2E?= Message-ID: <3dcr64405Pz7LjS@mail.python.org> http://hg.python.org/cpython/rev/55a748f6e396 changeset: 87832:55a748f6e396 branch: 2.7 parent: 87797:9bf55766d935 user: Nadeem Vawda date: Sun Dec 08 15:31:50 2013 +0100 summary: Closes #19878: Fix segfault in bz2 module. Initial patch by Vajrasky Kok. files: Lib/test/test_bz2.py | 12 ++++++++++++ Misc/NEWS | 3 +++ Modules/bz2module.c | 17 ++++++++++------- 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/Lib/test/test_bz2.py b/Lib/test/test_bz2.py --- a/Lib/test/test_bz2.py +++ b/Lib/test/test_bz2.py @@ -325,6 +325,18 @@ self.assertRaises(ValueError, f.readline) self.assertRaises(ValueError, f.readlines) + def testInitNonExistentFile(self): + # Issue #19878: Should not segfault when __init__ with non-existent + # file for the second time. + self.createTempFile() + # Test close(): + with BZ2File(self.filename, "wb") as f: + self.assertRaises(IOError, f.__init__, "non-existent-file") + # Test object deallocation without call to close(): + f = bz2.BZ2File(self.filename) + self.assertRaises(IOError, f.__init__, "non-existent-file") + del f + class BZ2CompressorTest(BaseTest): def testCompress(self): # "Test BZ2Compressor.compress()/flush()" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,9 @@ Library ------- +- Issue #19878: Fix segfault in bz2 module after calling __init__ twice with + non-existent filename. Initial patch by Vajrasky Kok. + - Issue #16373: Prevent infinite recursion for ABC Set class comparisons. - Issue #19138: doctest's IGNORE_EXCEPTION_DETAIL now allows a match when diff --git a/Modules/bz2module.c b/Modules/bz2module.c --- a/Modules/bz2module.c +++ b/Modules/bz2module.c @@ -1206,12 +1206,16 @@ 0, NULL, NULL); break; } - if (self->fp) { - PyFile_DecUseCount((PyFileObject *)self->file); - self->fp = NULL; + if (self->file) { + if (self->fp) + PyFile_DecUseCount((PyFileObject *)self->file); + ret = PyObject_CallMethod(self->file, "close", NULL); + } else { + Py_INCREF(Py_None); + ret = Py_None; } + self->fp = NULL; self->mode = MODE_CLOSED; - ret = PyObject_CallMethod(self->file, "close", NULL); if (bzerror != BZ_OK) { Util_CatchBZ2Error(bzerror); Py_XDECREF(ret); @@ -1479,10 +1483,9 @@ 0, NULL, NULL); break; } - if (self->fp) { + if (self->fp != NULL && self->file != NULL) PyFile_DecUseCount((PyFileObject *)self->file); - self->fp = NULL; - } + self->fp = NULL; Util_DropReadAhead(self); Py_XDECREF(self->file); Py_TYPE(self)->tp_free((PyObject *)self); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 16:45:39 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 8 Dec 2013 16:45:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5MDk5?= =?utf-8?q?=3A_The_struct_module_now_supports_Unicode_format_strings=2E?= Message-ID: <3dcsLq00nNz7LjS@mail.python.org> http://hg.python.org/cpython/rev/42d3afd29460 changeset: 87833:42d3afd29460 branch: 2.7 user: Serhiy Storchaka date: Sun Dec 08 17:44:50 2013 +0200 summary: Issue #19099: The struct module now supports Unicode format strings. files: Lib/test/test_struct.py | 12 ++++++++++++ Misc/NEWS | 2 ++ Modules/_struct.c | 23 +++++++++++++++++++---- 3 files changed, 33 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_struct.py b/Lib/test/test_struct.py --- a/Lib/test/test_struct.py +++ b/Lib/test/test_struct.py @@ -574,6 +574,18 @@ self.check_sizeof('0s', 1) self.check_sizeof('0c', 0) + def test_unicode_format(self): + try: + unicode + except NameError: + self.skipTest('no unicode support') + # Issue #19099 + s = struct.Struct(unichr(ord('I'))) + self.assertEqual(s.format, 'I') + self.assertIs(type(s.format), str) + self.assertRaises(ValueError, struct.Struct, unichr(0x80)) + + def test_main(): support.run_unittest(StructTest) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -15,6 +15,8 @@ Library ------- +- Issue #19099: The struct module now supports Unicode format strings. + - Issue #19878: Fix segfault in bz2 module after calling __init__ twice with non-existent filename. Initial patch by Vajrasky Kok. diff --git a/Modules/_struct.c b/Modules/_struct.c --- a/Modules/_struct.c +++ b/Modules/_struct.c @@ -1371,13 +1371,28 @@ assert(PyStruct_Check(self)); - if (!PyArg_ParseTupleAndKeywords(args, kwds, "S:Struct", kwlist, + if (!PyArg_ParseTupleAndKeywords(args, kwds, "O:Struct", kwlist, &o_format)) return -1; - Py_INCREF(o_format); - Py_CLEAR(soself->s_format); - soself->s_format = o_format; + if (PyString_Check(o_format)) { + Py_INCREF(o_format); + Py_CLEAR(soself->s_format); + soself->s_format = o_format; + } + else if (PyUnicode_Check(o_format)) { + PyObject *str = PyUnicode_AsEncodedString(o_format, "ascii", NULL); + if (str == NULL) + return -1; + Py_CLEAR(soself->s_format); + soself->s_format = str; + } + else { + PyErr_Format(PyExc_TypeError, + "Struct() argument 1 must be string, not %s", + Py_TYPE(o_format)->tp_name); + return -1; + } ret = prepare_s(soself); return ret; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 17:16:37 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 8 Dec 2013 17:16:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5NTM1?= =?utf-8?q?=3A_Fixed_test=5Fdocxmlrpc_when_python_is_run_with_-OO=2E?= Message-ID: <3dct2Y0lKpz7LjS@mail.python.org> http://hg.python.org/cpython/rev/910b1cb5176c changeset: 87834:910b1cb5176c branch: 3.3 parent: 87826:3580c718add3 user: Serhiy Storchaka date: Sun Dec 08 18:14:49 2013 +0200 summary: Issue #19535: Fixed test_docxmlrpc when python is run with -OO. files: Lib/test/test_docxmlrpc.py | 6 ++++-- Misc/NEWS | 2 ++ 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_docxmlrpc.py b/Lib/test/test_docxmlrpc.py --- a/Lib/test/test_docxmlrpc.py +++ b/Lib/test/test_docxmlrpc.py @@ -202,10 +202,12 @@ """ Test that annotations works as expected """ self.client.request("GET", "/") response = self.client.getresponse() + docstring = (b'' if sys.flags.optimize >= 2 else + b'
Use function annotations.
') self.assertIn( (b'
annotation' - b'(x: int)
Use function annotations.' - b'
\n
' + b'(x: int)
' + docstring + b'
\n' + b'
' b'method_annotation(x: bytes)
'), response.read()) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -97,6 +97,8 @@ Tests ----- +- Issue #19535: Fixed test_docxmlrpc when python is run with -OO. + - Issue #19926: Removed unneeded test_main from test_abstract_numbers. Patch by Vajrasky Kok. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 17:16:38 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sun, 8 Dec 2013 17:16:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319535=3A_Fixed_test=5Fdocxmlrpc=2C_test=5Ffunct?= =?utf-8?q?ools=2C_test=5Finspect=2C_and?= Message-ID: <3dct2Z2nsjz7Ll3@mail.python.org> http://hg.python.org/cpython/rev/e71142abf8b6 changeset: 87835:e71142abf8b6 parent: 87831:1d0b7e90da4d parent: 87834:910b1cb5176c user: Serhiy Storchaka date: Sun Dec 08 18:16:18 2013 +0200 summary: Issue #19535: Fixed test_docxmlrpc, test_functools, test_inspect, and test_statistics when python is run with -OO. files: Lib/test/test_docxmlrpc.py | 6 ++++-- Lib/test/test_functools.py | 3 ++- Lib/test/test_inspect.py | 3 ++- Lib/test/test_statistics.py | 3 +++ Misc/NEWS | 3 +++ 5 files changed, 14 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_docxmlrpc.py b/Lib/test/test_docxmlrpc.py --- a/Lib/test/test_docxmlrpc.py +++ b/Lib/test/test_docxmlrpc.py @@ -202,10 +202,12 @@ """ Test that annotations works as expected """ self.client.request("GET", "/") response = self.client.getresponse() + docstring = (b'' if sys.flags.optimize >= 2 else + b'
Use function annotations.
') self.assertIn( (b'
annotation' - b'(x: int)
Use function annotations.' - b'
\n
' + b'(x: int)
' + docstring + b'
\n' + b'
' b'method_annotation(x: bytes)
'), response.read()) diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -1126,7 +1126,8 @@ "Simple test" return "Test" self.assertEqual(g.__name__, "g") - self.assertEqual(g.__doc__, "Simple test") + if sys.flags.optimize < 2: + self.assertEqual(g.__doc__, "Simple test") @unittest.skipUnless(decimal, 'requires _decimal') @support.cpython_only diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -2529,7 +2529,8 @@ # Just a quick sanity check on the output self.assertIn(module.__name__, output) self.assertIn(module.__file__, output) - self.assertIn(module.__cached__, output) + if not sys.flags.optimize: + self.assertIn(module.__cached__, output) self.assertEqual(err, b'') diff --git a/Lib/test/test_statistics.py b/Lib/test/test_statistics.py --- a/Lib/test/test_statistics.py +++ b/Lib/test/test_statistics.py @@ -8,6 +8,7 @@ import doctest import math import random +import sys import types import unittest @@ -625,6 +626,8 @@ class DocTests(unittest.TestCase): + @unittest.skipIf(sys.flags.optimize >= 2, + "Docstrings are omitted with -OO and above") def test_doc_tests(self): failed, tried = doctest.testmod(statistics) self.assertGreater(tried, 0) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -112,6 +112,9 @@ Tests ----- +- Issue #19535: Fixed test_docxmlrpc, test_functools, test_inspect, and + test_statistics when python is run with -OO. + - Issue #19926: Removed unneeded test_main from test_abstract_numbers. Patch by Vajrasky Kok. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 19:50:27 2013 From: python-checkins at python.org (nadeem.vawda) Date: Sun, 8 Dec 2013 19:50:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzE4NDMwOiBEb2N1?= =?utf-8?q?ment_that_peek=28=29_may_change_the_position_of_the_underlying_?= =?utf-8?q?file_for?= Message-ID: <3dcxS354LNz7LjM@mail.python.org> http://hg.python.org/cpython/rev/5ca6e8af0aab changeset: 87836:5ca6e8af0aab branch: 3.3 parent: 87834:910b1cb5176c user: Nadeem Vawda date: Sun Dec 08 19:47:22 2013 +0100 summary: #18430: Document that peek() may change the position of the underlying file for the BZ2File, GzipFile and LZMAFile classes. files: Doc/library/bz2.rst | 5 +++++ Doc/library/gzip.rst | 5 +++++ Doc/library/lzma.rst | 5 +++++ 3 files changed, 15 insertions(+), 0 deletions(-) diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst --- a/Doc/library/bz2.rst +++ b/Doc/library/bz2.rst @@ -91,6 +91,11 @@ byte of data will be returned (unless at EOF). The exact number of bytes returned is unspecified. + .. note:: While calling :meth:`peek` does not change the file position of + the :class:`BZ2File`, it may change the position of the underlying file + object (e.g. if the :class:`BZ2File` was constructed by passing a file + object for *filename*). + .. versionadded:: 3.3 .. versionchanged:: 3.1 diff --git a/Doc/library/gzip.rst b/Doc/library/gzip.rst --- a/Doc/library/gzip.rst +++ b/Doc/library/gzip.rst @@ -113,6 +113,11 @@ the call. The number of bytes returned may be more or less than requested. + .. note:: While calling :meth:`peek` does not change the file position of + the :class:`GzipFile`, it may change the position of the underlying + file object (e.g. if the :class:`GzipFile` was constructed with the + *fileobj* parameter). + .. versionadded:: 3.2 .. versionchanged:: 3.1 diff --git a/Doc/library/lzma.rst b/Doc/library/lzma.rst --- a/Doc/library/lzma.rst +++ b/Doc/library/lzma.rst @@ -98,6 +98,11 @@ byte of data will be returned, unless EOF has been reached. The exact number of bytes returned is unspecified (the *size* argument is ignored). + .. note:: While calling :meth:`peek` does not change the file position of + the :class:`LZMAFile`, it may change the position of the underlying + file object (e.g. if the :class:`LZMAFile` was constructed by passing a + file object for *filename*). + Compressing and decompressing data in memory -------------------------------------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 19:50:29 2013 From: python-checkins at python.org (nadeem.vawda) Date: Sun, 8 Dec 2013 19:50:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Closes_=2318430=3A_Document_that_peek=28=29_may_change_t?= =?utf-8?q?he_position_of_the_underlying?= Message-ID: <3dcxS516zSz7Ljn@mail.python.org> http://hg.python.org/cpython/rev/0f587fe304be changeset: 87837:0f587fe304be parent: 87835:e71142abf8b6 parent: 87836:5ca6e8af0aab user: Nadeem Vawda date: Sun Dec 08 19:50:05 2013 +0100 summary: Closes #18430: Document that peek() may change the position of the underlying file for the BZ2File, GzipFile and LZMAFile classes. files: Doc/library/bz2.rst | 5 +++++ Doc/library/gzip.rst | 5 +++++ Doc/library/lzma.rst | 5 +++++ 3 files changed, 15 insertions(+), 0 deletions(-) diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst --- a/Doc/library/bz2.rst +++ b/Doc/library/bz2.rst @@ -95,6 +95,11 @@ byte of data will be returned (unless at EOF). The exact number of bytes returned is unspecified. + .. note:: While calling :meth:`peek` does not change the file position of + the :class:`BZ2File`, it may change the position of the underlying file + object (e.g. if the :class:`BZ2File` was constructed by passing a file + object for *filename*). + .. versionadded:: 3.3 .. versionchanged:: 3.1 diff --git a/Doc/library/gzip.rst b/Doc/library/gzip.rst --- a/Doc/library/gzip.rst +++ b/Doc/library/gzip.rst @@ -117,6 +117,11 @@ the call. The number of bytes returned may be more or less than requested. + .. note:: While calling :meth:`peek` does not change the file position of + the :class:`GzipFile`, it may change the position of the underlying + file object (e.g. if the :class:`GzipFile` was constructed with the + *fileobj* parameter). + .. versionadded:: 3.2 .. versionchanged:: 3.1 diff --git a/Doc/library/lzma.rst b/Doc/library/lzma.rst --- a/Doc/library/lzma.rst +++ b/Doc/library/lzma.rst @@ -102,6 +102,11 @@ byte of data will be returned, unless EOF has been reached. The exact number of bytes returned is unspecified (the *size* argument is ignored). + .. note:: While calling :meth:`peek` does not change the file position of + the :class:`LZMAFile`, it may change the position of the underlying + file object (e.g. if the :class:`LZMAFile` was constructed by passing a + file object for *filename*). + .. versionchanged:: 3.4 Added support for the ``"x"`` and ``"xb"`` modes. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 19:58:40 2013 From: python-checkins at python.org (gregory.p.smith) Date: Sun, 8 Dec 2013 19:58:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Fixes_issue_?= =?utf-8?q?=2319929=3A_Call_os=2Eread_with_32768_within_subprocess=2EPopen?= Message-ID: <3dcxdX0BHsz7LlJ@mail.python.org> http://hg.python.org/cpython/rev/03a056c3b88e changeset: 87838:03a056c3b88e branch: 3.3 parent: 87836:5ca6e8af0aab user: Gregory P. Smith date: Sun Dec 08 10:56:07 2013 -0800 summary: Fixes issue #19929: Call os.read with 32768 within subprocess.Popen communicate rather than 4096 for efficiency. A microbenchmark shows Linux and OS X both using ~50% less cpu time this way. files: Lib/subprocess.py | 2 +- Misc/NEWS | 4 ++++ 2 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -1654,7 +1654,7 @@ if self._input_offset >= len(self._input): close_unregister_and_remove(fd) elif mode & select_POLLIN_POLLPRI: - data = os.read(fd, 4096) + data = os.read(fd, 32768) if not data: close_unregister_and_remove(fd) self._fd2output[fd].append(data) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,10 @@ Library ------- +- Issue #19929: Call os.read with 32768 within subprocess.Popen.communicate + rather than 4096 for efficiency. A microbenchmark shows Linux and OS X + both using ~50% less cpu time this way. + - Issue #19506: Use a memoryview to avoid a data copy when piping data to stdin within subprocess.Popen.communicate. 5-10% less cpu usage. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 8 19:58:41 2013 From: python-checkins at python.org (gregory.p.smith) Date: Sun, 8 Dec 2013 19:58:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Fixes_issue_=2319929=3A_Call_os=2Eread_with_32768_within?= =?utf-8?q?_subprocess=2EPopen?= Message-ID: <3dcxdY2TYGz7Ljn@mail.python.org> http://hg.python.org/cpython/rev/4de4b5a4e405 changeset: 87839:4de4b5a4e405 parent: 87837:0f587fe304be parent: 87838:03a056c3b88e user: Gregory P. Smith date: Sun Dec 08 10:58:28 2013 -0800 summary: Fixes issue #19929: Call os.read with 32768 within subprocess.Popen communicate rather than 4096 for efficiency. A microbenchmark shows Linux and OS X both using ~50% less cpu time this way. files: Lib/subprocess.py | 2 +- Misc/NEWS | 4 ++++ 2 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -1601,7 +1601,7 @@ selector.unregister(key.fileobj) key.fileobj.close() elif key.fileobj in (self.stdout, self.stderr): - data = os.read(key.fd, 4096) + data = os.read(key.fd, 32768) if not data: selector.unregister(key.fileobj) key.fileobj.close() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -24,6 +24,10 @@ - Issue #19343: Expose FreeBSD-specific APIs in resource module. Original patch by Koobs. +- Issue #19929: Call os.read with 32768 within subprocess.Popen.communicate + rather than 4096 for efficiency. A microbenchmark shows Linux and OS X + both using ~50% less cpu time this way. + - Issue #19506: Use a memoryview to avoid a data copy when piping data to stdin within subprocess.Popen.communicate. 5-10% less cpu usage. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 00:04:41 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 9 Dec 2013 00:04:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE3NDI5?= =?utf-8?q?=3A_platform=2Elinux=5Fdistribution=28=29_now_decodes_files_fro?= =?utf-8?q?m_the_UTF-8?= Message-ID: <3dd35P3mTbz7LjT@mail.python.org> http://hg.python.org/cpython/rev/4580976c07cb changeset: 87840:4580976c07cb branch: 3.3 parent: 87838:03a056c3b88e user: Victor Stinner date: Mon Dec 09 00:01:27 2013 +0100 summary: Issue #17429: platform.linux_distribution() now decodes files from the UTF-8 encoding with the surrogateescape error handler, instead of decoding from the locale encoding in strict mode. It fixes the function on Fedora 19 which is probably the first major distribution release with a non-ASCII name. Patch written by Toshio Kuratomi. files: Lib/platform.py | 9 +++++++-- Lib/test/test_platform.py | 16 ++++++++++++++++ Misc/ACKS | 1 + Misc/NEWS | 6 ++++++ 4 files changed, 30 insertions(+), 2 deletions(-) diff --git a/Lib/platform.py b/Lib/platform.py --- a/Lib/platform.py +++ b/Lib/platform.py @@ -129,6 +129,10 @@ # Standard Unix uses /dev/null DEV_NULL = '/dev/null' +# Directory to search for configuration information on Unix. +# Constant used by test_platform to test linux_distribution(). +_UNIXCONFDIR = '/etc' + ### Platform specific APIs _libc_search = re.compile(b'(__libc_init)' @@ -315,7 +319,7 @@ """ try: - etc = os.listdir('/etc') + etc = os.listdir(_UNIXCONFDIR) except os.error: # Probably not a Unix system return distname,version,id @@ -331,7 +335,8 @@ return _dist_try_harder(distname,version,id) # Read the first line - with open('/etc/'+file, 'r') as f: + with open(os.path.join(_UNIXCONFDIR, file), 'r', + encoding='utf-8', errors='surrogateescape') as f: firstline = f.readline() _distname, _version, _id = _parse_release_file(firstline) diff --git a/Lib/test/test_platform.py b/Lib/test/test_platform.py --- a/Lib/test/test_platform.py +++ b/Lib/test/test_platform.py @@ -1,7 +1,10 @@ +from unittest import mock +import contextlib import os import platform import subprocess import sys +import tempfile import unittest import warnings @@ -295,6 +298,19 @@ returncode = ret >> 8 self.assertEqual(returncode, len(data)) + def test_linux_distribution_encoding(self): + # Issue #17429 + with tempfile.TemporaryDirectory() as tempdir: + filename = os.path.join(tempdir, 'fedora-release') + with open(filename, 'w', encoding='utf-8') as f: + f.write('Fedora release 19 (Schr\xf6dinger\u2019s Cat)\n') + + with mock.patch('platform._UNIXCONFDIR', tempdir): + distname, version, distid = platform.linux_distribution() + + self.assertEqual(distname, 'Fedora') + self.assertEqual(version, '19') + self.assertEqual(distid, 'Schr\xf6dinger\u2019s Cat') def test_main(): support.run_unittest( diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -689,6 +689,7 @@ Andrew Kuchling Dave Kuhlman Jon Kuhn +Toshio Kuratomi Vladimir Kushnir Erno Kuusela Ross Lagerwall diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -18,6 +18,12 @@ Library ------- +- Issue #17429: platform.linux_distribution() now decodes files from the UTF-8 + encoding with the surrogateescape error handler, instead of decoding from the + locale encoding in strict mode. It fixes the function on Fedora 19 which is + probably the first major distribution release with a non-ASCII name. Patch + written by Toshio Kuratomi. + - Issue #19929: Call os.read with 32768 within subprocess.Popen.communicate rather than 4096 for efficiency. A microbenchmark shows Linux and OS X both using ~50% less cpu time this way. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 00:04:42 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 9 Dec 2013 00:04:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogKE1lcmdlIDMuMykgSXNzdWUgIzE3NDI5OiBwbGF0Zm9ybS5saW51eF9k?= =?utf-8?q?istribution=28=29_now_decodes_files_from?= Message-ID: <3dd35Q5ntMz7LjS@mail.python.org> http://hg.python.org/cpython/rev/407f18c8ce8a changeset: 87841:407f18c8ce8a parent: 87839:4de4b5a4e405 parent: 87840:4580976c07cb user: Victor Stinner date: Mon Dec 09 00:04:09 2013 +0100 summary: (Merge 3.3) Issue #17429: platform.linux_distribution() now decodes files from the UTF-8 encoding with the surrogateescape error handler, instead of decoding from the locale encoding in strict mode. It fixes the function on Fedora 19 which is probably the first major distribution release with a non-ASCII name. Patch written by Toshio Kuratomi. files: Lib/platform.py | 9 +++++++-- Lib/test/test_platform.py | 16 ++++++++++++++++ Misc/ACKS | 1 + Misc/NEWS | 6 ++++++ 4 files changed, 30 insertions(+), 2 deletions(-) diff --git a/Lib/platform.py b/Lib/platform.py --- a/Lib/platform.py +++ b/Lib/platform.py @@ -129,6 +129,10 @@ # Standard Unix uses /dev/null DEV_NULL = '/dev/null' +# Directory to search for configuration information on Unix. +# Constant used by test_platform to test linux_distribution(). +_UNIXCONFDIR = '/etc' + ### Platform specific APIs _libc_search = re.compile(b'(__libc_init)' @@ -315,7 +319,7 @@ """ try: - etc = os.listdir('/etc') + etc = os.listdir(_UNIXCONFDIR) except OSError: # Probably not a Unix system return distname,version,id @@ -331,7 +335,8 @@ return _dist_try_harder(distname,version,id) # Read the first line - with open('/etc/'+file, 'r') as f: + with open(os.path.join(_UNIXCONFDIR, file), 'r', + encoding='utf-8', errors='surrogateescape') as f: firstline = f.readline() _distname, _version, _id = _parse_release_file(firstline) diff --git a/Lib/test/test_platform.py b/Lib/test/test_platform.py --- a/Lib/test/test_platform.py +++ b/Lib/test/test_platform.py @@ -1,7 +1,10 @@ +from unittest import mock +import contextlib import os import platform import subprocess import sys +import tempfile import unittest import warnings @@ -295,6 +298,19 @@ returncode = ret >> 8 self.assertEqual(returncode, len(data)) + def test_linux_distribution_encoding(self): + # Issue #17429 + with tempfile.TemporaryDirectory() as tempdir: + filename = os.path.join(tempdir, 'fedora-release') + with open(filename, 'w', encoding='utf-8') as f: + f.write('Fedora release 19 (Schr\xf6dinger\u2019s Cat)\n') + + with mock.patch('platform._UNIXCONFDIR', tempdir): + distname, version, distid = platform.linux_distribution() + + self.assertEqual(distname, 'Fedora') + self.assertEqual(version, '19') + self.assertEqual(distid, 'Schr\xf6dinger\u2019s Cat') def test_main(): support.run_unittest( diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -711,6 +711,7 @@ Andrew Kuchling Dave Kuhlman Jon Kuhn +Toshio Kuratomi Vladimir Kushnir Erno Kuusela Ross Lagerwall diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -21,6 +21,12 @@ Library ------- +- Issue #17429: platform.linux_distribution() now decodes files from the UTF-8 + encoding with the surrogateescape error handler, instead of decoding from the + locale encoding in strict mode. It fixes the function on Fedora 19 which is + probably the first major distribution release with a non-ASCII name. Patch + written by Toshio Kuratomi. + - Issue #19343: Expose FreeBSD-specific APIs in resource module. Original patch by Koobs. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 00:20:34 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 9 Dec 2013 00:20:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2317429=3A_some_PEP?= =?utf-8?q?_8_compliance_fixes_for_the_platform_modules=2C_add?= Message-ID: <3dd3Rk2YTVz7Ljt@mail.python.org> http://hg.python.org/cpython/rev/831b2c80a9c9 changeset: 87842:831b2c80a9c9 user: Victor Stinner date: Mon Dec 09 00:14:52 2013 +0100 summary: Issue #17429: some PEP 8 compliance fixes for the platform modules, add whitespaces files: Lib/platform.py | 224 ++++++++++++++++++----------------- 1 files changed, 113 insertions(+), 111 deletions(-) diff --git a/Lib/platform.py b/Lib/platform.py --- a/Lib/platform.py +++ b/Lib/platform.py @@ -122,7 +122,7 @@ except AttributeError: # os.devnull was added in Python 2.4, so emulate it for earlier # Python versions - if sys.platform in ('dos','win32','win16'): + if sys.platform in ('dos', 'win32', 'win16'): # Use the old CP/M NUL as device name DEV_NULL = 'NUL' else: @@ -141,7 +141,7 @@ b'|' br'(libc(_\w+)?\.so(?:\.(\d[0-9.]*))?)', re.ASCII) -def libc_ver(executable=sys.executable,lib='',version='', +def libc_ver(executable=sys.executable, lib='', version='', chunksize=16384): @@ -163,12 +163,12 @@ # here to work around problems with Cygwin not being # able to open symlinks for reading executable = os.path.realpath(executable) - f = open(executable,'rb') + f = open(executable, 'rb') binary = f.read(chunksize) pos = 0 while 1: if b'libc' in binary or b'GLIBC' in binary: - m = _libc_search.search(binary,pos) + m = _libc_search.search(binary, pos) else: m = None if not m: @@ -177,7 +177,7 @@ break pos = 0 continue - libcinit,glibc,glibcversion,so,threads,soversion = [ + libcinit, glibc, glibcversion, so, threads, soversion = [ s.decode('latin1') if s is not None else s for s in m.groups()] if libcinit and not lib: @@ -197,9 +197,9 @@ version = version + threads pos = m.end() f.close() - return lib,version + return lib, version -def _dist_try_harder(distname,version,id): +def _dist_try_harder(distname, version, id): """ Tries some special tricks to get the distribution information in case the default method fails. @@ -214,7 +214,7 @@ for line in open('/var/adm/inst-log/info'): tv = line.split() if len(tv) == 2: - tag,value = tv + tag, value = tv else: continue if tag == 'MIN_DIST_VERSION': @@ -222,7 +222,7 @@ elif tag == 'DIST_IDENT': values = value.split('-') id = values[2] - return distname,version,id + return distname, version, id if os.path.exists('/etc/.installed'): # Caldera OpenLinux has some infos in that file (thanks to Colin Kong) @@ -231,7 +231,7 @@ if len(pkg) >= 2 and pkg[0] == 'OpenLinux': # XXX does Caldera support non Intel platforms ? If yes, # where can we find the needed id ? - return 'OpenLinux',pkg[1],id + return 'OpenLinux', pkg[1], id if os.path.isdir('/usr/lib/setup'): # Check for slackware version tag file (thanks to Greg Andruk) @@ -243,9 +243,9 @@ verfiles.sort() distname = 'slackware' version = verfiles[-1][14:] - return distname,version,id + return distname, version, id - return distname,version,id + return distname, version, id _release_filename = re.compile(r'(\w+)[-_](release|version)', re.ASCII) _lsb_release_version = re.compile(r'(.+)' @@ -314,7 +314,7 @@ distribution read from the OS is returned. Otherwise the short name taken from supported_dists is used. - Returns a tuple (distname,version,id) which default to the + Returns a tuple (distname, version, id) which default to the args given as parameters. """ @@ -322,17 +322,17 @@ etc = os.listdir(_UNIXCONFDIR) except OSError: # Probably not a Unix system - return distname,version,id + return distname, version, id etc.sort() for file in etc: m = _release_filename.match(file) if m is not None: - _distname,dummy = m.groups() + _distname, dummy = m.groups() if _distname in supported_dists: distname = _distname break else: - return _dist_try_harder(distname,version,id) + return _dist_try_harder(distname, version, id) # Read the first line with open(os.path.join(_UNIXCONFDIR, file), 'r', @@ -350,7 +350,7 @@ # To maintain backwards compatibility: -def dist(distname='',version='',id='', +def dist(distname='', version='', id='', supported_dists=_supported_dists): @@ -360,7 +360,7 @@ /etc and then reverts to _dist_try_harder() in case no suitable files are found. - Returns a tuple (distname,version,id) which default to the + Returns a tuple (distname, version, id) which default to the args given as parameters. """ @@ -385,11 +385,11 @@ if build: l.append(build) try: - ints = map(int,l) + ints = map(int, l) except ValueError: strings = l else: - strings = list(map(str,ints)) + strings = list(map(str, ints)) version = '.'.join(strings[:3]) return version @@ -408,10 +408,10 @@ def _syscmd_ver(system='', release='', version='', - supported_platforms=('win32','win16','dos')): + supported_platforms=('win32', 'win16', 'dos')): """ Tries to figure out the OS version used and returns - a tuple (system,release,version). + a tuple (system, release, version). It uses the "ver" shell command for this which is known to exists on Windows, DOS. XXX Others too ? @@ -421,10 +421,10 @@ """ if sys.platform not in supported_platforms: - return system,release,version + return system, release, version # Try some common cmd strings - for cmd in ('ver','command /c ver','cmd /c ver'): + for cmd in ('ver', 'command /c ver', 'cmd /c ver'): try: pipe = popen(cmd) info = pipe.read() @@ -433,18 +433,18 @@ # XXX How can I suppress shell errors from being written # to stderr ? except OSError as why: - #print 'Command %s failed: %s' % (cmd,why) + #print 'Command %s failed: %s' % (cmd, why) continue else: break else: - return system,release,version + return system, release, version # Parse the output info = info.strip() m = _ver_output.match(info) if m is not None: - system,release,version = m.groups() + system, release, version = m.groups() # Strip trailing dots from version and release if release[-1] == '.': release = release[:-1] @@ -453,9 +453,9 @@ # Normalize the version and build strings (eliminating additional # zeros) version = _norm_version(version) - return system,release,version + return system, release, version -def _win32_getvalue(key,name,default=''): +def _win32_getvalue(key, name, default=''): """ Read a value for name from the registry key. @@ -470,14 +470,14 @@ import winreg RegQueryValueEx = winreg.QueryValueEx try: - return RegQueryValueEx(key,name) + return RegQueryValueEx(key, name) except: return default -def win32_ver(release='',version='',csd='',ptype=''): +def win32_ver(release='', version='', csd='', ptype=''): """ Get additional version information from the Windows Registry - and return a tuple (version,csd,ptype) referring to version + and return a tuple (version, csd, ptype) referring to version number, CSD level (service pack), and OS type (multi/single processor). @@ -514,7 +514,7 @@ sys.getwindowsversion except AttributeError: # No emulation possible, so return the defaults... - return release,version,csd,ptype + return release, version, csd, ptype else: # Emulation using winreg (added in Python 2.0) and # sys.getwindowsversion() (added in Python 2.3) @@ -532,8 +532,8 @@ # Find out the registry key and some general version infos winver = GetVersionEx() - maj,min,buildno,plat,csd = winver - version = '%i.%i.%i' % (maj,min,buildno & 0xFFFF) + maj, min, buildno, plat, csd = winver + version = '%i.%i.%i' % (maj, min, buildno & 0xFFFF) if hasattr(winver, "service_pack"): if winver.service_pack != "": csd = 'SP%s' % winver.service_pack_major @@ -608,8 +608,8 @@ else: if not release: # E.g. Win3.1 with win32s - release = '%i.%i' % (maj,min) - return release,version,csd,ptype + release = '%i.%i' % (maj, min) + return release, version, csd, ptype # Open the registry key try: @@ -617,7 +617,7 @@ # Get a value to make sure the key exists... RegQueryValueEx(keyCurVer, 'SystemRoot') except: - return release,version,csd,ptype + return release, version, csd, ptype # Parse values #subversion = _win32_getvalue(keyCurVer, @@ -627,17 +627,17 @@ # release = release + subversion # 95a, 95b, etc. build = _win32_getvalue(keyCurVer, 'CurrentBuildNumber', - ('',1))[0] + ('', 1))[0] ptype = _win32_getvalue(keyCurVer, 'CurrentType', - (ptype,1))[0] + (ptype, 1))[0] # Normalize version - version = _norm_version(version,build) + version = _norm_version(version, build) # Close key RegCloseKey(keyCurVer) - return release,version,csd,ptype + return release, version, csd, ptype def _mac_ver_xml(): fn = '/System/Library/CoreServices/SystemVersion.plist' @@ -651,16 +651,16 @@ pl = plistlib.readPlist(fn) release = pl['ProductVersion'] - versioninfo=('', '', '') + versioninfo = ('', '', '') machine = os.uname().machine if machine in ('ppc', 'Power Macintosh'): # Canonical name machine = 'PowerPC' - return release,versioninfo,machine + return release, versioninfo, machine -def mac_ver(release='',versioninfo=('','',''),machine=''): +def mac_ver(release='', versioninfo=('', '', ''), machine=''): """ Get MacOS version information and return it as tuple (release, versioninfo, machine) with versioninfo being a tuple (version, @@ -677,9 +677,9 @@ return info # If that also doesn't work return the default values - return release,versioninfo,machine + return release, versioninfo, machine -def _java_getprop(name,default): +def _java_getprop(name, default): from java.lang import System try: @@ -690,13 +690,13 @@ except AttributeError: return default -def java_ver(release='',vendor='',vminfo=('','',''),osinfo=('','','')): +def java_ver(release='', vendor='', vminfo=('', '', ''), osinfo=('', '', '')): """ Version interface for Jython. - Returns a tuple (release,vendor,vminfo,osinfo) with vminfo being - a tuple (vm_name,vm_release,vm_vendor) and osinfo being a - tuple (os_name,os_version,os_arch). + Returns a tuple (release, vendor, vminfo, osinfo) with vminfo being + a tuple (vm_name, vm_release, vm_vendor) and osinfo being a + tuple (os_name, os_version, os_arch). Values which cannot be determined are set to the defaults given as parameters (which all default to ''). @@ -706,7 +706,7 @@ try: import java.lang except ImportError: - return release,vendor,vminfo,osinfo + return release, vendor, vminfo, osinfo vendor = _java_getprop('java.vendor', vendor) release = _java_getprop('java.version', release) @@ -725,9 +725,9 @@ ### System name aliasing -def system_alias(system,release,version): +def system_alias(system, release, version): - """ Returns (system,release,version) aliased to common + """ Returns (system, release, version) aliased to common marketing names used for some systems. It also does some reordering of the information in some cases @@ -737,13 +737,13 @@ if system == 'Rhapsody': # Apple's BSD derivative # XXX How can we determine the marketing release number ? - return 'MacOS X Server',system+release,version + return 'MacOS X Server', system+release, version elif system == 'SunOS': # Sun's OS if release < '5': # These releases use the old name SunOS - return system,release,version + return system, release, version # Modify release (marketing release = SunOS release - 3) l = release.split('.') if l: @@ -771,11 +771,11 @@ else: version = '64bit' - elif system in ('win32','win16'): + elif system in ('win32', 'win16'): # In case one of the other tricks system = 'Windows' - return system,release,version + return system, release, version ### Various internal helpers @@ -788,21 +788,21 @@ platform = '-'.join(x.strip() for x in filter(len, args)) # Cleanup some possible filename obstacles... - platform = platform.replace(' ','_') - platform = platform.replace('/','-') - platform = platform.replace('\\','-') - platform = platform.replace(':','-') - platform = platform.replace(';','-') - platform = platform.replace('"','-') - platform = platform.replace('(','-') - platform = platform.replace(')','-') + platform = platform.replace(' ', '_') + platform = platform.replace('/', '-') + platform = platform.replace('\\', '-') + platform = platform.replace(':', '-') + platform = platform.replace(';', '-') + platform = platform.replace('"', '-') + platform = platform.replace('(', '-') + platform = platform.replace(')', '-') # No need to report 'unknown' information... - platform = platform.replace('unknown','') + platform = platform.replace('unknown', '') # Fold '--'s and remove trailing '-' while 1: - cleaned = platform.replace('--','-') + cleaned = platform.replace('--', '-') if cleaned == platform: break platform = cleaned @@ -834,14 +834,14 @@ filepath = os.path.abspath(filepath) while os.path.islink(filepath): filepath = os.path.normpath( - os.path.join(os.path.dirname(filepath),os.readlink(filepath))) + os.path.join(os.path.dirname(filepath), os.readlink(filepath))) return filepath -def _syscmd_uname(option,default=''): +def _syscmd_uname(option, default=''): """ Interface to the system's uname command. """ - if sys.platform in ('dos','win32','win16'): + if sys.platform in ('dos', 'win32', 'win16'): # XXX Others too ? return default try: @@ -855,7 +855,7 @@ else: return output -def _syscmd_file(target,default=''): +def _syscmd_file(target, default=''): """ Interface to the system's file command. @@ -864,7 +864,7 @@ default in case the command should fail. """ - if sys.platform in ('dos','win32','win16'): + if sys.platform in ('dos', 'win32', 'win16'): # XXX Others too ? return default target = _follow_symlinks(target) @@ -886,17 +886,17 @@ # Default values for architecture; non-empty strings override the # defaults given as parameters _default_architecture = { - 'win32': ('','WindowsPE'), - 'win16': ('','Windows'), - 'dos': ('','MSDOS'), + 'win32': ('', 'WindowsPE'), + 'win16': ('', 'Windows'), + 'dos': ('', 'MSDOS'), } -def architecture(executable=sys.executable,bits='',linkage=''): +def architecture(executable=sys.executable, bits='', linkage=''): """ Queries the given executable (defaults to the Python interpreter binary) for various architecture information. - Returns a tuple (bits,linkage) which contains information about + Returns a tuple (bits, linkage) which contains information about the bit architecture and the linkage format used for the executable. Both values are returned as strings. @@ -934,16 +934,16 @@ # "file" command did not return anything; we'll try to provide # some sensible defaults then... if sys.platform in _default_architecture: - b,l = _default_architecture[sys.platform] + b, l = _default_architecture[sys.platform] if b: bits = b if l: linkage = l - return bits,linkage + return bits, linkage if 'executable' not in fileout: # Format not supported - return bits,linkage + return bits, linkage # Bits if '32-bit' in fileout: @@ -971,7 +971,7 @@ # XXX the A.OUT format also falls under this class... pass - return bits,linkage + return bits, linkage ### Portable uname() interface @@ -983,7 +983,7 @@ def uname(): """ Fairly portable uname interface. Returns a tuple - of strings (system,node,release,version,machine,processor) + of strings (system, node, release, version, machine, processor) identifying the underlying platform. Note that unlike the os.uname function this also returns @@ -1002,7 +1002,7 @@ # Get some infos from the builtin os.uname API... try: - system,node,release,version,machine = os.uname() + system, node, release, version, machine = os.uname() except AttributeError: no_os_uname = 1 @@ -1020,7 +1020,7 @@ # Try win32_ver() on win32 platforms if system == 'win32': - release,version,csd,ptype = win32_ver() + release, version, csd, ptype = win32_ver() if release and version: use_syscmd_ver = 0 # Try to use the PROCESSOR_* environment variables @@ -1039,7 +1039,7 @@ # Try the 'ver' system command available on some # platforms if use_syscmd_ver: - system,release,version = _syscmd_ver(system) + system, release, version = _syscmd_ver(system) # Normalize system to what win32_ver() normally returns # (_syscmd_ver() tends to return the vendor name as well) if system == 'Microsoft Windows': @@ -1057,7 +1057,7 @@ # In case we still don't know anything useful, we'll try to # help ourselves - if system in ('win32','win16'): + if system in ('win32', 'win16'): if not version: if system == 'win32': version = '32bit' @@ -1066,7 +1066,7 @@ system = 'Windows' elif system[:4] == 'java': - release,vendor,vminfo,osinfo = java_ver() + release, vendor, vminfo, osinfo = java_ver() system = 'Java' version = ', '.join(vminfo) if not version: @@ -1084,14 +1084,14 @@ except ImportError: pass else: - csid, cpu_number = vms_lib.getsyi('SYI$_CPU',0) + csid, cpu_number = vms_lib.getsyi('SYI$_CPU', 0) if (cpu_number >= 128): processor = 'Alpha' else: processor = 'VAX' if not processor: # Get processor information from the uname system command - processor = _syscmd_uname('-p','') + processor = _syscmd_uname('-p', '') #If any unknowns still exist, replace them with ''s, which are more portable if system == 'unknown': @@ -1112,7 +1112,8 @@ system = 'Windows' release = 'Vista' - _uname_cache = uname_result(system,node,release,version,machine,processor) + _uname_cache = uname_result(system, node, release, version, + machine, processor) return _uname_cache ### Direct interfaces to some of the uname() return values @@ -1409,57 +1410,58 @@ # Get uname information and then apply platform specific cosmetics # to it... - system,node,release,version,machine,processor = uname() + system, node, release, version, machine, processor = uname() if machine == processor: processor = '' if aliased: - system,release,version = system_alias(system,release,version) + system, release, version = system_alias(system, release, version) if system == 'Windows': # MS platforms - rel,vers,csd,ptype = win32_ver(version) + rel, vers, csd, ptype = win32_ver(version) if terse: - platform = _platform(system,release) + platform = _platform(system, release) else: - platform = _platform(system,release,version,csd) + platform = _platform(system, release, version, csd) elif system in ('Linux',): # Linux based systems - distname,distversion,distid = dist('') + distname, distversion, distid = dist('') if distname and not terse: - platform = _platform(system,release,machine,processor, + platform = _platform(system, release, machine, processor, 'with', - distname,distversion,distid) + distname, distversion, distid) else: # If the distribution name is unknown check for libc vs. glibc - libcname,libcversion = libc_ver(sys.executable) - platform = _platform(system,release,machine,processor, + libcname, libcversion = libc_ver(sys.executable) + platform = _platform(system, release, machine, processor, 'with', libcname+libcversion) elif system == 'Java': # Java platforms - r,v,vminfo,(os_name,os_version,os_arch) = java_ver() + r, v, vminfo, (os_name, os_version, os_arch) = java_ver() if terse or not os_name: - platform = _platform(system,release,version) + platform = _platform(system, release, version) else: - platform = _platform(system,release,version, + platform = _platform(system, release, version, 'on', - os_name,os_version,os_arch) + os_name, os_version, os_arch) elif system == 'MacOS': # MacOS platforms if terse: - platform = _platform(system,release) + platform = _platform(system, release) else: - platform = _platform(system,release,machine) + platform = _platform(system, release, machine) else: # Generic handler if terse: - platform = _platform(system,release) + platform = _platform(system, release) else: - bits,linkage = architecture(sys.executable) - platform = _platform(system,release,machine,processor,bits,linkage) + bits, linkage = architecture(sys.executable) + platform = _platform(system, release, machine, + processor, bits, linkage) _platform_cache[(aliased, terse)] = platform return platform @@ -1470,5 +1472,5 @@ # Default is to print the aliased verbose platform string terse = ('terse' in sys.argv or '--terse' in sys.argv) aliased = (not 'nonaliased' in sys.argv and not '--nonaliased' in sys.argv) - print(platform(aliased,terse)) + print(platform(aliased, terse)) sys.exit(0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 00:26:46 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 9 Dec 2013 00:26:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE3NDI5?= =?utf-8?q?=3A_Oops=2C_remove_unused_import?= Message-ID: <3dd3Zt0vzVz7LjS@mail.python.org> http://hg.python.org/cpython/rev/a951ab03bda0 changeset: 87843:a951ab03bda0 branch: 3.3 parent: 87840:4580976c07cb user: Victor Stinner date: Mon Dec 09 00:25:57 2013 +0100 summary: Issue #17429: Oops, remove unused import files: Lib/test/test_platform.py | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_platform.py b/Lib/test/test_platform.py --- a/Lib/test/test_platform.py +++ b/Lib/test/test_platform.py @@ -1,5 +1,4 @@ from unittest import mock -import contextlib import os import platform import subprocess -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 00:26:47 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 9 Dec 2013 00:26:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E3=29_Issue_=2317429=3A_Oops=2C_remove_unuse?= =?utf-8?q?d_import?= Message-ID: <3dd3Zv2l8tz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/209bf9576dc8 changeset: 87844:209bf9576dc8 parent: 87842:831b2c80a9c9 parent: 87843:a951ab03bda0 user: Victor Stinner date: Mon Dec 09 00:26:12 2013 +0100 summary: (Merge 3.3) Issue #17429: Oops, remove unused import files: Lib/test/test_platform.py | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_platform.py b/Lib/test/test_platform.py --- a/Lib/test/test_platform.py +++ b/Lib/test/test_platform.py @@ -1,5 +1,4 @@ from unittest import mock -import contextlib import os import platform import subprocess -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 01:15:33 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 9 Dec 2013 01:15:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_=2319830=3A_Fix_a_Reso?= =?utf-8?q?urceWarning_in_test=5Fpoplib=2E?= Message-ID: <3dd4g93cckz7LjT@mail.python.org> http://hg.python.org/cpython/rev/1e3c7153a14d changeset: 87845:1e3c7153a14d user: Victor Stinner date: Mon Dec 09 01:15:10 2013 +0100 summary: Fix #19830: Fix a ResourceWarning in test_poplib. Patch written by Vajrasky Kok. files: Lib/test/test_poplib.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_poplib.py b/Lib/test/test_poplib.py --- a/Lib/test/test_poplib.py +++ b/Lib/test/test_poplib.py @@ -421,7 +421,7 @@ # happens in the test_too_long_lines case; the overlong # response will be treated as response to QUIT and raise # this exception - pass + self.client.close() self.server.stop() def test_stls(self): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 01:56:41 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 9 Dec 2013 01:56:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Close_=2319880=3A_Fix_a_re?= =?utf-8?q?ference_leak_in_unittest=2ETestCase=2E_Explicitly_break?= Message-ID: <3dd5Zd3P8Lz7LjS@mail.python.org> http://hg.python.org/cpython/rev/09658ea0b93d changeset: 87846:09658ea0b93d user: Victor Stinner date: Mon Dec 09 01:52:50 2013 +0100 summary: Close #19880: Fix a reference leak in unittest.TestCase. Explicitly break reference cycles between frames and the _Outcome instance. files: Lib/unittest/case.py | 14 +++++++++++- Lib/unittest/test/test_case.py | 26 ++++++++++++++++++++++ Misc/NEWS | 3 ++ 3 files changed, 42 insertions(+), 1 deletions(-) diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py --- a/Lib/unittest/case.py +++ b/Lib/unittest/case.py @@ -69,6 +69,9 @@ else: self.success = False self.errors.append((test_case, exc_info)) + # explicitly break a reference cycle: + # exc_info -> frame -> exc_info + exc_info = None else: if self.result_supports_subtests and self.success: self.errors.append((test_case, None)) @@ -559,8 +562,8 @@ return expecting_failure = getattr(testMethod, "__unittest_expecting_failure__", False) + outcome = _Outcome(result) try: - outcome = _Outcome(result) self._outcome = outcome with outcome.testPartExecutor(self): @@ -593,6 +596,15 @@ if stopTestRun is not None: stopTestRun() + # explicitly break reference cycles: + # outcome.errors -> frame -> outcome -> outcome.errors + # outcome.expectedFailure -> frame -> outcome -> outcome.expectedFailure + outcome.errors.clear() + outcome.expectedFailure = None + + # clear the outcome, no more needed + self._outcome = None + def doCleanups(self): """Execute all cleanup functions. Normally called for you after tearDown.""" diff --git a/Lib/unittest/test/test_case.py b/Lib/unittest/test/test_case.py --- a/Lib/unittest/test/test_case.py +++ b/Lib/unittest/test/test_case.py @@ -1533,6 +1533,32 @@ del case self.assertFalse(wr()) + def test_no_exception_leak(self): + # Issue #19880: TestCase.run() should not keep a reference + # to the exception + class MyException(Exception): + ninstance = 0 + + def __init__(self): + MyException.ninstance += 1 + Exception.__init__(self) + + def __del__(self): + MyException.ninstance -= 1 + + class TestCase(unittest.TestCase): + def test1(self): + raise MyException() + + @unittest.expectedFailure + def test2(self): + raise MyException() + + for method_name in ('test1', 'test2'): + testcase = TestCase(method_name) + testcase.run() + self.assertEqual(MyException.ninstance, 0) + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -21,6 +21,9 @@ Library ------- +- Issue #19880: Fix a reference leak in unittest.TestCase. Explicitly break + reference cycles between frames and the _Outcome instance. + - Issue #17429: platform.linux_distribution() now decodes files from the UTF-8 encoding with the surrogateescape error handler, instead of decoding from the locale encoding in strict mode. It fixes the function on Fedora 19 which is -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 01:57:33 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 9 Dec 2013 01:57:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319876=3A_Run_also?= Message-ID: <3dd5bd232tz7LjS@mail.python.org> http://hg.python.org/cpython/rev/c4c1c4bc8086 changeset: 87847:c4c1c4bc8086 user: Victor Stinner date: Mon Dec 09 01:57:14 2013 +0100 summary: Issue #19876: Run also test_selectors.test_unregister_after_fd_close_and_reuse() on Windows os.dup2() is available on Windows. files: Lib/test/test_selectors.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_selectors.py b/Lib/test/test_selectors.py --- a/Lib/test/test_selectors.py +++ b/Lib/test/test_selectors.py @@ -109,7 +109,7 @@ s.unregister(r) s.unregister(w) - @unittest.skipUnless(os.name == 'posix', "requires posix") + @unittest.skipUnless(hasattr(os, 'dup2'), "need os.dup2()") def test_unregister_after_fd_close_and_reuse(self): s = self.SELECTOR() self.addCleanup(s.close) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 01:59:20 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 9 Dec 2013 01:59:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Backed_out_changeset_c4c1c?= =?utf-8?q?4bc8086?= Message-ID: <3dd5dh13NBz7LjS@mail.python.org> http://hg.python.org/cpython/rev/07a6d4c30c4e changeset: 87848:07a6d4c30c4e user: Victor Stinner date: Mon Dec 09 01:59:07 2013 +0100 summary: Backed out changeset c4c1c4bc8086 files: Lib/test/test_selectors.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_selectors.py b/Lib/test/test_selectors.py --- a/Lib/test/test_selectors.py +++ b/Lib/test/test_selectors.py @@ -109,7 +109,7 @@ s.unregister(r) s.unregister(w) - @unittest.skipUnless(hasattr(os, 'dup2'), "need os.dup2()") + @unittest.skipUnless(os.name == 'posix', "requires posix") def test_unregister_after_fd_close_and_reuse(self): s = self.SELECTOR() self.addCleanup(s.close) -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Mon Dec 9 09:40:03 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 09 Dec 2013 09:40:03 +0100 Subject: [Python-checkins] Daily reference leaks (07a6d4c30c4e): sum=4 Message-ID: results for 07a6d4c30c4e on branch "default" -------------------------------------------- test_asyncio leaked [0, 4, 0] memory blocks, sum=4 test_site leaked [-2, 0, 2] references, sum=0 test_site leaked [-2, 0, 2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogjrcrZb', '-x'] From python-checkins at python.org Mon Dec 9 12:40:37 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 9 Dec 2013 12:40:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319817=3A_Fix_prin?= =?utf-8?q?t=5Fexception=28=29=2C_clear_the_exception_on_error?= Message-ID: <3ddMsd54Bdz7Lkg@mail.python.org> http://hg.python.org/cpython/rev/45442f2a2494 changeset: 87849:45442f2a2494 user: Victor Stinner date: Mon Dec 09 02:10:08 2013 +0100 summary: Issue #19817: Fix print_exception(), clear the exception on error files: Python/pythonrun.c | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Python/pythonrun.c b/Python/pythonrun.c --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -1888,9 +1888,11 @@ _Py_IDENTIFIER(print_file_and_line); if (!PyExceptionInstance_Check(value)) { - PyFile_WriteString("TypeError: print_exception(): Exception expected for value, ", f); - PyFile_WriteString(Py_TYPE(value)->tp_name, f); - PyFile_WriteString(" found\n", f); + err = PyFile_WriteString("TypeError: print_exception(): Exception expected for value, ", f); + err += PyFile_WriteString(Py_TYPE(value)->tp_name, f); + err += PyFile_WriteString(" found\n", f); + if (err) + PyErr_Clear(); return; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 12:40:39 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 9 Dec 2013 12:40:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_add_an_exam?= =?utf-8?q?ple_with_Future?= Message-ID: <3ddMsg06Ldz7Lkx@mail.python.org> http://hg.python.org/cpython/rev/40e8485530bc changeset: 87850:40e8485530bc user: Victor Stinner date: Mon Dec 09 12:40:17 2013 +0100 summary: asyncio doc: add an example with Future files: Doc/library/asyncio-task.rst | 24 ++++++++++++++++++++++++ 1 files changed, 24 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -408,3 +408,27 @@ * ``wait_task()`` stops the event loop when ``print_sum()`` is done. + +Example: Future and get result +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Example combining a :class:`Future` and a :ref:`coroutine `:: + + import asyncio + + @asyncio.coroutine + def slow_operation(future): + yield from asyncio.sleep(1) + future.set_result('Future in done!') + + loop = asyncio.get_event_loop() + future = asyncio.Future() + asyncio.Task(slow_operation(future)) + loop.run_until_complete(future) + print(future.result()) + loop.close() + +The example waits for the completion of the future (which takes 1 second). The +coroutine is responsible of the computation. The event loop is notified when +the future is done (see the :meth:`Future.set_result` method). + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 13:04:23 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 9 Dec 2013 13:04:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_another_Future_?= =?utf-8?q?example_using_add=5Fdone=5Fcallback=28=29?= Message-ID: <3ddNP32tFYz7Lk1@mail.python.org> http://hg.python.org/cpython/rev/f4129412b11a changeset: 87851:f4129412b11a user: Victor Stinner date: Mon Dec 09 13:04:12 2013 +0100 summary: asyncio: another Future example using add_done_callback() files: Doc/library/asyncio-task.rst | 36 ++++++++++++++++++++++- 1 files changed, 34 insertions(+), 2 deletions(-) diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -409,8 +409,8 @@ * ``wait_task()`` stops the event loop when ``print_sum()`` is done. -Example: Future and get result -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Example: Future with run_until_complete() +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Example combining a :class:`Future` and a :ref:`coroutine `:: @@ -432,3 +432,35 @@ coroutine is responsible of the computation. The event loop is notified when the future is done (see the :meth:`Future.set_result` method). +Example: Future with run_until_complete() +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The previous example can be written differently using the +:meth:`Future.add_done_callback` method:: + + import asyncio + + @asyncio.coroutine + def slow_operation(future): + yield from asyncio.sleep(1) + future.set_result('Future in done!') + + def exit(future): + print(future.result()) + loop.stop() + + loop = asyncio.get_event_loop() + future = asyncio.Future() + asyncio.Task(slow_operation(future)) + future.add_done_callback(exit) + loop.run_forever() + loop.close() + +The future is now responsible to display the result and stop the loop using the +``exit()`` callback. + +.. note:: + The coroutine is only executed when the event loop starts running, so it is + possible to add a "done callback" to the future after creating the task + scheduling the coroutine. + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 13:19:38 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 9 Dec 2013 13:19:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_explain_why?= =?utf-8?q?_the_loop_is_running_twice?= Message-ID: <3ddNkf4JL6z7Lkw@mail.python.org> http://hg.python.org/cpython/rev/ebe28dba4a78 changeset: 87852:ebe28dba4a78 user: Victor Stinner date: Mon Dec 09 13:19:23 2013 +0100 summary: asyncio doc: explain why the loop is running twice files: Doc/library/asyncio-protocol.rst | 89 ++++++++++--------- 1 files changed, 48 insertions(+), 41 deletions(-) diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -582,45 +582,6 @@ Protocol example: TCP echo server and client ============================================ -Echo server ------------ - -TCP echo server example, send back received data and close the connection:: - - import asyncio - - class EchoServer(asyncio.Protocol): - def connection_made(self, transport): - peername = transport.get_extra_info('peername') - print('connection from {}'.format(peername)) - self.transport = transport - - def data_received(self, data): - print('data received: {}'.format(data.decode())) - self.transport.write(data) - - # close the socket - self.transport.close() - - loop = asyncio.get_event_loop() - task = loop.create_server(EchoServer, '127.0.0.1', 8888) - server = loop.run_until_complete(task) - print('serving on {}'.format(server.sockets[0].getsockname())) - - try: - loop.run_forever() - except KeyboardInterrupt: - print("exit") - finally: - server.close() - loop.close() - -:meth:`Transport.close` can be called immediatly after -:meth:`WriteTransport.write` even if data are not sent yet on the socket: both -methods are asynchronous. ``yield from`` is not needed because these transport -methods don't return coroutines. - - Echo client ----------- @@ -643,8 +604,54 @@ asyncio.get_event_loop().stop() loop = asyncio.get_event_loop() - task = loop.create_connection(EchoClient, '127.0.0.1', 8888) - loop.run_until_complete(task) + coro = loop.create_connection(EchoClient, '127.0.0.1', 8888) + loop.run_until_complete(coro) loop.run_forever() loop.close() +The event loop is running twice. The +:meth:`~BaseEventLoop.run_until_complete` method is preferred in this short +example to raise an exception if the server is not listening, instead of +having to write a short coroutine to handle the exception and stop the +running loop. At :meth:`~BaseEventLoop.run_until_complete` exit, the loop is +no more running, so there is no need to stop the loop in case of an error. + +Echo server +----------- + +TCP echo server example, send back received data and close the connection:: + + import asyncio + + class EchoServer(asyncio.Protocol): + def connection_made(self, transport): + peername = transport.get_extra_info('peername') + print('connection from {}'.format(peername)) + self.transport = transport + + def data_received(self, data): + print('data received: {}'.format(data.decode())) + self.transport.write(data) + + # close the socket + self.transport.close() + + loop = asyncio.get_event_loop() + coro = loop.create_server(EchoServer, '127.0.0.1', 8888) + server = loop.run_until_complete(coro) + print('serving on {}'.format(server.sockets[0].getsockname())) + + try: + loop.run_forever() + except KeyboardInterrupt: + print("exit") + finally: + server.close() + loop.close() + +:meth:`Transport.close` can be called immediatly after +:meth:`WriteTransport.write` even if data are not sent yet on the socket: both +methods are asynchronous. ``yield from`` is not needed because these transport +methods don't return coroutines. + + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 16:46:03 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 9 Dec 2013 16:46:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2315475=3A_Add_=5F?= =?utf-8?q?=5Fsizeof=5F=5F_implementations_for_itertools_objects=2E?= Message-ID: <3ddTJq4L35z7LlD@mail.python.org> http://hg.python.org/cpython/rev/9bce03920afe changeset: 87853:9bce03920afe user: Serhiy Storchaka date: Mon Dec 09 17:45:57 2013 +0200 summary: Issue #15475: Add __sizeof__ implementations for itertools objects. files: Lib/test/test_itertools.py | 43 +++++++++++++++++++++- Misc/NEWS | 2 + Modules/itertoolsmodule.c | 51 ++++++++++++++++++++++++++ 3 files changed, 95 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py --- a/Lib/test/test_itertools.py +++ b/Lib/test/test_itertools.py @@ -10,6 +10,8 @@ import copy import pickle from functools import reduce +import sys +import struct maxsize = support.MAX_Py_ssize_t minsize = -maxsize-1 @@ -1807,6 +1809,44 @@ # we expect type errors because of wrong argument count self.assertNotIn("does not take keyword arguments", err.args[0]) + at support.cpython_only +class SizeofTest(unittest.TestCase): + def setUp(self): + self.ssize_t = struct.calcsize('n') + + check_sizeof = support.check_sizeof + + def test_product_sizeof(self): + basesize = support.calcobjsize('3Pi') + check = self.check_sizeof + check(product('ab', '12'), basesize + 2 * self.ssize_t) + check(product(*(('abc',) * 10)), basesize + 10 * self.ssize_t) + + def test_combinations_sizeof(self): + basesize = support.calcobjsize('3Pni') + check = self.check_sizeof + check(combinations('abcd', 3), basesize + 3 * self.ssize_t) + check(combinations(range(10), 4), basesize + 4 * self.ssize_t) + + def test_combinations_with_replacement_sizeof(self): + cwr = combinations_with_replacement + basesize = support.calcobjsize('3Pni') + check = self.check_sizeof + check(cwr('abcd', 3), basesize + 3 * self.ssize_t) + check(cwr(range(10), 4), basesize + 4 * self.ssize_t) + + def test_permutations_sizeof(self): + basesize = support.calcobjsize('4Pni') + check = self.check_sizeof + check(permutations('abcd'), + basesize + 4 * self.ssize_t + 4 * self.ssize_t) + check(permutations('abcd', 3), + basesize + 4 * self.ssize_t + 3 * self.ssize_t) + check(permutations('abcde', 3), + basesize + 5 * self.ssize_t + 3 * self.ssize_t) + check(permutations(range(10), 4), + basesize + 10 * self.ssize_t + 4 * self.ssize_t) + libreftest = """ Doctest for examples in the library reference: libitertools.tex @@ -2042,7 +2082,8 @@ def test_main(verbose=None): test_classes = (TestBasicOps, TestVariousIteratorArgs, TestGC, RegressionTests, LengthTransparency, - SubclassWithKwargsTest, TestExamples) + SubclassWithKwargsTest, TestExamples, + SizeofTest) support.run_unittest(*test_classes) # verify reference counting diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -21,6 +21,8 @@ Library ------- +- Issue #15475: Add __sizeof__ implementations for itertools objects. + - Issue #19880: Fix a reference leak in unittest.TestCase. Explicitly break reference cycles between frames and the _Outcome instance. diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c --- a/Modules/itertoolsmodule.c +++ b/Modules/itertoolsmodule.c @@ -2057,6 +2057,18 @@ Py_TYPE(lz)->tp_free(lz); } +static PyObject * +product_sizeof(productobject *lz, void *unused) +{ + Py_ssize_t res; + + res = sizeof(productobject); + res += PyTuple_GET_SIZE(lz->pools) * sizeof(Py_ssize_t); + return PyLong_FromSsize_t(res); +} + +PyDoc_STRVAR(sizeof_doc, "Returns size in memory, in bytes."); + static int product_traverse(productobject *lz, visitproc visit, void *arg) { @@ -2226,6 +2238,8 @@ reduce_doc}, {"__setstate__", (PyCFunction)product_setstate, METH_O, setstate_doc}, + {"__sizeof__", (PyCFunction)product_sizeof, METH_NOARGS, + sizeof_doc}, {NULL, NULL} /* sentinel */ }; @@ -2366,6 +2380,16 @@ Py_TYPE(co)->tp_free(co); } +static PyObject * +combinations_sizeof(combinationsobject *co, void *unused) +{ + Py_ssize_t res; + + res = sizeof(combinationsobject); + res += co->r * sizeof(Py_ssize_t); + return PyLong_FromSsize_t(res); +} + static int combinations_traverse(combinationsobject *co, visitproc visit, void *arg) { @@ -2537,6 +2561,8 @@ reduce_doc}, {"__setstate__", (PyCFunction)combinations_setstate, METH_O, setstate_doc}, + {"__sizeof__", (PyCFunction)combinations_sizeof, METH_NOARGS, + sizeof_doc}, {NULL, NULL} /* sentinel */ }; @@ -2695,6 +2721,16 @@ Py_TYPE(co)->tp_free(co); } +static PyObject * +cwr_sizeof(cwrobject *co, void *unused) +{ + Py_ssize_t res; + + res = sizeof(cwrobject); + res += co->r * sizeof(Py_ssize_t); + return PyLong_FromSsize_t(res); +} + static int cwr_traverse(cwrobject *co, visitproc visit, void *arg) { @@ -2854,6 +2890,8 @@ reduce_doc}, {"__setstate__", (PyCFunction)cwr_setstate, METH_O, setstate_doc}, + {"__sizeof__", (PyCFunction)cwr_sizeof, METH_NOARGS, + sizeof_doc}, {NULL, NULL} /* sentinel */ }; @@ -3030,6 +3068,17 @@ Py_TYPE(po)->tp_free(po); } +static PyObject * +permutations_sizeof(permutationsobject *po, void *unused) +{ + Py_ssize_t res; + + res = sizeof(permutationsobject); + res += PyTuple_GET_SIZE(po->pool) * sizeof(Py_ssize_t); + res += po->r * sizeof(Py_ssize_t); + return PyLong_FromSsize_t(res); +} + static int permutations_traverse(permutationsobject *po, visitproc visit, void *arg) { @@ -3235,6 +3284,8 @@ reduce_doc}, {"__setstate__", (PyCFunction)permutations_setstate, METH_O, setstate_doc}, + {"__sizeof__", (PyCFunction)permutations_sizeof, METH_NOARGS, + sizeof_doc}, {NULL, NULL} /* sentinel */ }; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 19:11:32 2013 From: python-checkins at python.org (stefan.krah) Date: Mon, 9 Dec 2013 19:11:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Update_copyrig?= =?utf-8?q?ht=2E_The_four_year_increment_is_intentional_=28to_save_work=29?= =?utf-8?q?=2E?= Message-ID: <3ddXXh2WXKz7Lk3@mail.python.org> http://hg.python.org/cpython/rev/07a1b6c225b7 changeset: 87854:07a1b6c225b7 branch: 3.3 parent: 87843:a951ab03bda0 user: Stefan Krah date: Sun Dec 08 19:54:05 2013 +0100 summary: Update copyright. The four year increment is intentional (to save work). files: Modules/_decimal/libmpdec/basearith.c | 2 +- Modules/_decimal/libmpdec/basearith.h | 2 +- Modules/_decimal/libmpdec/bits.h | 2 +- Modules/_decimal/libmpdec/constants.c | 2 +- Modules/_decimal/libmpdec/constants.h | 2 +- Modules/_decimal/libmpdec/context.c | 2 +- Modules/_decimal/libmpdec/convolute.c | 2 +- Modules/_decimal/libmpdec/convolute.h | 2 +- Modules/_decimal/libmpdec/crt.c | 2 +- Modules/_decimal/libmpdec/crt.h | 2 +- Modules/_decimal/libmpdec/difradix2.c | 2 +- Modules/_decimal/libmpdec/difradix2.h | 2 +- Modules/_decimal/libmpdec/fnt.c | 2 +- Modules/_decimal/libmpdec/fnt.h | 2 +- Modules/_decimal/libmpdec/fourstep.c | 2 +- Modules/_decimal/libmpdec/fourstep.h | 2 +- Modules/_decimal/libmpdec/io.c | 2 +- Modules/_decimal/libmpdec/io.h | 2 +- Modules/_decimal/libmpdec/literature/fnt.py | 2 +- Modules/_decimal/libmpdec/memory.c | 2 +- Modules/_decimal/libmpdec/memory.h | 2 +- Modules/_decimal/libmpdec/mpdecimal.c | 2 +- Modules/_decimal/libmpdec/mpdecimal.h | 2 +- Modules/_decimal/libmpdec/numbertheory.c | 2 +- Modules/_decimal/libmpdec/numbertheory.h | 2 +- Modules/_decimal/libmpdec/sixstep.c | 2 +- Modules/_decimal/libmpdec/sixstep.h | 2 +- Modules/_decimal/libmpdec/transpose.c | 2 +- Modules/_decimal/libmpdec/transpose.h | 2 +- Modules/_decimal/libmpdec/typearith.h | 2 +- Modules/_decimal/libmpdec/umodarith.h | 2 +- Modules/_decimal/libmpdec/vccompat.h | 2 +- Modules/_decimal/libmpdec/vcdiv64.asm | 2 +- 33 files changed, 33 insertions(+), 33 deletions(-) diff --git a/Modules/_decimal/libmpdec/basearith.c b/Modules/_decimal/libmpdec/basearith.c --- a/Modules/_decimal/libmpdec/basearith.c +++ b/Modules/_decimal/libmpdec/basearith.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/basearith.h b/Modules/_decimal/libmpdec/basearith.h --- a/Modules/_decimal/libmpdec/basearith.h +++ b/Modules/_decimal/libmpdec/basearith.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/bits.h b/Modules/_decimal/libmpdec/bits.h --- a/Modules/_decimal/libmpdec/bits.h +++ b/Modules/_decimal/libmpdec/bits.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/constants.c b/Modules/_decimal/libmpdec/constants.c --- a/Modules/_decimal/libmpdec/constants.c +++ b/Modules/_decimal/libmpdec/constants.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/constants.h b/Modules/_decimal/libmpdec/constants.h --- a/Modules/_decimal/libmpdec/constants.h +++ b/Modules/_decimal/libmpdec/constants.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/context.c b/Modules/_decimal/libmpdec/context.c --- a/Modules/_decimal/libmpdec/context.c +++ b/Modules/_decimal/libmpdec/context.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/convolute.c b/Modules/_decimal/libmpdec/convolute.c --- a/Modules/_decimal/libmpdec/convolute.c +++ b/Modules/_decimal/libmpdec/convolute.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/convolute.h b/Modules/_decimal/libmpdec/convolute.h --- a/Modules/_decimal/libmpdec/convolute.h +++ b/Modules/_decimal/libmpdec/convolute.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/crt.c b/Modules/_decimal/libmpdec/crt.c --- a/Modules/_decimal/libmpdec/crt.c +++ b/Modules/_decimal/libmpdec/crt.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/crt.h b/Modules/_decimal/libmpdec/crt.h --- a/Modules/_decimal/libmpdec/crt.h +++ b/Modules/_decimal/libmpdec/crt.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/difradix2.c b/Modules/_decimal/libmpdec/difradix2.c --- a/Modules/_decimal/libmpdec/difradix2.c +++ b/Modules/_decimal/libmpdec/difradix2.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/difradix2.h b/Modules/_decimal/libmpdec/difradix2.h --- a/Modules/_decimal/libmpdec/difradix2.h +++ b/Modules/_decimal/libmpdec/difradix2.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/fnt.c b/Modules/_decimal/libmpdec/fnt.c --- a/Modules/_decimal/libmpdec/fnt.c +++ b/Modules/_decimal/libmpdec/fnt.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/fnt.h b/Modules/_decimal/libmpdec/fnt.h --- a/Modules/_decimal/libmpdec/fnt.h +++ b/Modules/_decimal/libmpdec/fnt.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/fourstep.c b/Modules/_decimal/libmpdec/fourstep.c --- a/Modules/_decimal/libmpdec/fourstep.c +++ b/Modules/_decimal/libmpdec/fourstep.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/fourstep.h b/Modules/_decimal/libmpdec/fourstep.h --- a/Modules/_decimal/libmpdec/fourstep.h +++ b/Modules/_decimal/libmpdec/fourstep.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/io.c b/Modules/_decimal/libmpdec/io.c --- a/Modules/_decimal/libmpdec/io.c +++ b/Modules/_decimal/libmpdec/io.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/io.h b/Modules/_decimal/libmpdec/io.h --- a/Modules/_decimal/libmpdec/io.h +++ b/Modules/_decimal/libmpdec/io.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/literature/fnt.py b/Modules/_decimal/libmpdec/literature/fnt.py --- a/Modules/_decimal/libmpdec/literature/fnt.py +++ b/Modules/_decimal/libmpdec/literature/fnt.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2008-2012 Stefan Krah. All rights reserved. +# Copyright (c) 2008-2016 Stefan Krah. All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/memory.c b/Modules/_decimal/libmpdec/memory.c --- a/Modules/_decimal/libmpdec/memory.c +++ b/Modules/_decimal/libmpdec/memory.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/memory.h b/Modules/_decimal/libmpdec/memory.h --- a/Modules/_decimal/libmpdec/memory.h +++ b/Modules/_decimal/libmpdec/memory.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/mpdecimal.h b/Modules/_decimal/libmpdec/mpdecimal.h --- a/Modules/_decimal/libmpdec/mpdecimal.h +++ b/Modules/_decimal/libmpdec/mpdecimal.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/numbertheory.c b/Modules/_decimal/libmpdec/numbertheory.c --- a/Modules/_decimal/libmpdec/numbertheory.c +++ b/Modules/_decimal/libmpdec/numbertheory.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/numbertheory.h b/Modules/_decimal/libmpdec/numbertheory.h --- a/Modules/_decimal/libmpdec/numbertheory.h +++ b/Modules/_decimal/libmpdec/numbertheory.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/sixstep.c b/Modules/_decimal/libmpdec/sixstep.c --- a/Modules/_decimal/libmpdec/sixstep.c +++ b/Modules/_decimal/libmpdec/sixstep.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/sixstep.h b/Modules/_decimal/libmpdec/sixstep.h --- a/Modules/_decimal/libmpdec/sixstep.h +++ b/Modules/_decimal/libmpdec/sixstep.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/transpose.c b/Modules/_decimal/libmpdec/transpose.c --- a/Modules/_decimal/libmpdec/transpose.c +++ b/Modules/_decimal/libmpdec/transpose.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/transpose.h b/Modules/_decimal/libmpdec/transpose.h --- a/Modules/_decimal/libmpdec/transpose.h +++ b/Modules/_decimal/libmpdec/transpose.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/typearith.h b/Modules/_decimal/libmpdec/typearith.h --- a/Modules/_decimal/libmpdec/typearith.h +++ b/Modules/_decimal/libmpdec/typearith.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/umodarith.h b/Modules/_decimal/libmpdec/umodarith.h --- a/Modules/_decimal/libmpdec/umodarith.h +++ b/Modules/_decimal/libmpdec/umodarith.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/vccompat.h b/Modules/_decimal/libmpdec/vccompat.h --- a/Modules/_decimal/libmpdec/vccompat.h +++ b/Modules/_decimal/libmpdec/vccompat.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/vcdiv64.asm b/Modules/_decimal/libmpdec/vcdiv64.asm --- a/Modules/_decimal/libmpdec/vcdiv64.asm +++ b/Modules/_decimal/libmpdec/vcdiv64.asm @@ -1,5 +1,5 @@ ; -; Copyright (c) 2008-2012 Stefan Krah. All rights reserved. +; Copyright (c) 2008-2016 Stefan Krah. All rights reserved. ; ; Redistribution and use in source and binary forms, with or without ; modification, are permitted provided that the following conditions -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 19:11:33 2013 From: python-checkins at python.org (stefan.krah) Date: Mon, 9 Dec 2013 19:11:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Missed_one_cop?= =?utf-8?q?yright=2E?= Message-ID: <3ddXXj4Dsxz7LlF@mail.python.org> http://hg.python.org/cpython/rev/2823e54e4958 changeset: 87855:2823e54e4958 branch: 3.3 user: Stefan Krah date: Sun Dec 08 20:00:56 2013 +0100 summary: Missed one copyright. files: Modules/_decimal/libmpdec/literature/umodarith.lisp | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/_decimal/libmpdec/literature/umodarith.lisp b/Modules/_decimal/libmpdec/literature/umodarith.lisp --- a/Modules/_decimal/libmpdec/literature/umodarith.lisp +++ b/Modules/_decimal/libmpdec/literature/umodarith.lisp @@ -1,5 +1,5 @@ ; -; Copyright (c) 2008-2012 Stefan Krah. All rights reserved. +; Copyright (c) 2008-2016 Stefan Krah. All rights reserved. ; ; Redistribution and use in source and binary forms, with or without ; modification, are permitted provided that the following conditions -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 19:11:34 2013 From: python-checkins at python.org (stefan.krah) Date: Mon, 9 Dec 2013 19:11:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Fix_two_typos?= =?utf-8?q?=2E?= Message-ID: <3ddXXk5sxlz7Lm1@mail.python.org> http://hg.python.org/cpython/rev/2f0493b85426 changeset: 87856:2f0493b85426 branch: 3.3 user: Stefan Krah date: Sun Dec 08 20:08:32 2013 +0100 summary: Fix two typos. files: Modules/_decimal/libmpdec/literature/mulmod-64.txt | 2 +- Modules/_decimal/libmpdec/transpose.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Modules/_decimal/libmpdec/literature/mulmod-64.txt b/Modules/_decimal/libmpdec/literature/mulmod-64.txt --- a/Modules/_decimal/libmpdec/literature/mulmod-64.txt +++ b/Modules/_decimal/libmpdec/literature/mulmod-64.txt @@ -59,7 +59,7 @@ Maximum numbers of step b): --------------------------- -# To avoid unneccessary formalism, define: +# To avoid unnecessary formalism, define: def R(hi, lo, z): return divmod(hi * z - hi + lo, 2**64) diff --git a/Modules/_decimal/libmpdec/transpose.c b/Modules/_decimal/libmpdec/transpose.c --- a/Modules/_decimal/libmpdec/transpose.c +++ b/Modules/_decimal/libmpdec/transpose.c @@ -169,7 +169,7 @@ /* * Transpose 2^n * 2^n matrix. For cache efficiency, the matrix is split into * square blocks with side length 'SIDE'. First, the blocks are transposed, - * then a square tranposition is done on each individual block. + * then a square transposition is done on each individual block. */ static void squaretrans_pow2(mpd_uint_t *matrix, mpd_size_t size) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 19:11:36 2013 From: python-checkins at python.org (stefan.krah) Date: Mon, 9 Dec 2013 19:11:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Add_libmpdec_l?= =?utf-8?q?icense=2E?= Message-ID: <3ddXXm0X1xz7Lmr@mail.python.org> http://hg.python.org/cpython/rev/219f56029927 changeset: 87857:219f56029927 branch: 3.3 user: Stefan Krah date: Sun Dec 08 20:54:02 2013 +0100 summary: Add libmpdec license. files: Doc/license.rst | 33 +++++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-) diff --git a/Doc/license.rst b/Doc/license.rst --- a/Doc/license.rst +++ b/Doc/license.rst @@ -845,3 +845,36 @@ Jean-loup Gailly Mark Adler jloup at gzip.org madler at alumni.caltech.edu + +libmpdec +-------- + +The :mod:`_decimal` Module is built using an included copy of the libmpdec +library unless the build is configured ``--with-system-libmpdec``:: + + Copyright (c) 2008-2016 Stefan Krah. All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + 1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + + THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS "AS IS" AND + ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + SUCH DAMAGE. + + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 19:11:37 2013 From: python-checkins at python.org (stefan.krah) Date: Mon, 9 Dec 2013 19:11:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgMy4zLg==?= Message-ID: <3ddXXn53Ynz7Ln6@mail.python.org> http://hg.python.org/cpython/rev/be7f928c15e9 changeset: 87858:be7f928c15e9 parent: 87853:9bce03920afe parent: 87857:219f56029927 user: Stefan Krah date: Mon Dec 09 19:03:24 2013 +0100 summary: Merge 3.3. files: Doc/license.rst | 33 ++++++++++ Modules/_decimal/libmpdec/basearith.c | 2 +- Modules/_decimal/libmpdec/basearith.h | 2 +- Modules/_decimal/libmpdec/bits.h | 2 +- Modules/_decimal/libmpdec/constants.c | 2 +- Modules/_decimal/libmpdec/constants.h | 2 +- Modules/_decimal/libmpdec/context.c | 2 +- Modules/_decimal/libmpdec/convolute.c | 2 +- Modules/_decimal/libmpdec/convolute.h | 2 +- Modules/_decimal/libmpdec/crt.c | 2 +- Modules/_decimal/libmpdec/crt.h | 2 +- Modules/_decimal/libmpdec/difradix2.c | 2 +- Modules/_decimal/libmpdec/difradix2.h | 2 +- Modules/_decimal/libmpdec/fnt.c | 2 +- Modules/_decimal/libmpdec/fnt.h | 2 +- Modules/_decimal/libmpdec/fourstep.c | 2 +- Modules/_decimal/libmpdec/fourstep.h | 2 +- Modules/_decimal/libmpdec/io.c | 2 +- Modules/_decimal/libmpdec/io.h | 2 +- Modules/_decimal/libmpdec/literature/fnt.py | 2 +- Modules/_decimal/libmpdec/literature/mulmod-64.txt | 2 +- Modules/_decimal/libmpdec/literature/umodarith.lisp | 2 +- Modules/_decimal/libmpdec/memory.c | 2 +- Modules/_decimal/libmpdec/memory.h | 2 +- Modules/_decimal/libmpdec/mpdecimal.c | 2 +- Modules/_decimal/libmpdec/mpdecimal.h | 2 +- Modules/_decimal/libmpdec/numbertheory.c | 2 +- Modules/_decimal/libmpdec/numbertheory.h | 2 +- Modules/_decimal/libmpdec/sixstep.c | 2 +- Modules/_decimal/libmpdec/sixstep.h | 2 +- Modules/_decimal/libmpdec/transpose.c | 4 +- Modules/_decimal/libmpdec/transpose.h | 2 +- Modules/_decimal/libmpdec/typearith.h | 2 +- Modules/_decimal/libmpdec/umodarith.h | 2 +- Modules/_decimal/libmpdec/vccompat.h | 2 +- Modules/_decimal/libmpdec/vcdiv64.asm | 2 +- 36 files changed, 69 insertions(+), 36 deletions(-) diff --git a/Doc/license.rst b/Doc/license.rst --- a/Doc/license.rst +++ b/Doc/license.rst @@ -934,3 +934,36 @@ ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +libmpdec +-------- + +The :mod:`_decimal` Module is built using an included copy of the libmpdec +library unless the build is configured ``--with-system-libmpdec``:: + + Copyright (c) 2008-2016 Stefan Krah. All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + 1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + + THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS "AS IS" AND + ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + SUCH DAMAGE. + + diff --git a/Modules/_decimal/libmpdec/basearith.c b/Modules/_decimal/libmpdec/basearith.c --- a/Modules/_decimal/libmpdec/basearith.c +++ b/Modules/_decimal/libmpdec/basearith.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/basearith.h b/Modules/_decimal/libmpdec/basearith.h --- a/Modules/_decimal/libmpdec/basearith.h +++ b/Modules/_decimal/libmpdec/basearith.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/bits.h b/Modules/_decimal/libmpdec/bits.h --- a/Modules/_decimal/libmpdec/bits.h +++ b/Modules/_decimal/libmpdec/bits.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/constants.c b/Modules/_decimal/libmpdec/constants.c --- a/Modules/_decimal/libmpdec/constants.c +++ b/Modules/_decimal/libmpdec/constants.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/constants.h b/Modules/_decimal/libmpdec/constants.h --- a/Modules/_decimal/libmpdec/constants.h +++ b/Modules/_decimal/libmpdec/constants.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/context.c b/Modules/_decimal/libmpdec/context.c --- a/Modules/_decimal/libmpdec/context.c +++ b/Modules/_decimal/libmpdec/context.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/convolute.c b/Modules/_decimal/libmpdec/convolute.c --- a/Modules/_decimal/libmpdec/convolute.c +++ b/Modules/_decimal/libmpdec/convolute.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/convolute.h b/Modules/_decimal/libmpdec/convolute.h --- a/Modules/_decimal/libmpdec/convolute.h +++ b/Modules/_decimal/libmpdec/convolute.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/crt.c b/Modules/_decimal/libmpdec/crt.c --- a/Modules/_decimal/libmpdec/crt.c +++ b/Modules/_decimal/libmpdec/crt.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/crt.h b/Modules/_decimal/libmpdec/crt.h --- a/Modules/_decimal/libmpdec/crt.h +++ b/Modules/_decimal/libmpdec/crt.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/difradix2.c b/Modules/_decimal/libmpdec/difradix2.c --- a/Modules/_decimal/libmpdec/difradix2.c +++ b/Modules/_decimal/libmpdec/difradix2.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/difradix2.h b/Modules/_decimal/libmpdec/difradix2.h --- a/Modules/_decimal/libmpdec/difradix2.h +++ b/Modules/_decimal/libmpdec/difradix2.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/fnt.c b/Modules/_decimal/libmpdec/fnt.c --- a/Modules/_decimal/libmpdec/fnt.c +++ b/Modules/_decimal/libmpdec/fnt.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/fnt.h b/Modules/_decimal/libmpdec/fnt.h --- a/Modules/_decimal/libmpdec/fnt.h +++ b/Modules/_decimal/libmpdec/fnt.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/fourstep.c b/Modules/_decimal/libmpdec/fourstep.c --- a/Modules/_decimal/libmpdec/fourstep.c +++ b/Modules/_decimal/libmpdec/fourstep.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/fourstep.h b/Modules/_decimal/libmpdec/fourstep.h --- a/Modules/_decimal/libmpdec/fourstep.h +++ b/Modules/_decimal/libmpdec/fourstep.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/io.c b/Modules/_decimal/libmpdec/io.c --- a/Modules/_decimal/libmpdec/io.c +++ b/Modules/_decimal/libmpdec/io.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/io.h b/Modules/_decimal/libmpdec/io.h --- a/Modules/_decimal/libmpdec/io.h +++ b/Modules/_decimal/libmpdec/io.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/literature/fnt.py b/Modules/_decimal/libmpdec/literature/fnt.py --- a/Modules/_decimal/libmpdec/literature/fnt.py +++ b/Modules/_decimal/libmpdec/literature/fnt.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2008-2012 Stefan Krah. All rights reserved. +# Copyright (c) 2008-2016 Stefan Krah. All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/literature/mulmod-64.txt b/Modules/_decimal/libmpdec/literature/mulmod-64.txt --- a/Modules/_decimal/libmpdec/literature/mulmod-64.txt +++ b/Modules/_decimal/libmpdec/literature/mulmod-64.txt @@ -59,7 +59,7 @@ Maximum numbers of step b): --------------------------- -# To avoid unneccessary formalism, define: +# To avoid unnecessary formalism, define: def R(hi, lo, z): return divmod(hi * z - hi + lo, 2**64) diff --git a/Modules/_decimal/libmpdec/literature/umodarith.lisp b/Modules/_decimal/libmpdec/literature/umodarith.lisp --- a/Modules/_decimal/libmpdec/literature/umodarith.lisp +++ b/Modules/_decimal/libmpdec/literature/umodarith.lisp @@ -1,5 +1,5 @@ ; -; Copyright (c) 2008-2012 Stefan Krah. All rights reserved. +; Copyright (c) 2008-2016 Stefan Krah. All rights reserved. ; ; Redistribution and use in source and binary forms, with or without ; modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/memory.c b/Modules/_decimal/libmpdec/memory.c --- a/Modules/_decimal/libmpdec/memory.c +++ b/Modules/_decimal/libmpdec/memory.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/memory.h b/Modules/_decimal/libmpdec/memory.h --- a/Modules/_decimal/libmpdec/memory.h +++ b/Modules/_decimal/libmpdec/memory.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/mpdecimal.h b/Modules/_decimal/libmpdec/mpdecimal.h --- a/Modules/_decimal/libmpdec/mpdecimal.h +++ b/Modules/_decimal/libmpdec/mpdecimal.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/numbertheory.c b/Modules/_decimal/libmpdec/numbertheory.c --- a/Modules/_decimal/libmpdec/numbertheory.c +++ b/Modules/_decimal/libmpdec/numbertheory.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/numbertheory.h b/Modules/_decimal/libmpdec/numbertheory.h --- a/Modules/_decimal/libmpdec/numbertheory.h +++ b/Modules/_decimal/libmpdec/numbertheory.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/sixstep.c b/Modules/_decimal/libmpdec/sixstep.c --- a/Modules/_decimal/libmpdec/sixstep.c +++ b/Modules/_decimal/libmpdec/sixstep.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/sixstep.h b/Modules/_decimal/libmpdec/sixstep.h --- a/Modules/_decimal/libmpdec/sixstep.h +++ b/Modules/_decimal/libmpdec/sixstep.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/transpose.c b/Modules/_decimal/libmpdec/transpose.c --- a/Modules/_decimal/libmpdec/transpose.c +++ b/Modules/_decimal/libmpdec/transpose.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions @@ -169,7 +169,7 @@ /* * Transpose 2^n * 2^n matrix. For cache efficiency, the matrix is split into * square blocks with side length 'SIDE'. First, the blocks are transposed, - * then a square tranposition is done on each individual block. + * then a square transposition is done on each individual block. */ static void squaretrans_pow2(mpd_uint_t *matrix, mpd_size_t size) diff --git a/Modules/_decimal/libmpdec/transpose.h b/Modules/_decimal/libmpdec/transpose.h --- a/Modules/_decimal/libmpdec/transpose.h +++ b/Modules/_decimal/libmpdec/transpose.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/typearith.h b/Modules/_decimal/libmpdec/typearith.h --- a/Modules/_decimal/libmpdec/typearith.h +++ b/Modules/_decimal/libmpdec/typearith.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/umodarith.h b/Modules/_decimal/libmpdec/umodarith.h --- a/Modules/_decimal/libmpdec/umodarith.h +++ b/Modules/_decimal/libmpdec/umodarith.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/vccompat.h b/Modules/_decimal/libmpdec/vccompat.h --- a/Modules/_decimal/libmpdec/vccompat.h +++ b/Modules/_decimal/libmpdec/vccompat.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/vcdiv64.asm b/Modules/_decimal/libmpdec/vcdiv64.asm --- a/Modules/_decimal/libmpdec/vcdiv64.asm +++ b/Modules/_decimal/libmpdec/vcdiv64.asm @@ -1,5 +1,5 @@ ; -; Copyright (c) 2008-2012 Stefan Krah. All rights reserved. +; Copyright (c) 2008-2016 Stefan Krah. All rights reserved. ; ; Redistribution and use in source and binary forms, with or without ; modification, are permitted provided that the following conditions -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 19:11:39 2013 From: python-checkins at python.org (stefan.krah) Date: Mon, 9 Dec 2013 19:11:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_whitespace=2E?= Message-ID: <3ddXXq08Lnz7LmN@mail.python.org> http://hg.python.org/cpython/rev/6b2577ef4f0f changeset: 87859:6b2577ef4f0f user: Stefan Krah date: Mon Dec 09 19:11:05 2013 +0100 summary: Fix whitespace. files: Doc/license.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/license.rst b/Doc/license.rst --- a/Doc/license.rst +++ b/Doc/license.rst @@ -934,7 +934,7 @@ ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - + libmpdec -------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 9 19:19:22 2013 From: python-checkins at python.org (stefan.krah) Date: Mon, 9 Dec 2013 19:19:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_typo=2E?= Message-ID: <3ddXjk37cpz7LmN@mail.python.org> http://hg.python.org/cpython/rev/ec8d2f54dcb2 changeset: 87860:ec8d2f54dcb2 user: Stefan Krah date: Mon Dec 09 19:18:59 2013 +0100 summary: Fix typo. files: Doc/license.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/license.rst b/Doc/license.rst --- a/Doc/license.rst +++ b/Doc/license.rst @@ -897,7 +897,7 @@ cfuhash ------- -The implementtation of the hash table used by the :mod:`tracemalloc` is based +The implementation of the hash table used by the :mod:`tracemalloc` is based on the cfuhash project:: Copyright (c) 2005 Don Owens -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 01:21:27 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 10 Dec 2013 01:21:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5OTMy?= =?utf-8?q?=3A_Fix_typo_in_import=2Eh=2C_missing_whitespaces_in_function_p?= =?utf-8?q?rototypes=2E?= Message-ID: <3ddhlW0N5Jz7Lk3@mail.python.org> http://hg.python.org/cpython/rev/7b077c691fef changeset: 87861:7b077c691fef branch: 3.3 parent: 87857:219f56029927 user: Victor Stinner date: Tue Dec 10 01:19:58 2013 +0100 summary: Issue #19932: Fix typo in import.h, missing whitespaces in function prototypes. files: Include/import.h | 8 ++++---- Misc/NEWS | 2 ++ 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/Include/import.h b/Include/import.h --- a/Include/import.h +++ b/Include/import.h @@ -86,15 +86,15 @@ PyAPI_FUNC(void) _PyImport_ReInitLock(void); -PyAPI_FUNC(PyObject *)_PyImport_FindBuiltin( +PyAPI_FUNC(PyObject *) _PyImport_FindBuiltin( const char *name /* UTF-8 encoded string */ ); -PyAPI_FUNC(PyObject *)_PyImport_FindExtensionObject(PyObject *, PyObject *); -PyAPI_FUNC(int)_PyImport_FixupBuiltin( +PyAPI_FUNC(PyObject *) _PyImport_FindExtensionObject(PyObject *, PyObject *); +PyAPI_FUNC(int) _PyImport_FixupBuiltin( PyObject *mod, char *name /* UTF-8 encoded string */ ); -PyAPI_FUNC(int)_PyImport_FixupExtensionObject(PyObject*, PyObject *, PyObject *); +PyAPI_FUNC(int) _PyImport_FixupExtensionObject(PyObject*, PyObject *, PyObject *); struct _inittab { char *name; /* ASCII encoded string */ diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,8 @@ Core and Builtins ----------------- +- Issue #19932: Fix typo in import.h, missing whitespaces in function prototypes. + - Issue #19729: In str.format(), fix recursive expansion in format spec. - Issue #19638: Fix possible crash / undefined behaviour from huge (more than 2 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 01:21:28 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 10 Dec 2013 01:21:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E3=29_Issue_=2319932=3A_Fix_typo_in_import?= =?utf-8?q?=2Eh=2C_missing_whitespaces_in_function?= Message-ID: <3ddhlX2NbSz7Lkt@mail.python.org> http://hg.python.org/cpython/rev/e017cd46009a changeset: 87862:e017cd46009a parent: 87860:ec8d2f54dcb2 parent: 87861:7b077c691fef user: Victor Stinner date: Tue Dec 10 01:20:39 2013 +0100 summary: (Merge 3.3) Issue #19932: Fix typo in import.h, missing whitespaces in function prototypes. files: Include/import.h | 8 ++++---- Misc/NEWS | 2 ++ 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/Include/import.h b/Include/import.h --- a/Include/import.h +++ b/Include/import.h @@ -86,15 +86,15 @@ PyAPI_FUNC(void) _PyImport_ReInitLock(void); -PyAPI_FUNC(PyObject *)_PyImport_FindBuiltin( +PyAPI_FUNC(PyObject *) _PyImport_FindBuiltin( const char *name /* UTF-8 encoded string */ ); -PyAPI_FUNC(PyObject *)_PyImport_FindExtensionObject(PyObject *, PyObject *); -PyAPI_FUNC(int)_PyImport_FixupBuiltin( +PyAPI_FUNC(PyObject *) _PyImport_FindExtensionObject(PyObject *, PyObject *); +PyAPI_FUNC(int) _PyImport_FixupBuiltin( PyObject *mod, const char *name /* UTF-8 encoded string */ ); -PyAPI_FUNC(int)_PyImport_FixupExtensionObject(PyObject*, PyObject *, PyObject *); +PyAPI_FUNC(int) _PyImport_FixupExtensionObject(PyObject*, PyObject *, PyObject *); struct _inittab { const char *name; /* ASCII encoded string */ diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,8 @@ Core and Builtins ----------------- +- Issue #19932: Fix typo in import.h, missing whitespaces in function prototypes. + - Issue #19736: Add module-level statvfs constants defined for GNU/glibc based systems. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 01:24:05 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 10 Dec 2013 01:24:05 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5OTMy?= =?utf-8?q?=3A_Fix_typo_in_import=2Eh=2C_missing_whitespaces_in_function_p?= =?utf-8?q?rototypes=2E?= Message-ID: <3ddhpY365Fz7LkC@mail.python.org> http://hg.python.org/cpython/rev/eb1039fe090c changeset: 87863:eb1039fe090c branch: 2.7 parent: 87833:42d3afd29460 user: Victor Stinner date: Tue Dec 10 01:23:22 2013 +0100 summary: Issue #19932: Fix typo in import.h, missing whitespaces in function prototypes. files: Include/import.h | 4 ++-- Misc/NEWS | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Include/import.h b/Include/import.h --- a/Include/import.h +++ b/Include/import.h @@ -40,8 +40,8 @@ PyAPI_FUNC(int) _PyImport_IsScript(struct filedescr *); PyAPI_FUNC(void) _PyImport_ReInitLock(void); -PyAPI_FUNC(PyObject *)_PyImport_FindExtension(char *, char *); -PyAPI_FUNC(PyObject *)_PyImport_FixupExtension(char *, char *); +PyAPI_FUNC(PyObject *) _PyImport_FindExtension(char *, char *); +PyAPI_FUNC(PyObject *) _PyImport_FixupExtension(char *, char *); struct _inittab { char *name; diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -9,6 +9,8 @@ Core and Builtins ----------------- +- Issue #19932: Fix typo in import.h, missing whitespaces in function prototypes. + - Issue #19638: Fix possible crash / undefined behaviour from huge (more than 2 billion characters) input strings in _Py_dg_strtod. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 02:45:09 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 10 Dec 2013 02:45:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_assyncio_doc=3A_rewrite=2C?= =?utf-8?q?_improve_and_move_coroutine=2C_Future_and_Task_examples?= Message-ID: <3ddkc54jSkz7LkC@mail.python.org> http://hg.python.org/cpython/rev/09960bf7d432 changeset: 87864:09960bf7d432 parent: 87862:e017cd46009a user: Victor Stinner date: Tue Dec 10 02:09:46 2013 +0100 summary: assyncio doc: rewrite, improve and move coroutine, Future and Task examples files: Doc/library/asyncio-task.rst | 294 +++++++++++----------- 1 files changed, 152 insertions(+), 142 deletions(-) diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -56,6 +56,55 @@ Coroutines (and tasks) can only run when the event loop is running. +.. _asyncio-hello-world-coroutine: + +Example: "Hello World" coroutine +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Print ``"Hello World"`` every two seconds using a coroutine:: + + import asyncio + + @asyncio.coroutine + def greet_every_two_seconds(): + while True: + print('Hello World') + yield from asyncio.sleep(2) + + loop = asyncio.get_event_loop() + loop.run_until_complete(greet_every_two_seconds()) + + +.. seealso:: + + :ref:`Hello World example using a callback `. + + +Example: Chain coroutines +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Example chaining coroutines:: + + import asyncio + + @asyncio.coroutine + def compute(x, y): + print("Compute %s + %s ..." % (x, y)) + yield from asyncio.sleep(1.0) + return x + y + + @asyncio.coroutine + def print_sum(x, y): + result = yield from compute(x, y) + print("%s + %s = %s" % (x, y, result)) + + loop = asyncio.get_event_loop() + loop.run_until_complete(print_sum(1, 2)) + +``compute()`` is chained to ``print_sum()``: ``print_sum()`` coroutine waits +until ``compute()`` is completed before returing its result. + + InvalidStateError ----------------- @@ -148,6 +197,69 @@ :exc:`InvalidStateError`. +Example: Future with run_until_complete() +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Example combining a :class:`Future` and a :ref:`coroutine `:: + + import asyncio + + @asyncio.coroutine + def slow_operation(future): + yield from asyncio.sleep(1) + future.set_result('Future in done!') + + loop = asyncio.get_event_loop() + future = asyncio.Future() + asyncio.Task(slow_operation(future)) + loop.run_until_complete(future) + print(future.result()) + +The coroutine is responsible of the computation (which takes 1 second) and +it stores the result into the future. The +:meth:`~BaseEventLoop.run_until_complete` method waits for the completion of +the future. + +.. note:: + The :meth:`~BaseEventLoop.run_until_complete` method uses internally the + :meth:`~Future.add_done_callback` method to be notified when the future is + done. + + +Example: Future with run_forever() +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The previous example can be written differently using the +:meth:`Future.add_done_callback` method to describe explicitly the control +flow:: + + import asyncio + + @asyncio.coroutine + def slow_operation(future): + yield from asyncio.sleep(1) + future.set_result('Future in done!') + + def got_result(future): + print(future.result()) + loop.stop() + + loop = asyncio.get_event_loop() + future = asyncio.Future() + asyncio.Task(slow_operation(future)) + future.add_done_callback(got_result) + loop.run_forever() + +In this example, the future is responsible to display the result and to stop +the loop. + +.. note:: + The coroutine is only executed when the event loop starts running, so it is + possible to add a "done callback" to the future after creating the task + scheduling the coroutine. + + + Task ---- @@ -195,6 +307,46 @@ goes; by default it goes to sys.stderr. +Example: Parallel execution of tasks +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Example executing 3 tasks (A, B, C) in parallel:: + + import asyncio + + @asyncio.coroutine + def factorial(task, n): + f = 1 + for i in range(2, n+1): + print("[%s] Compute factorial(%s)..." % (task, i)) + yield from asyncio.sleep(1) + f *= n + print("[%s] factorial(%s) = %s" % (task, n, f)) + + task_a = asyncio.Task(factorial("A", 2)) + task_b = asyncio.Task(factorial("B", 3)) + task_c = asyncio.Task(factorial("C", 4)) + tasks = [task_a, task_b, task_c] + + loop = asyncio.get_event_loop() + loop.run_until_complete(asyncio.wait(tasks)) + +Output:: + + [A] Compute factorial(2)... + [B] Compute factorial(2)... + [C] Compute factorial(2)... + [A] factorial(2) = 2 + [B] Compute factorial(3)... + [C] Compute factorial(3)... + [B] factorial(3) = 9 + [C] Compute factorial(4)... + [C] factorial(4) = 64 + +When a task is created, it is automatically scheduled for execution. The event +loop stops when all tasks are done. + + Task functions -------------- @@ -322,145 +474,3 @@ the timeout occurs are returned in the second set. -Examples --------- - - -.. _asyncio-hello-world-coroutine: - -Example: Hello World (coroutine) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Print ``Hello World`` every two seconds, using a coroutine:: - - import asyncio - - @asyncio.coroutine - def greet_every_two_seconds(): - while True: - print('Hello World') - yield from asyncio.sleep(2) - - loop = asyncio.get_event_loop() - loop.run_until_complete(greet_every_two_seconds()) - - -.. seealso:: - - :ref:`Hello World example using a callback `. - -Example: Chains coroutines and parallel execution -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Example chaining coroutines and executing multiple coroutines in parallel:: - - import asyncio - - @asyncio.coroutine - def compute(x, y): - print("Start computing %s + %s" % (x, y)) - yield from asyncio.sleep(3.0) - return x + y - - @asyncio.coroutine - def print_sum(x, y): - result = yield from compute(x, y) - print("%s + %s = %s" % (x, y, result)) - - @asyncio.coroutine - def wait_task(task): - while 1: - done, pending = yield from asyncio.wait([task], timeout=1.0) - if done: - break - print("Compute in progress...") - asyncio.get_event_loop().stop() - - print("Schedule tasks") - task = asyncio.async(print_sum(1, 2)) - asyncio.async(wait_task(task)) - - print("Execute tasks") - loop = asyncio.get_event_loop() - loop.run_forever() - loop.close() - - - -Output:: - - Schedule tasks - Execute tasks - Start computing 1 + 2 - Compute in progress... - Compute in progress... - 1 + 2 = 3 - -Details: - -* ``compute()`` is chained to ``print_sum()``: ``print_sum()`` coroutine waits - until ``compute()`` is complete. Coroutines are executed in parallel: - ``wait_task()`` is executed while ``compute()`` is blocked in - ``asyncio.sleep(3.0)``. - -* Coroutines are not executed before the loop is running: ``"Execute tasks"`` - is written before ``"Start computing 1 + 2"``. - -* ``wait_task()`` stops the event loop when ``print_sum()`` is done. - - -Example: Future with run_until_complete() -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Example combining a :class:`Future` and a :ref:`coroutine `:: - - import asyncio - - @asyncio.coroutine - def slow_operation(future): - yield from asyncio.sleep(1) - future.set_result('Future in done!') - - loop = asyncio.get_event_loop() - future = asyncio.Future() - asyncio.Task(slow_operation(future)) - loop.run_until_complete(future) - print(future.result()) - loop.close() - -The example waits for the completion of the future (which takes 1 second). The -coroutine is responsible of the computation. The event loop is notified when -the future is done (see the :meth:`Future.set_result` method). - -Example: Future with run_until_complete() -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The previous example can be written differently using the -:meth:`Future.add_done_callback` method:: - - import asyncio - - @asyncio.coroutine - def slow_operation(future): - yield from asyncio.sleep(1) - future.set_result('Future in done!') - - def exit(future): - print(future.result()) - loop.stop() - - loop = asyncio.get_event_loop() - future = asyncio.Future() - asyncio.Task(slow_operation(future)) - future.add_done_callback(exit) - loop.run_forever() - loop.close() - -The future is now responsible to display the result and stop the loop using the -``exit()`` callback. - -.. note:: - The coroutine is only executed when the event loop starts running, so it is - possible to add a "done callback" to the future after creating the task - scheduling the coroutine. - -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 02:51:23 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 10 Dec 2013 02:51:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_rewrite_the?= =?utf-8?q?_callback_hello_world_to_use_call=5Fsoon=28=29_instead_of_a?= Message-ID: <3ddklH0dBJz7Ljq@mail.python.org> http://hg.python.org/cpython/rev/9eb247762a91 changeset: 87865:9eb247762a91 user: Victor Stinner date: Tue Dec 10 02:47:22 2013 +0100 summary: asyncio doc: rewrite the callback hello world to use call_soon() instead of a direct call. files: Doc/library/asyncio-eventloop.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -348,7 +348,7 @@ loop.call_later(2, print_and_repeat, loop) loop = asyncio.get_event_loop() - print_and_repeat(loop) + loop.call_soon(print_and_repeat, loop) loop.run_forever() .. seealso:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 02:51:24 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 10 Dec 2013 02:51:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_fix_2nd_task_ex?= =?utf-8?q?ample?= Message-ID: <3ddklJ2Ttjz7Ll6@mail.python.org> http://hg.python.org/cpython/rev/240dc7622087 changeset: 87866:240dc7622087 user: Victor Stinner date: Tue Dec 10 02:51:05 2013 +0100 summary: asyncio: fix 2nd task example files: Doc/library/asyncio-task.rst | 30 ++++++++++++------------ 1 files changed, 15 insertions(+), 15 deletions(-) diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -315,13 +315,13 @@ import asyncio @asyncio.coroutine - def factorial(task, n): + def factorial(name, number): f = 1 - for i in range(2, n+1): - print("[%s] Compute factorial(%s)..." % (task, i)) + for i in range(2, number+1): + print("Task %s: Compute factorial(%s)..." % (name, i)) yield from asyncio.sleep(1) - f *= n - print("[%s] factorial(%s) = %s" % (task, n, f)) + f *= i + print("Task %s: factorial(%s) = %s" % (name, number, f)) task_a = asyncio.Task(factorial("A", 2)) task_b = asyncio.Task(factorial("B", 3)) @@ -333,17 +333,17 @@ Output:: - [A] Compute factorial(2)... - [B] Compute factorial(2)... - [C] Compute factorial(2)... - [A] factorial(2) = 2 - [B] Compute factorial(3)... - [C] Compute factorial(3)... - [B] factorial(3) = 9 - [C] Compute factorial(4)... - [C] factorial(4) = 64 + Task A: Compute factorial(2)... + Task B: Compute factorial(2)... + Task C: Compute factorial(2)... + Task A: factorial(2) = 2 + Task B: Compute factorial(3)... + Task C: Compute factorial(3)... + Task B: factorial(3) = 6 + Task C: Compute factorial(4)... + Task C: factorial(4) = 24 -When a task is created, it is automatically scheduled for execution. The event +A task is automatically scheduled for execution when it is created. The event loop stops when all tasks are done. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 02:52:59 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 10 Dec 2013 02:52:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_don=27t_documen?= =?utf-8?q?t_private_functions?= Message-ID: <3ddkn74VYPz7Ljq@mail.python.org> http://hg.python.org/cpython/rev/1cecfc642333 changeset: 87867:1cecfc642333 user: Victor Stinner date: Tue Dec 10 02:52:49 2013 +0100 summary: asyncio: don't document private functions files: Doc/library/asyncio-task.rst | 8 -------- 1 files changed, 0 insertions(+), 8 deletions(-) diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -391,14 +391,6 @@ outer Future is *not* cancelled in this case. (This is to prevent the cancellation of one child to cause other children to be cancelled.) -.. function:: tasks.iscoroutinefunction(func) - - Return ``True`` if *func* is a decorated coroutine function. - -.. function:: tasks.iscoroutine(obj) - - Return ``True`` if *obj* is a coroutine object. - .. function:: sleep(delay, result=None, \*, loop=None) Create a :ref:`coroutine ` that completes after a given time -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 04:05:32 2013 From: python-checkins at python.org (eric.snow) Date: Tue, 10 Dec 2013 04:05:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_19851=3A_Fix_a_regre?= =?utf-8?q?ssion_in_reloading_submodules=2E?= Message-ID: <3ddmNr13pNz7Ljq@mail.python.org> http://hg.python.org/cpython/rev/1d67eb1df5a9 changeset: 87868:1d67eb1df5a9 user: Eric Snow date: Mon Dec 09 19:59:10 2013 -0700 summary: Issue 19851: Fix a regression in reloading submodules. files: Lib/importlib/__init__.py | 15 ++- Lib/test/test_importlib/test_api.py | 68 +++++++++++++++++ Misc/NEWS | 2 + 3 files changed, 81 insertions(+), 4 deletions(-) diff --git a/Lib/importlib/__init__.py b/Lib/importlib/__init__.py --- a/Lib/importlib/__init__.py +++ b/Lib/importlib/__init__.py @@ -153,10 +153,17 @@ _RELOADING[name] = module try: parent_name = name.rpartition('.')[0] - if parent_name and parent_name not in sys.modules: - msg = "parent {!r} not in sys.modules" - raise ImportError(msg.format(parent_name), name=parent_name) - spec = module.__spec__ = _bootstrap._find_spec(name, None, module) + if parent_name: + try: + parent = sys.modules[parent_name] + except KeyError: + msg = "parent {!r} not in sys.modules" + raise ImportError(msg.format(parent_name), name=parent_name) + else: + pkgpath = parent.__path__ + else: + pkgpath = None + spec = module.__spec__ = _bootstrap._find_spec(name, pkgpath, module) methods = _bootstrap._SpecMethods(spec) methods.exec(module) # The module may have replaced itself in sys.modules! diff --git a/Lib/test/test_importlib/test_api.py b/Lib/test/test_importlib/test_api.py --- a/Lib/test/test_importlib/test_api.py +++ b/Lib/test/test_importlib/test_api.py @@ -4,6 +4,7 @@ frozen_util, source_util = util.import_importlib('importlib.util') frozen_machinery, source_machinery = util.import_importlib('importlib.machinery') +from contextlib import contextmanager import os.path import sys from test import support @@ -11,6 +12,37 @@ import unittest + at contextmanager +def temp_module(name, content='', *, pkg=False): + conflicts = [n for n in sys.modules if n.partition('.')[0] == name] + with support.temp_cwd(None) as cwd: + with util.uncache(name, *conflicts): + with support.DirsOnSysPath(cwd): + frozen_init.invalidate_caches() + + location = os.path.join(cwd, name) + if pkg: + modpath = os.path.join(location, '__init__.py') + os.mkdir(name) + else: + modpath = location + '.py' + if content is None: + # Make sure the module file gets created. + content = '' + if content is not None: + # not a namespace package + with open(modpath, 'w') as modfile: + modfile.write(content) + yield location + + +def submodule(parent, name, pkg_dir, content=''): + path = os.path.join(pkg_dir, name + '.py') + with open(path, 'w') as subfile: + subfile.write(content) + return '{}.{}'.format(parent, name), path + + class ImportModuleTests: """Test importlib.import_module.""" @@ -246,6 +278,32 @@ # None is returned upon failure to find a loader. self.assertIsNone(self.init.find_spec('nevergoingtofindthismodule')) + def test_find_submodule(self): + name = 'spam' + subname = 'ham' + with temp_module(name, pkg=True) as pkg_dir: + fullname, _ = submodule(name, subname, pkg_dir) + spec = self.init.find_spec(fullname, [pkg_dir]) + self.assertIsNot(spec, None) + self.assertNotIn(name, sorted(sys.modules)) + # Ensure successive calls behave the same. + spec_again = self.init.find_spec(fullname, [pkg_dir]) + # XXX Once #19927 is resolved, uncomment this line. + #self.assertEqual(spec_again, spec) + + def test_find_submodule_missing_path(self): + name = 'spam' + subname = 'ham' + with temp_module(name, pkg=True) as pkg_dir: + fullname, _ = submodule(name, subname, pkg_dir) + spec = self.init.find_spec(fullname) + self.assertIs(spec, None) + self.assertNotIn(name, sorted(sys.modules)) + # Ensure successive calls behave the same. + spec = self.init.find_spec(fullname) + self.assertIs(spec, None) + + class Frozen_FindSpecTests(FindSpecTests, unittest.TestCase): init = frozen_init machinery = frozen_machinery @@ -410,6 +468,16 @@ self.assertEqual(loader.path, init_path) self.assertEqual(ns, expected) + def test_reload_submodule(self): + # See #19851. + name = 'spam' + subname = 'ham' + with temp_module(name, pkg=True) as pkg_dir: + fullname, _ = submodule(name, subname, pkg_dir) + ham = self.init.import_module(fullname) + reloaded = self.init.reload(ham) + self.assertIs(reloaded, ham) + class Frozen_ReloadTests(ReloadTests, unittest.TestCase): init = frozen_init diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -114,6 +114,8 @@ - Issue #6477: Added support for pickling the types of built-in singletons (i.e., Ellipsis, NotImplemented, None). +- Issue #19851: Fixed a regression in reloading sub-modules. + - ssl.create_default_context() sets OP_NO_COMPRESSION to prevent CRIME. - Issue #19802: Add socket.SO_PRIORITY. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 04:34:27 2013 From: python-checkins at python.org (eric.snow) Date: Tue, 10 Dec 2013 04:34:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Remove_commented-out_modul?= =?utf-8?q?e_spec_test_code_and_an_out-of-date_note=2E?= Message-ID: <3ddn2C0lbLz7LjT@mail.python.org> http://hg.python.org/cpython/rev/96a68e369d13 changeset: 87869:96a68e369d13 user: Eric Snow date: Mon Dec 09 20:33:33 2013 -0700 summary: Remove commented-out module spec test code and an out-of-date note. files: Lib/test/test_importlib/test_spec.py | 10 ---------- 1 files changed, 0 insertions(+), 10 deletions(-) diff --git a/Lib/test/test_importlib/test_spec.py b/Lib/test/test_importlib/test_spec.py --- a/Lib/test/test_importlib/test_spec.py +++ b/Lib/test/test_importlib/test_spec.py @@ -17,13 +17,6 @@ class TestLoader: def __init__(self, path=None, is_package=None): -# if path: -# if is_package: -# if not path.endswith('.py'): -# path = os.path.join(path, '__init__.py') -# elif is_package is None: -# is_package = path.endswith('__init__.py') - self.path = path self.package = is_package @@ -554,9 +547,6 @@ class ModuleReprTests: - # XXX Add more tests for repr(module) once ModuleSpec._module_repr() - # is in place? - def setUp(self): self.module = type(os)('spam') self.spec = self.machinery.ModuleSpec('spam', TestLoader()) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 09:07:29 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 10 Dec 2013 09:07:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5NDgx?= =?utf-8?q?=3A_print=28=29_of_unicode=2C_str_or_bytearray_subclass_instanc?= =?utf-8?q?e_in_IDLE?= Message-ID: <3ddv5F0qqJz7Ljq@mail.python.org> http://hg.python.org/cpython/rev/df9596ca838c changeset: 87870:df9596ca838c branch: 2.7 parent: 87863:eb1039fe090c user: Serhiy Storchaka date: Tue Dec 10 10:04:41 2013 +0200 summary: Issue #19481: print() of unicode, str or bytearray subclass instance in IDLE no more hangs. files: Lib/idlelib/PyShell.py | 12 ++++++++++-- Misc/NEWS | 6 ++++++ 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -1338,8 +1338,16 @@ def write(self, s): if self.closed: raise ValueError("write to closed file") - if not isinstance(s, (basestring, bytearray)): - raise TypeError('must be string, not ' + type(s).__name__) + if type(s) not in (unicode, str, bytearray): + # See issue #19481 + if isinstance(s, unicode): + s = unicode.__getslice__(s, None, None) + elif isinstance(s, str): + s = str.__str__(s) + elif isinstance(s, bytearray): + s = bytearray.__str__(s) + else: + raise TypeError('must be string, not ' + type(s).__name__) return self.shell.write(s, self.tags) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -68,6 +68,12 @@ - Issue #19286: Directories in ``package_data`` are no longer added to the filelist, preventing failure outlined in the ticket. +IDLE +---- + +- Issue #19481: print() of unicode, str or bytearray subclass instance in IDLE + no more hangs. + Tests ----- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 09:07:30 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 10 Dec 2013 09:07:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5NDgx?= =?utf-8?q?=3A_print=28=29_of_string_subclass_instance_in_IDLE_no_more_han?= =?utf-8?b?Z3Mu?= Message-ID: <3ddv5G2sGnz7Lk5@mail.python.org> http://hg.python.org/cpython/rev/d462b2bf875b changeset: 87871:d462b2bf875b branch: 3.3 parent: 87861:7b077c691fef user: Serhiy Storchaka date: Tue Dec 10 10:05:19 2013 +0200 summary: Issue #19481: print() of string subclass instance in IDLE no more hangs. files: Lib/idlelib/PyShell.py | 7 +++++-- Misc/NEWS | 5 +++++ 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -1331,8 +1331,11 @@ def write(self, s): if self.closed: raise ValueError("write to closed file") - if not isinstance(s, str): - raise TypeError('must be str, not ' + type(s).__name__) + if type(s) is not str: + if not isinstance(s, str): + raise TypeError('must be str, not ' + type(s).__name__) + # See issue #19481 + s = str.__str__(s) return self.shell.write(s, self.tags) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -106,6 +106,11 @@ - Issue #19286: Directories in ``package_data`` are no longer added to the filelist, preventing failure outlined in the ticket. +IDLE +---- + +- Issue #19481: print() of string subclass instance in IDLE no more hangs. + Tests ----- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 09:07:31 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 10 Dec 2013 09:07:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319481=3A_print=28=29_of_string_subclass_instanc?= =?utf-8?q?e_in_IDLE_no_more_hangs=2E?= Message-ID: <3ddv5H4fRDz7Lk5@mail.python.org> http://hg.python.org/cpython/rev/1d68ea8148ce changeset: 87872:1d68ea8148ce parent: 87869:96a68e369d13 parent: 87871:d462b2bf875b user: Serhiy Storchaka date: Tue Dec 10 10:06:35 2013 +0200 summary: Issue #19481: print() of string subclass instance in IDLE no more hangs. files: Lib/idlelib/PyShell.py | 7 +++++-- Misc/NEWS | 5 +++++ 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -1334,8 +1334,11 @@ def write(self, s): if self.closed: raise ValueError("write to closed file") - if not isinstance(s, str): - raise TypeError('must be str, not ' + type(s).__name__) + if type(s) is not str: + if not isinstance(s, str): + raise TypeError('must be str, not ' + type(s).__name__) + # See issue #19481 + s = str.__str__(s) return self.shell.write(s, self.tags) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -128,6 +128,11 @@ - Issue #19545: Avoid chained exceptions while passing stray % to time.strptime(). Initial patch by Claudiu Popa. +IDLE +---- + +- Issue #19481: print() of string subclass instance in IDLE no more hangs. + Tests ----- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 09:26:00 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 10 Dec 2013 09:26:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5OTI4?= =?utf-8?q?=3A_Implemented_a_test_for_repr=28=29_of_cell_objects=2E?= Message-ID: <3ddvVc392vz7Ljm@mail.python.org> http://hg.python.org/cpython/rev/d98c5806c33c changeset: 87873:d98c5806c33c branch: 2.7 parent: 87870:df9596ca838c user: Serhiy Storchaka date: Tue Dec 10 10:20:11 2013 +0200 summary: Issue #19928: Implemented a test for repr() of cell objects. files: Lib/test/test_repr.py | 11 +++++++++-- Misc/NEWS | 2 ++ 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_repr.py b/Lib/test/test_repr.py --- a/Lib/test/test_repr.py +++ b/Lib/test/test_repr.py @@ -179,8 +179,15 @@ self.assertTrue(repr(x).startswith('') + self.assertRegexpMatches(r(x), r'') def test_descriptors(self): eq = self.assertEqual diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -77,6 +77,8 @@ Tests ----- +- Issue #19928: Implemented a test for repr() of cell objects. + - Issue #19595: Re-enabled a long-disabled test in test_winsound. - Issue #19588: Fixed tests in test_random that were silently skipped most -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 09:26:01 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 10 Dec 2013 09:26:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5OTI4?= =?utf-8?q?=3A_Implemented_a_test_for_repr=28=29_of_cell_objects=2E?= Message-ID: <3ddvVd5D8Bz7Lk5@mail.python.org> http://hg.python.org/cpython/rev/49eb895be796 changeset: 87874:49eb895be796 branch: 3.3 parent: 87871:d462b2bf875b user: Serhiy Storchaka date: Tue Dec 10 10:20:31 2013 +0200 summary: Issue #19928: Implemented a test for repr() of cell objects. files: Lib/test/test_reprlib.py | 12 +++++++++--- Misc/NEWS | 2 ++ 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_reprlib.py b/Lib/test/test_reprlib.py --- a/Lib/test/test_reprlib.py +++ b/Lib/test/test_reprlib.py @@ -166,10 +166,16 @@ eq(r([[[[[[{}]]]]]]), "[[[[[[{}]]]]]]") eq(r([[[[[[[{}]]]]]]]), "[[[[[[[...]]]]]]]") - @unittest.skip('hard to catch a cell object') def test_cell(self): - # XXX Hmm? How to get at a cell object? - pass + def get_cell(): + x = 42 + def inner(): + return x + return inner + x = get_cell().__closure__[0] + self.assertRegex(repr(x), + r'') + self.assertRegex(r(x), r'') def test_descriptors(self): eq = self.assertEqual diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -114,6 +114,8 @@ Tests ----- +- Issue #19928: Implemented a test for repr() of cell objects. + - Issue #19535: Fixed test_docxmlrpc when python is run with -OO. - Issue #19926: Removed unneeded test_main from test_abstract_numbers. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 09:26:02 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 10 Dec 2013 09:26:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319928=3A_Implemented_a_test_for_repr=28=29_of_c?= =?utf-8?q?ell_objects=2E?= Message-ID: <3ddvVf750Dz7Lkh@mail.python.org> http://hg.python.org/cpython/rev/af9f3d737d4a changeset: 87875:af9f3d737d4a parent: 87872:1d68ea8148ce parent: 87874:49eb895be796 user: Serhiy Storchaka date: Tue Dec 10 10:21:51 2013 +0200 summary: Issue #19928: Implemented a test for repr() of cell objects. files: Lib/test/test_reprlib.py | 12 +++++++++--- Misc/NEWS | 2 ++ 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_reprlib.py b/Lib/test/test_reprlib.py --- a/Lib/test/test_reprlib.py +++ b/Lib/test/test_reprlib.py @@ -166,10 +166,16 @@ eq(r([[[[[[{}]]]]]]), "[[[[[[{}]]]]]]") eq(r([[[[[[[{}]]]]]]]), "[[[[[[[...]]]]]]]") - @unittest.skip('hard to catch a cell object') def test_cell(self): - # XXX Hmm? How to get at a cell object? - pass + def get_cell(): + x = 42 + def inner(): + return x + return inner + x = get_cell().__closure__[0] + self.assertRegex(repr(x), + r'') + self.assertRegex(r(x), r'') def test_descriptors(self): eq = self.assertEqual diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -136,6 +136,8 @@ Tests ----- +- Issue #19928: Implemented a test for repr() of cell objects. + - Issue #19535: Fixed test_docxmlrpc, test_functools, test_inspect, and test_statistics when python is run with -OO. -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Tue Dec 10 09:40:50 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 10 Dec 2013 09:40:50 +0100 Subject: [Python-checkins] Daily reference leaks (1cecfc642333): sum=-4 Message-ID: results for 1cecfc642333 on branch "default" -------------------------------------------- test_site leaked [0, -2, 0] references, sum=-2 test_site leaked [0, -2, 0] memory blocks, sum=-2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogmh4VnQ', '-x'] From python-checkins at python.org Tue Dec 10 12:15:05 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 10 Dec 2013 12:15:05 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_document_Task?= =?utf-8?b?LmN1cnJlbnRfdGFzaygp?= Message-ID: <3ddzFj0Zs3z7Lk4@mail.python.org> http://hg.python.org/cpython/rev/3f2d4bb34fd6 changeset: 87876:3f2d4bb34fd6 user: Victor Stinner date: Tue Dec 10 12:14:50 2013 +0100 summary: asyncio: document Task.current_task() files: Doc/library/asyncio-task.rst | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -273,6 +273,14 @@ By default all tasks for the current event loop are returned. + .. classmethod:: current_task(loop=None) + + Return the currently running task in an event loop or ``None``. + + By default the current task for the current event loop is returned. + + ``None`` is returned when called not in the context of a :class:`Task`. + .. method:: cancel() Cancel the task. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 12:19:26 2013 From: python-checkins at python.org (nick.coghlan) Date: Tue, 10 Dec 2013 12:19:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5NDA3?= =?utf-8?q?=3A_add_Python_Packaging_User_Guide_notes?= Message-ID: <3ddzLk0Xyxz7Lk4@mail.python.org> http://hg.python.org/cpython/rev/a9f91a38a265 changeset: 87877:a9f91a38a265 branch: 2.7 parent: 87873:d98c5806c33c user: Nick Coghlan date: Tue Dec 10 21:18:32 2013 +1000 summary: Issue #19407: add Python Packaging User Guide notes The stdlib docs for package distribution and building extensions are rather dated, and that isn't expected to change for 2.7 and 3.3. The Python Packaging User Guide isn't complete either, but it's already a much better road map for new users than the existing stdlib docs. files: Doc/distutils/index.rst | 10 ++++++++++ Doc/extending/index.rst | 9 +++++++++ Doc/install/index.rst | 18 +++++++++++++----- 3 files changed, 32 insertions(+), 5 deletions(-) diff --git a/Doc/distutils/index.rst b/Doc/distutils/index.rst --- a/Doc/distutils/index.rst +++ b/Doc/distutils/index.rst @@ -12,6 +12,16 @@ make Python modules and extensions easily available to a wider audience with very little overhead for build/release/install mechanics. +.. note:: + + This guide only covers the basic tools for building and distributing + extensions that are provided as part of this version of Python. Third + party tools offer easier to use and more secure alternatives. Refer to the + `quick recommendations section + `__ + in the Python Packaging User Guide for more information. + + .. toctree:: :maxdepth: 2 :numbered: diff --git a/Doc/extending/index.rst b/Doc/extending/index.rst --- a/Doc/extending/index.rst +++ b/Doc/extending/index.rst @@ -21,6 +21,15 @@ For a detailed description of the whole Python/C API, see the separate :ref:`c-api-index`. +.. note:: + + This guide only covers the basic tools for creating extensions provided + as part of this version of CPython. Third party tools may offer simpler + alternatives. Refer to the `binary extensions section + `__ + in the Python Packaging User Guide for more information. + + .. toctree:: :maxdepth: 2 :numbered: diff --git a/Doc/install/index.rst b/Doc/install/index.rst --- a/Doc/install/index.rst +++ b/Doc/install/index.rst @@ -20,12 +20,20 @@ Finally, it might be useful to include all the material from my "Care and Feeding of a Python Installation" talk in here somewhere. Yow! -.. topic:: Abstract +This document describes the Python Distribution Utilities ("Distutils") from the +end-user's point-of-view, describing how to extend the capabilities of a +standard Python installation by building and installing third-party Python +modules and extensions. - This document describes the Python Distribution Utilities ("Distutils") from the - end-user's point-of-view, describing how to extend the capabilities of a - standard Python installation by building and installing third-party Python - modules and extensions. + +.. note:: + + This guide only covers the basic tools for installing extensions that are + provided as part of this version of Python. Third party tools offer easier + to use and more secure alternatives. Refer to the + `quick recommendations section + `__ + in the Python Packaging User Guide for more information. .. _inst-intro: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 12:22:08 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 10 Dec 2013 12:22:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_don=27t_doc?= =?utf-8?q?ument_Task=2Ecancel=28=29=2C_already_documented_in_Future=2Ecan?= =?utf-8?b?Y2VsKCk=?= Message-ID: <3ddzPr3cjKz7LjT@mail.python.org> http://hg.python.org/cpython/rev/8da1729fd602 changeset: 87878:8da1729fd602 parent: 87876:3f2d4bb34fd6 user: Victor Stinner date: Tue Dec 10 12:18:15 2013 +0100 summary: asyncio doc: don't document Task.cancel(), already documented in Future.cancel() files: Doc/library/asyncio-task.rst | 4 ---- 1 files changed, 0 insertions(+), 4 deletions(-) diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -281,10 +281,6 @@ ``None`` is returned when called not in the context of a :class:`Task`. - .. method:: cancel() - - Cancel the task. - .. method:: get_stack(self, \*, limit=None) Return the list of stack frames for this task's coroutine. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 12:22:09 2013 From: python-checkins at python.org (victor.stinner) Date: Tue, 10 Dec 2013 12:22:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_simplify_ta?= =?utf-8?q?sk_example?= Message-ID: <3ddzPs5GQ7z7Lkf@mail.python.org> http://hg.python.org/cpython/rev/03e875beda37 changeset: 87879:03e875beda37 user: Victor Stinner date: Tue Dec 10 12:20:14 2013 +0100 summary: asyncio doc: simplify task example files: Doc/library/asyncio-task.rst | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -327,10 +327,10 @@ f *= i print("Task %s: factorial(%s) = %s" % (name, number, f)) - task_a = asyncio.Task(factorial("A", 2)) - task_b = asyncio.Task(factorial("B", 3)) - task_c = asyncio.Task(factorial("C", 4)) - tasks = [task_a, task_b, task_c] + tasks = [ + asyncio.Task(factorial("A", 2)), + asyncio.Task(factorial("B", 3)), + asyncio.Task(factorial("C", 4))] loop = asyncio.get_event_loop() loop.run_until_complete(asyncio.wait(tasks)) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 12:32:39 2013 From: python-checkins at python.org (nick.coghlan) Date: Tue, 10 Dec 2013 12:32:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5NDA3?= =?utf-8?q?=3A_add_Python_Packaging_User_Guide_notes?= Message-ID: <3ddzdz3Z1Gz7LkV@mail.python.org> http://hg.python.org/cpython/rev/16b7536e418b changeset: 87880:16b7536e418b branch: 3.3 parent: 87874:49eb895be796 user: Nick Coghlan date: Tue Dec 10 21:24:55 2013 +1000 summary: Issue #19407: add Python Packaging User Guide notes The stdlib docs for package distribution and building extensions are rather dated, and that isn't expected to change for 2.7 and 3.3. The Python Packaging User Guide isn't complete either, but it's already a much better road map for new users than the existing stdlib docs. files: Doc/distutils/index.rst | 10 ++++++++++ Doc/extending/index.rst | 9 +++++++++ Doc/install/index.rst | 18 +++++++++++++----- 3 files changed, 32 insertions(+), 5 deletions(-) diff --git a/Doc/distutils/index.rst b/Doc/distutils/index.rst --- a/Doc/distutils/index.rst +++ b/Doc/distutils/index.rst @@ -12,6 +12,16 @@ make Python modules and extensions easily available to a wider audience with very little overhead for build/release/install mechanics. +.. note:: + + This guide only covers the basic tools for building and distributing + extensions that are provided as part of this version of Python. Third + party tools offer easier to use and more secure alternatives. Refer to the + `quick recommendations section + `__ + in the Python Packaging User Guide for more information. + + .. toctree:: :maxdepth: 2 :numbered: diff --git a/Doc/extending/index.rst b/Doc/extending/index.rst --- a/Doc/extending/index.rst +++ b/Doc/extending/index.rst @@ -21,6 +21,15 @@ For a detailed description of the whole Python/C API, see the separate :ref:`c-api-index`. +.. note:: + + This guide only covers the basic tools for creating extensions provided + as part of this version of CPython. Third party tools may offer simpler + alternatives. Refer to the `binary extensions section + `__ + in the Python Packaging User Guide for more information. + + .. toctree:: :maxdepth: 2 :numbered: diff --git a/Doc/install/index.rst b/Doc/install/index.rst --- a/Doc/install/index.rst +++ b/Doc/install/index.rst @@ -20,12 +20,20 @@ Finally, it might be useful to include all the material from my "Care and Feeding of a Python Installation" talk in here somewhere. Yow! -.. topic:: Abstract +This document describes the Python Distribution Utilities ("Distutils") from the +end-user's point-of-view, describing how to extend the capabilities of a +standard Python installation by building and installing third-party Python +modules and extensions. - This document describes the Python Distribution Utilities ("Distutils") from the - end-user's point-of-view, describing how to extend the capabilities of a - standard Python installation by building and installing third-party Python - modules and extensions. + +.. note:: + + This guide only covers the basic tools for installing extensions that are + provided as part of this version of Python. Third party tools offer easier + to use and more secure alternatives. Refer to the + `quick recommendations section + `__ + in the Python Packaging User Guide for more information. .. _inst-intro: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 12:32:40 2013 From: python-checkins at python.org (nick.coghlan) Date: Tue, 10 Dec 2013 12:32:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319407=3A_merge_PPUG_notes_from_3=2E3?= Message-ID: <3ddzf05MJhz7Llx@mail.python.org> http://hg.python.org/cpython/rev/bc21da9727ad changeset: 87881:bc21da9727ad parent: 87879:03e875beda37 parent: 87880:16b7536e418b user: Nick Coghlan date: Tue Dec 10 21:31:14 2013 +1000 summary: Issue #19407: merge PPUG notes from 3.3 files: Doc/distutils/index.rst | 10 ++++++++++ Doc/extending/index.rst | 9 +++++++++ Doc/install/index.rst | 18 +++++++++++++----- 3 files changed, 32 insertions(+), 5 deletions(-) diff --git a/Doc/distutils/index.rst b/Doc/distutils/index.rst --- a/Doc/distutils/index.rst +++ b/Doc/distutils/index.rst @@ -12,6 +12,16 @@ make Python modules and extensions easily available to a wider audience with very little overhead for build/release/install mechanics. +.. note:: + + This guide only covers the basic tools for building and distributing + extensions that are provided as part of this version of Python. Third + party tools offer easier to use and more secure alternatives. Refer to the + `quick recommendations section + `__ + in the Python Packaging User Guide for more information. + + .. toctree:: :maxdepth: 2 :numbered: diff --git a/Doc/extending/index.rst b/Doc/extending/index.rst --- a/Doc/extending/index.rst +++ b/Doc/extending/index.rst @@ -21,6 +21,15 @@ For a detailed description of the whole Python/C API, see the separate :ref:`c-api-index`. +.. note:: + + This guide only covers the basic tools for creating extensions provided + as part of this version of CPython. Third party tools may offer simpler + alternatives. Refer to the `binary extensions section + `__ + in the Python Packaging User Guide for more information. + + .. toctree:: :maxdepth: 2 :numbered: diff --git a/Doc/install/index.rst b/Doc/install/index.rst --- a/Doc/install/index.rst +++ b/Doc/install/index.rst @@ -20,12 +20,20 @@ Finally, it might be useful to include all the material from my "Care and Feeding of a Python Installation" talk in here somewhere. Yow! -.. topic:: Abstract +This document describes the Python Distribution Utilities ("Distutils") from the +end-user's point-of-view, describing how to extend the capabilities of a +standard Python installation by building and installing third-party Python +modules and extensions. - This document describes the Python Distribution Utilities ("Distutils") from the - end-user's point-of-view, describing how to extend the capabilities of a - standard Python installation by building and installing third-party Python - modules and extensions. + +.. note:: + + This guide only covers the basic tools for installing extensions that are + provided as part of this version of Python. Third party tools offer easier + to use and more secure alternatives. Refer to the + `quick recommendations section + `__ + in the Python Packaging User Guide for more information. .. _inst-intro: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 12:54:22 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 10 Dec 2013 12:54:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Remove_commented_out_debug?= =?utf-8?q?ging_code_=28remnants_of_issue_=2318996=29=2E?= Message-ID: <3df0721W8Cz7Ljg@mail.python.org> http://hg.python.org/cpython/rev/a0c687dc0039 changeset: 87882:a0c687dc0039 user: Serhiy Storchaka date: Tue Dec 10 13:53:56 2013 +0200 summary: Remove commented out debugging code (remnants of issue #18996). files: Lib/unittest/test/test_case.py | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) diff --git a/Lib/unittest/test/test_case.py b/Lib/unittest/test/test_case.py --- a/Lib/unittest/test/test_case.py +++ b/Lib/unittest/test/test_case.py @@ -876,8 +876,6 @@ with self.assertRaises(self.failureException) as cm: self.assertEqual(s1, s2) c = 'xxxx[85 chars]xxxxxxxxxxx' - #print() - #print(str(cm.exception)) self.assertEqual(str(cm.exception), "'%sa%s' != '%sb%s'" % (c, p, c, p)) p = 'y' * 100 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 13:06:52 2013 From: python-checkins at python.org (ezio.melotti) Date: Tue, 10 Dec 2013 13:06:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogIzE5OTQzOiBmaXgg?= =?utf-8?q?typo_noticed_by_Jakub_Wilk=2E?= Message-ID: <3df0PS07Bkz7Ljg@mail.python.org> http://hg.python.org/cpython/rev/b96a6493cecf changeset: 87883:b96a6493cecf branch: 2.7 parent: 87877:a9f91a38a265 user: Ezio Melotti date: Tue Dec 10 14:05:46 2013 +0200 summary: #19943: fix typo noticed by Jakub Wilk. files: Lib/lib2to3/fixes/fix_import.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/lib2to3/fixes/fix_import.py b/Lib/lib2to3/fixes/fix_import.py --- a/Lib/lib2to3/fixes/fix_import.py +++ b/Lib/lib2to3/fixes/fix_import.py @@ -32,7 +32,7 @@ elif node.type == syms.dotted_as_names: pending.extend(node.children[::-2]) else: - raise AssertionError("unkown node type") + raise AssertionError("unknown node type") class FixImport(fixer_base.BaseFix): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 13:06:53 2013 From: python-checkins at python.org (ezio.melotti) Date: Tue, 10 Dec 2013 13:06:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzE5OTQzOiBmaXgg?= =?utf-8?q?typo_noticed_by_Jakub_Wilk=2E?= Message-ID: <3df0PT24RMz7Lkw@mail.python.org> http://hg.python.org/cpython/rev/9f38bbd4e041 changeset: 87884:9f38bbd4e041 branch: 3.3 parent: 87880:16b7536e418b user: Ezio Melotti date: Tue Dec 10 14:05:46 2013 +0200 summary: #19943: fix typo noticed by Jakub Wilk. files: Lib/lib2to3/fixes/fix_import.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/lib2to3/fixes/fix_import.py b/Lib/lib2to3/fixes/fix_import.py --- a/Lib/lib2to3/fixes/fix_import.py +++ b/Lib/lib2to3/fixes/fix_import.py @@ -32,7 +32,7 @@ elif node.type == syms.dotted_as_names: pending.extend(node.children[::-2]) else: - raise AssertionError("unkown node type") + raise AssertionError("unknown node type") class FixImport(fixer_base.BaseFix): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 13:06:54 2013 From: python-checkins at python.org (ezio.melotti) Date: Tue, 10 Dec 2013 13:06:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogIzE5OTQzOiBtZXJnZSB3aXRoIDMuMy4=?= Message-ID: <3df0PV4M9kz7Lm1@mail.python.org> http://hg.python.org/cpython/rev/a3bdbe220f8a changeset: 87885:a3bdbe220f8a parent: 87882:a0c687dc0039 parent: 87884:9f38bbd4e041 user: Ezio Melotti date: Tue Dec 10 14:06:18 2013 +0200 summary: #19943: merge with 3.3. files: Lib/lib2to3/fixes/fix_import.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/lib2to3/fixes/fix_import.py b/Lib/lib2to3/fixes/fix_import.py --- a/Lib/lib2to3/fixes/fix_import.py +++ b/Lib/lib2to3/fixes/fix_import.py @@ -32,7 +32,7 @@ elif node.type == syms.dotted_as_names: pending.extend(node.children[::-2]) else: - raise AssertionError("unkown node type") + raise AssertionError("unknown node type") class FixImport(fixer_base.BaseFix): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 17:23:20 2013 From: python-checkins at python.org (nadeem.vawda) Date: Tue, 10 Dec 2013 17:23:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Skip_test_for_?= =?utf-8?q?=2319878_on_Windows=2E?= Message-ID: <3df65N2G7wz7Llg@mail.python.org> http://hg.python.org/cpython/rev/3337298f5c75 changeset: 87886:3337298f5c75 branch: 2.7 parent: 87883:b96a6493cecf user: Nadeem Vawda date: Tue Dec 10 17:23:00 2013 +0100 summary: Skip test for #19878 on Windows. files: Lib/test/test_bz2.py | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_bz2.py b/Lib/test/test_bz2.py --- a/Lib/test/test_bz2.py +++ b/Lib/test/test_bz2.py @@ -325,6 +325,9 @@ self.assertRaises(ValueError, f.readline) self.assertRaises(ValueError, f.readlines) + @unittest.skipIf(sys.platform == 'win32', + 'test depends on being able to delete a still-open file,' + ' which is not possible on Windows') def testInitNonExistentFile(self): # Issue #19878: Should not segfault when __init__ with non-existent # file for the second time. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 21:09:42 2013 From: python-checkins at python.org (zach.ware) Date: Tue, 10 Dec 2013 21:09:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5NTcy?= =?utf-8?q?=3A_More_silently_skipped_tests_explicitly_skipped=2E?= Message-ID: <3dfC6Z5X3Kz7Lmm@mail.python.org> http://hg.python.org/cpython/rev/423e09aedf79 changeset: 87887:423e09aedf79 branch: 2.7 user: Zachary Ware date: Tue Dec 10 14:09:20 2013 -0600 summary: Issue #19572: More silently skipped tests explicitly skipped. files: Lib/test/string_tests.py | 44 ++-- Lib/test/test_aepack.py | 8 +- Lib/test/test_array.py | 2 +- Lib/test/test_bsddb.py | 23 +- Lib/test/test_builtin.py | 109 +++++----- Lib/test/test_cfgparser.py | 12 +- Lib/test/test_cmath.py | 2 +- Lib/test/test_codecencodings_iso2022.py | 1 + Lib/test/test_decimal.py | 18 +- Lib/test/test_descr.py | 3 +- Lib/test/test_dis.py | 2 + Lib/test/test_dumbdbm.py | 6 +- Lib/test/test_fileio.py | 42 ++-- Lib/test/test_float.py | 2 +- Lib/test/test_functools.py | 8 +- Lib/test/test_getargs2.py | 17 +- Lib/test/test_grp.py | 4 +- Lib/test/test_imp.py | 12 +- Lib/test/test_index.py | 7 +- Lib/test/test_io.py | 12 +- Lib/test/test_iter.py | 4 +- Lib/test/test_macos.py | 11 +- Lib/test/test_macostools.py | 48 ++-- Lib/test/test_memoryview.py | 10 +- Lib/test/test_multibytecodec.py | 88 ++++---- Lib/test/test_multibytecodec_support.py | 4 +- Lib/test/test_multiprocessing.py | 14 +- Lib/test/test_nis.py | 6 +- Lib/test/test_os.py | 5 +- Lib/test/test_posix.py | 9 +- Lib/test/test_pwd.py | 17 +- Lib/test/test_re.py | 6 +- Lib/test/test_repr.py | 2 + Lib/test/test_resource.py | 126 +++++------ Lib/test/test_shutil.py | 12 +- Lib/test/test_site.py | 1 + Lib/test/test_socket.py | 18 +- Lib/test/test_ssl.py | 3 +- Lib/test/test_str.py | 12 +- Lib/test/test_strptime.py | 6 +- Lib/test/test_tarfile.py | 17 +- Lib/test/test_telnetlib.py | 1 - Lib/test/test_tempfile.py | 12 +- Lib/test/test_thread.py | 42 +-- Lib/test/test_threading.py | 16 +- Lib/test/test_timeout.py | 3 + Lib/test/test_unicode.py | 10 +- Lib/test/test_urllibnet.py | 7 +- Lib/test/test_warnings.py | 2 +- Lib/test/test_xmlrpc.py | 12 +- Lib/test/test_zipimport.py | 2 +- Misc/NEWS | 2 +- 52 files changed, 420 insertions(+), 442 deletions(-) diff --git a/Lib/test/string_tests.py b/Lib/test/string_tests.py --- a/Lib/test/string_tests.py +++ b/Lib/test/string_tests.py @@ -750,10 +750,10 @@ self.checkraises(TypeError, 'hello', 'replace', 42, 'h') self.checkraises(TypeError, 'hello', 'replace', 'h', 42) + @unittest.skipIf(sys.maxint > (1 << 32) or struct.calcsize('P') != 4, + 'only applies to 32-bit platforms') def test_replace_overflow(self): # Check for overflow checking on 32 bit machines - if sys.maxint != 2147483647 or struct.calcsize("P") > 4: - return A2_16 = "A" * (2**16) self.checkraises(OverflowError, A2_16, "replace", "", A2_16) self.checkraises(OverflowError, A2_16, "replace", "A", A2_16) @@ -1286,27 +1286,27 @@ # Additional tests that only work with # 8bit compatible object, i.e. str and UserString - if test_support.have_unicode: - def test_encoding_decoding(self): - codecs = [('rot13', 'uryyb jbeyq'), - ('base64', 'aGVsbG8gd29ybGQ=\n'), - ('hex', '68656c6c6f20776f726c64'), - ('uu', 'begin 666 \n+:&5L;&\\@=V]R;&0 \n \nend\n')] - for encoding, data in codecs: - self.checkequal(data, 'hello world', 'encode', encoding) - self.checkequal('hello world', data, 'decode', encoding) - # zlib is optional, so we make the test optional too... - try: - import zlib - except ImportError: - pass - else: - data = 'x\x9c\xcbH\xcd\xc9\xc9W(\xcf/\xcaI\x01\x00\x1a\x0b\x04]' - self.checkequal(data, 'hello world', 'encode', 'zlib') - self.checkequal('hello world', data, 'decode', 'zlib') + @unittest.skipUnless(test_support.have_unicode, 'no unicode support') + def test_encoding_decoding(self): + codecs = [('rot13', 'uryyb jbeyq'), + ('base64', 'aGVsbG8gd29ybGQ=\n'), + ('hex', '68656c6c6f20776f726c64'), + ('uu', 'begin 666 \n+:&5L;&\\@=V]R;&0 \n \nend\n')] + for encoding, data in codecs: + self.checkequal(data, 'hello world', 'encode', encoding) + self.checkequal('hello world', data, 'decode', encoding) + # zlib is optional, so we make the test optional too... + try: + import zlib + except ImportError: + pass + else: + data = 'x\x9c\xcbH\xcd\xc9\xc9W(\xcf/\xcaI\x01\x00\x1a\x0b\x04]' + self.checkequal(data, 'hello world', 'encode', 'zlib') + self.checkequal('hello world', data, 'decode', 'zlib') - self.checkraises(TypeError, 'xyz', 'decode', 42) - self.checkraises(TypeError, 'xyz', 'encode', 42) + self.checkraises(TypeError, 'xyz', 'decode', 42) + self.checkraises(TypeError, 'xyz', 'encode', 42) class MixinStrUnicodeTest: diff --git a/Lib/test/test_aepack.py b/Lib/test/test_aepack.py --- a/Lib/test/test_aepack.py +++ b/Lib/test/test_aepack.py @@ -59,10 +59,10 @@ try: import Carbon.File except: - return + self.skipTest('Carbon.File not available') if not hasattr(Carbon.File, "FSSpec"): - return + self.skipTest('Carbon.File.FSSpec not available') o = Carbon.File.FSSpec(os.curdir) packed = aepack.pack(o) unpacked = aepack.unpack(packed) @@ -72,9 +72,9 @@ try: import Carbon.File except: - return + self.skipTest('Carbon.File not available') if not hasattr(Carbon.File, "FSSpec"): - return + self.skipTest('Carbon.File.FSSpec not available') o = Carbon.File.FSSpec(os.curdir).NewAliasMinimal() packed = aepack.pack(o) unpacked = aepack.unpack(packed) diff --git a/Lib/test/test_array.py b/Lib/test/test_array.py --- a/Lib/test/test_array.py +++ b/Lib/test/test_array.py @@ -754,7 +754,7 @@ try: import gc except ImportError: - return + self.skipTest('gc module not available') a = array.array(self.typecode) l = [iter(a)] l.append(l) diff --git a/Lib/test/test_bsddb.py b/Lib/test/test_bsddb.py --- a/Lib/test/test_bsddb.py +++ b/Lib/test/test_bsddb.py @@ -47,10 +47,7 @@ self.assertIn('discovered', self.f.values()) def test_close_and_reopen(self): - if self.fname is None: - # if we're using an in-memory only db, we can't reopen it - # so finish here. - return + self.assertIsNotNone(self.fname) self.f.close() self.f = self.openmethod[0](self.fname, 'w') for k, v in self.d.iteritems(): @@ -309,8 +306,7 @@ self.assertEqual(self.f[k], v) def test_keyordering(self): - if self.openmethod[0] is not bsddb.btopen: - return + self.assertIs(self.openmethod[0], bsddb.btopen) keys = self.d.keys() keys.sort() self.assertEqual(self.f.first()[0], keys[0]) @@ -327,19 +323,34 @@ fname = None openmethod = [bsddb.btopen] + # if we're using an in-memory only db, we can't reopen it + test_close_and_reopen = None + class TestBTree_InMemory_Truncate(TestBSDDB): fname = None openflag = 'n' openmethod = [bsddb.btopen] + # if we're using an in-memory only db, we can't reopen it + test_close_and_reopen = None + class TestHashTable(TestBSDDB): fname = test_support.TESTFN openmethod = [bsddb.hashopen] + # keyordering is specific to btopen method + test_keyordering = None + class TestHashTable_InMemory(TestBSDDB): fname = None openmethod = [bsddb.hashopen] + # if we're using an in-memory only db, we can't reopen it + test_close_and_reopen = None + + # keyordering is specific to btopen method + test_keyordering = None + ## # (bsddb.rnopen,'Record Numbers'), 'put' for RECNO for bsddb 1.85 ## # appears broken... at least on ## # Solaris Intel - rmasse 1/97 diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py --- a/Lib/test/test_builtin.py +++ b/Lib/test/test_builtin.py @@ -447,59 +447,6 @@ return 'a' self.assertRaises(TypeError, eval, 'dir()', globals(), C()) - # Done outside of the method test_z to get the correct scope - z = 0 - f = open(TESTFN, 'w') - f.write('z = z+1\n') - f.write('z = z*2\n') - f.close() - with check_py3k_warnings(("execfile.. not supported in 3.x", - DeprecationWarning)): - execfile(TESTFN) - - def test_execfile(self): - global numruns - if numruns: - return - numruns += 1 - - globals = {'a': 1, 'b': 2} - locals = {'b': 200, 'c': 300} - - self.assertEqual(self.__class__.z, 2) - globals['z'] = 0 - execfile(TESTFN, globals) - self.assertEqual(globals['z'], 2) - locals['z'] = 0 - execfile(TESTFN, globals, locals) - self.assertEqual(locals['z'], 2) - - class M: - "Test mapping interface versus possible calls from execfile()." - def __init__(self): - self.z = 10 - def __getitem__(self, key): - if key == 'z': - return self.z - raise KeyError - def __setitem__(self, key, value): - if key == 'z': - self.z = value - return - raise KeyError - - locals = M() - locals['z'] = 0 - execfile(TESTFN, globals, locals) - self.assertEqual(locals['z'], 2) - - unlink(TESTFN) - self.assertRaises(TypeError, execfile) - self.assertRaises(TypeError, execfile, TESTFN, {}, ()) - import os - self.assertRaises(IOError, execfile, os.curdir) - self.assertRaises(IOError, execfile, "I_dont_exist") - def test_filter(self): self.assertEqual(filter(lambda c: 'a' <= c <= 'z', 'Hello World'), 'elloorld') self.assertEqual(filter(None, [1, 'hello', [], [3], '', None, 9, 0]), [1, 'hello', [3], 9]) @@ -1646,6 +1593,56 @@ self.assertRaises(ValueError, x.translate, "1", 1) self.assertRaises(TypeError, x.translate, "1"*256, 1) +class TestExecFile(unittest.TestCase): + # Done outside of the method test_z to get the correct scope + z = 0 + f = open(TESTFN, 'w') + f.write('z = z+1\n') + f.write('z = z*2\n') + f.close() + with check_py3k_warnings(("execfile.. not supported in 3.x", + DeprecationWarning)): + execfile(TESTFN) + + def test_execfile(self): + globals = {'a': 1, 'b': 2} + locals = {'b': 200, 'c': 300} + + self.assertEqual(self.__class__.z, 2) + globals['z'] = 0 + execfile(TESTFN, globals) + self.assertEqual(globals['z'], 2) + locals['z'] = 0 + execfile(TESTFN, globals, locals) + self.assertEqual(locals['z'], 2) + + class M: + "Test mapping interface versus possible calls from execfile()." + def __init__(self): + self.z = 10 + def __getitem__(self, key): + if key == 'z': + return self.z + raise KeyError + def __setitem__(self, key, value): + if key == 'z': + self.z = value + return + raise KeyError + + locals = M() + locals['z'] = 0 + execfile(TESTFN, globals, locals) + self.assertEqual(locals['z'], 2) + + unlink(TESTFN) + self.assertRaises(TypeError, execfile) + self.assertRaises(TypeError, execfile, TESTFN, {}, ()) + import os + self.assertRaises(IOError, execfile, os.curdir) + self.assertRaises(IOError, execfile, "I_dont_exist") + + class TestSorted(unittest.TestCase): def test_basic(self): @@ -1693,6 +1690,12 @@ run_unittest(*args) def test_main(verbose=None): + global numruns + if not numruns: + with check_py3k_warnings( + (".+ not supported in 3.x", DeprecationWarning)): + run_unittest(TestExecFile) + numruns += 1 test_classes = (BuiltinTest, TestSorted) _run_unittest(*test_classes) diff --git a/Lib/test/test_cfgparser.py b/Lib/test/test_cfgparser.py --- a/Lib/test/test_cfgparser.py +++ b/Lib/test/test_cfgparser.py @@ -284,13 +284,17 @@ cf.set("sect", "option1", mystr("splat")) cf.set("sect", "option2", "splat") cf.set("sect", "option2", mystr("splat")) + + def test_set_unicode(self): try: unicode except NameError: - pass - else: - cf.set("sect", "option1", unicode("splat")) - cf.set("sect", "option2", unicode("splat")) + self.skipTest('no unicode support') + + cf = self.fromstring("[sect]\n" + "option1=foo\n") + cf.set("sect", "option1", unicode("splat")) + cf.set("sect", "option2", unicode("splat")) def test_read_returns_file_list(self): file1 = test_support.findfile("cfgparser.1") diff --git a/Lib/test/test_cmath.py b/Lib/test/test_cmath.py --- a/Lib/test/test_cmath.py +++ b/Lib/test/test_cmath.py @@ -282,7 +282,7 @@ def test_specific_values(self): if not float.__getformat__("double").startswith("IEEE"): - return + self.skipTest('needs IEEE double') def rect_complex(z): """Wrapped version of rect that accepts a complex number instead of diff --git a/Lib/test/test_codecencodings_iso2022.py b/Lib/test/test_codecencodings_iso2022.py --- a/Lib/test/test_codecencodings_iso2022.py +++ b/Lib/test/test_codecencodings_iso2022.py @@ -36,6 +36,7 @@ # iso2022_kr.txt cannot be used to test "chunk coding": the escape # sequence is only written on the first line + @unittest.skip('iso2022_kr.txt cannot be used to test "chunk coding"') def test_chunkcoding(self): pass diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -223,7 +223,6 @@ global skip_expected if skip_expected: raise unittest.SkipTest - return with open(file) as f: for line in f: line = line.replace('\r\n', '').replace('\n', '') @@ -234,7 +233,6 @@ #Exception raised where there shouldn't have been one. self.fail('Exception "'+exception.__class__.__name__ + '" raised on line '+line) - return def eval_line(self, s): if s.find(' -> ') >= 0 and s[:2] != '--' and not s.startswith(' --'): @@ -391,7 +389,6 @@ 'Incorrect answer for ' + s + ' -- got ' + result) self.assertItemsEqual(myexceptions, theirexceptions, 'Incorrect flags set in ' + s + ' -- got ' + str(myexceptions)) - return def getexceptions(self): return [e for e in Signals if self.context.flags[e]] @@ -834,7 +831,7 @@ try: from locale import CHAR_MAX except ImportError: - return + self.skipTest('locale.CHAR_MAX not available') # Set up some localeconv-like dictionaries en_US = { @@ -1196,7 +1193,6 @@ cls.assertEqual(test1, Decimal('0.3333333333333333333333333333')) cls.assertEqual(test2, Decimal('0.3333333333333333333333333333')) - return def thfunc2(cls): d1 = Decimal(1) @@ -1210,17 +1206,12 @@ cls.assertEqual(test1, Decimal('0.3333333333333333333333333333')) cls.assertEqual(test2, Decimal('0.333333333333333333')) - return + at unittest.skipUnless(threading, 'threading required') class DecimalUseOfContextTest(unittest.TestCase): '''Unit tests for Use of Context cases in Decimal.''' - try: - import threading - except ImportError: - threading = None - # Take care executing this test from IDLE, there's an issue in threading # that hangs IDLE and I couldn't find it @@ -1239,10 +1230,6 @@ self.finish1.wait() self.finish2.wait() - return - - if threading is None: - del test_threading class DecimalUsabilityTest(unittest.TestCase): @@ -1540,7 +1527,6 @@ self.assertEqual(d1._sign, b1._sign) self.assertEqual(d1._int, b1._int) self.assertEqual(d1._exp, b1._exp) - return Decimal(d1) self.assertEqual(d1._sign, b1._sign) diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -1059,11 +1059,12 @@ c.abc = 5 self.assertEqual(c.abc, 5) + def test_unicode_slots(self): # Test unicode slot names try: unicode except NameError: - pass + self.skipTest('no unicode support') else: # Test a single unicode string is not expanded as a sequence. class C(object): diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py --- a/Lib/test/test_dis.py +++ b/Lib/test/test_dis.py @@ -125,6 +125,8 @@ # so fails if the tests are run with -O. Skip this test then. if __debug__: self.do_disassembly_test(bug1333982, dis_bug1333982) + else: + self.skipTest('need asserts, run without -O') def test_big_linenos(self): def func(count): diff --git a/Lib/test/test_dumbdbm.py b/Lib/test/test_dumbdbm.py --- a/Lib/test/test_dumbdbm.py +++ b/Lib/test/test_dumbdbm.py @@ -38,11 +38,9 @@ self.read_helper(f) f.close() + @unittest.skipUnless(hasattr(os, 'chmod'), 'os.chmod not available') + @unittest.skipUnless(hasattr(os, 'umask'), 'os.umask not available') def test_dumbdbm_creation_mode(self): - # On platforms without chmod, don't do anything. - if not (hasattr(os, 'chmod') and hasattr(os, 'umask')): - return - try: old_umask = os.umask(0002) f = dumbdbm.open(_fname, 'c', 0637) diff --git a/Lib/test/test_fileio.py b/Lib/test/test_fileio.py --- a/Lib/test/test_fileio.py +++ b/Lib/test/test_fileio.py @@ -283,28 +283,29 @@ self.assertEqual(f.seekable(), True) self.assertEqual(f.isatty(), False) f.close() - - if sys.platform != "win32": - try: - f = _FileIO("/dev/tty", "a") - except EnvironmentError: - # When run in a cron job there just aren't any - # ttys, so skip the test. This also handles other - # OS'es that don't support /dev/tty. - pass - else: - self.assertEqual(f.readable(), False) - self.assertEqual(f.writable(), True) - if sys.platform != "darwin" and \ - 'bsd' not in sys.platform and \ - not sys.platform.startswith('sunos'): - # Somehow /dev/tty appears seekable on some BSDs - self.assertEqual(f.seekable(), False) - self.assertEqual(f.isatty(), True) - f.close() finally: os.unlink(TESTFN) + @unittest.skipIf(sys.platform == 'win32', 'no ttys on Windows') + def testAblesOnTTY(self): + try: + f = _FileIO("/dev/tty", "a") + except EnvironmentError: + # When run in a cron job there just aren't any + # ttys, so skip the test. This also handles other + # OS'es that don't support /dev/tty. + self.skipTest('need /dev/tty') + else: + self.assertEqual(f.readable(), False) + self.assertEqual(f.writable(), True) + if sys.platform != "darwin" and \ + 'bsd' not in sys.platform and \ + not sys.platform.startswith('sunos'): + # Somehow /dev/tty appears seekable on some BSDs + self.assertEqual(f.seekable(), False) + self.assertEqual(f.isatty(), True) + f.close() + def testInvalidModeStrings(self): # check invalid mode strings for mode in ("", "aU", "wU+", "rw", "rt"): @@ -342,8 +343,7 @@ try: fn = TESTFN.encode("ascii") except UnicodeEncodeError: - # Skip test - return + self.skipTest('could not encode %r to ascii' % TESTFN) f = _FileIO(fn, "w") try: f.write(b"abc") diff --git a/Lib/test/test_float.py b/Lib/test/test_float.py --- a/Lib/test/test_float.py +++ b/Lib/test/test_float.py @@ -101,7 +101,7 @@ # it still has to accept the normal python syntax import locale if not locale.localeconv()['decimal_point'] == ',': - return + self.skipTest('decimal_point is not ","') self.assertEqual(float(" 3.14 "), 3.14) self.assertEqual(float("+3.14 "), 3.14) diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -43,8 +43,6 @@ self.assertEqual(p.args, (1, 2)) self.assertEqual(p.keywords, dict(a=10, b=20)) # attributes should not be writable - if not isinstance(self.thetype, type): - return self.assertRaises(TypeError, setattr, p, 'func', map) self.assertRaises(TypeError, setattr, p, 'args', (1, 2)) self.assertRaises(TypeError, setattr, p, 'keywords', dict(a=1, b=2)) @@ -180,8 +178,10 @@ thetype = PythonPartial # the python version isn't picklable - def test_pickle(self): pass - def test_setstate_refcount(self): pass + test_pickle = test_setstate_refcount = None + + # the python version isn't a type + test_attributes = None class TestUpdateWrapper(unittest.TestCase): diff --git a/Lib/test/test_getargs2.py b/Lib/test/test_getargs2.py --- a/Lib/test/test_getargs2.py +++ b/Lib/test/test_getargs2.py @@ -42,6 +42,13 @@ INT_MIN, LONG_MIN, LONG_MAX, PY_SSIZE_T_MIN, PY_SSIZE_T_MAX, \ SHRT_MIN, SHRT_MAX +try: + from _testcapi import getargs_L, getargs_K +except ImportError: + _PY_LONG_LONG_available = False +else: + _PY_LONG_LONG_available = True + # fake, they are not defined in Python's header files LLONG_MAX = 2**63-1 LLONG_MIN = -2**63 @@ -208,6 +215,7 @@ self.assertRaises(OverflowError, getargs_n, VERY_LARGE) + at unittest.skipUnless(_PY_LONG_LONG_available, 'PY_LONG_LONG not available') class LongLong_TestCase(unittest.TestCase): def test_L(self): from _testcapi import getargs_L @@ -322,13 +330,8 @@ self.fail('TypeError should have been raised') def test_main(): - tests = [Signed_TestCase, Unsigned_TestCase, Tuple_TestCase, Keywords_TestCase] - try: - from _testcapi import getargs_L, getargs_K - except ImportError: - pass # PY_LONG_LONG not available - else: - tests.append(LongLong_TestCase) + tests = [Signed_TestCase, Unsigned_TestCase, LongLong_TestCase, + Tuple_TestCase, Keywords_TestCase] test_support.run_unittest(*tests) if __name__ == "__main__": diff --git a/Lib/test/test_grp.py b/Lib/test/test_grp.py --- a/Lib/test/test_grp.py +++ b/Lib/test/test_grp.py @@ -26,8 +26,10 @@ for e in entries: self.check_value(e) + def test_values_extended(self): + entries = grp.getgrall() if len(entries) > 1000: # Huge group file (NIS?) -- skip the rest - return + self.skipTest('huge group file, extended test skipped') for e in entries: e2 = grp.getgrgid(e.gr_gid) diff --git a/Lib/test/test_imp.py b/Lib/test/test_imp.py --- a/Lib/test/test_imp.py +++ b/Lib/test/test_imp.py @@ -2,7 +2,12 @@ import unittest from test import test_support +try: + import thread +except ImportError: + thread = None + at unittest.skipUnless(thread, 'threading not available') class LockTests(unittest.TestCase): """Very basic test of import lock functions.""" @@ -68,13 +73,8 @@ def test_main(): tests = [ ReloadTests, + LockTests, ] - try: - import thread - except ImportError: - pass - else: - tests.append(LockTests) test_support.run_unittest(*tests) if __name__ == "__main__": diff --git a/Lib/test/test_index.py b/Lib/test/test_index.py --- a/Lib/test/test_index.py +++ b/Lib/test/test_index.py @@ -91,7 +91,7 @@ self.assertEqual(self.seq[self.o:self.o2], self.seq[1:3]) self.assertEqual(self.seq[self.n:self.n2], self.seq[2:4]) - def test_slice_bug7532(self): + def test_slice_bug7532a(self): seqlen = len(self.seq) self.o.ind = int(seqlen * 1.5) self.n.ind = seqlen + 2 @@ -99,9 +99,12 @@ self.assertEqual(self.seq[:self.o], self.seq) self.assertEqual(self.seq[self.n:], self.seq[0:0]) self.assertEqual(self.seq[:self.n], self.seq) + + def test_slice_bug7532b(self): if isinstance(self.seq, ClassicSeq): - return + self.skipTest('test fails for ClassicSeq') # These tests fail for ClassicSeq (see bug #7532) + seqlen = len(self.seq) self.o2.ind = -seqlen - 2 self.n2.ind = -int(seqlen * 1.5) self.assertEqual(self.seq[self.o2:], self.seq) diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -393,14 +393,9 @@ # a long time to build the >2GB file and takes >2GB of disk space # therefore the resource must be enabled to run this test. if sys.platform[:3] == 'win' or sys.platform == 'darwin': - if not support.is_resource_enabled("largefile"): - print("\nTesting large file ops skipped on %s." % sys.platform, - file=sys.stderr) - print("It requires %d bytes and a long time." % self.LARGE, - file=sys.stderr) - print("Use 'regrtest.py -u largefile test_io' to run it.", - file=sys.stderr) - return + support.requires( + 'largefile', + 'test requires %s bytes and a long time to run' % self.LARGE) with self.open(support.TESTFN, "w+b", 0) as f: self.large_file_ops(f) with self.open(support.TESTFN, "w+b") as f: @@ -650,6 +645,7 @@ self.assertEqual(42, bufio.fileno()) + @unittest.skip('test having existential crisis') def test_no_fileno(self): # XXX will we always have fileno() function? If so, kill # this test. Else, write it. diff --git a/Lib/test/test_iter.py b/Lib/test/test_iter.py --- a/Lib/test/test_iter.py +++ b/Lib/test/test_iter.py @@ -526,7 +526,7 @@ d = {"one": 1, "two": 2, "three": 3} self.assertEqual(reduce(add, d), "".join(d.keys())) - # This test case will be removed if we don't have Unicode + @unittest.skipUnless(have_unicode, 'needs unicode support') def test_unicode_join_endcase(self): # This class inserts a Unicode object into its argument's natural @@ -567,8 +567,6 @@ unlink(TESTFN) except OSError: pass - if not have_unicode: - def test_unicode_join_endcase(self): pass # Test iterators with 'x in y' and 'x not in y'. def test_in_and_not_in(self): diff --git a/Lib/test/test_macos.py b/Lib/test/test_macos.py --- a/Lib/test/test_macos.py +++ b/Lib/test/test_macos.py @@ -8,11 +8,9 @@ TESTFN2 = test_support.TESTFN + '2' class TestMacOS(unittest.TestCase): - + @unittest.skipUnless(os.path.exists('/Developer/Tools/SetFile'), + '/Developer/Tools/SetFile does not exist') def testGetCreatorAndType(self): - if not os.path.exists('/Developer/Tools/SetFile'): - return - try: fp = open(test_support.TESTFN, 'w') fp.write('\n') @@ -29,10 +27,9 @@ finally: os.unlink(test_support.TESTFN) + @unittest.skipUnless(os.path.exists('/Developer/Tools/GetFileInfo'), + '/Developer/Tools/GetFileInfo does not exist') def testSetCreatorAndType(self): - if not os.path.exists('/Developer/Tools/GetFileInfo'): - return - try: fp = open(test_support.TESTFN, 'w') fp.write('\n') diff --git a/Lib/test/test_macostools.py b/Lib/test/test_macostools.py --- a/Lib/test/test_macostools.py +++ b/Lib/test/test_macostools.py @@ -12,6 +12,8 @@ TESTFN2 = test_support.TESTFN + '2' +requires_32bit = unittest.skipUnless(sys.maxint < 2**32, '32-bit only test') + class TestMacostools(unittest.TestCase): def setUp(self): @@ -51,30 +53,32 @@ DeprecationWarning), quiet=True): macostools.touched(test_support.TESTFN) - if sys.maxint < 2**32: - def test_copy(self): - test_support.unlink(TESTFN2) - macostools.copy(test_support.TESTFN, TESTFN2) - self.assertEqual(self.compareData(), '') + @requires_32bit + def test_copy(self): + test_support.unlink(TESTFN2) + macostools.copy(test_support.TESTFN, TESTFN2) + self.assertEqual(self.compareData(), '') - if sys.maxint < 2**32: - def test_mkalias(self): - test_support.unlink(TESTFN2) - macostools.mkalias(test_support.TESTFN, TESTFN2) - fss, _, _ = Carbon.File.ResolveAliasFile(TESTFN2, 0) - self.assertEqual(fss.as_pathname(), os.path.realpath(test_support.TESTFN)) + @requires_32bit + def test_mkalias(self): + test_support.unlink(TESTFN2) + macostools.mkalias(test_support.TESTFN, TESTFN2) + fss, _, _ = Carbon.File.ResolveAliasFile(TESTFN2, 0) + self.assertEqual(fss.as_pathname(), os.path.realpath(test_support.TESTFN)) - def test_mkalias_relative(self): - test_support.unlink(TESTFN2) - # If the directory doesn't exist, then chances are this is a new - # install of Python so don't create it since the user might end up - # running ``sudo make install`` and creating the directory here won't - # leave it with the proper permissions. - if not os.path.exists(sys.prefix): - return - macostools.mkalias(test_support.TESTFN, TESTFN2, sys.prefix) - fss, _, _ = Carbon.File.ResolveAliasFile(TESTFN2, 0) - self.assertEqual(fss.as_pathname(), os.path.realpath(test_support.TESTFN)) + @requires_32bit + # If the directory doesn't exist, then chances are this is a new + # install of Python so don't create it since the user might end up + # running ``sudo make install`` and creating the directory here won't + # leave it with the proper permissions. + @unittest.skipUnless(os.path.exists(sys.prefix), + "%r doesn't exist" % sys.prefix) + def test_mkalias_relative(self): + test_support.unlink(TESTFN2) + + macostools.mkalias(test_support.TESTFN, TESTFN2, sys.prefix) + fss, _, _ = Carbon.File.ResolveAliasFile(TESTFN2, 0) + self.assertEqual(fss.as_pathname(), os.path.realpath(test_support.TESTFN)) def test_main(): diff --git a/Lib/test/test_memoryview.py b/Lib/test/test_memoryview.py --- a/Lib/test/test_memoryview.py +++ b/Lib/test/test_memoryview.py @@ -63,7 +63,7 @@ def test_setitem_readonly(self): if not self.ro_type: - return + self.skipTest("no read-only type to test") b = self.ro_type(self._source) oldrefcount = sys.getrefcount(b) m = self._view(b) @@ -77,7 +77,7 @@ def test_setitem_writable(self): if not self.rw_type: - return + self.skipTest("no writable type to test") tp = self.rw_type b = self.rw_type(self._source) oldrefcount = sys.getrefcount(b) @@ -183,13 +183,13 @@ def test_attributes_readonly(self): if not self.ro_type: - return + self.skipTest("no read-only type to test") m = self.check_attributes_with_type(self.ro_type) self.assertEqual(m.readonly, True) def test_attributes_writable(self): if not self.rw_type: - return + self.skipTest("no writable type to test") m = self.check_attributes_with_type(self.rw_type) self.assertEqual(m.readonly, False) @@ -236,7 +236,7 @@ # buffer as writable causing a segfault if using mmap tp = self.ro_type if tp is None: - return + self.skipTest("no read-only type to test") b = tp(self._source) m = self._view(b) i = io.BytesIO(b'ZZZZ') diff --git a/Lib/test/test_multibytecodec.py b/Lib/test/test_multibytecodec.py --- a/Lib/test/test_multibytecodec.py +++ b/Lib/test/test_multibytecodec.py @@ -157,57 +157,55 @@ os.unlink(TESTFN) class Test_StreamWriter(unittest.TestCase): - if len(u'\U00012345') == 2: # UCS2 - def test_gb18030(self): - s = StringIO.StringIO() - c = codecs.getwriter('gb18030')(s) - c.write(u'123') - self.assertEqual(s.getvalue(), '123') - c.write(u'\U00012345') - self.assertEqual(s.getvalue(), '123\x907\x959') + @unittest.skipUnless(len(u'\U00012345') == 2, 'need a narrow build') + def test_gb18030(self): + s = StringIO.StringIO() + c = codecs.getwriter('gb18030')(s) + c.write(u'123') + self.assertEqual(s.getvalue(), '123') + c.write(u'\U00012345') + self.assertEqual(s.getvalue(), '123\x907\x959') + c.write(u'\U00012345'[0]) + self.assertEqual(s.getvalue(), '123\x907\x959') + c.write(u'\U00012345'[1] + u'\U00012345' + u'\uac00\u00ac') + self.assertEqual(s.getvalue(), + '123\x907\x959\x907\x959\x907\x959\x827\xcf5\x810\x851') + c.write(u'\U00012345'[0]) + self.assertEqual(s.getvalue(), + '123\x907\x959\x907\x959\x907\x959\x827\xcf5\x810\x851') + self.assertRaises(UnicodeError, c.reset) + self.assertEqual(s.getvalue(), + '123\x907\x959\x907\x959\x907\x959\x827\xcf5\x810\x851') + + @unittest.skipUnless(len(u'\U00012345') == 2, 'need a narrow build') + def test_utf_8(self): + s= StringIO.StringIO() + c = codecs.getwriter('utf-8')(s) + c.write(u'123') + self.assertEqual(s.getvalue(), '123') + c.write(u'\U00012345') + self.assertEqual(s.getvalue(), '123\xf0\x92\x8d\x85') + + # Python utf-8 codec can't buffer surrogate pairs yet. + if 0: c.write(u'\U00012345'[0]) - self.assertEqual(s.getvalue(), '123\x907\x959') + self.assertEqual(s.getvalue(), '123\xf0\x92\x8d\x85') c.write(u'\U00012345'[1] + u'\U00012345' + u'\uac00\u00ac') self.assertEqual(s.getvalue(), - '123\x907\x959\x907\x959\x907\x959\x827\xcf5\x810\x851') + '123\xf0\x92\x8d\x85\xf0\x92\x8d\x85\xf0\x92\x8d\x85' + '\xea\xb0\x80\xc2\xac') c.write(u'\U00012345'[0]) self.assertEqual(s.getvalue(), - '123\x907\x959\x907\x959\x907\x959\x827\xcf5\x810\x851') - self.assertRaises(UnicodeError, c.reset) + '123\xf0\x92\x8d\x85\xf0\x92\x8d\x85\xf0\x92\x8d\x85' + '\xea\xb0\x80\xc2\xac') + c.reset() self.assertEqual(s.getvalue(), - '123\x907\x959\x907\x959\x907\x959\x827\xcf5\x810\x851') - - def test_utf_8(self): - s= StringIO.StringIO() - c = codecs.getwriter('utf-8')(s) - c.write(u'123') - self.assertEqual(s.getvalue(), '123') - c.write(u'\U00012345') - self.assertEqual(s.getvalue(), '123\xf0\x92\x8d\x85') - - # Python utf-8 codec can't buffer surrogate pairs yet. - if 0: - c.write(u'\U00012345'[0]) - self.assertEqual(s.getvalue(), '123\xf0\x92\x8d\x85') - c.write(u'\U00012345'[1] + u'\U00012345' + u'\uac00\u00ac') - self.assertEqual(s.getvalue(), - '123\xf0\x92\x8d\x85\xf0\x92\x8d\x85\xf0\x92\x8d\x85' - '\xea\xb0\x80\xc2\xac') - c.write(u'\U00012345'[0]) - self.assertEqual(s.getvalue(), - '123\xf0\x92\x8d\x85\xf0\x92\x8d\x85\xf0\x92\x8d\x85' - '\xea\xb0\x80\xc2\xac') - c.reset() - self.assertEqual(s.getvalue(), - '123\xf0\x92\x8d\x85\xf0\x92\x8d\x85\xf0\x92\x8d\x85' - '\xea\xb0\x80\xc2\xac\xed\xa0\x88') - c.write(u'\U00012345'[1]) - self.assertEqual(s.getvalue(), - '123\xf0\x92\x8d\x85\xf0\x92\x8d\x85\xf0\x92\x8d\x85' - '\xea\xb0\x80\xc2\xac\xed\xa0\x88\xed\xbd\x85') - - else: # UCS4 - pass + '123\xf0\x92\x8d\x85\xf0\x92\x8d\x85\xf0\x92\x8d\x85' + '\xea\xb0\x80\xc2\xac\xed\xa0\x88') + c.write(u'\U00012345'[1]) + self.assertEqual(s.getvalue(), + '123\xf0\x92\x8d\x85\xf0\x92\x8d\x85\xf0\x92\x8d\x85' + '\xea\xb0\x80\xc2\xac\xed\xa0\x88\xed\xbd\x85') def test_streamwriter_strwrite(self): s = StringIO.StringIO() diff --git a/Lib/test/test_multibytecodec_support.py b/Lib/test/test_multibytecodec_support.py --- a/Lib/test/test_multibytecodec_support.py +++ b/Lib/test/test_multibytecodec_support.py @@ -67,7 +67,7 @@ def test_xmlcharrefreplace(self): if self.has_iso10646: - return + self.skipTest('encoding contains full ISO 10646 map') s = u"\u0b13\u0b23\u0b60 nd eggs" self.assertEqual( @@ -77,7 +77,7 @@ def test_customreplace_encode(self): if self.has_iso10646: - return + self.skipTest('encoding contains full ISO 10646 map') from htmlentitydefs import codepoint2name diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -182,7 +182,7 @@ def test_current(self): if self.TYPE == 'threads': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) current = self.current_process() authkey = current.authkey @@ -249,7 +249,7 @@ def test_terminate(self): if self.TYPE == 'threads': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) p = self.Process(target=self._test_terminate) p.daemon = True @@ -334,7 +334,7 @@ def test_sys_exit(self): # See Issue 13854 if self.TYPE == 'threads': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) testfn = test_support.TESTFN self.addCleanup(test_support.unlink, testfn) @@ -582,7 +582,7 @@ try: self.assertEqual(q.qsize(), 0) except NotImplementedError: - return + self.skipTest('qsize method not implemented') q.put(1) self.assertEqual(q.qsize(), 1) q.put(5) @@ -683,7 +683,7 @@ def test_timeout(self): if self.TYPE != 'processes': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) sem = self.Semaphore(0) acquire = TimingWrapper(sem.acquire) @@ -1120,7 +1120,7 @@ def test_map_unplicklable(self): # Issue #19425 -- failure to pickle should not cause a hang if self.TYPE == 'threads': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) class A(object): def __reduce__(self): raise RuntimeError('cannot pickle') @@ -1573,7 +1573,7 @@ def test_sendbytes(self): if self.TYPE != 'processes': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) msg = latin('abcdefghijklmnopqrstuvwxyz') a, b = self.Pipe() diff --git a/Lib/test/test_nis.py b/Lib/test/test_nis.py --- a/Lib/test/test_nis.py +++ b/Lib/test/test_nis.py @@ -9,11 +9,7 @@ maps = nis.maps() except nis.error, msg: # NIS is probably not active, so this test isn't useful - if test_support.verbose: - print "Test Skipped:", msg - # Can't raise SkipTest as regrtest only recognizes the exception - # import time. - return + self.skipTest(str(msg)) try: # On some systems, this map is only accessible to the # super user diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -134,7 +134,6 @@ self.assertEqual(first.args, second.args) else: self.fail("expected os.tmpfile() to raise OSError") - return else: # open() worked, therefore, tmpfile() should work. Close our # dummy file and proceed with the test as normal. @@ -258,7 +257,7 @@ except OSError, e: # On AtheOS, glibc always returns ENOSYS if e.errno == errno.ENOSYS: - return + self.skipTest('glibc always returns ENOSYS on AtheOS') # Make sure direct access works self.assertEqual(result.f_bfree, result[3]) @@ -338,7 +337,7 @@ os.stat(r"c:\pagefile.sys") except WindowsError, e: if e.errno == 2: # file does not exist; cannot run test - return + self.skipTest(r'c:\pagefile.sys does not exist') self.fail("Could not stat pagefile.sys") from test import mapping_tests diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -461,18 +461,15 @@ os.mkdir(base_path) os.chdir(base_path) except: -# Just returning nothing instead of the SkipTest exception, -# because the test results in Error in that case. -# Is that ok? -# raise unittest.SkipTest, "cannot create directory for testing" - return + self.skipTest("cannot create directory for testing") try: def _create_and_do_getcwd(dirname, current_path_length = 0): try: os.mkdir(dirname) except: - raise unittest.SkipTest, "mkdir cannot create directory sufficiently deep for getcwd test" + self.skipTest("mkdir cannot create directory sufficiently " + "deep for getcwd test") os.chdir(dirname) try: diff --git a/Lib/test/test_pwd.py b/Lib/test/test_pwd.py --- a/Lib/test/test_pwd.py +++ b/Lib/test/test_pwd.py @@ -8,8 +8,6 @@ def test_values(self): entries = pwd.getpwall() - entriesbyname = {} - entriesbyuid = {} for e in entries: self.assertEqual(len(e), 7) @@ -32,13 +30,20 @@ # for one uid # self.assertEqual(pwd.getpwuid(e.pw_uid), e) # instead of this collect all entries for one uid - # and check afterwards + # and check afterwards (done in test_values_extended) + + def test_values_extended(self): + entries = pwd.getpwall() + entriesbyname = {} + entriesbyuid = {} + + if len(entries) > 1000: # Huge passwd file (NIS?) -- skip this test + self.skipTest('passwd file is huge; extended test skipped') + + for e in entries: entriesbyname.setdefault(e.pw_name, []).append(e) entriesbyuid.setdefault(e.pw_uid, []).append(e) - if len(entries) > 1000: # Huge passwd file (NIS?) -- skip the rest - return - # check whether the entry returned by getpwuid() # for each uid is among those from getpwall() for this uid for e in entries: diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -700,7 +700,7 @@ try: unicode except NameError: - return # no problem if we have no unicode + self.skipTest('no problem if we have no unicode') class my_unicode(unicode): pass pat = re.compile(my_unicode("abc")) self.assertEqual(pat.match("xyz"), None) @@ -714,7 +714,7 @@ try: unicode except NameError: - return # no problem if we have no unicode + self.skipTest('no problem if we have no unicode') self.assertTrue(re.compile('bug_926075') is not re.compile(eval("u'bug_926075'"))) @@ -722,7 +722,7 @@ try: unicode except NameError: - pass + self.skipTest('no problem if we have no unicode') pattern = eval('u"[\u002E\u3002\uFF0E\uFF61]"') self.assertEqual(re.compile(pattern).split("a.b.c"), ['a','b','c']) diff --git a/Lib/test/test_repr.py b/Lib/test/test_repr.py --- a/Lib/test/test_repr.py +++ b/Lib/test/test_repr.py @@ -268,6 +268,7 @@ eq(repr(foo.foo), "" % foo.__name__) + @unittest.skip('need a suitable object') def test_object(self): # XXX Test the repr of a type with a really long tp_name but with no # tp_repr. WIBNI we had ::Inline? :) @@ -309,6 +310,7 @@ '= 0, "Error resolving host to ip.") try: hname, aliases, ipaddrs = socket.gethostbyaddr(ip) except socket.error: # Probably a similar problem as above; skip this test - return + self.skipTest('address lookup failure') all_host_names = [hostname, hname] + aliases fqhn = socket.getfqdn(ip) if not fqhn in all_host_names: @@ -491,9 +491,9 @@ try: from socket import inet_pton, AF_INET6, has_ipv6 if not has_ipv6: - return + self.skipTest('IPv6 not available') except ImportError: - return + self.skipTest('could not import needed symbols from socket') f = lambda a: inet_pton(AF_INET6, a) self.assertEqual('\x00' * 16, f('::')) @@ -525,9 +525,9 @@ try: from socket import inet_ntop, AF_INET6, has_ipv6 if not has_ipv6: - return + self.skipTest('IPv6 not available') except ImportError: - return + self.skipTest('could not import needed symbols from socket') f = lambda a: inet_ntop(AF_INET6, a) self.assertEqual('::', f('\x00' * 16)) @@ -567,7 +567,7 @@ my_ip_addr = socket.gethostbyname(socket.gethostname()) except socket.error: # Probably name lookup wasn't set up right; skip this test - return + self.skipTest('name lookup failure') self.assertIn(name[0], ("0.0.0.0", my_ip_addr), '%s invalid' % name[0]) self.assertEqual(name[1], port) @@ -786,10 +786,10 @@ big_chunk = 'f' * 2048 self.serv_conn.sendall(big_chunk) + @unittest.skipUnless(hasattr(socket, 'fromfd'), + 'socket.fromfd not availble') def testFromFd(self): # Testing fromfd() - if not hasattr(socket, "fromfd"): - return # On Windows, this doesn't exist fd = self.cli_conn.fileno() sock = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM) self.addCleanup(sock.close) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -192,9 +192,8 @@ self.assertTrue(s.startswith("OpenSSL {:d}.{:d}.{:d}".format(major, minor, fix)), (s, t)) + @test_support.requires_resource('network') def test_ciphers(self): - if not test_support.is_resource_enabled('network'): - return remote = ("svn.python.org", 443) with test_support.transient_internet(remote[0]): s = ssl.wrap_socket(socket.socket(socket.AF_INET), diff --git a/Lib/test/test_str.py b/Lib/test/test_str.py --- a/Lib/test/test_str.py +++ b/Lib/test/test_str.py @@ -1,4 +1,4 @@ - +import unittest import struct import sys from test import test_support, string_tests @@ -110,12 +110,12 @@ self.assertEqual(str(Foo9("foo")), "string") self.assertEqual(unicode(Foo9("foo")), u"not unicode") + # This test only affects 32-bit platforms because expandtabs can only take + # an int as the max value, not a 64-bit C long. If expandtabs is changed + # to take a 64-bit long, this test should apply to all platforms. + @unittest.skipIf(sys.maxint > (1 << 32) or struct.calcsize('P') != 4, + 'only applies to 32-bit platforms') def test_expandtabs_overflows_gracefully(self): - # This test only affects 32-bit platforms because expandtabs can only take - # an int as the max value, not a 64-bit C long. If expandtabs is changed - # to take a 64-bit long, this test should apply to all platforms. - if sys.maxint > (1 << 32) or struct.calcsize('P') != 4: - return self.assertRaises(OverflowError, 't\tt\t'.expandtabs, sys.maxint) def test__format__(self): diff --git a/Lib/test/test_strptime.py b/Lib/test/test_strptime.py --- a/Lib/test/test_strptime.py +++ b/Lib/test/test_strptime.py @@ -313,7 +313,7 @@ # when time.tzname[0] == time.tzname[1] and time.daylight tz_name = time.tzname[0] if tz_name.upper() in ("UTC", "GMT"): - return + self.skipTest('need non-UTC/GMT timezone') try: original_tzname = time.tzname original_daylight = time.daylight @@ -526,7 +526,7 @@ try: locale.setlocale(locale.LC_TIME, ('en_US', 'UTF8')) except locale.Error: - return + self.skipTest('test needs en_US.UTF8 locale') try: _strptime._strptime_time('10', '%d') # Get id of current cache object. @@ -543,7 +543,7 @@ # If this is the case just suppress the exception and fall-through # to the resetting to the original locale. except locale.Error: - pass + self.skipTest('test needs de_DE.UTF8 locale') # Make sure we don't trample on the locale setting once we leave the # test. finally: diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py --- a/Lib/test/test_tarfile.py +++ b/Lib/test/test_tarfile.py @@ -260,7 +260,7 @@ def test_fail_comp(self): # For Gzip and Bz2 Tests: fail with a ReadError on an uncompressed file. if self.mode == "r:": - return + self.skipTest('needs a gz or bz2 mode') self.assertRaises(tarfile.ReadError, tarfile.open, tarname, self.mode) fobj = open(tarname, "rb") self.assertRaises(tarfile.ReadError, tarfile.open, fileobj=fobj, mode=self.mode) @@ -446,14 +446,12 @@ def test_detect_fileobj(self): self._test_modes(self._testfunc_fileobj) + @unittest.skipUnless(bz2, 'requires bz2') def test_detect_stream_bz2(self): # Originally, tarfile's stream detection looked for the string # "BZh91" at the start of the file. This is incorrect because # the '9' represents the blocksize (900kB). If the file was # compressed using another blocksize autodetection fails. - if not bz2: - return - with open(tarname, "rb") as fobj: data = fobj.read() @@ -982,12 +980,11 @@ self.assertTrue(data.count("\0") == tarfile.RECORDSIZE, "incorrect zero padding") + @unittest.skipIf(sys.platform == 'win32', 'not appropriate for Windows') + @unittest.skipUnless(hasattr(os, 'umask'), 'requires os.umask') def test_file_mode(self): # Test for issue #8464: Create files with correct # permissions. - if sys.platform == "win32" or not hasattr(os, "umask"): - return - if os.path.exists(tmpname): os.remove(tmpname) @@ -1360,15 +1357,13 @@ self._add_testfile() self._test(names=["foo", "bar"]) + @unittest.skipUnless(gzip, 'requires gzip') def test_append_gz(self): - if gzip is None: - return self._create_testtar("w:gz") self.assertRaises(tarfile.ReadError, tarfile.open, tmpname, "a") + @unittest.skipUnless(bz2, 'requires bz2') def test_append_bz2(self): - if bz2 is None: - return self._create_testtar("w:bz2") self.assertRaises(tarfile.ReadError, tarfile.open, tmpname, "a") diff --git a/Lib/test/test_telnetlib.py b/Lib/test/test_telnetlib.py --- a/Lib/test/test_telnetlib.py +++ b/Lib/test/test_telnetlib.py @@ -177,7 +177,6 @@ self.dataq.join() data = telnet.read_all() self.assertEqual(data, ''.join(want[:-1])) - return def _test_blocking(self, func): self.dataq.put([self.block_long, EOF_sigil]) diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -342,10 +342,9 @@ finally: os.rmdir(dir) + @unittest.skipUnless(has_stat, 'os.stat not available') def test_file_mode(self): # _mkstemp_inner creates files with the proper mode - if not has_stat: - return # ugh, can't use SkipTest. file = self.do_create() mode = stat.S_IMODE(os.stat(file.name).st_mode) @@ -357,10 +356,9 @@ expected = user * (1 + 8 + 64) self.assertEqual(mode, expected) + @unittest.skipUnless(has_spawnl, 'os.spawnl not available') def test_noinherit(self): # _mkstemp_inner file handles are not inherited by child processes - if not has_spawnl: - return # ugh, can't use SkipTest. if support.verbose: v="v" @@ -395,10 +393,9 @@ "child process caught fatal signal %d" % -retval) self.assertFalse(retval > 0, "child process reports failure %d"%retval) + @unittest.skipUnless(has_textmode, "text mode not available") def test_textmode(self): # _mkstemp_inner can create files in text mode - if not has_textmode: - return # ugh, can't use SkipTest. self.do_create(bin=0).write("blat\n") # XXX should test that the file really is a text file @@ -590,10 +587,9 @@ finally: os.rmdir(dir) + @unittest.skipUnless(has_stat, 'os.stat not available') def test_mode(self): # mkdtemp creates directories with the proper mode - if not has_stat: - return # ugh, can't use SkipTest. dir = self.do_create() try: diff --git a/Lib/test/test_thread.py b/Lib/test/test_thread.py --- a/Lib/test/test_thread.py +++ b/Lib/test/test_thread.py @@ -70,39 +70,35 @@ thread.stack_size(0) self.assertEqual(thread.stack_size(), 0, "stack_size not reset to default") - if os.name not in ("nt", "os2", "posix"): - return - - tss_supported = True + @unittest.skipIf(os.name not in ("nt", "os2", "posix"), 'test meant for nt, os2, and posix') + def test_nt_and_posix_stack_size(self): try: thread.stack_size(4096) except ValueError: verbose_print("caught expected ValueError setting " "stack_size(4096)") except thread.error: - tss_supported = False - verbose_print("platform does not support changing thread stack " - "size") + self.skipTest("platform does not support changing thread stack " + "size") - if tss_supported: - fail_msg = "stack_size(%d) failed - should succeed" - for tss in (262144, 0x100000, 0): - thread.stack_size(tss) - self.assertEqual(thread.stack_size(), tss, fail_msg % tss) - verbose_print("successfully set stack_size(%d)" % tss) + fail_msg = "stack_size(%d) failed - should succeed" + for tss in (262144, 0x100000, 0): + thread.stack_size(tss) + self.assertEqual(thread.stack_size(), tss, fail_msg % tss) + verbose_print("successfully set stack_size(%d)" % tss) - for tss in (262144, 0x100000): - verbose_print("trying stack_size = (%d)" % tss) - self.next_ident = 0 - self.created = 0 - for i in range(NUMTASKS): - self.newtask() + for tss in (262144, 0x100000): + verbose_print("trying stack_size = (%d)" % tss) + self.next_ident = 0 + self.created = 0 + for i in range(NUMTASKS): + self.newtask() - verbose_print("waiting for all tasks to complete") - self.done_mutex.acquire() - verbose_print("all tasks done") + verbose_print("waiting for all tasks to complete") + self.done_mutex.acquire() + verbose_print("all tasks done") - thread.stack_size(0) + thread.stack_size(0) def test__count(self): # Test the _count() function. diff --git a/Lib/test/test_threading.py b/Lib/test/test_threading.py --- a/Lib/test/test_threading.py +++ b/Lib/test/test_threading.py @@ -125,9 +125,7 @@ try: threading.stack_size(262144) except thread.error: - if verbose: - print 'platform does not support changing thread stack size' - return + self.skipTest('platform does not support changing thread stack size') self.test_various_ops() threading.stack_size(0) @@ -138,9 +136,7 @@ try: threading.stack_size(0x100000) except thread.error: - if verbose: - print 'platform does not support changing thread stack size' - return + self.skipTest('platform does not support changing thread stack size') self.test_various_ops() threading.stack_size(0) @@ -167,9 +163,7 @@ try: import ctypes except ImportError: - if verbose: - print "test_PyThreadState_SetAsyncExc can't import ctypes" - return # can't do anything + self.skipTest('requires ctypes') set_async_exc = ctypes.pythonapi.PyThreadState_SetAsyncExc @@ -275,9 +269,7 @@ try: import ctypes except ImportError: - if verbose: - print("test_finalize_with_runnning_thread can't import ctypes") - return # can't do anything + self.skipTest('requires ctypes') rc = subprocess.call([sys.executable, "-c", """if 1: import ctypes, sys, time, thread diff --git a/Lib/test/test_timeout.py b/Lib/test/test_timeout.py --- a/Lib/test/test_timeout.py +++ b/Lib/test/test_timeout.py @@ -178,16 +178,19 @@ "timeout (%g) is %g seconds more than expected (%g)" %(_delta, self.fuzz, _timeout)) + @unittest.skip('test not implemented') def testSend(self): # Test send() timeout # couldn't figure out how to test it pass + @unittest.skip('test not implemented') def testSendto(self): # Test sendto() timeout # couldn't figure out how to test it pass + @unittest.skip('test not implemented') def testSendall(self): # Test sendall() timeout # couldn't figure out how to test it diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -1278,12 +1278,12 @@ self.assertEqual(repr(s1()), '\\n') self.assertEqual(repr(s2()), '\\n') + # This test only affects 32-bit platforms because expandtabs can only take + # an int as the max value, not a 64-bit C long. If expandtabs is changed + # to take a 64-bit long, this test should apply to all platforms. + @unittest.skipIf(sys.maxint > (1 << 32) or struct.calcsize('P') != 4, + 'only applies to 32-bit platforms') def test_expandtabs_overflows_gracefully(self): - # This test only affects 32-bit platforms because expandtabs can only take - # an int as the max value, not a 64-bit C long. If expandtabs is changed - # to take a 64-bit long, this test should apply to all platforms. - if sys.maxint > (1 << 32) or struct.calcsize('P') != 4: - return self.assertRaises(OverflowError, u't\tt\t'.expandtabs, sys.maxint) def test__format__(self): diff --git a/Lib/test/test_urllibnet.py b/Lib/test/test_urllibnet.py --- a/Lib/test/test_urllibnet.py +++ b/Lib/test/test_urllibnet.py @@ -112,12 +112,9 @@ open_url.close() self.assertEqual(code, 404) + @unittest.skipIf(sys.platform in ('win32',), 'not appropriate for Windows') + @unittest.skipUnless(hasattr(os, 'fdopen'), 'os.fdopen not available') def test_fileno(self): - if (sys.platform in ('win32',) or - not hasattr(os, 'fdopen')): - # On Windows, socket handles are not file descriptors; this - # test can't pass on Windows. - return # Make sure fd returned by fileno is valid. open_url = self.urlopen("http://www.python.org/") fd = open_url.fileno() diff --git a/Lib/test/test_warnings.py b/Lib/test/test_warnings.py --- a/Lib/test/test_warnings.py +++ b/Lib/test/test_warnings.py @@ -668,7 +668,7 @@ # Explicit tests for the test_support convenience wrapper wmod = self.module if wmod is not sys.modules['warnings']: - return + self.skipTest('module to test is not loaded warnings module') with test_support.check_warnings(quiet=False) as w: self.assertEqual(w.warnings, []) wmod.simplefilter("always") diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py --- a/Lib/test/test_xmlrpc.py +++ b/Lib/test/test_xmlrpc.py @@ -19,6 +19,11 @@ threading = None try: + import gzip +except ImportError: + gzip = None + +try: unicode except NameError: have_unicode = False @@ -681,6 +686,7 @@ #A test case that verifies that gzip encoding works in both directions #(for a request and the response) + at unittest.skipUnless(gzip, 'gzip not available') class GzipServerTestCase(BaseServerTestCase): #a request handler that supports keep-alive and logs requests into a #class variable @@ -1011,11 +1017,7 @@ xmlrpc_tests.append(SimpleServerTestCase) xmlrpc_tests.append(KeepaliveServerTestCase1) xmlrpc_tests.append(KeepaliveServerTestCase2) - try: - import gzip - xmlrpc_tests.append(GzipServerTestCase) - except ImportError: - pass #gzip not supported in this build + xmlrpc_tests.append(GzipServerTestCase) xmlrpc_tests.append(MultiPathServerTestCase) xmlrpc_tests.append(ServerProxyTestCase) xmlrpc_tests.append(FailingServerTestCase) diff --git a/Lib/test/test_zipimport.py b/Lib/test/test_zipimport.py --- a/Lib/test/test_zipimport.py +++ b/Lib/test/test_zipimport.py @@ -123,7 +123,7 @@ # so we'll simply skip it then. Bug #765456. # if "zlib" in sys.builtin_module_names: - return + self.skipTest('zlib is a builtin module') if "zlib" in sys.modules: del sys.modules["zlib"] files = {"zlib.py": (NOW, test_src)} diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -91,7 +91,7 @@ - Issue #17883: Backported _is_gui_available() in test.test_support to avoid hanging Windows buildbots on test_ttk_guionly. -- Issue #18702: All skipped tests now reported as skipped. +- Issue #18702, #19572: All skipped tests now reported as skipped. - Issue #19085: Added basic tests for all tkinter widget options. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 21:18:56 2013 From: python-checkins at python.org (zach.ware) Date: Tue, 10 Dec 2013 21:18:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5OTI4?= =?utf-8?q?=3A_Fix_test_on_Windows?= Message-ID: <3dfCKD1lWBz7Lk7@mail.python.org> http://hg.python.org/cpython/rev/21a9abf6d428 changeset: 87888:21a9abf6d428 branch: 2.7 user: Zachary Ware date: Tue Dec 10 14:14:28 2013 -0600 summary: Issue #19928: Fix test on Windows files: Lib/test/test_repr.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_repr.py b/Lib/test/test_repr.py --- a/Lib/test/test_repr.py +++ b/Lib/test/test_repr.py @@ -185,8 +185,8 @@ return x return inner x = get_cell().__closure__[0] - self.assertRegexpMatches(repr(x), r'') + self.assertRegexpMatches(repr(x), r'') self.assertRegexpMatches(r(x), r'') def test_descriptors(self): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 21:18:57 2013 From: python-checkins at python.org (zach.ware) Date: Tue, 10 Dec 2013 21:18:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5OTI4?= =?utf-8?q?=3A_Fix_test_on_Windows?= Message-ID: <3dfCKF4f1vz7LnT@mail.python.org> http://hg.python.org/cpython/rev/a0f9f0778ce3 changeset: 87889:a0f9f0778ce3 branch: 3.3 parent: 87884:9f38bbd4e041 user: Zachary Ware date: Tue Dec 10 14:17:22 2013 -0600 summary: Issue #19928: Fix test on Windows files: Lib/test/test_reprlib.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_reprlib.py b/Lib/test/test_reprlib.py --- a/Lib/test/test_reprlib.py +++ b/Lib/test/test_reprlib.py @@ -173,8 +173,8 @@ return x return inner x = get_cell().__closure__[0] - self.assertRegex(repr(x), - r'') + self.assertRegex(repr(x), r'') self.assertRegex(r(x), r'') def test_descriptors(self): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 21:18:59 2013 From: python-checkins at python.org (zach.ware) Date: Tue, 10 Dec 2013 21:18:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319928=3A_Fix_test_on_Windows?= Message-ID: <3dfCKH0MBvz7Lp1@mail.python.org> http://hg.python.org/cpython/rev/eb0622dc8376 changeset: 87890:eb0622dc8376 parent: 87885:a3bdbe220f8a parent: 87889:a0f9f0778ce3 user: Zachary Ware date: Tue Dec 10 14:18:30 2013 -0600 summary: Issue #19928: Fix test on Windows files: Lib/test/test_reprlib.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_reprlib.py b/Lib/test/test_reprlib.py --- a/Lib/test/test_reprlib.py +++ b/Lib/test/test_reprlib.py @@ -173,8 +173,8 @@ return x return inner x = get_cell().__closure__[0] - self.assertRegex(repr(x), - r'') + self.assertRegex(repr(x), r'') self.assertRegex(r(x), r'') def test_descriptors(self): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 10 23:07:00 2013 From: python-checkins at python.org (zach.ware) Date: Tue, 10 Dec 2013 23:07:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5NTcy?= =?utf-8?q?=3A_Replace_a_return_that_shouldn=27t_have_been_removed_from_te?= =?utf-8?b?c3Rfb3Mu?= Message-ID: <3dfFjw3BHhz7Ljg@mail.python.org> http://hg.python.org/cpython/rev/ca9bca7aecda changeset: 87891:ca9bca7aecda branch: 2.7 parent: 87888:21a9abf6d428 user: Zachary Ware date: Tue Dec 10 16:06:46 2013 -0600 summary: Issue #19572: Replace a return that shouldn't have been removed from test_os. This should fix the test_os failure on the AMD64 Windows 7 buildbot. files: Lib/test/test_os.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -134,6 +134,7 @@ self.assertEqual(first.args, second.args) else: self.fail("expected os.tmpfile() to raise OSError") + return else: # open() worked, therefore, tmpfile() should work. Close our # dummy file and proceed with the test as normal. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 11 01:33:46 2013 From: python-checkins at python.org (ned.deily) Date: Wed, 11 Dec 2013 01:33:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE4Mjcw?= =?utf-8?q?=3A_Prevent_possible_IDLE_AttributeError_on_OS_X_when_no_initia?= =?utf-8?q?l?= Message-ID: <3dfJzG20hBz7LkV@mail.python.org> http://hg.python.org/cpython/rev/5becf8b612ee changeset: 87892:5becf8b612ee branch: 2.7 user: Ned Deily date: Tue Dec 10 16:21:58 2013 -0800 summary: Issue #18270: Prevent possible IDLE AttributeError on OS X when no initial shell window is present. (Original patch by Terry Reedy) files: Lib/idlelib/PyShell.py | 25 ++++++++++++++----------- Misc/NEWS | 3 +++ 2 files changed, 17 insertions(+), 11 deletions(-) diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py old mode 100644 new mode 100755 --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -1546,20 +1546,22 @@ args.remove(filename) if not args: flist.new() + if enable_shell: shell = flist.open_shell() if not shell: return # couldn't open shell - if macosxSupport.runningAsOSXApp() and flist.dict: # On OSX: when the user has double-clicked on a file that causes # IDLE to be launched the shell window will open just in front of # the file she wants to see. Lower the interpreter window when # there are open files. shell.top.lower() + else: + shell = flist.pyshell - shell = flist.pyshell - # handle remaining options: + # Handle remaining options. If any of these are set, enable_shell + # was set also, so shell must be true to reach here. if debug: shell.open_debugger() if startup: @@ -1567,7 +1569,7 @@ os.environ.get("PYTHONSTARTUP") if filename and os.path.isfile(filename): shell.interp.execfile(filename) - if shell and cmd or script: + if cmd or script: shell.interp.runcommand("""if 1: import sys as _sys _sys.argv = %r @@ -1578,13 +1580,14 @@ elif script: shell.interp.prepend_syspath(script) shell.interp.execfile(script) - - # Check for problematic OS X Tk versions and print a warning message - # in the IDLE shell window; this is less intrusive than always opening - # a separate window. - tkversionwarning = macosxSupport.tkVersionWarning(root) - if tkversionwarning: - shell.interp.runcommand(''.join(("print('", tkversionwarning, "')"))) + elif shell: + # If there is a shell window and no cmd or script in progress, + # check for problematic OS X Tk versions and print a warning + # message in the IDLE shell window; this is less intrusive + # than always opening a separate window. + tkversionwarning = macosxSupport.tkVersionWarning(root) + if tkversionwarning: + shell.interp.runcommand("print('%s')" % tkversionwarning) while flist.inversedict: # keep IDLE running while files are open. root.mainloop() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -74,6 +74,9 @@ - Issue #19481: print() of unicode, str or bytearray subclass instance in IDLE no more hangs. +- Issue #18270: Prevent possible IDLE AttributeError on OS X when no initial + shell window is present. + Tests ----- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 11 01:33:47 2013 From: python-checkins at python.org (ned.deily) Date: Wed, 11 Dec 2013 01:33:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE4Mjcw?= =?utf-8?q?=3A_Prevent_possible_IDLE_AttributeError_on_OS_X_when_no_initia?= =?utf-8?q?l?= Message-ID: <3dfJzH3yyFz7LkX@mail.python.org> http://hg.python.org/cpython/rev/016c64e66a9e changeset: 87893:016c64e66a9e branch: 3.3 parent: 87889:a0f9f0778ce3 user: Ned Deily date: Tue Dec 10 16:24:01 2013 -0800 summary: Issue #18270: Prevent possible IDLE AttributeError on OS X when no initial shell window is present. (Original patch by Terry Reedy) files: Lib/idlelib/PyShell.py | 25 ++++++++++++++----------- Misc/NEWS | 5 ++++- 2 files changed, 18 insertions(+), 12 deletions(-) diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py old mode 100644 new mode 100755 --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -1534,20 +1534,22 @@ args.remove(filename) if not args: flist.new() + if enable_shell: shell = flist.open_shell() if not shell: return # couldn't open shell - if macosxSupport.runningAsOSXApp() and flist.dict: # On OSX: when the user has double-clicked on a file that causes # IDLE to be launched the shell window will open just in front of # the file she wants to see. Lower the interpreter window when # there are open files. shell.top.lower() + else: + shell = flist.pyshell - shell = flist.pyshell - # handle remaining options: + # Handle remaining options. If any of these are set, enable_shell + # was set also, so shell must be true to reach here. if debug: shell.open_debugger() if startup: @@ -1555,7 +1557,7 @@ os.environ.get("PYTHONSTARTUP") if filename and os.path.isfile(filename): shell.interp.execfile(filename) - if shell and cmd or script: + if cmd or script: shell.interp.runcommand("""if 1: import sys as _sys _sys.argv = %r @@ -1566,13 +1568,14 @@ elif script: shell.interp.prepend_syspath(script) shell.interp.execfile(script) - - # Check for problematic OS X Tk versions and print a warning message - # in the IDLE shell window; this is less intrusive than always opening - # a separate window. - tkversionwarning = macosxSupport.tkVersionWarning(root) - if tkversionwarning: - shell.interp.runcommand(''.join(("print('", tkversionwarning, "')"))) + elif shell: + # If there is a shell window and no cmd or script in progress, + # check for problematic OS X Tk versions and print a warning + # message in the IDLE shell window; this is less intrusive + # than always opening a separate window. + tkversionwarning = macosxSupport.tkVersionWarning(root) + if tkversionwarning: + shell.interp.runcommand("print('%s')" % tkversionwarning) while flist.inversedict: # keep IDLE running while files are open. root.mainloop() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -109,7 +109,10 @@ IDLE ---- -- Issue #19481: print() of string subclass instance in IDLE no more hangs. +- Issue #19481: print() of string subclass instance in IDLE no longer hangs. + +- Issue #18270: Prevent possible IDLE AttributeError on OS X when no initial + shell window is present. Tests ----- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 11 01:33:48 2013 From: python-checkins at python.org (ned.deily) Date: Wed, 11 Dec 2013 01:33:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2318270=3A_merge_from_3=2E3?= Message-ID: <3dfJzJ5rP8z7Llc@mail.python.org> http://hg.python.org/cpython/rev/9d1fb265b88a changeset: 87894:9d1fb265b88a parent: 87890:eb0622dc8376 parent: 87893:016c64e66a9e user: Ned Deily date: Tue Dec 10 16:32:57 2013 -0800 summary: Issue #18270: merge from 3.3 files: Lib/idlelib/PyShell.py | 25 ++++++++++++++----------- Misc/NEWS | 5 ++++- 2 files changed, 18 insertions(+), 12 deletions(-) diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py old mode 100644 new mode 100755 --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -1540,20 +1540,22 @@ args.remove(filename) if not args: flist.new() + if enable_shell: shell = flist.open_shell() if not shell: return # couldn't open shell - if macosxSupport.runningAsOSXApp() and flist.dict: # On OSX: when the user has double-clicked on a file that causes # IDLE to be launched the shell window will open just in front of # the file she wants to see. Lower the interpreter window when # there are open files. shell.top.lower() + else: + shell = flist.pyshell - shell = flist.pyshell - # handle remaining options: + # Handle remaining options. If any of these are set, enable_shell + # was set also, so shell must be true to reach here. if debug: shell.open_debugger() if startup: @@ -1561,7 +1563,7 @@ os.environ.get("PYTHONSTARTUP") if filename and os.path.isfile(filename): shell.interp.execfile(filename) - if shell and cmd or script: + if cmd or script: shell.interp.runcommand("""if 1: import sys as _sys _sys.argv = %r @@ -1572,13 +1574,14 @@ elif script: shell.interp.prepend_syspath(script) shell.interp.execfile(script) - - # Check for problematic OS X Tk versions and print a warning message - # in the IDLE shell window; this is less intrusive than always opening - # a separate window. - tkversionwarning = macosxSupport.tkVersionWarning(root) - if tkversionwarning: - shell.interp.runcommand(''.join(("print('", tkversionwarning, "')"))) + elif shell: + # If there is a shell window and no cmd or script in progress, + # check for problematic OS X Tk versions and print a warning + # message in the IDLE shell window; this is less intrusive + # than always opening a separate window. + tkversionwarning = macosxSupport.tkVersionWarning(root) + if tkversionwarning: + shell.interp.runcommand("print('%s')" % tkversionwarning) while flist.inversedict: # keep IDLE running while files are open. root.mainloop() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -131,7 +131,10 @@ IDLE ---- -- Issue #19481: print() of string subclass instance in IDLE no more hangs. +- Issue #19481: print() of string subclass instance in IDLE no longer hangs. + +- Issue #18270: Prevent possible IDLE AttributeError on OS X when no initial + shell window is present. Tests ----- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 11 03:26:27 2013 From: python-checkins at python.org (gregory.p.smith) Date: Wed, 11 Dec 2013 03:26:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Fixes_Issue_?= =?utf-8?q?=2317200=3A_telnetlib=27s_read=5Funtil_and_expect_timeout_was_b?= =?utf-8?q?roken_by_the?= Message-ID: <3dfMTH2d0jz7LlN@mail.python.org> http://hg.python.org/cpython/rev/d61e8050b7d7 changeset: 87895:d61e8050b7d7 branch: 2.7 parent: 87892:5becf8b612ee user: Gregory P. Smith date: Tue Dec 10 18:22:03 2013 -0800 summary: Fixes Issue #17200: telnetlib's read_until and expect timeout was broken by the fix to Issue #14635 in Python 2.7.4 to be interpreted as milliseconds instead of seconds when the platform supports select.poll (ie: everywhere). It is now treated as seconds once again. files: Lib/telnetlib.py | 7 +++++-- Misc/NEWS | 5 +++++ 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/Lib/telnetlib.py b/Lib/telnetlib.py --- a/Lib/telnetlib.py +++ b/Lib/telnetlib.py @@ -312,7 +312,9 @@ poller.register(self, poll_in_or_priority_flags) while i < 0 and not self.eof: try: - ready = poller.poll(call_timeout) + # Poll takes its timeout in milliseconds. + ready = poller.poll(None if timeout is None + else 1000 * call_timeout) except select.error as e: if e.errno == errno.EINTR: if timeout is not None: @@ -682,7 +684,8 @@ poller.register(self, poll_in_or_priority_flags) while not m and not self.eof: try: - ready = poller.poll(call_timeout) + ready = poller.poll(None if timeout is None + else 1000 * call_timeout) except select.error as e: if e.errno == errno.EINTR: if timeout is not None: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -17,6 +17,11 @@ Library ------- +- Issue #17200: telnetlib's read_until and expect timeout was broken by the + fix to Issue #14635 in Python 2.7.4 to be interpreted as milliseconds + instead of seconds when the platform supports select.poll (ie: everywhere). + It is now treated as seconds once again. + - Issue #19099: The struct module now supports Unicode format strings. - Issue #19878: Fix segfault in bz2 module after calling __init__ twice with -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 11 03:26:28 2013 From: python-checkins at python.org (gregory.p.smith) Date: Wed, 11 Dec 2013 03:26:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Fixes_Issue_?= =?utf-8?q?=2317200=3A_telnetlib=27s_read=5Funtil_and_expect_timeout_was_b?= =?utf-8?q?roken_by_the?= Message-ID: <3dfMTJ4W70z7Llc@mail.python.org> http://hg.python.org/cpython/rev/46186736e91c changeset: 87896:46186736e91c branch: 3.3 parent: 87893:016c64e66a9e user: Gregory P. Smith date: Tue Dec 10 18:25:21 2013 -0800 summary: Fixes Issue #17200: telnetlib's read_until and expect timeout was broken by the fix to Issue #14635 in Python 3.3.0 to be interpreted as milliseconds instead of seconds when the platform supports select.poll (ie: everywhere). It is now treated as seconds once again. files: Lib/telnetlib.py | 6 ++++-- Misc/NEWS | 5 +++++ 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/Lib/telnetlib.py b/Lib/telnetlib.py --- a/Lib/telnetlib.py +++ b/Lib/telnetlib.py @@ -315,7 +315,8 @@ poller.register(self, poll_in_or_priority_flags) while i < 0 and not self.eof: try: - ready = poller.poll(call_timeout) + ready = poller.poll(None if timeout is None + else 1000 * call_timeout) except select.error as e: if e.errno == errno.EINTR: if timeout is not None: @@ -683,7 +684,8 @@ poller.register(self, poll_in_or_priority_flags) while not m and not self.eof: try: - ready = poller.poll(call_timeout) + ready = poller.poll(None if timeout is None + else 1000 * call_timeout) except select.error as e: if e.errno == errno.EINTR: if timeout is not None: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -20,6 +20,11 @@ Library ------- +- Issue #17200: telnetlib's read_until and expect timeout was broken by the + fix to Issue #14635 in Python 3.3.0 to be interpreted as milliseconds + instead of seconds when the platform supports select.poll (ie: everywhere). + It is now treated as seconds once again. + - Issue #17429: platform.linux_distribution() now decodes files from the UTF-8 encoding with the surrogateescape error handler, instead of decoding from the locale encoding in strict mode. It fixes the function on Fedora 19 which is -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 11 03:26:29 2013 From: python-checkins at python.org (gregory.p.smith) Date: Wed, 11 Dec 2013 03:26:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_null_merge_=283=2E4_doesn=27t_need_this_fix=29?= Message-ID: <3dfMTK6HXyz7Lm2@mail.python.org> http://hg.python.org/cpython/rev/34dd1ae8c00e changeset: 87897:34dd1ae8c00e parent: 87894:9d1fb265b88a parent: 87896:46186736e91c user: Gregory P. Smith date: Tue Dec 10 18:26:02 2013 -0800 summary: null merge (3.4 doesn't need this fix) files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 11 06:17:21 2013 From: python-checkins at python.org (eric.snow) Date: Wed, 11 Dec 2013 06:17:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2318864=3A_Add_a_se?= =?utf-8?q?tter_for_ModuleSpec=2Ehas=5Flocation=2E?= Message-ID: <3dfRGT0K4Nz7Llb@mail.python.org> http://hg.python.org/cpython/rev/e961a166dc70 changeset: 87898:e961a166dc70 user: Eric Snow date: Tue Dec 10 22:16:41 2013 -0700 summary: Issue #18864: Add a setter for ModuleSpec.has_location. files: Doc/library/importlib.rst | 2 +- Lib/importlib/_bootstrap.py | 4 + Lib/test/test_importlib/test_spec.py | 7 + Misc/NEWS | 2 + Python/importlib.h | 5849 +++++++------ 5 files changed, 2943 insertions(+), 2921 deletions(-) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -901,7 +901,7 @@ .. attribute:: has_location - (Read-only) Boolean indicating whether or not the module's "origin" + Boolean indicating whether or not the module's "origin" attribute refers to a loadable location. :mod:`importlib.util` -- Utility code for importers diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -841,6 +841,10 @@ def has_location(self): return self._set_fileattr + @has_location.setter + def has_location(self, value): + self._set_fileattr = bool(value) + def spec_from_loader(name, loader, *, origin=None, is_package=None): """Return a module spec based on various loader methods.""" diff --git a/Lib/test/test_importlib/test_spec.py b/Lib/test/test_importlib/test_spec.py --- a/Lib/test/test_importlib/test_spec.py +++ b/Lib/test/test_importlib/test_spec.py @@ -116,6 +116,13 @@ self.assertIs(spec.cached, None) self.assertFalse(spec.has_location) + def test_has_location_setter(self): + spec = self.machinery.ModuleSpec(self.name, self.loader, + origin='somewhere') + self.assertFalse(spec.has_location) + spec.has_location = True + self.assertTrue(spec.has_location) + def test_equality(self): other = type(sys.implementation)(name=self.name, loader=self.loader, diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -105,6 +105,8 @@ - Issue #19698: Removed exec_module() methods from importlib.machinery.BuiltinImporter and ExtensionFileLoader. +- Issue #18864: Added a setter for ModuleSpec.has_location. + - Fixed _pickle.Unpickler to not fail when loading empty strings as persistent IDs. diff --git a/Python/importlib.h b/Python/importlib.h --- a/Python/importlib.h +++ b/Python/importlib.h [stripped] -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Wed Dec 11 09:41:43 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 11 Dec 2013 09:41:43 +0100 Subject: [Python-checkins] Daily reference leaks (34dd1ae8c00e): sum=0 Message-ID: results for 34dd1ae8c00e on branch "default" -------------------------------------------- test_site leaked [2, -2, 0] references, sum=0 test_site leaked [2, -2, 0] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogKYtyJ8', '-x'] From python-checkins at python.org Wed Dec 11 20:28:19 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 11 Dec 2013 20:28:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE3NTc2?= =?utf-8?q?=3A_Deprecation_warning_emitted_now_when_=5F=5Fint=5F=5F=28=29_?= =?utf-8?b?b3IgX19pbmRleF9fKCk=?= Message-ID: <3dfp8M5ZFMz7Lmp@mail.python.org> http://hg.python.org/cpython/rev/618cca51a27e changeset: 87899:618cca51a27e branch: 3.3 parent: 87896:46186736e91c user: Serhiy Storchaka date: Wed Dec 11 21:07:54 2013 +0200 summary: Issue #17576: Deprecation warning emitted now when __int__() or __index__() return not int instance. Introduced _PyLong_FromNbInt() and refactored PyLong_As*() functions. files: Include/longobject.h | 6 + Lib/test/test_getargs2.py | 70 +++++++ Lib/test/test_index.py | 36 +++- Lib/test/test_int.py | 76 +++++-- Misc/NEWS | 3 + Objects/abstract.c | 92 ++++------ Objects/longobject.c | 229 +++++++++++++------------ 7 files changed, 319 insertions(+), 193 deletions(-) diff --git a/Include/longobject.h b/Include/longobject.h --- a/Include/longobject.h +++ b/Include/longobject.h @@ -152,6 +152,12 @@ unsigned char* bytes, size_t n, int little_endian, int is_signed); +/* _PyLong_FromNbInt: Convert the given object to a PyLongObject + using the nb_int slot, if available. Raise TypeError if either the + nb_int slot is not available or the result of the call to nb_int + returns something not of type int. +*/ +PyAPI_FUNC(PyLongObject *)_PyLong_FromNbInt(PyObject *); /* _PyLong_Format: Convert the long to a string object with given base, appending a base prefix of 0[box] if base is 2, 8 or 16. */ diff --git a/Lib/test/test_getargs2.py b/Lib/test/test_getargs2.py --- a/Lib/test/test_getargs2.py +++ b/Lib/test/test_getargs2.py @@ -50,12 +50,34 @@ def __int__(self): return 99 +class IntSubclass(int): + def __int__(self): + return 99 + +class BadInt: + def __int__(self): + return 1.0 + +class BadInt2: + def __int__(self): + return True + +class BadInt3(int): + def __int__(self): + return True + + class Unsigned_TestCase(unittest.TestCase): def test_b(self): from _testcapi import getargs_b # b returns 'unsigned char', and does range checking (0 ... UCHAR_MAX) self.assertRaises(TypeError, getargs_b, 3.14) self.assertEqual(99, getargs_b(Int())) + self.assertEqual(0, getargs_b(IntSubclass())) + self.assertRaises(TypeError, getargs_b, BadInt()) + with self.assertWarns(DeprecationWarning): + self.assertEqual(1, getargs_b(BadInt2())) + self.assertEqual(0, getargs_b(BadInt3())) self.assertRaises(OverflowError, getargs_b, -1) self.assertEqual(0, getargs_b(0)) @@ -70,6 +92,11 @@ # B returns 'unsigned char', no range checking self.assertRaises(TypeError, getargs_B, 3.14) self.assertEqual(99, getargs_B(Int())) + self.assertEqual(0, getargs_B(IntSubclass())) + self.assertRaises(TypeError, getargs_B, BadInt()) + with self.assertWarns(DeprecationWarning): + self.assertEqual(1, getargs_B(BadInt2())) + self.assertEqual(0, getargs_B(BadInt3())) self.assertEqual(UCHAR_MAX, getargs_B(-1)) self.assertEqual(0, getargs_B(0)) @@ -84,6 +111,11 @@ # H returns 'unsigned short', no range checking self.assertRaises(TypeError, getargs_H, 3.14) self.assertEqual(99, getargs_H(Int())) + self.assertEqual(0, getargs_H(IntSubclass())) + self.assertRaises(TypeError, getargs_H, BadInt()) + with self.assertWarns(DeprecationWarning): + self.assertEqual(1, getargs_H(BadInt2())) + self.assertEqual(0, getargs_H(BadInt3())) self.assertEqual(USHRT_MAX, getargs_H(-1)) self.assertEqual(0, getargs_H(0)) @@ -99,6 +131,11 @@ # I returns 'unsigned int', no range checking self.assertRaises(TypeError, getargs_I, 3.14) self.assertEqual(99, getargs_I(Int())) + self.assertEqual(0, getargs_I(IntSubclass())) + self.assertRaises(TypeError, getargs_I, BadInt()) + with self.assertWarns(DeprecationWarning): + self.assertEqual(1, getargs_I(BadInt2())) + self.assertEqual(0, getargs_I(BadInt3())) self.assertEqual(UINT_MAX, getargs_I(-1)) self.assertEqual(0, getargs_I(0)) @@ -115,6 +152,10 @@ # it does not accept float, or instances with __int__ self.assertRaises(TypeError, getargs_k, 3.14) self.assertRaises(TypeError, getargs_k, Int()) + self.assertEqual(0, getargs_k(IntSubclass())) + self.assertRaises(TypeError, getargs_k, BadInt()) + self.assertRaises(TypeError, getargs_k, BadInt2()) + self.assertEqual(0, getargs_k(BadInt3())) self.assertEqual(ULONG_MAX, getargs_k(-1)) self.assertEqual(0, getargs_k(0)) @@ -131,6 +172,11 @@ # h returns 'short', and does range checking (SHRT_MIN ... SHRT_MAX) self.assertRaises(TypeError, getargs_h, 3.14) self.assertEqual(99, getargs_h(Int())) + self.assertEqual(0, getargs_h(IntSubclass())) + self.assertRaises(TypeError, getargs_h, BadInt()) + with self.assertWarns(DeprecationWarning): + self.assertEqual(1, getargs_h(BadInt2())) + self.assertEqual(0, getargs_h(BadInt3())) self.assertRaises(OverflowError, getargs_h, SHRT_MIN-1) self.assertEqual(SHRT_MIN, getargs_h(SHRT_MIN)) @@ -145,6 +191,11 @@ # i returns 'int', and does range checking (INT_MIN ... INT_MAX) self.assertRaises(TypeError, getargs_i, 3.14) self.assertEqual(99, getargs_i(Int())) + self.assertEqual(0, getargs_i(IntSubclass())) + self.assertRaises(TypeError, getargs_i, BadInt()) + with self.assertWarns(DeprecationWarning): + self.assertEqual(1, getargs_i(BadInt2())) + self.assertEqual(0, getargs_i(BadInt3())) self.assertRaises(OverflowError, getargs_i, INT_MIN-1) self.assertEqual(INT_MIN, getargs_i(INT_MIN)) @@ -159,6 +210,11 @@ # l returns 'long', and does range checking (LONG_MIN ... LONG_MAX) self.assertRaises(TypeError, getargs_l, 3.14) self.assertEqual(99, getargs_l(Int())) + self.assertEqual(0, getargs_l(IntSubclass())) + self.assertRaises(TypeError, getargs_l, BadInt()) + with self.assertWarns(DeprecationWarning): + self.assertEqual(1, getargs_l(BadInt2())) + self.assertEqual(0, getargs_l(BadInt3())) self.assertRaises(OverflowError, getargs_l, LONG_MIN-1) self.assertEqual(LONG_MIN, getargs_l(LONG_MIN)) @@ -174,6 +230,10 @@ # (PY_SSIZE_T_MIN ... PY_SSIZE_T_MAX) self.assertRaises(TypeError, getargs_n, 3.14) self.assertRaises(TypeError, getargs_n, Int()) + self.assertEqual(0, getargs_n(IntSubclass())) + self.assertRaises(TypeError, getargs_n, BadInt()) + self.assertRaises(TypeError, getargs_n, BadInt2()) + self.assertEqual(0, getargs_n(BadInt3())) self.assertRaises(OverflowError, getargs_n, PY_SSIZE_T_MIN-1) self.assertEqual(PY_SSIZE_T_MIN, getargs_n(PY_SSIZE_T_MIN)) @@ -192,6 +252,11 @@ self.assertRaises(TypeError, getargs_L, 3.14) self.assertRaises(TypeError, getargs_L, "Hello") self.assertEqual(99, getargs_L(Int())) + self.assertEqual(0, getargs_L(IntSubclass())) + self.assertRaises(TypeError, getargs_L, BadInt()) + with self.assertWarns(DeprecationWarning): + self.assertEqual(1, getargs_L(BadInt2())) + self.assertEqual(0, getargs_L(BadInt3())) self.assertRaises(OverflowError, getargs_L, LLONG_MIN-1) self.assertEqual(LLONG_MIN, getargs_L(LLONG_MIN)) @@ -206,6 +271,11 @@ # K return 'unsigned long long', no range checking self.assertRaises(TypeError, getargs_K, 3.14) self.assertRaises(TypeError, getargs_K, Int()) + self.assertEqual(0, getargs_K(IntSubclass())) + self.assertRaises(TypeError, getargs_K, BadInt()) + self.assertRaises(TypeError, getargs_K, BadInt2()) + self.assertEqual(0, getargs_K(BadInt3())) + self.assertEqual(ULLONG_MAX, getargs_K(ULLONG_MAX)) self.assertEqual(0, getargs_K(0)) self.assertEqual(0, getargs_K(ULLONG_MAX+1)) diff --git a/Lib/test/test_index.py b/Lib/test/test_index.py --- a/Lib/test/test_index.py +++ b/Lib/test/test_index.py @@ -9,7 +9,7 @@ class TrapInt(int): def __index__(self): - return self + return int(self) class BaseTestCase(unittest.TestCase): def setUp(self): @@ -55,6 +55,40 @@ self.assertRaises(TypeError, slice(self.o).indices, 0) self.assertRaises(TypeError, slice(self.n).indices, 0) + def test_int_subclass_with_index(self): + # __index__ should be used when computing indices, even for int + # subclasses. See issue #17576. + class MyInt(int): + def __index__(self): + return int(self) + 1 + + my_int = MyInt(7) + direct_index = my_int.__index__() + operator_index = operator.index(my_int) + self.assertEqual(direct_index, 8) + self.assertEqual(operator_index, 7) + # Both results should be of exact type int. + self.assertIs(type(direct_index), int) + #self.assertIs(type(operator_index), int) + + def test_index_returns_int_subclass(self): + class BadInt: + def __index__(self): + return True + + class BadInt2(int): + def __index__(self): + return True + + bad_int = BadInt() + with self.assertWarns(DeprecationWarning): + n = operator.index(bad_int) + self.assertEqual(n, 1) + + bad_int = BadInt2() + n = operator.index(bad_int) + self.assertEqual(n, 0) + class SeqTestCase: # This test case isn't run directly. It just defines common tests diff --git a/Lib/test/test_int.py b/Lib/test/test_int.py --- a/Lib/test/test_int.py +++ b/Lib/test/test_int.py @@ -263,32 +263,7 @@ def __int__(self): return 42 - class Foo1(object): - def __int__(self): - return 42 - - class Foo2(int): - def __int__(self): - return 42 - - class Foo3(int): - def __int__(self): - return self - - class Foo4(int): - def __int__(self): - return 42 - - class Foo5(int): - def __int__(self): - return 42. - self.assertEqual(int(Foo0()), 42) - self.assertEqual(int(Foo1()), 42) - self.assertEqual(int(Foo2()), 42) - self.assertEqual(int(Foo3()), 0) - self.assertEqual(int(Foo4()), 42) - self.assertRaises(TypeError, int, Foo5()) class Classic: pass @@ -351,6 +326,57 @@ with self.assertRaises(TypeError): int(TruncReturnsBadInt()) + def test_int_subclass_with_int(self): + class MyInt(int): + def __int__(self): + return 42 + + class BadInt(int): + def __int__(self): + return 42.0 + + my_int = MyInt(7) + self.assertEqual(my_int, 7) + self.assertEqual(int(my_int), 42) + + self.assertRaises(TypeError, int, BadInt()) + + def test_int_returns_int_subclass(self): + class BadInt: + def __int__(self): + return True + + class BadInt2(int): + def __int__(self): + return True + + class TruncReturnsBadInt: + def __trunc__(self): + return BadInt() + + class TruncReturnsIntSubclass: + def __trunc__(self): + return True + + bad_int = BadInt() + with self.assertWarns(DeprecationWarning): + n = int(bad_int) + self.assertEqual(n, 1) + + bad_int = BadInt2() + with self.assertWarns(DeprecationWarning): + n = int(bad_int) + self.assertEqual(n, 1) + + bad_int = TruncReturnsBadInt() + with self.assertWarns(DeprecationWarning): + n = int(bad_int) + self.assertEqual(n, 1) + + good_int = TruncReturnsIntSubclass() + n = int(good_int) + self.assertEqual(n, 1) + def test_error_message(self): def check(s, base=None): with self.assertRaises(ValueError, diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #17576: Deprecation warning emitted now when __int__() or __index__() + return not int instance. + - Issue #19932: Fix typo in import.h, missing whitespaces in function prototypes. - Issue #19729: In str.format(), fix recursive expansion in format spec. diff --git a/Objects/abstract.c b/Objects/abstract.c --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -1132,7 +1132,7 @@ return type_error("bad operand type for abs(): '%.200s'", o); } -/* Return a Python int from the object item +/* Return a Python int from the object item. Raise TypeError if the result is not an int or if the object cannot be interpreted as an index. */ @@ -1146,21 +1146,30 @@ Py_INCREF(item); return item; } - if (PyIndex_Check(item)) { - result = item->ob_type->tp_as_number->nb_index(item); - if (result && !PyLong_Check(result)) { - PyErr_Format(PyExc_TypeError, - "__index__ returned non-int " - "(type %.200s)", - result->ob_type->tp_name); - Py_DECREF(result); - return NULL; - } - } - else { + if (!PyIndex_Check(item)) { PyErr_Format(PyExc_TypeError, "'%.200s' object cannot be interpreted " "as an integer", item->ob_type->tp_name); + return NULL; + } + result = item->ob_type->tp_as_number->nb_index(item); + if (!result || PyLong_CheckExact(result)) + return result; + if (!PyLong_Check(result)) { + PyErr_Format(PyExc_TypeError, + "__index__ returned non-int (type %.200s)", + result->ob_type->tp_name); + Py_DECREF(result); + return NULL; + } + /* Issue #17576: warn if 'result' not of exact type int. */ + if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, + "__index__ returned non-int (type %.200s). " + "The ability to return an instance of a strict subclass of int " + "is deprecated, and may be removed in a future version of Python.", + result->ob_type->tp_name)) { + Py_DECREF(result); + return NULL; } return result; } @@ -1212,34 +1221,6 @@ } -/* - Returns the Integral instance converted to an int. The instance is expected - to be an int or have an __int__ method. Steals integral's - reference. error_format will be used to create the TypeError if integral - isn't actually an Integral instance. error_format should be a format string - that can accept a char* naming integral's type. -*/ -static PyObject * -convert_integral_to_int(PyObject *integral, const char *error_format) -{ - PyNumberMethods *nb; - if (PyLong_Check(integral)) - return integral; - nb = Py_TYPE(integral)->tp_as_number; - if (nb->nb_int) { - PyObject *as_int = nb->nb_int(integral); - if (!as_int || PyLong_Check(as_int)) { - Py_DECREF(integral); - return as_int; - } - Py_DECREF(as_int); - } - PyErr_Format(PyExc_TypeError, error_format, Py_TYPE(integral)->tp_name); - Py_DECREF(integral); - return NULL; -} - - PyObject * PyNumber_Long(PyObject *o) { @@ -1257,29 +1238,28 @@ } m = o->ob_type->tp_as_number; if (m && m->nb_int) { /* This should include subclasses of int */ - PyObject *res = m->nb_int(o); - if (res && !PyLong_Check(res)) { - PyErr_Format(PyExc_TypeError, - "__int__ returned non-int (type %.200s)", - res->ob_type->tp_name); - Py_DECREF(res); - return NULL; - } - return res; + return (PyObject *)_PyLong_FromNbInt(o); } - if (PyLong_Check(o)) /* An int subclass without nb_int */ - return _PyLong_Copy((PyLongObject *)o); trunc_func = _PyObject_LookupSpecial(o, &PyId___trunc__); if (trunc_func) { PyObject *truncated = PyEval_CallObject(trunc_func, NULL); PyObject *int_instance; Py_DECREF(trunc_func); - if (truncated == NULL) - return NULL; + if (truncated == NULL || PyLong_Check(truncated)) + return truncated; /* __trunc__ is specified to return an Integral type, but int() needs to return a int. */ - int_instance = convert_integral_to_int(truncated, - "__trunc__ returned non-Integral (type %.200s)"); + m = truncated->ob_type->tp_as_number; + if (m == NULL || m->nb_int == NULL) { + PyErr_Format( + PyExc_TypeError, + "__trunc__ returned non-Integral (type %.200s)", + truncated->ob_type->tp_name); + Py_DECREF(truncated); + return NULL; + } + int_instance = (PyObject *)_PyLong_FromNbInt(truncated); + Py_DECREF(truncated); return int_instance; } if (PyErr_Occurred()) diff --git a/Objects/longobject.c b/Objects/longobject.c --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -116,6 +116,56 @@ return v; } +/* _PyLong_FromNbInt: Convert the given object to a PyLongObject + using the nb_int slot, if available. Raise TypeError if either the + nb_int slot is not available or the result of the call to nb_int + returns something not of type int. +*/ +PyLongObject * +_PyLong_FromNbInt(PyObject *integral) +{ + PyNumberMethods *nb; + PyObject *result; + + /* Fast path for the case that we already have an int. */ + if (PyLong_CheckExact(integral)) { + Py_INCREF(integral); + return (PyLongObject *)integral; + } + + nb = Py_TYPE(integral)->tp_as_number; + if (nb == NULL || nb->nb_int == NULL) { + PyErr_Format(PyExc_TypeError, + "an integer is required (got type %.200s)", + Py_TYPE(integral)->tp_name); + return NULL; + } + + /* Convert using the nb_int slot, which should return something + of exact type int. */ + result = nb->nb_int(integral); + if (!result || PyLong_CheckExact(result)) + return (PyLongObject *)result; + if (!PyLong_Check(result)) { + PyErr_Format(PyExc_TypeError, + "__int__ returned non-int (type %.200s)", + result->ob_type->tp_name); + Py_DECREF(result); + return NULL; + } + /* Issue #17576: warn if 'result' not of exact type int. */ + if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, + "__int__ returned non-int (type %.200s). " + "The ability to return an instance of a strict subclass of int " + "is deprecated, and may be removed in a future version of Python.", + result->ob_type->tp_name)) { + Py_DECREF(result); + return NULL; + } + return (PyLongObject *)result; +} + + /* Allocate a new int object with size digits. Return NULL and set exception if we run out of memory. */ @@ -347,28 +397,17 @@ return -1; } - if (!PyLong_Check(vv)) { - PyNumberMethods *nb; - nb = vv->ob_type->tp_as_number; - if (nb == NULL || nb->nb_int == NULL) { - PyErr_SetString(PyExc_TypeError, - "an integer is required"); - return -1; - } - vv = (*nb->nb_int) (vv); - if (vv == NULL) + if (PyLong_Check(vv)) { + v = (PyLongObject *)vv; + } + else { + v = _PyLong_FromNbInt(vv); + if (v == NULL) return -1; do_decref = 1; - if (!PyLong_Check(vv)) { - Py_DECREF(vv); - PyErr_SetString(PyExc_TypeError, - "nb_int should return int object"); - return -1; - } } res = -1; - v = (PyLongObject *)vv; i = Py_SIZE(v); switch (i) { @@ -412,7 +451,7 @@ } exit: if (do_decref) { - Py_DECREF(vv); + Py_DECREF(v); } return res; } @@ -630,36 +669,25 @@ unsigned long PyLong_AsUnsignedLongMask(register PyObject *op) { - PyNumberMethods *nb; PyLongObject *lo; unsigned long val; - if (op && PyLong_Check(op)) + if (op == NULL) { + PyErr_BadInternalCall(); + return (unsigned long)-1; + } + + if (PyLong_Check(op)) { return _PyLong_AsUnsignedLongMask(op); - - if (op == NULL || (nb = op->ob_type->tp_as_number) == NULL || - nb->nb_int == NULL) { - PyErr_SetString(PyExc_TypeError, "an integer is required"); - return (unsigned long)-1; - } - - lo = (PyLongObject*) (*nb->nb_int) (op); + } + + lo = _PyLong_FromNbInt(op); if (lo == NULL) return (unsigned long)-1; - if (PyLong_Check(lo)) { - val = _PyLong_AsUnsignedLongMask((PyObject *)lo); - Py_DECREF(lo); - if (PyErr_Occurred()) - return (unsigned long)-1; - return val; - } - else - { - Py_DECREF(lo); - PyErr_SetString(PyExc_TypeError, - "nb_int should return int object"); - return (unsigned long)-1; - } + + val = _PyLong_AsUnsignedLongMask((PyObject *)lo); + Py_DECREF(lo); + return val; } int @@ -1169,40 +1197,41 @@ PY_LONG_LONG bytes; int one = 1; int res; + int do_decref = 0; /* if nb_int was called */ if (vv == NULL) { PyErr_BadInternalCall(); return -1; } - if (!PyLong_Check(vv)) { - PyNumberMethods *nb; - PyObject *io; - if ((nb = vv->ob_type->tp_as_number) == NULL || - nb->nb_int == NULL) { - PyErr_SetString(PyExc_TypeError, "an integer is required"); + + if (PyLong_Check(vv)) { + v = (PyLongObject *)vv; + } + else { + v = _PyLong_FromNbInt(vv); + if (v == NULL) return -1; - } - io = (*nb->nb_int) (vv); - if (io == NULL) - return -1; - if (PyLong_Check(io)) { - bytes = PyLong_AsLongLong(io); - Py_DECREF(io); - return bytes; - } - Py_DECREF(io); - PyErr_SetString(PyExc_TypeError, "integer conversion failed"); - return -1; - } - - v = (PyLongObject*)vv; + do_decref = 1; + } + + res = 0; switch(Py_SIZE(v)) { - case -1: return -(sdigit)v->ob_digit[0]; - case 0: return 0; - case 1: return v->ob_digit[0]; - } - res = _PyLong_AsByteArray((PyLongObject *)vv, (unsigned char *)&bytes, - SIZEOF_LONG_LONG, IS_LITTLE_ENDIAN, 1); + case -1: + bytes = -(sdigit)v->ob_digit[0]; + break; + case 0: + bytes = 0; + break; + case 1: + bytes = v->ob_digit[0]; + break; + default: + res = _PyLong_AsByteArray((PyLongObject *)v, (unsigned char *)&bytes, + SIZEOF_LONG_LONG, IS_LITTLE_ENDIAN, 1); + } + if (do_decref) { + Py_DECREF(v); + } /* Plan 9 can't handle PY_LONG_LONG in ? : expressions */ if (res < 0) @@ -1283,36 +1312,25 @@ unsigned PY_LONG_LONG PyLong_AsUnsignedLongLongMask(register PyObject *op) { - PyNumberMethods *nb; PyLongObject *lo; unsigned PY_LONG_LONG val; - if (op && PyLong_Check(op)) + if (op == NULL) { + PyErr_BadInternalCall(); + return (unsigned long)-1; + } + + if (PyLong_Check(op)) { return _PyLong_AsUnsignedLongLongMask(op); - - if (op == NULL || (nb = op->ob_type->tp_as_number) == NULL || - nb->nb_int == NULL) { - PyErr_SetString(PyExc_TypeError, "an integer is required"); - return (unsigned PY_LONG_LONG)-1; - } - - lo = (PyLongObject*) (*nb->nb_int) (op); + } + + lo = _PyLong_FromNbInt(op); if (lo == NULL) return (unsigned PY_LONG_LONG)-1; - if (PyLong_Check(lo)) { - val = _PyLong_AsUnsignedLongLongMask((PyObject *)lo); - Py_DECREF(lo); - if (PyErr_Occurred()) - return (unsigned PY_LONG_LONG)-1; - return val; - } - else - { - Py_DECREF(lo); - PyErr_SetString(PyExc_TypeError, - "nb_int should return int object"); - return (unsigned PY_LONG_LONG)-1; - } + + val = _PyLong_AsUnsignedLongLongMask((PyObject *)lo); + Py_DECREF(lo); + return val; } #undef IS_LITTLE_ENDIAN @@ -1343,28 +1361,17 @@ return -1; } - if (!PyLong_Check(vv)) { - PyNumberMethods *nb; - nb = vv->ob_type->tp_as_number; - if (nb == NULL || nb->nb_int == NULL) { - PyErr_SetString(PyExc_TypeError, - "an integer is required"); - return -1; - } - vv = (*nb->nb_int) (vv); - if (vv == NULL) + if (PyLong_Check(vv)) { + v = (PyLongObject *)vv; + } + else { + v = _PyLong_FromNbInt(vv); + if (v == NULL) return -1; do_decref = 1; - if (!PyLong_Check(vv)) { - Py_DECREF(vv); - PyErr_SetString(PyExc_TypeError, - "nb_int should return int object"); - return -1; - } } res = -1; - v = (PyLongObject *)vv; i = Py_SIZE(v); switch (i) { @@ -1408,7 +1415,7 @@ } exit: if (do_decref) { - Py_DECREF(vv); + Py_DECREF(v); } return res; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 11 20:28:21 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 11 Dec 2013 20:28:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2317576=3A_Deprecation_warning_emitted_now_when_?= =?utf-8?b?X19pbnRfXygpIG9yIF9faW5kZXhfXygp?= Message-ID: <3dfp8P32KSz7LmZ@mail.python.org> http://hg.python.org/cpython/rev/59fb79d0411e changeset: 87900:59fb79d0411e parent: 87898:e961a166dc70 parent: 87899:618cca51a27e user: Serhiy Storchaka date: Wed Dec 11 21:26:36 2013 +0200 summary: Issue #17576: Deprecation warning emitted now when __int__() or __index__() return not int instance. Introduced _PyLong_FromNbInt() and refactored PyLong_As*() functions. files: Include/longobject.h | 6 + Lib/test/test_getargs2.py | 70 +++++++ Lib/test/test_index.py | 36 +++- Lib/test/test_int.py | 76 +++++-- Misc/NEWS | 3 + Objects/abstract.c | 92 ++++------ Objects/longobject.c | 229 +++++++++++++------------ 7 files changed, 319 insertions(+), 193 deletions(-) diff --git a/Include/longobject.h b/Include/longobject.h --- a/Include/longobject.h +++ b/Include/longobject.h @@ -165,6 +165,12 @@ unsigned char* bytes, size_t n, int little_endian, int is_signed); +/* _PyLong_FromNbInt: Convert the given object to a PyLongObject + using the nb_int slot, if available. Raise TypeError if either the + nb_int slot is not available or the result of the call to nb_int + returns something not of type int. +*/ +PyAPI_FUNC(PyLongObject *)_PyLong_FromNbInt(PyObject *); /* _PyLong_Format: Convert the long to a string object with given base, appending a base prefix of 0[box] if base is 2, 8 or 16. */ diff --git a/Lib/test/test_getargs2.py b/Lib/test/test_getargs2.py --- a/Lib/test/test_getargs2.py +++ b/Lib/test/test_getargs2.py @@ -52,12 +52,34 @@ def __int__(self): return 99 +class IntSubclass(int): + def __int__(self): + return 99 + +class BadInt: + def __int__(self): + return 1.0 + +class BadInt2: + def __int__(self): + return True + +class BadInt3(int): + def __int__(self): + return True + + class Unsigned_TestCase(unittest.TestCase): def test_b(self): from _testcapi import getargs_b # b returns 'unsigned char', and does range checking (0 ... UCHAR_MAX) self.assertRaises(TypeError, getargs_b, 3.14) self.assertEqual(99, getargs_b(Int())) + self.assertEqual(0, getargs_b(IntSubclass())) + self.assertRaises(TypeError, getargs_b, BadInt()) + with self.assertWarns(DeprecationWarning): + self.assertEqual(1, getargs_b(BadInt2())) + self.assertEqual(0, getargs_b(BadInt3())) self.assertRaises(OverflowError, getargs_b, -1) self.assertEqual(0, getargs_b(0)) @@ -72,6 +94,11 @@ # B returns 'unsigned char', no range checking self.assertRaises(TypeError, getargs_B, 3.14) self.assertEqual(99, getargs_B(Int())) + self.assertEqual(0, getargs_B(IntSubclass())) + self.assertRaises(TypeError, getargs_B, BadInt()) + with self.assertWarns(DeprecationWarning): + self.assertEqual(1, getargs_B(BadInt2())) + self.assertEqual(0, getargs_B(BadInt3())) self.assertEqual(UCHAR_MAX, getargs_B(-1)) self.assertEqual(0, getargs_B(0)) @@ -86,6 +113,11 @@ # H returns 'unsigned short', no range checking self.assertRaises(TypeError, getargs_H, 3.14) self.assertEqual(99, getargs_H(Int())) + self.assertEqual(0, getargs_H(IntSubclass())) + self.assertRaises(TypeError, getargs_H, BadInt()) + with self.assertWarns(DeprecationWarning): + self.assertEqual(1, getargs_H(BadInt2())) + self.assertEqual(0, getargs_H(BadInt3())) self.assertEqual(USHRT_MAX, getargs_H(-1)) self.assertEqual(0, getargs_H(0)) @@ -101,6 +133,11 @@ # I returns 'unsigned int', no range checking self.assertRaises(TypeError, getargs_I, 3.14) self.assertEqual(99, getargs_I(Int())) + self.assertEqual(0, getargs_I(IntSubclass())) + self.assertRaises(TypeError, getargs_I, BadInt()) + with self.assertWarns(DeprecationWarning): + self.assertEqual(1, getargs_I(BadInt2())) + self.assertEqual(0, getargs_I(BadInt3())) self.assertEqual(UINT_MAX, getargs_I(-1)) self.assertEqual(0, getargs_I(0)) @@ -117,6 +154,10 @@ # it does not accept float, or instances with __int__ self.assertRaises(TypeError, getargs_k, 3.14) self.assertRaises(TypeError, getargs_k, Int()) + self.assertEqual(0, getargs_k(IntSubclass())) + self.assertRaises(TypeError, getargs_k, BadInt()) + self.assertRaises(TypeError, getargs_k, BadInt2()) + self.assertEqual(0, getargs_k(BadInt3())) self.assertEqual(ULONG_MAX, getargs_k(-1)) self.assertEqual(0, getargs_k(0)) @@ -133,6 +174,11 @@ # h returns 'short', and does range checking (SHRT_MIN ... SHRT_MAX) self.assertRaises(TypeError, getargs_h, 3.14) self.assertEqual(99, getargs_h(Int())) + self.assertEqual(0, getargs_h(IntSubclass())) + self.assertRaises(TypeError, getargs_h, BadInt()) + with self.assertWarns(DeprecationWarning): + self.assertEqual(1, getargs_h(BadInt2())) + self.assertEqual(0, getargs_h(BadInt3())) self.assertRaises(OverflowError, getargs_h, SHRT_MIN-1) self.assertEqual(SHRT_MIN, getargs_h(SHRT_MIN)) @@ -147,6 +193,11 @@ # i returns 'int', and does range checking (INT_MIN ... INT_MAX) self.assertRaises(TypeError, getargs_i, 3.14) self.assertEqual(99, getargs_i(Int())) + self.assertEqual(0, getargs_i(IntSubclass())) + self.assertRaises(TypeError, getargs_i, BadInt()) + with self.assertWarns(DeprecationWarning): + self.assertEqual(1, getargs_i(BadInt2())) + self.assertEqual(0, getargs_i(BadInt3())) self.assertRaises(OverflowError, getargs_i, INT_MIN-1) self.assertEqual(INT_MIN, getargs_i(INT_MIN)) @@ -161,6 +212,11 @@ # l returns 'long', and does range checking (LONG_MIN ... LONG_MAX) self.assertRaises(TypeError, getargs_l, 3.14) self.assertEqual(99, getargs_l(Int())) + self.assertEqual(0, getargs_l(IntSubclass())) + self.assertRaises(TypeError, getargs_l, BadInt()) + with self.assertWarns(DeprecationWarning): + self.assertEqual(1, getargs_l(BadInt2())) + self.assertEqual(0, getargs_l(BadInt3())) self.assertRaises(OverflowError, getargs_l, LONG_MIN-1) self.assertEqual(LONG_MIN, getargs_l(LONG_MIN)) @@ -176,6 +232,10 @@ # (PY_SSIZE_T_MIN ... PY_SSIZE_T_MAX) self.assertRaises(TypeError, getargs_n, 3.14) self.assertRaises(TypeError, getargs_n, Int()) + self.assertEqual(0, getargs_n(IntSubclass())) + self.assertRaises(TypeError, getargs_n, BadInt()) + self.assertRaises(TypeError, getargs_n, BadInt2()) + self.assertEqual(0, getargs_n(BadInt3())) self.assertRaises(OverflowError, getargs_n, PY_SSIZE_T_MIN-1) self.assertEqual(PY_SSIZE_T_MIN, getargs_n(PY_SSIZE_T_MIN)) @@ -195,6 +255,11 @@ self.assertRaises(TypeError, getargs_L, 3.14) self.assertRaises(TypeError, getargs_L, "Hello") self.assertEqual(99, getargs_L(Int())) + self.assertEqual(0, getargs_L(IntSubclass())) + self.assertRaises(TypeError, getargs_L, BadInt()) + with self.assertWarns(DeprecationWarning): + self.assertEqual(1, getargs_L(BadInt2())) + self.assertEqual(0, getargs_L(BadInt3())) self.assertRaises(OverflowError, getargs_L, LLONG_MIN-1) self.assertEqual(LLONG_MIN, getargs_L(LLONG_MIN)) @@ -209,6 +274,11 @@ # K return 'unsigned long long', no range checking self.assertRaises(TypeError, getargs_K, 3.14) self.assertRaises(TypeError, getargs_K, Int()) + self.assertEqual(0, getargs_K(IntSubclass())) + self.assertRaises(TypeError, getargs_K, BadInt()) + self.assertRaises(TypeError, getargs_K, BadInt2()) + self.assertEqual(0, getargs_K(BadInt3())) + self.assertEqual(ULLONG_MAX, getargs_K(ULLONG_MAX)) self.assertEqual(0, getargs_K(0)) self.assertEqual(0, getargs_K(ULLONG_MAX+1)) diff --git a/Lib/test/test_index.py b/Lib/test/test_index.py --- a/Lib/test/test_index.py +++ b/Lib/test/test_index.py @@ -9,7 +9,7 @@ class TrapInt(int): def __index__(self): - return self + return int(self) class BaseTestCase(unittest.TestCase): def setUp(self): @@ -55,6 +55,40 @@ self.assertRaises(TypeError, slice(self.o).indices, 0) self.assertRaises(TypeError, slice(self.n).indices, 0) + def test_int_subclass_with_index(self): + # __index__ should be used when computing indices, even for int + # subclasses. See issue #17576. + class MyInt(int): + def __index__(self): + return int(self) + 1 + + my_int = MyInt(7) + direct_index = my_int.__index__() + operator_index = operator.index(my_int) + self.assertEqual(direct_index, 8) + self.assertEqual(operator_index, 7) + # Both results should be of exact type int. + self.assertIs(type(direct_index), int) + #self.assertIs(type(operator_index), int) + + def test_index_returns_int_subclass(self): + class BadInt: + def __index__(self): + return True + + class BadInt2(int): + def __index__(self): + return True + + bad_int = BadInt() + with self.assertWarns(DeprecationWarning): + n = operator.index(bad_int) + self.assertEqual(n, 1) + + bad_int = BadInt2() + n = operator.index(bad_int) + self.assertEqual(n, 0) + class SeqTestCase: # This test case isn't run directly. It just defines common tests diff --git a/Lib/test/test_int.py b/Lib/test/test_int.py --- a/Lib/test/test_int.py +++ b/Lib/test/test_int.py @@ -304,32 +304,7 @@ def __int__(self): return 42 - class Foo1(object): - def __int__(self): - return 42 - - class Foo2(int): - def __int__(self): - return 42 - - class Foo3(int): - def __int__(self): - return self - - class Foo4(int): - def __int__(self): - return 42 - - class Foo5(int): - def __int__(self): - return 42. - self.assertEqual(int(Foo0()), 42) - self.assertEqual(int(Foo1()), 42) - self.assertEqual(int(Foo2()), 42) - self.assertEqual(int(Foo3()), 0) - self.assertEqual(int(Foo4()), 42) - self.assertRaises(TypeError, int, Foo5()) class Classic: pass @@ -392,6 +367,57 @@ with self.assertRaises(TypeError): int(TruncReturnsBadInt()) + def test_int_subclass_with_int(self): + class MyInt(int): + def __int__(self): + return 42 + + class BadInt(int): + def __int__(self): + return 42.0 + + my_int = MyInt(7) + self.assertEqual(my_int, 7) + self.assertEqual(int(my_int), 42) + + self.assertRaises(TypeError, int, BadInt()) + + def test_int_returns_int_subclass(self): + class BadInt: + def __int__(self): + return True + + class BadInt2(int): + def __int__(self): + return True + + class TruncReturnsBadInt: + def __trunc__(self): + return BadInt() + + class TruncReturnsIntSubclass: + def __trunc__(self): + return True + + bad_int = BadInt() + with self.assertWarns(DeprecationWarning): + n = int(bad_int) + self.assertEqual(n, 1) + + bad_int = BadInt2() + with self.assertWarns(DeprecationWarning): + n = int(bad_int) + self.assertEqual(n, 1) + + bad_int = TruncReturnsBadInt() + with self.assertWarns(DeprecationWarning): + n = int(bad_int) + self.assertEqual(n, 1) + + good_int = TruncReturnsIntSubclass() + n = int(good_int) + self.assertEqual(n, 1) + def test_error_message(self): def check(s, base=None): with self.assertRaises(ValueError, diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #17576: Deprecation warning emitted now when __int__() or __index__() + return not int instance. + - Issue #19932: Fix typo in import.h, missing whitespaces in function prototypes. - Issue #19736: Add module-level statvfs constants defined for GNU/glibc diff --git a/Objects/abstract.c b/Objects/abstract.c --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -1155,7 +1155,7 @@ return type_error("bad operand type for abs(): '%.200s'", o); } -/* Return a Python int from the object item +/* Return a Python int from the object item. Raise TypeError if the result is not an int or if the object cannot be interpreted as an index. */ @@ -1169,21 +1169,30 @@ Py_INCREF(item); return item; } - if (PyIndex_Check(item)) { - result = item->ob_type->tp_as_number->nb_index(item); - if (result && !PyLong_Check(result)) { - PyErr_Format(PyExc_TypeError, - "__index__ returned non-int " - "(type %.200s)", - result->ob_type->tp_name); - Py_DECREF(result); - return NULL; - } - } - else { + if (!PyIndex_Check(item)) { PyErr_Format(PyExc_TypeError, "'%.200s' object cannot be interpreted " "as an integer", item->ob_type->tp_name); + return NULL; + } + result = item->ob_type->tp_as_number->nb_index(item); + if (!result || PyLong_CheckExact(result)) + return result; + if (!PyLong_Check(result)) { + PyErr_Format(PyExc_TypeError, + "__index__ returned non-int (type %.200s)", + result->ob_type->tp_name); + Py_DECREF(result); + return NULL; + } + /* Issue #17576: warn if 'result' not of exact type int. */ + if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, + "__index__ returned non-int (type %.200s). " + "The ability to return an instance of a strict subclass of int " + "is deprecated, and may be removed in a future version of Python.", + result->ob_type->tp_name)) { + Py_DECREF(result); + return NULL; } return result; } @@ -1235,34 +1244,6 @@ } -/* - Returns the Integral instance converted to an int. The instance is expected - to be an int or have an __int__ method. Steals integral's - reference. error_format will be used to create the TypeError if integral - isn't actually an Integral instance. error_format should be a format string - that can accept a char* naming integral's type. -*/ -static PyObject * -convert_integral_to_int(PyObject *integral, const char *error_format) -{ - PyNumberMethods *nb; - if (PyLong_Check(integral)) - return integral; - nb = Py_TYPE(integral)->tp_as_number; - if (nb->nb_int) { - PyObject *as_int = nb->nb_int(integral); - if (!as_int || PyLong_Check(as_int)) { - Py_DECREF(integral); - return as_int; - } - Py_DECREF(as_int); - } - PyErr_Format(PyExc_TypeError, error_format, Py_TYPE(integral)->tp_name); - Py_DECREF(integral); - return NULL; -} - - PyObject * PyNumber_Long(PyObject *o) { @@ -1280,29 +1261,28 @@ } m = o->ob_type->tp_as_number; if (m && m->nb_int) { /* This should include subclasses of int */ - PyObject *res = m->nb_int(o); - if (res && !PyLong_Check(res)) { - PyErr_Format(PyExc_TypeError, - "__int__ returned non-int (type %.200s)", - res->ob_type->tp_name); - Py_DECREF(res); - return NULL; - } - return res; + return (PyObject *)_PyLong_FromNbInt(o); } - if (PyLong_Check(o)) /* An int subclass without nb_int */ - return _PyLong_Copy((PyLongObject *)o); trunc_func = _PyObject_LookupSpecial(o, &PyId___trunc__); if (trunc_func) { PyObject *truncated = PyEval_CallObject(trunc_func, NULL); PyObject *int_instance; Py_DECREF(trunc_func); - if (truncated == NULL) - return NULL; + if (truncated == NULL || PyLong_Check(truncated)) + return truncated; /* __trunc__ is specified to return an Integral type, but int() needs to return a int. */ - int_instance = convert_integral_to_int(truncated, - "__trunc__ returned non-Integral (type %.200s)"); + m = truncated->ob_type->tp_as_number; + if (m == NULL || m->nb_int == NULL) { + PyErr_Format( + PyExc_TypeError, + "__trunc__ returned non-Integral (type %.200s)", + truncated->ob_type->tp_name); + Py_DECREF(truncated); + return NULL; + } + int_instance = (PyObject *)_PyLong_FromNbInt(truncated); + Py_DECREF(truncated); return int_instance; } if (PyErr_Occurred()) diff --git a/Objects/longobject.c b/Objects/longobject.c --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -122,6 +122,56 @@ return v; } +/* _PyLong_FromNbInt: Convert the given object to a PyLongObject + using the nb_int slot, if available. Raise TypeError if either the + nb_int slot is not available or the result of the call to nb_int + returns something not of type int. +*/ +PyLongObject * +_PyLong_FromNbInt(PyObject *integral) +{ + PyNumberMethods *nb; + PyObject *result; + + /* Fast path for the case that we already have an int. */ + if (PyLong_CheckExact(integral)) { + Py_INCREF(integral); + return (PyLongObject *)integral; + } + + nb = Py_TYPE(integral)->tp_as_number; + if (nb == NULL || nb->nb_int == NULL) { + PyErr_Format(PyExc_TypeError, + "an integer is required (got type %.200s)", + Py_TYPE(integral)->tp_name); + return NULL; + } + + /* Convert using the nb_int slot, which should return something + of exact type int. */ + result = nb->nb_int(integral); + if (!result || PyLong_CheckExact(result)) + return (PyLongObject *)result; + if (!PyLong_Check(result)) { + PyErr_Format(PyExc_TypeError, + "__int__ returned non-int (type %.200s)", + result->ob_type->tp_name); + Py_DECREF(result); + return NULL; + } + /* Issue #17576: warn if 'result' not of exact type int. */ + if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, + "__int__ returned non-int (type %.200s). " + "The ability to return an instance of a strict subclass of int " + "is deprecated, and may be removed in a future version of Python.", + result->ob_type->tp_name)) { + Py_DECREF(result); + return NULL; + } + return (PyLongObject *)result; +} + + /* Allocate a new int object with size digits. Return NULL and set exception if we run out of memory. */ @@ -353,28 +403,17 @@ return -1; } - if (!PyLong_Check(vv)) { - PyNumberMethods *nb; - nb = vv->ob_type->tp_as_number; - if (nb == NULL || nb->nb_int == NULL) { - PyErr_SetString(PyExc_TypeError, - "an integer is required"); - return -1; - } - vv = (*nb->nb_int) (vv); - if (vv == NULL) + if (PyLong_Check(vv)) { + v = (PyLongObject *)vv; + } + else { + v = _PyLong_FromNbInt(vv); + if (v == NULL) return -1; do_decref = 1; - if (!PyLong_Check(vv)) { - Py_DECREF(vv); - PyErr_SetString(PyExc_TypeError, - "nb_int should return int object"); - return -1; - } } res = -1; - v = (PyLongObject *)vv; i = Py_SIZE(v); switch (i) { @@ -418,7 +457,7 @@ } exit: if (do_decref) { - Py_DECREF(vv); + Py_DECREF(v); } return res; } @@ -636,36 +675,25 @@ unsigned long PyLong_AsUnsignedLongMask(PyObject *op) { - PyNumberMethods *nb; PyLongObject *lo; unsigned long val; - if (op && PyLong_Check(op)) + if (op == NULL) { + PyErr_BadInternalCall(); + return (unsigned long)-1; + } + + if (PyLong_Check(op)) { return _PyLong_AsUnsignedLongMask(op); - - if (op == NULL || (nb = op->ob_type->tp_as_number) == NULL || - nb->nb_int == NULL) { - PyErr_SetString(PyExc_TypeError, "an integer is required"); - return (unsigned long)-1; - } - - lo = (PyLongObject*) (*nb->nb_int) (op); + } + + lo = _PyLong_FromNbInt(op); if (lo == NULL) return (unsigned long)-1; - if (PyLong_Check(lo)) { - val = _PyLong_AsUnsignedLongMask((PyObject *)lo); - Py_DECREF(lo); - if (PyErr_Occurred()) - return (unsigned long)-1; - return val; - } - else - { - Py_DECREF(lo); - PyErr_SetString(PyExc_TypeError, - "nb_int should return int object"); - return (unsigned long)-1; - } + + val = _PyLong_AsUnsignedLongMask((PyObject *)lo); + Py_DECREF(lo); + return val; } int @@ -1167,40 +1195,41 @@ PyLongObject *v; PY_LONG_LONG bytes; int res; + int do_decref = 0; /* if nb_int was called */ if (vv == NULL) { PyErr_BadInternalCall(); return -1; } - if (!PyLong_Check(vv)) { - PyNumberMethods *nb; - PyObject *io; - if ((nb = vv->ob_type->tp_as_number) == NULL || - nb->nb_int == NULL) { - PyErr_SetString(PyExc_TypeError, "an integer is required"); + + if (PyLong_Check(vv)) { + v = (PyLongObject *)vv; + } + else { + v = _PyLong_FromNbInt(vv); + if (v == NULL) return -1; - } - io = (*nb->nb_int) (vv); - if (io == NULL) - return -1; - if (PyLong_Check(io)) { - bytes = PyLong_AsLongLong(io); - Py_DECREF(io); - return bytes; - } - Py_DECREF(io); - PyErr_SetString(PyExc_TypeError, "integer conversion failed"); - return -1; - } - - v = (PyLongObject*)vv; + do_decref = 1; + } + + res = 0; switch(Py_SIZE(v)) { - case -1: return -(sdigit)v->ob_digit[0]; - case 0: return 0; - case 1: return v->ob_digit[0]; - } - res = _PyLong_AsByteArray((PyLongObject *)vv, (unsigned char *)&bytes, - SIZEOF_LONG_LONG, PY_LITTLE_ENDIAN, 1); + case -1: + bytes = -(sdigit)v->ob_digit[0]; + break; + case 0: + bytes = 0; + break; + case 1: + bytes = v->ob_digit[0]; + break; + default: + res = _PyLong_AsByteArray((PyLongObject *)v, (unsigned char *)&bytes, + SIZEOF_LONG_LONG, PY_LITTLE_ENDIAN, 1); + } + if (do_decref) { + Py_DECREF(v); + } /* Plan 9 can't handle PY_LONG_LONG in ? : expressions */ if (res < 0) @@ -1280,36 +1309,25 @@ unsigned PY_LONG_LONG PyLong_AsUnsignedLongLongMask(PyObject *op) { - PyNumberMethods *nb; PyLongObject *lo; unsigned PY_LONG_LONG val; - if (op && PyLong_Check(op)) + if (op == NULL) { + PyErr_BadInternalCall(); + return (unsigned long)-1; + } + + if (PyLong_Check(op)) { return _PyLong_AsUnsignedLongLongMask(op); - - if (op == NULL || (nb = op->ob_type->tp_as_number) == NULL || - nb->nb_int == NULL) { - PyErr_SetString(PyExc_TypeError, "an integer is required"); - return (unsigned PY_LONG_LONG)-1; - } - - lo = (PyLongObject*) (*nb->nb_int) (op); + } + + lo = _PyLong_FromNbInt(op); if (lo == NULL) return (unsigned PY_LONG_LONG)-1; - if (PyLong_Check(lo)) { - val = _PyLong_AsUnsignedLongLongMask((PyObject *)lo); - Py_DECREF(lo); - if (PyErr_Occurred()) - return (unsigned PY_LONG_LONG)-1; - return val; - } - else - { - Py_DECREF(lo); - PyErr_SetString(PyExc_TypeError, - "nb_int should return int object"); - return (unsigned PY_LONG_LONG)-1; - } + + val = _PyLong_AsUnsignedLongLongMask((PyObject *)lo); + Py_DECREF(lo); + return val; } /* Get a C long long int from an int object or any object that has an @@ -1339,28 +1357,17 @@ return -1; } - if (!PyLong_Check(vv)) { - PyNumberMethods *nb; - nb = vv->ob_type->tp_as_number; - if (nb == NULL || nb->nb_int == NULL) { - PyErr_SetString(PyExc_TypeError, - "an integer is required"); - return -1; - } - vv = (*nb->nb_int) (vv); - if (vv == NULL) + if (PyLong_Check(vv)) { + v = (PyLongObject *)vv; + } + else { + v = _PyLong_FromNbInt(vv); + if (v == NULL) return -1; do_decref = 1; - if (!PyLong_Check(vv)) { - Py_DECREF(vv); - PyErr_SetString(PyExc_TypeError, - "nb_int should return int object"); - return -1; - } } res = -1; - v = (PyLongObject *)vv; i = Py_SIZE(v); switch (i) { @@ -1404,7 +1411,7 @@ } exit: if (do_decref) { - Py_DECREF(vv); + Py_DECREF(v); } return res; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 11 22:52:37 2013 From: python-checkins at python.org (r.david.murray) Date: Wed, 11 Dec 2013 22:52:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzE5MDYzOiBwYXJ0?= =?utf-8?q?ially_fix_set=5Fpayload_handling_of_non-ASCII_string_input=2E?= Message-ID: <3dfsLs5FGCz7LkC@mail.python.org> http://hg.python.org/cpython/rev/d842bc07d30b changeset: 87901:d842bc07d30b branch: 3.3 parent: 87899:618cca51a27e user: R David Murray date: Wed Dec 11 16:34:34 2013 -0500 summary: #19063: partially fix set_payload handling of non-ASCII string input. This is a backward compatible partial fix, the complete fix requires raising an error instead of accepting the invalid input, so the real fix is only suitable for 3.4. files: Lib/email/charset.py | 11 ++---- Lib/email/message.py | 26 +++++++++++++-- Lib/test/test_email/test_email.py | 32 +++++++++++++++++++ Misc/NEWS | 5 ++ 4 files changed, 63 insertions(+), 11 deletions(-) diff --git a/Lib/email/charset.py b/Lib/email/charset.py --- a/Lib/email/charset.py +++ b/Lib/email/charset.py @@ -386,7 +386,8 @@ string using the ascii codec produces the correct string version of the content. """ - # 7bit/8bit encodings return the string unchanged (module conversions) + if not string: + return string if self.body_encoding is BASE64: if isinstance(string, str): string = string.encode(self.output_charset) @@ -398,13 +399,9 @@ # character set, then, we must turn it into pseudo bytes via the # latin1 charset, which will encode any byte as a single code point # between 0 and 255, which is what body_encode is expecting. - # - # Note that this clause doesn't handle the case of a _payload that - # is already bytes. It never did, and the semantics of _payload - # being bytes has never been nailed down, so fixing that is a - # longer term TODO. if isinstance(string, str): - string = string.encode(self.output_charset).decode('latin1') + string = string.encode(self.output_charset) + string = string.decode('latin1') return email.quoprimime.body_encode(string) else: if isinstance(string, str): diff --git a/Lib/email/message.py b/Lib/email/message.py --- a/Lib/email/message.py +++ b/Lib/email/message.py @@ -275,9 +275,19 @@ Optional charset sets the message's default character set. See set_charset() for details. """ - if isinstance(payload, bytes): - payload = payload.decode('ascii', 'surrogateescape') - self._payload = payload + if hasattr(payload, 'encode'): + if charset is None: + # We should check for ASCII-only here, but we can't do that + # for backward compatibility reasons. Fixed in 3.4. + self._payload = payload + return + if not isinstance(charset, Charset): + charset = Charset(charset) + payload = payload.encode(charset.output_charset) + if hasattr(payload, 'decode'): + self._payload = payload.decode('ascii', 'surrogateescape') + else: + self._payload = payload if charset is not None: self.set_charset(charset) @@ -316,7 +326,15 @@ try: cte(self) except TypeError: - self._payload = charset.body_encode(self._payload) + # This if is for backward compatibility and will be removed + # in 3.4 when the ascii check is added to set_payload. + payload = self._payload + if payload: + try: + payload = payload.encode('ascii', 'surrogateescape') + except UnicodeError: + payload = payload.encode(charset.output_charset) + self._payload = charset.body_encode(payload) self.add_header('Content-Transfer-Encoding', cte) def get_charset(self): diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -92,6 +92,38 @@ msg.set_payload('This is a string payload', charset) self.assertEqual(msg.get_charset().input_charset, 'iso-8859-1') + def test_set_payload_with_8bit_data_and_charset(self): + data = b'\xd0\x90\xd0\x91\xd0\x92' + charset = Charset('utf-8') + msg = Message() + msg.set_payload(data, charset) + self.assertEqual(msg['content-transfer-encoding'], 'base64') + self.assertEqual(msg.get_payload(decode=True), data) + self.assertEqual(msg.get_payload(), '0JDQkdCS\n') + + def test_set_payload_with_non_ascii_and_charset_body_encoding_none(self): + data = b'\xd0\x90\xd0\x91\xd0\x92' + charset = Charset('utf-8') + charset.body_encoding = None # Disable base64 encoding + msg = Message() + msg.set_payload(data.decode('utf-8'), charset) + self.assertEqual(msg['content-transfer-encoding'], '8bit') + self.assertEqual(msg.get_payload(decode=True), data) + + def test_set_payload_with_8bit_data_and_charset_body_encoding_none(self): + data = b'\xd0\x90\xd0\x91\xd0\x92' + charset = Charset('utf-8') + charset.body_encoding = None # Disable base64 encoding + msg = Message() + msg.set_payload(data, charset) + self.assertEqual(msg['content-transfer-encoding'], '8bit') + self.assertEqual(msg.get_payload(decode=True), data) + + def test_set_payload_to_list(self): + msg = Message() + msg.set_payload([]) + self.assertEqual(msg.get_payload(), []) + def test_get_charsets(self): eq = self.assertEqual diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,11 @@ Library ------- +- Issue #19063: if a Charset's body_encoding was set to None, the email + package would generate a message claiming the Content-Transfer-Encoding + was 7bit, and produce garbage output for the content. This now works. + A couple of other set_payload mishandlings of non-ASCII are also fixed. + - Issue #17200: telnetlib's read_until and expect timeout was broken by the fix to Issue #14635 in Python 3.3.0 to be interpreted as milliseconds instead of seconds when the platform supports select.poll (ie: everywhere). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 11 22:52:39 2013 From: python-checkins at python.org (r.david.murray) Date: Wed, 11 Dec 2013 22:52:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Null_merge_for_=2319063_=283=2E4_fix_is_different=29=2E?= Message-ID: <3dfsLv06Hkz7LkC@mail.python.org> http://hg.python.org/cpython/rev/02cb48459b58 changeset: 87902:02cb48459b58 parent: 87900:59fb79d0411e parent: 87901:d842bc07d30b user: R David Murray date: Wed Dec 11 16:48:52 2013 -0500 summary: Null merge for #19063 (3.4 fix is different). files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 11 22:52:40 2013 From: python-checkins at python.org (r.david.murray) Date: Wed, 11 Dec 2013 22:52:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_=2319063=3A_fix_set=5Fpayl?= =?utf-8?q?oad_handling_of_non-ASCII_string_input=2E?= Message-ID: <3dfsLw39vpz7Ln7@mail.python.org> http://hg.python.org/cpython/rev/e20f98a8ed71 changeset: 87903:e20f98a8ed71 user: R David Murray date: Wed Dec 11 16:52:11 2013 -0500 summary: #19063: fix set_payload handling of non-ASCII string input. This version of the fix raises an error instead of accepting the invalid input (ie: if a non-ASCII string is used but no charset is specified). files: Doc/library/email.message.rst | 8 +- Lib/email/charset.py | 29 ++--- Lib/email/message.py | 22 +++- Lib/test/test_email/test_contentmanager.py | 5 +- Lib/test/test_email/test_email.py | 52 +++++++-- Misc/NEWS | 7 + 6 files changed, 85 insertions(+), 38 deletions(-) diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst --- a/Doc/library/email.message.rst +++ b/Doc/library/email.message.rst @@ -196,7 +196,13 @@ Set the entire message object's payload to *payload*. It is the client's responsibility to ensure the payload invariants. Optional *charset* sets - the message's default character set; see :meth:`set_charset` for details. + the message's character set; see :meth:`set_charset` for details. If + *payload* is a string containing non-ASCII characters, *charset* is + required. + + .. versionchanged:: 3.4 + Previous to 3.4 *charset* was not required when *payload* was a + non-ASCII string, but omitting it produced nonsense results. .. method:: set_charset(charset) diff --git a/Lib/email/charset.py b/Lib/email/charset.py --- a/Lib/email/charset.py +++ b/Lib/email/charset.py @@ -378,18 +378,19 @@ return None def body_encode(self, string): - """Body-encode a string by converting it first to bytes. + """Body-encode a string, converting it first to bytes if needed. The type of encoding (base64 or quoted-printable) will be based on - self.body_encoding. If body_encoding is None, we assume the - output charset is a 7bit encoding, so re-encoding the decoded - string using the ascii codec produces the correct string version - of the content. + self.body_encoding. If body_encoding is None, we perform no CTE + encoding (the CTE will be either 7bit or 8bit), we just encode the + binary representation to ascii using the surrogateescape error handler, + which will enable the Generators to produce the correct output. """ - # 7bit/8bit encodings return the string unchanged (module conversions) + if not string: + return string + if isinstance(string, str): + string = string.encode(self.output_charset) if self.body_encoding is BASE64: - if isinstance(string, str): - string = string.encode(self.output_charset) return email.base64mime.body_encode(string) elif self.body_encoding is QP: # quopromime.body_encode takes a string, but operates on it as if @@ -398,15 +399,7 @@ # character set, then, we must turn it into pseudo bytes via the # latin1 charset, which will encode any byte as a single code point # between 0 and 255, which is what body_encode is expecting. - # - # Note that this clause doesn't handle the case of a _payload that - # is already bytes. It never did, and the semantics of _payload - # being bytes has never been nailed down, so fixing that is a - # longer term TODO. - if isinstance(string, str): - string = string.encode(self.output_charset).decode('latin1') + string = string.decode('latin1') return email.quoprimime.body_encode(string) else: - if isinstance(string, str): - string = string.encode(self.output_charset).decode('ascii') - return string + return string.decode('ascii', 'surrogateescape') diff --git a/Lib/email/message.py b/Lib/email/message.py --- a/Lib/email/message.py +++ b/Lib/email/message.py @@ -301,9 +301,23 @@ Optional charset sets the message's default character set. See set_charset() for details. """ - if isinstance(payload, bytes): - payload = payload.decode('ascii', 'surrogateescape') - self._payload = payload + if hasattr(payload, 'encode'): + if charset is None: + try: + payload.encode('ascii', 'surrogateescape') + except UnicodeError: + raise TypeError("charset argument must be specified" + " when non-ASCII characters are used in the" + " payload") from None + self._payload = payload + return + if not isinstance(charset, Charset): + charset = Charset(charset) + payload = payload.encode(charset.output_charset) + if hasattr(payload, 'decode'): + self._payload = payload.decode('ascii', 'surrogateescape') + else: + self._payload = payload if charset is not None: self.set_charset(charset) @@ -342,7 +356,7 @@ try: cte(self) except TypeError: - self._payload = charset.body_encode(self._payload) + self._payload = charset.body_encode(self.get_payload(decode=True)) self.add_header('Content-Transfer-Encoding', cte) def get_charset(self): diff --git a/Lib/test/test_email/test_contentmanager.py b/Lib/test/test_email/test_contentmanager.py --- a/Lib/test/test_email/test_contentmanager.py +++ b/Lib/test/test_email/test_contentmanager.py @@ -208,12 +208,11 @@ "Bas?c t?xt.\n") def test_get_text_plain_utf8_base64_recoverable_bad_CTE_data(self): - m = self._str_msg(textwrap.dedent("""\ + m = self._bytes_msg(textwrap.dedent("""\ Content-Type: text/plain; charset="utf8" Content-Transfer-Encoding: base64 - QmFzw6xjIHTDq3h0Lgo\xFF= - """)) + QmFzw6xjIHTDq3h0Lgo""").encode('ascii') + b'\xFF=\n') self.assertEqual(raw_data_manager.get_content(m, errors='ignore'), "Bas?c t?xt.\n") diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -92,6 +92,44 @@ msg.set_payload('This is a string payload', charset) self.assertEqual(msg.get_charset().input_charset, 'iso-8859-1') + def test_set_payload_with_8bit_data_and_charset(self): + data = b'\xd0\x90\xd0\x91\xd0\x92' + charset = Charset('utf-8') + msg = Message() + msg.set_payload(data, charset) + self.assertEqual(msg['content-transfer-encoding'], 'base64') + self.assertEqual(msg.get_payload(decode=True), data) + self.assertEqual(msg.get_payload(), '0JDQkdCS\n') + + def test_set_payload_with_non_ascii_and_charset_body_encoding_none(self): + data = b'\xd0\x90\xd0\x91\xd0\x92' + charset = Charset('utf-8') + charset.body_encoding = None # Disable base64 encoding + msg = Message() + msg.set_payload(data.decode('utf-8'), charset) + self.assertEqual(msg['content-transfer-encoding'], '8bit') + self.assertEqual(msg.get_payload(decode=True), data) + + def test_set_payload_with_8bit_data_and_charset_body_encoding_none(self): + data = b'\xd0\x90\xd0\x91\xd0\x92' + charset = Charset('utf-8') + charset.body_encoding = None # Disable base64 encoding + msg = Message() + msg.set_payload(data, charset) + self.assertEqual(msg['content-transfer-encoding'], '8bit') + self.assertEqual(msg.get_payload(decode=True), data) + + def test_set_payload_to_list(self): + msg = Message() + msg.set_payload([]) + self.assertEqual(msg.get_payload(), []) + + def test_set_payload_with_non_ascii_and_no_charset_raises(self): + data = b'\xd0\x90\xd0\x91\xd0\x92'.decode('utf-8') + msg = Message() + with self.assertRaises(TypeError): + msg.set_payload(data) + def test_get_charsets(self): eq = self.assertEqual @@ -558,20 +596,10 @@ self.assertIsInstance(msg.defects[0], errors.InvalidBase64CharactersDefect) - def test_broken_unicode_payload(self): - # This test improves coverage but is not a compliance test. - # The behavior in this situation is currently undefined by the API. - x = 'this is a br\xf6ken thing to do' - msg = Message() - msg['content-type'] = 'text/plain' - msg['content-transfer-encoding'] = '8bit' - msg.set_payload(x) - self.assertEqual(msg.get_payload(decode=True), - bytes(x, 'raw-unicode-escape')) - def test_questionable_bytes_payload(self): # This test improves coverage but is not a compliance test, - # since it involves poking inside the black box. + # since it involves poking inside the black box in a way + # that actually breaks the model invariants. x = 'this is a qu?stionable thing to do'.encode('utf-8') msg = Message() msg['content-type'] = 'text/plain; charset="utf-8"' diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -26,6 +26,13 @@ Library ------- +- Issue #19063: if a Charset's body_encoding was set to None, the email + package would generate a message claiming the Content-Transfer-Encoding + was 7bit, and produce garbage output for the content. This now works. + A couple of other set_payload mishandlings of non-ASCII are also fixed. + In addition, calling set_payload with a string argument without + specifying a charset now raises an error (this is a new error in 3.4). + - Issue #15475: Add __sizeof__ implementations for itertools objects. - Issue #19880: Fix a reference leak in unittest.TestCase. Explicitly break -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 12 00:01:35 2013 From: python-checkins at python.org (zach.ware) Date: Thu, 12 Dec 2013 00:01:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5ODI4?= =?utf-8?q?=3A_Fixed_test=5Fsite_when_the_whole_suite_is_run_with_-S=2E?= Message-ID: <3dfttR3QN8z7Ljq@mail.python.org> http://hg.python.org/cpython/rev/40884256f8dd changeset: 87904:40884256f8dd branch: 3.3 parent: 87901:d842bc07d30b user: Zachary Ware date: Wed Dec 11 16:59:44 2013 -0600 summary: Issue #19828: Fixed test_site when the whole suite is run with -S. Also, cleaned up an unused import. files: Lib/test/test_site.py | 17 ++++++++--------- Misc/NEWS | 2 ++ 2 files changed, 10 insertions(+), 9 deletions(-) diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py --- a/Lib/test/test_site.py +++ b/Lib/test/test_site.py @@ -6,8 +6,7 @@ """ import unittest import test.support -from test.support import run_unittest, TESTFN, EnvironmentVarGuard -from test.support import captured_stderr +from test.support import captured_stderr, TESTFN, EnvironmentVarGuard import builtins import os import sys @@ -19,13 +18,13 @@ import sysconfig from copy import copy -# Need to make sure to not import 'site' if someone specified ``-S`` at the -# command-line. Detect this by just making sure 'site' has not been imported -# already. -if "site" in sys.modules: - import site -else: - raise unittest.SkipTest("importation of site.py suppressed") +# These tests are not particularly useful if Python was invoked with -S. +# If you add tests that are useful under -S, this skip should be moved +# to the class level. +if sys.flags.no_site: + raise unittest.SkipTest("Python was invoked with -S") + +import site if site.ENABLE_USER_SITE and not os.path.isdir(site.USER_SITE): # need to add user site directory for tests diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -130,6 +130,8 @@ Tests ----- +- Issue #19828: Fixed test_site when the whole suite is run with -S. + - Issue #19928: Implemented a test for repr() of cell objects. - Issue #19535: Fixed test_docxmlrpc when python is run with -OO. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 12 00:01:36 2013 From: python-checkins at python.org (zach.ware) Date: Thu, 12 Dec 2013 00:01:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319828=3A_Merge_with_3=2E3?= Message-ID: <3dfttS5Nhzz7Llc@mail.python.org> http://hg.python.org/cpython/rev/90834ad91d56 changeset: 87905:90834ad91d56 parent: 87903:e20f98a8ed71 parent: 87904:40884256f8dd user: Zachary Ware date: Wed Dec 11 17:01:21 2013 -0600 summary: Issue #19828: Merge with 3.3 files: Lib/test/test_site.py | 17 ++++++++--------- Misc/NEWS | 2 ++ 2 files changed, 10 insertions(+), 9 deletions(-) diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py --- a/Lib/test/test_site.py +++ b/Lib/test/test_site.py @@ -6,8 +6,7 @@ """ import unittest import test.support -from test.support import run_unittest, TESTFN, EnvironmentVarGuard -from test.support import captured_stderr +from test.support import captured_stderr, TESTFN, EnvironmentVarGuard import builtins import os import sys @@ -19,13 +18,13 @@ import sysconfig from copy import copy -# Need to make sure to not import 'site' if someone specified ``-S`` at the -# command-line. Detect this by just making sure 'site' has not been imported -# already. -if "site" in sys.modules: - import site -else: - raise unittest.SkipTest("importation of site.py suppressed") +# These tests are not particularly useful if Python was invoked with -S. +# If you add tests that are useful under -S, this skip should be moved +# to the class level. +if sys.flags.no_site: + raise unittest.SkipTest("Python was invoked with -S") + +import site if site.ENABLE_USER_SITE and not os.path.isdir(site.USER_SITE): # need to add user site directory for tests diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -151,6 +151,8 @@ Tests ----- +- Issue #19828: Fixed test_site when the whole suite is run with -S. + - Issue #19928: Implemented a test for repr() of cell objects. - Issue #19535: Fixed test_docxmlrpc, test_functools, test_inspect, and -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 12 00:21:52 2013 From: python-checkins at python.org (zach.ware) Date: Thu, 12 Dec 2013 00:21:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Filter_namespa?= =?utf-8?q?ceobject=27s_files_properly_in_the_pythoncore_VS_project=2E?= Message-ID: <3dfvKr72JHz7LmT@mail.python.org> http://hg.python.org/cpython/rev/222ade565dbd changeset: 87906:222ade565dbd branch: 3.3 parent: 87904:40884256f8dd user: Zachary Ware date: Wed Dec 11 17:12:34 2013 -0600 summary: Filter namespaceobject's files properly in the pythoncore VS project. files: PCbuild/pythoncore.vcxproj.filters | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-) diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -402,13 +402,15 @@ Python - Python Python + + Include + @@ -915,7 +917,9 @@ PC - + + Objects + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 12 00:21:54 2013 From: python-checkins at python.org (zach.ware) Date: Thu, 12 Dec 2013 00:21:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Null_merge_with_3=2E3=2E?= Message-ID: <3dfvKt1hQSz7Ln3@mail.python.org> http://hg.python.org/cpython/rev/aa19d80fe66a changeset: 87907:aa19d80fe66a parent: 87905:90834ad91d56 parent: 87906:222ade565dbd user: Zachary Ware date: Wed Dec 11 17:18:37 2013 -0600 summary: Null merge with 3.3. There are enough new things to make the merge harder than making the change in VS, and there are other things to add filters for in 3.4. files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 12 00:21:55 2013 From: python-checkins at python.org (zach.ware) Date: Thu, 12 Dec 2013 00:21:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Properly_filter_some_recen?= =?utf-8?q?t_additions_in_the_Windows_build_files=2E?= Message-ID: <3dfvKv3ThHz7LnG@mail.python.org> http://hg.python.org/cpython/rev/ae047ebe10f9 changeset: 87908:ae047ebe10f9 user: Zachary Ware date: Wed Dec 11 17:21:13 2013 -0600 summary: Properly filter some recent additions in the Windows build files. files: PCbuild/_overlapped.vcxproj.filters | 13 +++++++++++ PCbuild/_sha3.vcxproj.filters | 13 +++++++++++ PCbuild/pythoncore.vcxproj.filters | 20 +++++++++++++++- 3 files changed, 44 insertions(+), 2 deletions(-) diff --git a/PCbuild/_overlapped.vcxproj.filters b/PCbuild/_overlapped.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/_overlapped.vcxproj.filters @@ -0,0 +1,13 @@ +? + + + + {6f67c8db-7de7-4714-a967-2b0d4bc71f2e} + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/_sha3.vcxproj.filters b/PCbuild/_sha3.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/_sha3.vcxproj.filters @@ -0,0 +1,13 @@ +? + + + + {f1f1fd19-4f85-4a56-be41-af14f0bc2a9c} + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -414,7 +414,6 @@ Python - Python @@ -424,6 +423,12 @@ Include + + Include + + + Modules + @@ -930,13 +935,24 @@ PC - Modules Python + + Objects + + + Modules + + + Modules + + + Modules + -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Thu Dec 12 09:41:50 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 12 Dec 2013 09:41:50 +0100 Subject: [Python-checkins] Daily reference leaks (ae047ebe10f9): sum=0 Message-ID: results for ae047ebe10f9 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogolOmnF', '-x'] From python-checkins at python.org Thu Dec 12 12:36:34 2013 From: python-checkins at python.org (victor.stinner) Date: Thu, 12 Dec 2013 12:36:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_a_sequence_diagram_for?= =?utf-8?q?_the_chained_coroutines_example?= Message-ID: <3dgCdZ65sVz7LkS@mail.python.org> http://hg.python.org/cpython/rev/6864abd8e83a changeset: 87909:6864abd8e83a user: Victor Stinner date: Thu Dec 12 12:35:17 2013 +0100 summary: Add a sequence diagram for the chained coroutines example files: Doc/library/asyncio-task.rst | 11 ++++++++++- Doc/library/tulip_coro.dia | Bin Doc/library/tulip_coro.png | Bin 3 files changed, 10 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -74,7 +74,6 @@ loop = asyncio.get_event_loop() loop.run_until_complete(greet_every_two_seconds()) - .. seealso:: :ref:`Hello World example using a callback `. @@ -104,6 +103,16 @@ ``compute()`` is chained to ``print_sum()``: ``print_sum()`` coroutine waits until ``compute()`` is completed before returing its result. +Sequence diagram of the example: + +.. image:: tulip_coro.png + :align: center + +The diagram shows the logical links between the task and the two coroutines, it +does not describe exactly how things work internally. For example, the sleep +coroutine creates an internal future which uses +:meth:`BaseEventLoop.call_later` to wake up the task in 1 second. + InvalidStateError ----------------- diff --git a/Doc/library/tulip_coro.dia b/Doc/library/tulip_coro.dia new file mode 100644 index 0000000000000000000000000000000000000000..c4e19a066589b568b64cbc3cc9d5d8c60c4a2a08 GIT binary patch [stripped] diff --git a/Doc/library/tulip_coro.png b/Doc/library/tulip_coro.png new file mode 100644 index 0000000000000000000000000000000000000000..e083bc37de88d7fa55bf4124b077aef81b8c798a GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 12 17:33:38 2013 From: python-checkins at python.org (zach.ware) Date: Thu, 12 Dec 2013 17:33:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Avoid_UnicodeE?= =?utf-8?q?ncodeError_by_only_printing_ASCII=2E?= Message-ID: <3dgLDL6V88z7Ljs@mail.python.org> http://hg.python.org/cpython/rev/1ad2ff119356 changeset: 87910:1ad2ff119356 branch: 3.3 parent: 87906:222ade565dbd user: Zachary Ware date: Thu Dec 12 10:32:16 2013 -0600 summary: Avoid UnicodeEncodeError by only printing ASCII. This fixes running test_decimal in verbose mode on Windows, which I broke in issue #19572. files: Lib/test/test_decimal.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -1163,10 +1163,10 @@ thousands_sep = locale.localeconv()['thousands_sep'] if decimal_point != '\u066b': self.skipTest('inappropriate decimal point separator' - '({!r} not {!r})'.format(decimal_point, '\u066b')) + '({!a} not {!a})'.format(decimal_point, '\u066b')) if thousands_sep != '\u066c': self.skipTest('inappropriate thousands separator' - '({!r} not {!r})'.format(thousands_sep, '\u066c')) + '({!a} not {!a})'.format(thousands_sep, '\u066c')) self.assertEqual(format(Decimal('100000000.123'), 'n'), '100\u066c000\u066c000\u066b123') -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 12 17:33:40 2013 From: python-checkins at python.org (zach.ware) Date: Thu, 12 Dec 2013 17:33:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge_fix_from_3=2E3?= Message-ID: <3dgLDN1Fl6z7Ljw@mail.python.org> http://hg.python.org/cpython/rev/f9279a3accfe changeset: 87911:f9279a3accfe parent: 87909:6864abd8e83a parent: 87910:1ad2ff119356 user: Zachary Ware date: Thu Dec 12 10:33:27 2013 -0600 summary: Merge fix from 3.3 files: Lib/test/test_decimal.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -1163,10 +1163,10 @@ thousands_sep = locale.localeconv()['thousands_sep'] if decimal_point != '\u066b': self.skipTest('inappropriate decimal point separator' - '({!r} not {!r})'.format(decimal_point, '\u066b')) + '({!a} not {!a})'.format(decimal_point, '\u066b')) if thousands_sep != '\u066c': self.skipTest('inappropriate thousands separator' - '({!r} not {!r})'.format(thousands_sep, '\u066c')) + '({!a} not {!a})'.format(thousands_sep, '\u066c')) self.assertEqual(format(Decimal('100000000.123'), 'n'), '100\u066c000\u066c000\u066b123') -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 12 19:06:19 2013 From: python-checkins at python.org (stefan.krah) Date: Thu, 12 Dec 2013 19:06:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Do_not_discard?= =?utf-8?q?_const_qualifier_without_a_reason=2E?= Message-ID: <3dgNHH4Q9sz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/f95f5769f158 changeset: 87912:f95f5769f158 branch: 3.3 parent: 87910:1ad2ff119356 user: Stefan Krah date: Thu Dec 12 18:51:51 2013 +0100 summary: Do not discard const qualifier without a reason. files: Modules/_decimal/libmpdec/mpdecimal.c | 16 +++++++------- 1 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -3202,9 +3202,9 @@ } static inline void -_mpd_ptrswap(mpd_t **a, mpd_t **b) -{ - mpd_t *t = *a; +_mpd_ptrswap(const mpd_t **a, const mpd_t **b) +{ + const mpd_t *t = *a; *a = *b; *b = t; } @@ -3232,7 +3232,7 @@ _mpd_qaddsub(mpd_t *result, const mpd_t *a, const mpd_t *b, uint8_t sign_b, const mpd_context_t *ctx, uint32_t *status) { - mpd_t *big, *small; + const mpd_t *big, *small; MPD_NEW_STATIC(big_aligned,0,0,0,0); MPD_NEW_CONST(tiny,0,0,1,1,1,1); mpd_uint_t carry; @@ -3242,7 +3242,7 @@ /* compare exponents */ - big = (mpd_t *)a; small = (mpd_t *)b; + big = a; small = b; if (big->exp != small->exp) { if (small->exp > big->exp) { _mpd_ptrswap(&big, &small); @@ -4421,7 +4421,7 @@ const mpd_context_t *ctx, uint32_t *status) { uint32_t workstatus = 0; - mpd_t *cc = (mpd_t *)c; + const mpd_t *cc = c; if (result == c) { if ((cc = mpd_qncopy(c)) == NULL) { @@ -4435,7 +4435,7 @@ mpd_qadd(result, result, cc, ctx, &workstatus); } - if (cc != c) mpd_del(cc); + if (cc != c) mpd_del((mpd_t *)cc); *status |= workstatus; } @@ -5727,7 +5727,7 @@ _mpd_qmul(mpd_t *result, const mpd_t *a, const mpd_t *b, const mpd_context_t *ctx, uint32_t *status) { - mpd_t *big = (mpd_t *)a, *small = (mpd_t *)b; + const mpd_t *big = a, *small = b; mpd_uint_t *rdata = NULL; mpd_uint_t rbuf[MPD_MINALLOC_MAX]; mpd_size_t rsize, i; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 12 19:06:21 2013 From: python-checkins at python.org (stefan.krah) Date: Thu, 12 Dec 2013 19:06:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgZnJvbSAzLjMu?= Message-ID: <3dgNHK01M5z7Ljw@mail.python.org> http://hg.python.org/cpython/rev/0cb4ff55f850 changeset: 87913:0cb4ff55f850 parent: 87911:f9279a3accfe parent: 87912:f95f5769f158 user: Stefan Krah date: Thu Dec 12 19:05:08 2013 +0100 summary: Merge from 3.3. files: Modules/_decimal/libmpdec/mpdecimal.c | 16 +++++++------- 1 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -3202,9 +3202,9 @@ } static inline void -_mpd_ptrswap(mpd_t **a, mpd_t **b) -{ - mpd_t *t = *a; +_mpd_ptrswap(const mpd_t **a, const mpd_t **b) +{ + const mpd_t *t = *a; *a = *b; *b = t; } @@ -3232,7 +3232,7 @@ _mpd_qaddsub(mpd_t *result, const mpd_t *a, const mpd_t *b, uint8_t sign_b, const mpd_context_t *ctx, uint32_t *status) { - mpd_t *big, *small; + const mpd_t *big, *small; MPD_NEW_STATIC(big_aligned,0,0,0,0); MPD_NEW_CONST(tiny,0,0,1,1,1,1); mpd_uint_t carry; @@ -3242,7 +3242,7 @@ /* compare exponents */ - big = (mpd_t *)a; small = (mpd_t *)b; + big = a; small = b; if (big->exp != small->exp) { if (small->exp > big->exp) { _mpd_ptrswap(&big, &small); @@ -4421,7 +4421,7 @@ const mpd_context_t *ctx, uint32_t *status) { uint32_t workstatus = 0; - mpd_t *cc = (mpd_t *)c; + const mpd_t *cc = c; if (result == c) { if ((cc = mpd_qncopy(c)) == NULL) { @@ -4435,7 +4435,7 @@ mpd_qadd(result, result, cc, ctx, &workstatus); } - if (cc != c) mpd_del(cc); + if (cc != c) mpd_del((mpd_t *)cc); *status |= workstatus; } @@ -5727,7 +5727,7 @@ _mpd_qmul(mpd_t *result, const mpd_t *a, const mpd_t *b, const mpd_context_t *ctx, uint32_t *status) { - mpd_t *big = (mpd_t *)a, *small = (mpd_t *)b; + const mpd_t *big = a, *small = b; mpd_uint_t *rdata = NULL; mpd_uint_t rbuf[MPD_MINALLOC_MAX]; mpd_size_t rsize, i; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 12 23:13:26 2013 From: python-checkins at python.org (victor.stinner) Date: Thu, 12 Dec 2013 23:13:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319751=3A_Fix_typo?= =?utf-8?q?_in_configuration_option?= Message-ID: <3dgTmQ1ZsXz7LjM@mail.python.org> http://hg.python.org/cpython/rev/7c116d7c6c65 changeset: 87914:7c116d7c6c65 user: Victor Stinner date: Thu Dec 12 23:06:07 2013 +0100 summary: Issue #19751: Fix typo in configuration option files: Lib/test/test_sys.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -447,7 +447,7 @@ self.assertIsInstance(sys.hash_info.inf, int) self.assertIsInstance(sys.hash_info.nan, int) self.assertIsInstance(sys.hash_info.imag, int) - algo = sysconfig.get_config_var("PY_HASH_ALGORITHM") + algo = sysconfig.get_config_var("Py_HASH_ALGORITHM") if sys.hash_info.algorithm in {"fnv", "siphash24"}: self.assertIn(sys.hash_info.hash_bits, {32, 64}) self.assertIn(sys.hash_info.seed_bits, {32, 64, 128}) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 12 23:13:27 2013 From: python-checkins at python.org (victor.stinner) Date: Thu, 12 Dec 2013 23:13:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319751=3A_Fix_hash?= =?utf-8?q?=5Finfo_test_of_test=5Fsys_on_SPARC_Solaris?= Message-ID: <3dgTmR3J7pz7LjM@mail.python.org> http://hg.python.org/cpython/rev/c1a7ba57b4ff changeset: 87915:c1a7ba57b4ff user: Victor Stinner date: Thu Dec 12 23:07:40 2013 +0100 summary: Issue #19751: Fix hash_info test of test_sys on SPARC Solaris files: Lib/test/test_sys.py | 6 +----- 1 files changed, 1 insertions(+), 5 deletions(-) diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -457,11 +457,7 @@ elif algo == 2: self.assertEqual(sys.hash_info.algorithm, "fnv") else: - processor = platform.processor().lower() - if processor in {"sparc", "mips"}: - self.assertEqual(sys.hash_info.algorithm, "fnv") - else: - self.assertEqual(sys.hash_info.algorithm, "siphash24") + self.assertIn(sys.hash_info.algorithm, {"fnv", "siphash24"}) else: # PY_HASH_EXTERNAL self.assertEqual(algo, 0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 01:48:00 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 13 Dec 2013 01:48:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Close_=2319576=3A_PyGILSta?= =?utf-8?q?te=5FEnsure=28=29_now_initializes_threads=2E_At_startup=2C_Pyth?= =?utf-8?q?on?= Message-ID: <3dgYBm5Rbqz7LlS@mail.python.org> http://hg.python.org/cpython/rev/dc4e805ec68a changeset: 87916:dc4e805ec68a user: Victor Stinner date: Fri Dec 13 01:46:43 2013 +0100 summary: Close #19576: PyGILState_Ensure() now initializes threads. At startup, Python has no concrete GIL. If PyGILState_Ensure() is called from a new thread for the first time and PyEval_InitThreads() was not called yet, a GIL needs to be created. files: Misc/NEWS | 5 +++++ Python/pystate.c | 5 +++++ 2 files changed, 10 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,11 @@ Core and Builtins ----------------- +- Issue #19576: PyGILState_Ensure() now initializes threads. At startup, Python + has no concrete GIL. If PyGILState_Ensure() is called from a new thread for + the first time and PyEval_InitThreads() was not called yet, a GIL needs to be + created. + - Issue #17576: Deprecation warning emitted now when __int__() or __index__() return not int instance. diff --git a/Python/pystate.c b/Python/pystate.c --- a/Python/pystate.c +++ b/Python/pystate.c @@ -771,6 +771,11 @@ assert(autoInterpreterState); /* Py_Initialize() hasn't been called! */ tcur = (PyThreadState *)PyThread_get_key_value(autoTLSkey); if (tcur == NULL) { + /* At startup, Python has no concrete GIL. If PyGILState_Ensure() is + called from a new thread for the first time, we need the create the + GIL. */ + PyEval_InitThreads(); + /* Create a new thread state for this thread */ tcur = PyThreadState_New(autoInterpreterState); if (tcur == NULL) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 02:08:43 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 13 Dec 2013 02:08:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2314432=3A_Remove_t?= =?utf-8?q?he_thread_state_field_from_the_frame_structure=2E_Fix_a?= Message-ID: <3dgYfg2RvYz7Ljf@mail.python.org> http://hg.python.org/cpython/rev/d2560fd8a008 changeset: 87917:d2560fd8a008 user: Victor Stinner date: Fri Dec 13 02:01:38 2013 +0100 summary: Issue #14432: Remove the thread state field from the frame structure. Fix a crash when a generator is created in a C thread that is destroyed while the generator is still used. The issue was that a generator contains a frame, and the frame kept a reference to the Python state of the destroyed C thread. The crash occurs when a trace function is setup. files: Include/frameobject.h | 1 - Lib/test/test_sys.py | 2 +- Lib/test/test_threading.py | 38 +++++++++++ Misc/NEWS | 6 + Modules/_testcapimodule.c | 89 ++++++++++++++++++++++++++ Objects/frameobject.c | 1 - Python/ceval.c | 79 ++++++++++++---------- Python/sysmodule.c | 8 +- 8 files changed, 179 insertions(+), 45 deletions(-) diff --git a/Include/frameobject.h b/Include/frameobject.h --- a/Include/frameobject.h +++ b/Include/frameobject.h @@ -39,7 +39,6 @@ /* Borrowed reference to a generator, or NULL */ PyObject *f_gen; - PyThreadState *f_tstate; int f_lasti; /* Last instruction if called */ /* Call PyFrame_GetLineNumber() instead of reading this field directly. As of 2.3 f_lineno is only valid when tracing is diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -818,7 +818,7 @@ nfrees = len(x.f_code.co_freevars) extras = x.f_code.co_stacksize + x.f_code.co_nlocals +\ ncells + nfrees - 1 - check(x, vsize('13P3ic' + CO_MAXBLOCKS*'3i' + 'P' + extras*'P')) + check(x, vsize('12P3ic' + CO_MAXBLOCKS*'3i' + 'P' + extras*'P')) # function def func(): pass check(func, size('12P')) diff --git a/Lib/test/test_threading.py b/Lib/test/test_threading.py --- a/Lib/test/test_threading.py +++ b/Lib/test/test_threading.py @@ -662,6 +662,44 @@ self.assertRegex(err.rstrip(), b"^sys:1: ResourceWarning: unclosed file ") + def test_frame_tstate_tracing(self): + # Issue #14432: Crash when a generator is created in a C thread that is + # destroyed while the generator is still used. The issue was that a + # generator contains a frame, and the frame kept a reference to the + # Python state of the destroyed C thread. The crash occurs when a trace + # function is setup. + + def noop_trace(frame, event, arg): + # no operation + return noop_trace + + def generator(): + while 1: + yield "genereator" + + def callback(): + if callback.gen is None: + callback.gen = generator() + return next(callback.gen) + callback.gen = None + + old_trace = sys.gettrace() + sys.settrace(noop_trace) + try: + # Install a trace function + threading.settrace(noop_trace) + + # Create a generator in a C thread which exits after the call + _testcapi.call_in_temporary_c_thread(callback) + + # Call the generator in a different Python thread, check that the + # generator didn't keep a reference to the destroyed thread state + for test in range(3): + # The trace function is still called here + callback() + finally: + sys.settrace(old_trace) + class ThreadJoinOnShutdown(BaseTestCase): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,12 @@ Core and Builtins ----------------- +- Issue #14432: Remove the thread state field from the frame structure. Fix a + crash when a generator is created in a C thread that is destroyed while the + generator is still used. The issue was that a generator contains a frame, and + the frame kept a reference to the Python state of the destroyed C thread. The + crash occurs when a trace function is setup. + - Issue #19576: PyGILState_Ensure() now initializes threads. At startup, Python has no concrete GIL. If PyGILState_Ensure() is called from a new thread for the first time and PyEval_InitThreads() was not called yet, a GIL needs to be diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -2869,6 +2869,93 @@ "This docstring has a valid signature and some extra newlines." ); +typedef struct { + PyThread_type_lock start_event; + PyThread_type_lock exit_event; + PyObject *callback; +} test_c_thread_t; + +static void +temporary_c_thread(void *data) +{ + test_c_thread_t *test_c_thread = data; + PyGILState_STATE state; + PyObject *res; + + PyThread_release_lock(test_c_thread->start_event); + + /* Allocate a Python thread state for this thread */ + state = PyGILState_Ensure(); + + res = PyObject_CallFunction(test_c_thread->callback, "", NULL); + Py_CLEAR(test_c_thread->callback); + + if (res == NULL) { + PyErr_Print(); + } + else { + Py_DECREF(res); + } + + /* Destroy the Python thread state for this thread */ + PyGILState_Release(state); + + PyThread_release_lock(test_c_thread->exit_event); + + PyThread_exit_thread(); +} + +static PyObject * +call_in_temporary_c_thread(PyObject *self, PyObject *callback) +{ + PyObject *res = NULL; + test_c_thread_t test_c_thread; + long thread; + + PyEval_InitThreads(); + + test_c_thread.start_event = PyThread_allocate_lock(); + test_c_thread.exit_event = PyThread_allocate_lock(); + test_c_thread.callback = NULL; + if (!test_c_thread.start_event || !test_c_thread.exit_event) { + PyErr_SetString(PyExc_RuntimeError, "could not allocate lock"); + goto exit; + } + + Py_INCREF(callback); + test_c_thread.callback = callback; + + PyThread_acquire_lock(test_c_thread.start_event, 1); + PyThread_acquire_lock(test_c_thread.exit_event, 1); + + thread = PyThread_start_new_thread(temporary_c_thread, &test_c_thread); + if (thread == -1) { + PyErr_SetString(PyExc_RuntimeError, "unable to start the thread"); + PyThread_release_lock(test_c_thread.start_event); + PyThread_release_lock(test_c_thread.exit_event); + goto exit; + } + + PyThread_acquire_lock(test_c_thread.start_event, 1); + PyThread_release_lock(test_c_thread.start_event); + + Py_BEGIN_ALLOW_THREADS + PyThread_acquire_lock(test_c_thread.exit_event, 1); + PyThread_release_lock(test_c_thread.exit_event); + Py_END_ALLOW_THREADS + + Py_INCREF(Py_None); + res = Py_None; + +exit: + Py_CLEAR(test_c_thread.callback); + if (test_c_thread.start_event) + PyThread_free_lock(test_c_thread.start_event); + if (test_c_thread.exit_event) + PyThread_free_lock(test_c_thread.exit_event); + return res; +} + static PyMethodDef TestMethods[] = { {"raise_exception", raise_exception, METH_VARARGS}, {"raise_memoryerror", (PyCFunction)raise_memoryerror, METH_NOARGS}, @@ -2997,6 +3084,8 @@ {"docstring_with_signature_and_extra_newlines", (PyCFunction)test_with_docstring, METH_NOARGS, docstring_with_signature_and_extra_newlines}, + {"call_in_temporary_c_thread", call_in_temporary_c_thread, METH_O, + PyDoc_STR("set_error_class(error_class) -> None")}, {NULL, NULL} /* sentinel */ }; diff --git a/Objects/frameobject.c b/Objects/frameobject.c --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -726,7 +726,6 @@ Py_INCREF(locals); f->f_locals = locals; } - f->f_tstate = tstate; f->f_lasti = -1; f->f_lineno = code->co_firstlineno; diff --git a/Python/ceval.c b/Python/ceval.c --- a/Python/ceval.c +++ b/Python/ceval.c @@ -123,13 +123,16 @@ static int lltrace; static int prtrace(PyObject *, char *); #endif -static int call_trace(Py_tracefunc, PyObject *, PyFrameObject *, +static int call_trace(Py_tracefunc, PyObject *, + PyThreadState *, PyFrameObject *, int, PyObject *); static int call_trace_protected(Py_tracefunc, PyObject *, - PyFrameObject *, int, PyObject *); -static void call_exc_trace(Py_tracefunc, PyObject *, PyFrameObject *); + PyThreadState *, PyFrameObject *, + int, PyObject *); +static void call_exc_trace(Py_tracefunc, PyObject *, + PyThreadState *, PyFrameObject *); static int maybe_call_line_trace(Py_tracefunc, PyObject *, - PyFrameObject *, int *, int *, int *); + PyThreadState *, PyFrameObject *, int *, int *, int *); static PyObject * cmp_outcome(int, PyObject *, PyObject *); static PyObject * import_from(PyObject *, PyObject *); @@ -1136,7 +1139,7 @@ whenever an exception is detected. */ if (call_trace_protected(tstate->c_tracefunc, tstate->c_traceobj, - f, PyTrace_CALL, Py_None)) { + tstate, f, PyTrace_CALL, Py_None)) { /* Trace function raised an error */ goto exit_eval_frame; } @@ -1146,7 +1149,7 @@ return itself and isn't called for "line" events */ if (call_trace_protected(tstate->c_profilefunc, tstate->c_profileobj, - f, PyTrace_CALL, Py_None)) { + tstate, f, PyTrace_CALL, Py_None)) { /* Profile function raised an error */ goto exit_eval_frame; } @@ -1293,8 +1296,8 @@ err = maybe_call_line_trace(tstate->c_tracefunc, tstate->c_traceobj, - f, &instr_lb, &instr_ub, - &instr_prev); + tstate, f, + &instr_lb, &instr_ub, &instr_prev); /* Reload possibly changed frame fields */ JUMPTO(f->f_lasti); if (f->f_stacktop != NULL) { @@ -1906,7 +1909,7 @@ PyObject *val; if (tstate->c_tracefunc != NULL && PyErr_ExceptionMatches(PyExc_StopIteration)) - call_exc_trace(tstate->c_tracefunc, tstate->c_traceobj, f); + call_exc_trace(tstate->c_tracefunc, tstate->c_traceobj, tstate, f); err = _PyGen_FetchStopIterationValue(&val); if (err < 0) goto error; @@ -2658,7 +2661,7 @@ if (!PyErr_ExceptionMatches(PyExc_StopIteration)) goto error; else if (tstate->c_tracefunc != NULL) - call_exc_trace(tstate->c_tracefunc, tstate->c_traceobj, f); + call_exc_trace(tstate->c_tracefunc, tstate->c_traceobj, tstate, f); PyErr_Clear(); } /* iterator ended normally */ @@ -3054,7 +3057,8 @@ PyTraceBack_Here(f); if (tstate->c_tracefunc != NULL) - call_exc_trace(tstate->c_tracefunc, tstate->c_traceobj, f); + call_exc_trace(tstate->c_tracefunc, tstate->c_traceobj, + tstate, f); fast_block_end: assert(why != WHY_NOT); @@ -3182,8 +3186,8 @@ if (tstate->use_tracing) { if (tstate->c_tracefunc) { if (why == WHY_RETURN || why == WHY_YIELD) { - if (call_trace(tstate->c_tracefunc, - tstate->c_traceobj, f, + if (call_trace(tstate->c_tracefunc, tstate->c_traceobj, + tstate, f, PyTrace_RETURN, retval)) { Py_XDECREF(retval); retval = NULL; @@ -3191,18 +3195,19 @@ } } else if (why == WHY_EXCEPTION) { - call_trace_protected(tstate->c_tracefunc, - tstate->c_traceobj, f, + call_trace_protected(tstate->c_tracefunc, tstate->c_traceobj, + tstate, f, PyTrace_RETURN, NULL); } } if (tstate->c_profilefunc) { if (why == WHY_EXCEPTION) call_trace_protected(tstate->c_profilefunc, - tstate->c_profileobj, f, + tstate->c_profileobj, + tstate, f, PyTrace_RETURN, NULL); - else if (call_trace(tstate->c_profilefunc, - tstate->c_profileobj, f, + else if (call_trace(tstate->c_profilefunc, tstate->c_profileobj, + tstate, f, PyTrace_RETURN, retval)) { Py_XDECREF(retval); retval = NULL; @@ -3846,7 +3851,8 @@ #endif static void -call_exc_trace(Py_tracefunc func, PyObject *self, PyFrameObject *f) +call_exc_trace(Py_tracefunc func, PyObject *self, + PyThreadState *tstate, PyFrameObject *f) { PyObject *type, *value, *traceback, *orig_traceback, *arg; int err; @@ -3862,7 +3868,7 @@ PyErr_Restore(type, value, orig_traceback); return; } - err = call_trace(func, self, f, PyTrace_EXCEPTION, arg); + err = call_trace(func, self, tstate, f, PyTrace_EXCEPTION, arg); Py_DECREF(arg); if (err == 0) PyErr_Restore(type, value, orig_traceback); @@ -3874,13 +3880,14 @@ } static int -call_trace_protected(Py_tracefunc func, PyObject *obj, PyFrameObject *frame, +call_trace_protected(Py_tracefunc func, PyObject *obj, + PyThreadState *tstate, PyFrameObject *frame, int what, PyObject *arg) { PyObject *type, *value, *traceback; int err; PyErr_Fetch(&type, &value, &traceback); - err = call_trace(func, obj, frame, what, arg); + err = call_trace(func, obj, tstate, frame, what, arg); if (err == 0) { PyErr_Restore(type, value, traceback); @@ -3895,10 +3902,10 @@ } static int -call_trace(Py_tracefunc func, PyObject *obj, PyFrameObject *frame, +call_trace(Py_tracefunc func, PyObject *obj, + PyThreadState *tstate, PyFrameObject *frame, int what, PyObject *arg) { - PyThreadState *tstate = frame->f_tstate; int result; if (tstate->tracing) return 0; @@ -3914,8 +3921,7 @@ PyObject * _PyEval_CallTracing(PyObject *func, PyObject *args) { - PyFrameObject *frame = PyEval_GetFrame(); - PyThreadState *tstate = frame->f_tstate; + PyThreadState *tstate = PyThreadState_GET(); int save_tracing = tstate->tracing; int save_use_tracing = tstate->use_tracing; PyObject *result; @@ -3932,8 +3938,8 @@ /* See Objects/lnotab_notes.txt for a description of how tracing works. */ static int maybe_call_line_trace(Py_tracefunc func, PyObject *obj, - PyFrameObject *frame, int *instr_lb, int *instr_ub, - int *instr_prev) + PyThreadState *tstate, PyFrameObject *frame, + int *instr_lb, int *instr_ub, int *instr_prev) { int result = 0; int line = frame->f_lineno; @@ -3953,7 +3959,7 @@ number and call the trace function. */ if (frame->f_lasti == *instr_lb || frame->f_lasti < *instr_prev) { frame->f_lineno = line; - result = call_trace(func, obj, frame, PyTrace_LINE, Py_None); + result = call_trace(func, obj, tstate, frame, PyTrace_LINE, Py_None); } *instr_prev = frame->f_lasti; return result; @@ -4149,10 +4155,9 @@ #define C_TRACE(x, call) \ if (tstate->use_tracing && tstate->c_profilefunc) { \ - if (call_trace(tstate->c_profilefunc, \ - tstate->c_profileobj, \ - tstate->frame, PyTrace_C_CALL, \ - func)) { \ + if (call_trace(tstate->c_profilefunc, tstate->c_profileobj, \ + tstate, tstate->frame, \ + PyTrace_C_CALL, func)) { \ x = NULL; \ } \ else { \ @@ -4161,14 +4166,14 @@ if (x == NULL) { \ call_trace_protected(tstate->c_profilefunc, \ tstate->c_profileobj, \ - tstate->frame, PyTrace_C_EXCEPTION, \ - func); \ + tstate, tstate->frame, \ + PyTrace_C_EXCEPTION, func); \ /* XXX should pass (type, value, tb) */ \ } else { \ if (call_trace(tstate->c_profilefunc, \ tstate->c_profileobj, \ - tstate->frame, PyTrace_C_RETURN, \ - func)) { \ + tstate, tstate->frame, \ + PyTrace_C_RETURN, func)) { \ Py_DECREF(x); \ x = NULL; \ } \ diff --git a/Python/sysmodule.c b/Python/sysmodule.c --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -367,7 +367,7 @@ static PyObject * -call_trampoline(PyThreadState *tstate, PyObject* callback, +call_trampoline(PyObject* callback, PyFrameObject *frame, int what, PyObject *arg) { PyObject *args; @@ -405,12 +405,11 @@ profile_trampoline(PyObject *self, PyFrameObject *frame, int what, PyObject *arg) { - PyThreadState *tstate = frame->f_tstate; PyObject *result; if (arg == NULL) arg = Py_None; - result = call_trampoline(tstate, self, frame, what, arg); + result = call_trampoline(self, frame, what, arg); if (result == NULL) { PyEval_SetProfile(NULL, NULL); return -1; @@ -423,7 +422,6 @@ trace_trampoline(PyObject *self, PyFrameObject *frame, int what, PyObject *arg) { - PyThreadState *tstate = frame->f_tstate; PyObject *callback; PyObject *result; @@ -433,7 +431,7 @@ callback = frame->f_trace; if (callback == NULL) return 0; - result = call_trampoline(tstate, callback, frame, what, arg); + result = call_trampoline(callback, frame, what, arg); if (result == NULL) { PyEval_SetTrace(NULL, NULL); Py_XDECREF(frame->f_trace); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 02:19:33 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 13 Dec 2013 02:19:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE0NDMy?= =?utf-8?q?=3A_Generator_now_clears_the_borrowed_reference_to_the_thread_s?= =?utf-8?q?tate?= Message-ID: <3dgYv96cXRz7LlB@mail.python.org> http://hg.python.org/cpython/rev/0875e5bbe5f0 changeset: 87918:0875e5bbe5f0 branch: 3.3 parent: 87912:f95f5769f158 user: Victor Stinner date: Fri Dec 13 02:17:29 2013 +0100 summary: Issue #14432: Generator now clears the borrowed reference to the thread state Fix a crash when a generator is created in a C thread that is destroyed while the generator is still used. The issue was that a generator contains a frame, and the frame kept a reference to the Python state of the destroyed C thread. The crash occurs when a trace function is setup. files: Lib/test/test_threading.py | 43 ++++++++++++ Misc/NEWS | 6 + Modules/_testcapimodule.c | 89 ++++++++++++++++++++++++++ Objects/genobject.c | 3 + 4 files changed, 141 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_threading.py b/Lib/test/test_threading.py --- a/Lib/test/test_threading.py +++ b/Lib/test/test_threading.py @@ -17,6 +17,10 @@ import os from test.script_helper import assert_python_ok, assert_python_failure import subprocess +try: + import _testcapi +except ImportError: + _testcapi = None from test import lock_tests @@ -769,6 +773,45 @@ for t in threads: t.join() + @unittest.skipIf(_testcapi is None, "need _testcapi module") + def test_frame_tstate_tracing(self): + # Issue #14432: Crash when a generator is created in a C thread that is + # destroyed while the generator is still used. The issue was that a + # generator contains a frame, and the frame kept a reference to the + # Python state of the destroyed C thread. The crash occurs when a trace + # function is setup. + + def noop_trace(frame, event, arg): + # no operation + return noop_trace + + def generator(): + while 1: + yield "genereator" + + def callback(): + if callback.gen is None: + callback.gen = generator() + return next(callback.gen) + callback.gen = None + + old_trace = sys.gettrace() + sys.settrace(noop_trace) + try: + # Install a trace function + threading.settrace(noop_trace) + + # Create a generator in a C thread which exits after the call + _testcapi.call_in_temporary_c_thread(callback) + + # Call the generator in a different Python thread, check that the + # generator didn't keep a reference to the destroyed thread state + for test in range(3): + # The trace function is still called here + callback() + finally: + sys.settrace(old_trace) + class ThreadingExceptionTests(BaseTestCase): # A RuntimeError should be raised if Thread.start() is called diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,12 @@ Core and Builtins ----------------- +- Issue #14432: Generator now clears the borrowed reference to the thread + state. Fix a crash when a generator is created in a C thread that is + destroyed while the generator is still used. The issue was that a generator + contains a frame, and the frame kept a reference to the Python state of the + destroyed C thread. The crash occurs when a trace function is setup. + - Issue #17576: Deprecation warning emitted now when __int__() or __index__() return not int instance. diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -2477,6 +2477,93 @@ return Py_BuildValue("Nl", _PyLong_FromTime_t(sec), nsec); } +typedef struct { + PyThread_type_lock start_event; + PyThread_type_lock exit_event; + PyObject *callback; +} test_c_thread_t; + +static void +temporary_c_thread(void *data) +{ + test_c_thread_t *test_c_thread = data; + PyGILState_STATE state; + PyObject *res; + + PyThread_release_lock(test_c_thread->start_event); + + /* Allocate a Python thread state for this thread */ + state = PyGILState_Ensure(); + + res = PyObject_CallFunction(test_c_thread->callback, "", NULL); + Py_CLEAR(test_c_thread->callback); + + if (res == NULL) { + PyErr_Print(); + } + else { + Py_DECREF(res); + } + + /* Destroy the Python thread state for this thread */ + PyGILState_Release(state); + + PyThread_release_lock(test_c_thread->exit_event); + + PyThread_exit_thread(); +} + +static PyObject * +call_in_temporary_c_thread(PyObject *self, PyObject *callback) +{ + PyObject *res = NULL; + test_c_thread_t test_c_thread; + long thread; + + PyEval_InitThreads(); + + test_c_thread.start_event = PyThread_allocate_lock(); + test_c_thread.exit_event = PyThread_allocate_lock(); + test_c_thread.callback = NULL; + if (!test_c_thread.start_event || !test_c_thread.exit_event) { + PyErr_SetString(PyExc_RuntimeError, "could not allocate lock"); + goto exit; + } + + Py_INCREF(callback); + test_c_thread.callback = callback; + + PyThread_acquire_lock(test_c_thread.start_event, 1); + PyThread_acquire_lock(test_c_thread.exit_event, 1); + + thread = PyThread_start_new_thread(temporary_c_thread, &test_c_thread); + if (thread == -1) { + PyErr_SetString(PyExc_RuntimeError, "unable to start the thread"); + PyThread_release_lock(test_c_thread.start_event); + PyThread_release_lock(test_c_thread.exit_event); + goto exit; + } + + PyThread_acquire_lock(test_c_thread.start_event, 1); + PyThread_release_lock(test_c_thread.start_event); + + Py_BEGIN_ALLOW_THREADS + PyThread_acquire_lock(test_c_thread.exit_event, 1); + PyThread_release_lock(test_c_thread.exit_event); + Py_END_ALLOW_THREADS + + Py_INCREF(Py_None); + res = Py_None; + +exit: + Py_CLEAR(test_c_thread.callback); + if (test_c_thread.start_event) + PyThread_free_lock(test_c_thread.start_event); + if (test_c_thread.exit_event) + PyThread_free_lock(test_c_thread.exit_event); + return res; +} + static PyMethodDef TestMethods[] = { {"raise_exception", raise_exception, METH_VARARGS}, @@ -2574,6 +2661,8 @@ {"pytime_object_to_time_t", test_pytime_object_to_time_t, METH_VARARGS}, {"pytime_object_to_timeval", test_pytime_object_to_timeval, METH_VARARGS}, {"pytime_object_to_timespec", test_pytime_object_to_timespec, METH_VARARGS}, + {"call_in_temporary_c_thread", call_in_temporary_c_thread, METH_O, + PyDoc_STR("set_error_class(error_class) -> None")}, {NULL, NULL} /* sentinel */ }; diff --git a/Objects/genobject.c b/Objects/genobject.c --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -76,6 +76,7 @@ /* Generators always return to their most recent caller, not * necessarily their creator. */ + f->f_tstate = tstate; Py_XINCREF(tstate->frame); assert(f->f_back == NULL); f->f_back = tstate->frame; @@ -89,6 +90,8 @@ * cycle. */ assert(f->f_back == tstate->frame); Py_CLEAR(f->f_back); + /* Clear the borrowed reference to the thread state */ + f->f_tstate = NULL; /* If the generator just returned (as opposed to yielding), signal * that the generator is exhausted. */ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 02:19:35 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 13 Dec 2013 02:19:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2314432=3A_Null_merge_3=2E3=2C_Python_3=2E4_has_a?= =?utf-8?q?_different_fix?= Message-ID: <3dgYvC1d2zz7LlS@mail.python.org> http://hg.python.org/cpython/rev/55dd094a2b01 changeset: 87919:55dd094a2b01 parent: 87917:d2560fd8a008 parent: 87918:0875e5bbe5f0 user: Victor Stinner date: Fri Dec 13 02:18:41 2013 +0100 summary: Issue #14432: Null merge 3.3, Python 3.4 has a different fix files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 02:33:14 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 13 Dec 2013 02:33:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE0NDMy?= =?utf-8?q?=3A_Fix_compilation_when_thread_support_is_disabled?= Message-ID: <3dgZBy0dp4z7LjP@mail.python.org> http://hg.python.org/cpython/rev/2f975036cf39 changeset: 87920:2f975036cf39 branch: 3.3 parent: 87918:0875e5bbe5f0 user: Victor Stinner date: Fri Dec 13 02:30:12 2013 +0100 summary: Issue #14432: Fix compilation when thread support is disabled files: Modules/_testcapimodule.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -2477,6 +2477,7 @@ return Py_BuildValue("Nl", _PyLong_FromTime_t(sec), nsec); } +#ifdef WITH_THREAD typedef struct { PyThread_type_lock start_event; PyThread_type_lock exit_event; @@ -2563,6 +2564,7 @@ PyThread_free_lock(test_c_thread.exit_event); return res; } +#endif /* WITH_THREAD */ static PyMethodDef TestMethods[] = { @@ -2661,8 +2663,10 @@ {"pytime_object_to_time_t", test_pytime_object_to_time_t, METH_VARARGS}, {"pytime_object_to_timeval", test_pytime_object_to_timeval, METH_VARARGS}, {"pytime_object_to_timespec", test_pytime_object_to_timespec, METH_VARARGS}, +#ifdef WITH_THREAD {"call_in_temporary_c_thread", call_in_temporary_c_thread, METH_O, PyDoc_STR("set_error_class(error_class) -> None")}, +#endif {NULL, NULL} /* sentinel */ }; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 02:33:15 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 13 Dec 2013 02:33:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E3=29_Issue_=2314432=3A_Fix_compilation_when?= =?utf-8?q?_thread_support_is_disabled?= Message-ID: <3dgZBz2Qfhz7Llf@mail.python.org> http://hg.python.org/cpython/rev/9852637f05c3 changeset: 87921:9852637f05c3 parent: 87919:55dd094a2b01 parent: 87920:2f975036cf39 user: Victor Stinner date: Fri Dec 13 02:32:37 2013 +0100 summary: (Merge 3.3) Issue #14432: Fix compilation when thread support is disabled files: Modules/_testcapimodule.c | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -2869,6 +2869,7 @@ "This docstring has a valid signature and some extra newlines." ); +#ifdef WITH_THREAD typedef struct { PyThread_type_lock start_event; PyThread_type_lock exit_event; @@ -2955,6 +2956,8 @@ PyThread_free_lock(test_c_thread.exit_event); return res; } +#endif /* WITH_THREAD */ + static PyMethodDef TestMethods[] = { {"raise_exception", raise_exception, METH_VARARGS}, @@ -3084,8 +3087,10 @@ {"docstring_with_signature_and_extra_newlines", (PyCFunction)test_with_docstring, METH_NOARGS, docstring_with_signature_and_extra_newlines}, +#ifdef WITH_THREAD {"call_in_temporary_c_thread", call_in_temporary_c_thread, METH_O, PyDoc_STR("set_error_class(error_class) -> None")}, +#endif {NULL, NULL} /* sentinel */ }; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 02:39:40 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 13 Dec 2013 02:39:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE0NDMy?= =?utf-8?q?=3A_Generator_now_clears_the_borrowed_reference_to_the_thread_s?= =?utf-8?q?tate?= Message-ID: <3dgZLN0YHFz7LlP@mail.python.org> http://hg.python.org/cpython/rev/aa324af42c0e changeset: 87922:aa324af42c0e branch: 2.7 parent: 87895:d61e8050b7d7 user: Victor Stinner date: Fri Dec 13 02:37:09 2013 +0100 summary: Issue #14432: Generator now clears the borrowed reference to the thread state Fix a crash when a generator is created in a C thread that is destroyed while the generator is still used. The issue was that a generator contains a frame, and the frame kept a reference to the Python state of the destroyed C thread. The crash occurs when a trace function is setup. files: Lib/test/test_threading.py | 43 +++++++++++ Misc/NEWS | 6 + Modules/_testcapimodule.c | 94 ++++++++++++++++++++++++++ Objects/genobject.c | 3 + 4 files changed, 146 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_threading.py b/Lib/test/test_threading.py --- a/Lib/test/test_threading.py +++ b/Lib/test/test_threading.py @@ -14,6 +14,10 @@ import weakref import os import subprocess +try: + import _testcapi +except ImportError: + _testcapi = None from test import lock_tests @@ -720,6 +724,45 @@ for t in threads: t.join() + @unittest.skipIf(_testcapi is None, "need _testcapi module") + def test_frame_tstate_tracing(self): + # Issue #14432: Crash when a generator is created in a C thread that is + # destroyed while the generator is still used. The issue was that a + # generator contains a frame, and the frame kept a reference to the + # Python state of the destroyed C thread. The crash occurs when a trace + # function is setup. + + def noop_trace(frame, event, arg): + # no operation + return noop_trace + + def generator(): + while 1: + yield "genereator" + + def callback(): + if callback.gen is None: + callback.gen = generator() + return next(callback.gen) + callback.gen = None + + old_trace = sys.gettrace() + sys.settrace(noop_trace) + try: + # Install a trace function + threading.settrace(noop_trace) + + # Create a generator in a C thread which exits after the call + _testcapi.call_in_temporary_c_thread(callback) + + # Call the generator in a different Python thread, check that the + # generator didn't keep a reference to the destroyed thread state + for test in range(3): + # The trace function is still called here + callback() + finally: + sys.settrace(old_trace) + class ThreadingExceptionTests(BaseTestCase): # A RuntimeError should be raised if Thread.start() is called diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -9,6 +9,12 @@ Core and Builtins ----------------- +- Issue #14432: Generator now clears the borrowed reference to the thread + state. Fix a crash when a generator is created in a C thread that is + destroyed while the generator is still used. The issue was that a generator + contains a frame, and the frame kept a reference to the Python state of the + destroyed C thread. The crash occurs when a trace function is setup. + - Issue #19932: Fix typo in import.h, missing whitespaces in function prototypes. - Issue #19638: Fix possible crash / undefined behaviour from huge (more than 2 diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -1687,6 +1687,96 @@ Py_RETURN_NONE; } +#ifdef WITH_THREAD +typedef struct { + PyThread_type_lock start_event; + PyThread_type_lock exit_event; + PyObject *callback; +} test_c_thread_t; + +static void +temporary_c_thread(void *data) +{ + test_c_thread_t *test_c_thread = data; + PyGILState_STATE state; + PyObject *res; + + PyThread_release_lock(test_c_thread->start_event); + + /* Allocate a Python thread state for this thread */ + state = PyGILState_Ensure(); + + res = PyObject_CallFunction(test_c_thread->callback, "", NULL); + Py_CLEAR(test_c_thread->callback); + + if (res == NULL) { + PyErr_Print(); + } + else { + Py_DECREF(res); + } + + /* Destroy the Python thread state for this thread */ + PyGILState_Release(state); + + PyThread_release_lock(test_c_thread->exit_event); + + PyThread_exit_thread(); +} + +static PyObject * +call_in_temporary_c_thread(PyObject *self, PyObject *callback) +{ + PyObject *res = NULL; + test_c_thread_t test_c_thread; + long thread; + + PyEval_InitThreads(); + + test_c_thread.start_event = PyThread_allocate_lock(); + test_c_thread.exit_event = PyThread_allocate_lock(); + test_c_thread.callback = NULL; + if (!test_c_thread.start_event || !test_c_thread.exit_event) { + PyErr_SetString(PyExc_RuntimeError, "could not allocate lock"); + goto exit; + } + + Py_INCREF(callback); + test_c_thread.callback = callback; + + PyThread_acquire_lock(test_c_thread.start_event, 1); + PyThread_acquire_lock(test_c_thread.exit_event, 1); + + thread = PyThread_start_new_thread(temporary_c_thread, &test_c_thread); + if (thread == -1) { + PyErr_SetString(PyExc_RuntimeError, "unable to start the thread"); + PyThread_release_lock(test_c_thread.start_event); + PyThread_release_lock(test_c_thread.exit_event); + goto exit; + } + + PyThread_acquire_lock(test_c_thread.start_event, 1); + PyThread_release_lock(test_c_thread.start_event); + + Py_BEGIN_ALLOW_THREADS + PyThread_acquire_lock(test_c_thread.exit_event, 1); + PyThread_release_lock(test_c_thread.exit_event); + Py_END_ALLOW_THREADS + + Py_INCREF(Py_None); + res = Py_None; + +exit: + Py_CLEAR(test_c_thread.callback); + if (test_c_thread.start_event) + PyThread_free_lock(test_c_thread.start_event); + if (test_c_thread.exit_event) + PyThread_free_lock(test_c_thread.exit_event); + return res; +} +#endif /* WITH_THREAD */ + + static PyMethodDef TestMethods[] = { {"raise_exception", raise_exception, METH_VARARGS}, {"test_config", (PyCFunction)test_config, METH_NOARGS}, @@ -1745,6 +1835,10 @@ {"make_exception_with_doc", (PyCFunction)make_exception_with_doc, METH_VARARGS | METH_KEYWORDS}, {"sequence_delitem", (PyCFunction)sequence_delitem, METH_VARARGS}, +#ifdef WITH_THREAD + {"call_in_temporary_c_thread", call_in_temporary_c_thread, METH_O, + PyDoc_STR("set_error_class(error_class) -> None")}, +#endif {NULL, NULL} /* sentinel */ }; diff --git a/Objects/genobject.c b/Objects/genobject.c --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -76,6 +76,7 @@ /* Generators always return to their most recent caller, not * necessarily their creator. */ + f->f_tstate = tstate; Py_XINCREF(tstate->frame); assert(f->f_back == NULL); f->f_back = tstate->frame; @@ -89,6 +90,8 @@ * cycle. */ assert(f->f_back == tstate->frame); Py_CLEAR(f->f_back); + /* Clear the borrowed reference to the thread state */ + f->f_tstate = NULL; /* If the generator just returned (as opposed to yielding), signal * that the generator is exhausted. */ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 02:45:30 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 13 Dec 2013 02:45:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319952=3A_test=5Fa?= =?utf-8?q?syncio=3A_relax_timings_of_Windows_events=2C_buildbots_are?= Message-ID: <3dgZT66c2Vz7LlS@mail.python.org> http://hg.python.org/cpython/rev/a35b2d652449 changeset: 87923:a35b2d652449 parent: 87921:9852637f05c3 user: Victor Stinner date: Fri Dec 13 02:45:18 2013 +0100 summary: Issue #19952: test_asyncio: relax timings of Windows events, buildbots are sometimes busy files: Lib/test/test_asyncio/test_windows_events.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_asyncio/test_windows_events.py b/Lib/test/test_asyncio/test_windows_events.py --- a/Lib/test/test_asyncio/test_windows_events.py +++ b/Lib/test/test_asyncio/test_windows_events.py @@ -120,7 +120,7 @@ self.loop.run_until_complete(f) elapsed = self.loop.time() - start self.assertTrue(f.result()) - self.assertTrue(0 <= elapsed < 0.02, elapsed) + self.assertTrue(0 <= elapsed < 0.1, elapsed) _overlapped.ResetEvent(event) @@ -132,7 +132,7 @@ with self.assertRaises(futures.CancelledError): self.loop.run_until_complete(f) elapsed = self.loop.time() - start - self.assertTrue(0 <= elapsed < 0.02, elapsed) + self.assertTrue(0 <= elapsed < 0.1, elapsed) if __name__ == '__main__': -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 03:32:06 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 13 Dec 2013 03:32:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Close_=2319787=3A_PyThread?= =?utf-8?q?=5Fset=5Fkey=5Fvalue=28=29_now_always_set_the_value=2E_In_Pytho?= =?utf-8?b?biAzLjMs?= Message-ID: <3dgbVt4tDnz7Llv@mail.python.org> http://hg.python.org/cpython/rev/46393019b650 changeset: 87924:46393019b650 user: Victor Stinner date: Fri Dec 13 03:22:00 2013 +0100 summary: Close #19787: PyThread_set_key_value() now always set the value. In Python 3.3, the function did nothing if the key already exists (if the current value is a non-NULL pointer). _testcapi.run_in_subinterp() now correctly sets the new Python thread state of the current thread when a subinterpreter is created. files: Doc/whatsnew/3.4.rst | 5 +++++ Misc/NEWS | 4 ++++ Modules/_testcapimodule.c | 4 ++++ Modules/_tracemalloc.c | 9 +++------ Python/thread.c | 20 ++++++++------------ Python/thread_nt.h | 9 --------- Python/thread_pthread.h | 3 --- 7 files changed, 24 insertions(+), 30 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1068,3 +1068,8 @@ working directory will also now have an absolute path, including when using ``-m`` with the interpreter (this does not influence when the path to a file is specified on the command-line). + +* (C API) :c:func:`PyThread_set_key_value` now always set the value. In Python + 3.3, the function did nothing if the key already exists (if the current + value is a non-NULL pointer). + diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,10 @@ Core and Builtins ----------------- +- Issue #19787: PyThread_set_key_value() now always set the value. In Python + 3.3, the function did nothing if the key already exists (if the current value + is a non-NULL pointer). + - Issue #14432: Remove the thread state field from the frame structure. Fix a crash when a generator is created in a C thread that is destroyed while the generator is still used. The issue was that a generator contains a frame, and diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -2511,6 +2511,10 @@ r = PyRun_SimpleString(code); Py_EndInterpreter(substate); + /* restore previous thread safe. It was replaced by Py_NewInterpreter() + which creates a new thread state. */ + _PyThreadState_Init(mainstate); + PyThreadState_Swap(mainstate); return PyLong_FromLong(r); diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -168,14 +168,11 @@ assert(reentrant == 0 || reentrant == 1); if (reentrant) { assert(PyThread_get_key_value(tracemalloc_reentrant_key) == NULL); - PyThread_set_key_value(tracemalloc_reentrant_key, - REENTRANT); + PyThread_set_key_value(tracemalloc_reentrant_key, REENTRANT); } else { - /* FIXME: PyThread_set_key_value() cannot be used to set the flag - to zero, because it does nothing if the variable has already - a value set. */ - PyThread_delete_key_value(tracemalloc_reentrant_key); + assert(PyThread_get_key_value(tracemalloc_reentrant_key) == REENTRANT); + PyThread_set_key_value(tracemalloc_reentrant_key, NULL); } } diff --git a/Python/thread.c b/Python/thread.c --- a/Python/thread.c +++ b/Python/thread.c @@ -205,7 +205,7 @@ * segfaults. Now we lock the whole routine. */ static struct key * -find_key(int key, void *value) +find_key(int key, int update, void *value) { struct key *p, *prev_p; long id = PyThread_get_thread_ident(); @@ -215,8 +215,11 @@ PyThread_acquire_lock(keymutex, 1); prev_p = NULL; for (p = keyhead; p != NULL; p = p->next) { - if (p->id == id && p->key == key) + if (p->id == id && p->key == key) { + if (update) + p->value = value; goto Done; + } /* Sanity check. These states should never happen but if * they do we must abort. Otherwise we'll end up spinning in * in a tight loop with the lock held. A similar check is done @@ -227,7 +230,7 @@ if (p->next == keyhead) Py_FatalError("tls find_key: circular list(!)"); } - if (value == NULL) { + if (!update && value == NULL) { assert(p == NULL); goto Done; } @@ -279,19 +282,12 @@ PyThread_release_lock(keymutex); } -/* Confusing: If the current thread has an association for key, - * value is ignored, and 0 is returned. Else an attempt is made to create - * an association of key to value for the current thread. 0 is returned - * if that succeeds, but -1 is returned if there's not enough memory - * to create the association. value must not be NULL. - */ int PyThread_set_key_value(int key, void *value) { struct key *p; - assert(value != NULL); - p = find_key(key, value); + p = find_key(key, 1, value); if (p == NULL) return -1; else @@ -304,7 +300,7 @@ void * PyThread_get_key_value(int key) { - struct key *p = find_key(key, NULL); + struct key *p = find_key(key, 0, NULL); if (p == NULL) return NULL; diff --git a/Python/thread_nt.h b/Python/thread_nt.h --- a/Python/thread_nt.h +++ b/Python/thread_nt.h @@ -389,20 +389,11 @@ TlsFree(key); } -/* We must be careful to emulate the strange semantics implemented in thread.c, - * where the value is only set if it hasn't been set before. - */ int PyThread_set_key_value(int key, void *value) { BOOL ok; - void *oldvalue; - assert(value != NULL); - oldvalue = TlsGetValue(key); - if (oldvalue != NULL) - /* ignore value if already set */ - return 0; ok = TlsSetValue(key, value); if (!ok) return -1; diff --git a/Python/thread_pthread.h b/Python/thread_pthread.h --- a/Python/thread_pthread.h +++ b/Python/thread_pthread.h @@ -627,9 +627,6 @@ PyThread_set_key_value(int key, void *value) { int fail; - void *oldValue = pthread_getspecific(key); - if (oldValue != NULL) - return 0; fail = pthread_setspecific(key, value); return fail ? -1 : 0; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 03:41:19 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 13 Dec 2013 03:41:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_=2319957=3A_Simplify_encod?= =?utf-8?q?e=5F7or8bit_now_that_=5Fpayload_is_always_str=2E?= Message-ID: <3dgbjW1Tw7z7Lpm@mail.python.org> http://hg.python.org/cpython/rev/29e5dd1f608a changeset: 87925:29e5dd1f608a user: R David Murray date: Thu Dec 12 21:40:20 2013 -0500 summary: #19957: Simplify encode_7or8bit now that _payload is always str. Patch by Vajrasky Kok, test enhancement by me. files: Lib/email/encoders.py | 17 ++++------------- Lib/test/test_email/test_email.py | 9 ++++++++- 2 files changed, 12 insertions(+), 14 deletions(-) diff --git a/Lib/email/encoders.py b/Lib/email/encoders.py --- a/Lib/email/encoders.py +++ b/Lib/email/encoders.py @@ -54,21 +54,12 @@ # There's no payload. For backwards compatibility we use 7bit msg['Content-Transfer-Encoding'] = '7bit' return - # We play a trick to make this go fast. If encoding/decode to ASCII - # succeeds, we know the data must be 7bit, otherwise treat it as 8bit. + # We play a trick to make this go fast. If decoding from ASCII succeeds, + # we know the data must be 7bit, otherwise treat it as 8bit. try: - if isinstance(orig, str): - orig.encode('ascii') - else: - orig.decode('ascii') + orig.decode('ascii') except UnicodeError: - charset = msg.get_charset() - output_cset = charset and charset.output_charset - # iso-2022-* is non-ASCII but encodes to a 7-bit representation - if output_cset and output_cset.lower().startswith('iso-2022-'): - msg['Content-Transfer-Encoding'] = '7bit' - else: - msg['Content-Transfer-Encoding'] = '8bit' + msg['Content-Transfer-Encoding'] = '8bit' else: msg['Content-Transfer-Encoding'] = '7bit' diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -765,8 +765,15 @@ # whose output character set is 7bit gets a transfer-encoding # of 7bit. eq = self.assertEqual - msg = MIMEText('?', _charset='euc-jp') + msg = MIMEText('?\n', _charset='euc-jp') eq(msg['content-transfer-encoding'], '7bit') + eq(msg.as_string(), textwrap.dedent("""\ + MIME-Version: 1.0 + Content-Type: text/plain; charset="iso-2022-jp" + Content-Transfer-Encoding: 7bit + + \x1b$BJ8\x1b(B + """)) def test_qp_encode_latin1(self): msg = MIMEText('\xe1\xf6\n', 'text', 'ISO-8859-1') -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 04:15:20 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 13 Dec 2013 04:15:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Backout_changeset_46393019?= =?utf-8?q?b650?= Message-ID: <3dgcSm5zSNz7LmL@mail.python.org> http://hg.python.org/cpython/rev/a9feb69ec424 changeset: 87926:a9feb69ec424 user: Victor Stinner date: Fri Dec 13 04:14:41 2013 +0100 summary: Backout changeset 46393019b650 test_capi is failing and the fix is not trivial, I prefer to revert files: Doc/whatsnew/3.4.rst | 5 ----- Misc/NEWS | 4 ---- Modules/_testcapimodule.c | 4 ---- Modules/_tracemalloc.c | 9 ++++++--- Python/thread.c | 20 ++++++++++++-------- Python/thread_nt.h | 9 +++++++++ Python/thread_pthread.h | 3 +++ 7 files changed, 30 insertions(+), 24 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1068,8 +1068,3 @@ working directory will also now have an absolute path, including when using ``-m`` with the interpreter (this does not influence when the path to a file is specified on the command-line). - -* (C API) :c:func:`PyThread_set_key_value` now always set the value. In Python - 3.3, the function did nothing if the key already exists (if the current - value is a non-NULL pointer). - diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,10 +10,6 @@ Core and Builtins ----------------- -- Issue #19787: PyThread_set_key_value() now always set the value. In Python - 3.3, the function did nothing if the key already exists (if the current value - is a non-NULL pointer). - - Issue #14432: Remove the thread state field from the frame structure. Fix a crash when a generator is created in a C thread that is destroyed while the generator is still used. The issue was that a generator contains a frame, and diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -2511,10 +2511,6 @@ r = PyRun_SimpleString(code); Py_EndInterpreter(substate); - /* restore previous thread safe. It was replaced by Py_NewInterpreter() - which creates a new thread state. */ - _PyThreadState_Init(mainstate); - PyThreadState_Swap(mainstate); return PyLong_FromLong(r); diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -168,11 +168,14 @@ assert(reentrant == 0 || reentrant == 1); if (reentrant) { assert(PyThread_get_key_value(tracemalloc_reentrant_key) == NULL); - PyThread_set_key_value(tracemalloc_reentrant_key, REENTRANT); + PyThread_set_key_value(tracemalloc_reentrant_key, + REENTRANT); } else { - assert(PyThread_get_key_value(tracemalloc_reentrant_key) == REENTRANT); - PyThread_set_key_value(tracemalloc_reentrant_key, NULL); + /* FIXME: PyThread_set_key_value() cannot be used to set the flag + to zero, because it does nothing if the variable has already + a value set. */ + PyThread_delete_key_value(tracemalloc_reentrant_key); } } diff --git a/Python/thread.c b/Python/thread.c --- a/Python/thread.c +++ b/Python/thread.c @@ -205,7 +205,7 @@ * segfaults. Now we lock the whole routine. */ static struct key * -find_key(int key, int update, void *value) +find_key(int key, void *value) { struct key *p, *prev_p; long id = PyThread_get_thread_ident(); @@ -215,11 +215,8 @@ PyThread_acquire_lock(keymutex, 1); prev_p = NULL; for (p = keyhead; p != NULL; p = p->next) { - if (p->id == id && p->key == key) { - if (update) - p->value = value; + if (p->id == id && p->key == key) goto Done; - } /* Sanity check. These states should never happen but if * they do we must abort. Otherwise we'll end up spinning in * in a tight loop with the lock held. A similar check is done @@ -230,7 +227,7 @@ if (p->next == keyhead) Py_FatalError("tls find_key: circular list(!)"); } - if (!update && value == NULL) { + if (value == NULL) { assert(p == NULL); goto Done; } @@ -282,12 +279,19 @@ PyThread_release_lock(keymutex); } +/* Confusing: If the current thread has an association for key, + * value is ignored, and 0 is returned. Else an attempt is made to create + * an association of key to value for the current thread. 0 is returned + * if that succeeds, but -1 is returned if there's not enough memory + * to create the association. value must not be NULL. + */ int PyThread_set_key_value(int key, void *value) { struct key *p; - p = find_key(key, 1, value); + assert(value != NULL); + p = find_key(key, value); if (p == NULL) return -1; else @@ -300,7 +304,7 @@ void * PyThread_get_key_value(int key) { - struct key *p = find_key(key, 0, NULL); + struct key *p = find_key(key, NULL); if (p == NULL) return NULL; diff --git a/Python/thread_nt.h b/Python/thread_nt.h --- a/Python/thread_nt.h +++ b/Python/thread_nt.h @@ -389,11 +389,20 @@ TlsFree(key); } +/* We must be careful to emulate the strange semantics implemented in thread.c, + * where the value is only set if it hasn't been set before. + */ int PyThread_set_key_value(int key, void *value) { BOOL ok; + void *oldvalue; + assert(value != NULL); + oldvalue = TlsGetValue(key); + if (oldvalue != NULL) + /* ignore value if already set */ + return 0; ok = TlsSetValue(key, value); if (!ok) return -1; diff --git a/Python/thread_pthread.h b/Python/thread_pthread.h --- a/Python/thread_pthread.h +++ b/Python/thread_pthread.h @@ -627,6 +627,9 @@ PyThread_set_key_value(int key, void *value) { int fail; + void *oldValue = pthread_getspecific(key); + if (oldValue != NULL) + return 0; fail = pthread_setspecific(key, value); return fail ? -1 : 0; } -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Fri Dec 13 09:41:40 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 13 Dec 2013 09:41:40 +0100 Subject: [Python-checkins] Daily reference leaks (a35b2d652449): sum=4 Message-ID: results for a35b2d652449 on branch "default" -------------------------------------------- test_site leaked [2, -2, 2] references, sum=2 test_site leaked [2, -2, 2] memory blocks, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogvzvIOo', '-x'] From python-checkins at python.org Fri Dec 13 10:57:24 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 13 Dec 2013 10:57:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_remove_referenc?= =?utf-8?q?es_to_the_Tulip_project=2C_rename_Tulip_to_asyncio=2E?= Message-ID: <3dgnNh0MlSz7Lk0@mail.python.org> http://hg.python.org/cpython/rev/10378199e37b changeset: 87927:10378199e37b user: Victor Stinner date: Fri Dec 13 10:57:04 2013 +0100 summary: asyncio: remove references to the Tulip project, rename Tulip to asyncio. Patch written by Vajrasky Kok. files: Doc/library/asyncio-sync.rst | 2 +- Doc/whatsnew/3.4.rst | 5 +++++ Lib/asyncio/queues.py | 2 +- Lib/asyncio/test_utils.py | 2 +- 4 files changed, 8 insertions(+), 3 deletions(-) diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst --- a/Doc/library/asyncio-sync.rst +++ b/Doc/library/asyncio-sync.rst @@ -237,7 +237,7 @@ when the queue reaches *maxsize*, until an item is removed by :meth:`get`. Unlike the standard library :mod:`queue`, you can reliably know this Queue's - size with :meth:`qsize`, since your single-threaded Tulip application won't + size with :meth:`qsize`, since your single-threaded asyncio application won't be interrupted between calling :meth:`qsize` and doing an operation on the Queue. diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1068,3 +1068,8 @@ working directory will also now have an absolute path, including when using ``-m`` with the interpreter (this does not influence when the path to a file is specified on the command-line). + +* (C API) :c:func:`PyThread_set_key_value` now always set the value. In Python + 3.3, the function did nothing if the key already exists (if the current + value is a non-NULL pointer). + diff --git a/Lib/asyncio/queues.py b/Lib/asyncio/queues.py --- a/Lib/asyncio/queues.py +++ b/Lib/asyncio/queues.py @@ -26,7 +26,7 @@ queue reaches maxsize, until an item is removed by get(). Unlike the standard library Queue, you can reliably know this Queue's size - with qsize(), since your single-threaded Tulip application won't be + with qsize(), since your single-threaded asyncio application won't be interrupted between calling qsize() and doing an operation on the Queue. """ diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py --- a/Lib/asyncio/test_utils.py +++ b/Lib/asyncio/test_utils.py @@ -88,7 +88,7 @@ def finish_request(self, request, client_address): # The relative location of our test directory (which # contains the ssl key and certificate files) differs - # between the stdlib and stand-alone Tulip/asyncio. + # between the stdlib and stand-alone asyncio. # Prefer our own if we can find it. here = os.path.join(os.path.dirname(__file__), '..', 'tests') if not os.path.isdir(here): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 11:12:09 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 13 Dec 2013 11:12:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE3OTE5?= =?utf-8?q?=3A_select=2Epoll=2Epoll=28=29_again_works_with_poll=2EPOLLNVAL?= =?utf-8?q?_on_AIX=2E?= Message-ID: <3dgnjj4Vc6z7LnP@mail.python.org> http://hg.python.org/cpython/rev/08c95dd68cfc changeset: 87928:08c95dd68cfc branch: 3.3 parent: 87920:2f975036cf39 user: Serhiy Storchaka date: Fri Dec 13 12:08:01 2013 +0200 summary: Issue #17919: select.poll.poll() again works with poll.POLLNVAL on AIX. files: Lib/test/test_poll.py | 4 ---- Misc/NEWS | 2 ++ Modules/selectmodule.c | 5 ++--- 3 files changed, 4 insertions(+), 7 deletions(-) diff --git a/Lib/test/test_poll.py b/Lib/test/test_poll.py --- a/Lib/test/test_poll.py +++ b/Lib/test/test_poll.py @@ -161,10 +161,6 @@ pollster = select.poll() # Issue 15989 - self.assertRaises(OverflowError, pollster.register, 0, - _testcapi.SHRT_MAX + 1) - self.assertRaises(OverflowError, pollster.register, 0, - _testcapi.USHRT_MAX + 1) self.assertRaises(OverflowError, pollster.poll, _testcapi.INT_MAX + 1) self.assertRaises(OverflowError, pollster.poll, _testcapi.UINT_MAX + 1) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,8 @@ Library ------- +- Issue #17919: select.poll.poll() again works with poll.POLLNVAL on AIX. + - Issue #19063: if a Charset's body_encoding was set to None, the email package would generate a message claiming the Content-Transfer-Encoding was 7bit, and produce garbage output for the content. This now works. diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -380,11 +380,10 @@ poll_register(pollObject *self, PyObject *args) { PyObject *o, *key, *value; - int fd; - short events = POLLIN | POLLPRI | POLLOUT; + int fd, events = POLLIN | POLLPRI | POLLOUT; int err; - if (!PyArg_ParseTuple(args, "O|h:register", &o, &events)) { + if (!PyArg_ParseTuple(args, "O|i:register", &o, &events)) { return NULL; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 11:12:10 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 13 Dec 2013 11:12:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2317919=3A_select=2Epoll=2Epoll=28=29_again_works?= =?utf-8?q?_with_poll=2EPOLLNVAL_on_AIX=2E?= Message-ID: <3dgnjk6Lqxz7Lxy@mail.python.org> http://hg.python.org/cpython/rev/e78743e03c8f changeset: 87929:e78743e03c8f parent: 87927:10378199e37b parent: 87928:08c95dd68cfc user: Serhiy Storchaka date: Fri Dec 13 12:08:55 2013 +0200 summary: Issue #17919: select.poll.poll() again works with poll.POLLNVAL on AIX. files: Lib/test/test_poll.py | 4 ---- Misc/NEWS | 2 ++ Modules/selectmodule.c | 5 ++--- 3 files changed, 4 insertions(+), 7 deletions(-) diff --git a/Lib/test/test_poll.py b/Lib/test/test_poll.py --- a/Lib/test/test_poll.py +++ b/Lib/test/test_poll.py @@ -163,10 +163,6 @@ pollster = select.poll() # Issue 15989 - self.assertRaises(OverflowError, pollster.register, 0, - _testcapi.SHRT_MAX + 1) - self.assertRaises(OverflowError, pollster.register, 0, - _testcapi.USHRT_MAX + 1) self.assertRaises(OverflowError, pollster.poll, _testcapi.INT_MAX + 1) self.assertRaises(OverflowError, pollster.poll, _testcapi.UINT_MAX + 1) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -37,6 +37,8 @@ Library ------- +- Issue #17919: select.poll.poll() again works with poll.POLLNVAL on AIX. + - Issue #19063: if a Charset's body_encoding was set to None, the email package would generate a message claiming the Content-Transfer-Encoding was 7bit, and produce garbage output for the content. This now works. diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -375,11 +375,10 @@ poll_register(pollObject *self, PyObject *args) { PyObject *o, *key, *value; - int fd; - short events = POLLIN | POLLPRI | POLLOUT; + int fd, events = POLLIN | POLLPRI | POLLOUT; int err; - if (!PyArg_ParseTuple(args, "O|h:register", &o, &events)) { + if (!PyArg_ParseTuple(args, "O|i:register", &o, &events)) { return NULL; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 11:12:12 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 13 Dec 2013 11:12:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE3OTE5?= =?utf-8?q?=3A_select=2Epoll=2Epoll=28=29_again_works_with_poll=2EPOLLNVAL?= =?utf-8?q?_on_AIX=2E?= Message-ID: <3dgnjm1Hxwz7M34@mail.python.org> http://hg.python.org/cpython/rev/64f32a31cd49 changeset: 87930:64f32a31cd49 branch: 2.7 parent: 87922:aa324af42c0e user: Serhiy Storchaka date: Fri Dec 13 12:09:05 2013 +0200 summary: Issue #17919: select.poll.poll() again works with poll.POLLNVAL on AIX. files: Lib/test/test_poll.py | 4 ---- Misc/NEWS | 2 ++ Modules/selectmodule.c | 5 ++--- 3 files changed, 4 insertions(+), 7 deletions(-) diff --git a/Lib/test/test_poll.py b/Lib/test/test_poll.py --- a/Lib/test/test_poll.py +++ b/Lib/test/test_poll.py @@ -161,10 +161,6 @@ pollster = select.poll() # Issue 15989 - self.assertRaises(OverflowError, pollster.register, 0, - _testcapi.SHRT_MAX + 1) - self.assertRaises(OverflowError, pollster.register, 0, - _testcapi.USHRT_MAX + 1) self.assertRaises(OverflowError, pollster.poll, _testcapi.INT_MAX + 1) self.assertRaises(OverflowError, pollster.poll, _testcapi.UINT_MAX + 1) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,8 @@ Library ------- +- Issue #17919: select.poll.poll() again works with poll.POLLNVAL on AIX. + - Issue #17200: telnetlib's read_until and expect timeout was broken by the fix to Issue #14635 in Python 2.7.4 to be interpreted as milliseconds instead of seconds when the platform supports select.poll (ie: everywhere). diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -366,11 +366,10 @@ poll_register(pollObject *self, PyObject *args) { PyObject *o, *key, *value; - int fd; - short events = POLLIN | POLLPRI | POLLOUT; + int fd, events = POLLIN | POLLPRI | POLLOUT; int err; - if (!PyArg_ParseTuple(args, "O|h:register", &o, &events)) { + if (!PyArg_ParseTuple(args, "O|i:register", &o, &events)) { return NULL; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 11:14:22 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 13 Dec 2013 11:14:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319787=3A_PyThread?= =?utf-8?q?=5Fset=5Fkey=5Fvalue=28=29_now_always_set_the_value?= Message-ID: <3dgnmG2zDdz7Lmp@mail.python.org> http://hg.python.org/cpython/rev/5d078b0bae75 changeset: 87931:5d078b0bae75 parent: 87929:e78743e03c8f user: Victor Stinner date: Fri Dec 13 11:08:56 2013 +0100 summary: Issue #19787: PyThread_set_key_value() now always set the value In Python 3.3, PyThread_set_key_value() did nothing if the key already exists (if the current value is a non-NULL pointer). When _PyGILState_NoteThreadState() is called twice on the same thread with a different Python thread state, it still keeps the old Python thread state to keep the old behaviour. Replacing the Python thread state with the new state introduces new bugs: see issues #10915 and #15751. files: Misc/NEWS | 4 ++++ Modules/_tracemalloc.c | 9 +++------ Python/pystate.c | 18 +++++++++--------- Python/thread.c | 20 ++++++++------------ Python/thread_nt.h | 9 --------- Python/thread_pthread.h | 3 --- 6 files changed, 24 insertions(+), 39 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,10 @@ Core and Builtins ----------------- +- Issue #19787: PyThread_set_key_value() now always set the value. In Python + 3.3, the function did nothing if the key already exists (if the current value + is a non-NULL pointer). + - Issue #14432: Remove the thread state field from the frame structure. Fix a crash when a generator is created in a C thread that is destroyed while the generator is still used. The issue was that a generator contains a frame, and diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -168,14 +168,11 @@ assert(reentrant == 0 || reentrant == 1); if (reentrant) { assert(PyThread_get_key_value(tracemalloc_reentrant_key) == NULL); - PyThread_set_key_value(tracemalloc_reentrant_key, - REENTRANT); + PyThread_set_key_value(tracemalloc_reentrant_key, REENTRANT); } else { - /* FIXME: PyThread_set_key_value() cannot be used to set the flag - to zero, because it does nothing if the variable has already - a value set. */ - PyThread_delete_key_value(tracemalloc_reentrant_key); + assert(PyThread_get_key_value(tracemalloc_reentrant_key) == REENTRANT); + PyThread_set_key_value(tracemalloc_reentrant_key, NULL); } } diff --git a/Python/pystate.c b/Python/pystate.c --- a/Python/pystate.c +++ b/Python/pystate.c @@ -723,18 +723,18 @@ The only situation where you can legitimately have more than one thread state for an OS level thread is when there are multiple - interpreters, when: + interpreters. - a) You shouldn't really be using the PyGILState_ APIs anyway, - and: + You shouldn't really be using the PyGILState_ APIs anyway (see issues + #10915 and #15751). - b) The slightly odd way PyThread_set_key_value works (see - comments by its implementation) means that the first thread - state created for that given OS level thread will "win", - which seems reasonable behaviour. + The first thread state created for that given OS level thread will + "win", which seems reasonable behaviour. */ - if (PyThread_set_key_value(autoTLSkey, (void *)tstate) < 0) - Py_FatalError("Couldn't create autoTLSkey mapping"); + if (PyThread_get_key_value(autoTLSkey) == NULL) { + if (PyThread_set_key_value(autoTLSkey, (void *)tstate) < 0) + Py_FatalError("Couldn't create autoTLSkey mapping"); + } /* PyGILState_Release must not try to delete this thread state. */ tstate->gilstate_counter = 1; diff --git a/Python/thread.c b/Python/thread.c --- a/Python/thread.c +++ b/Python/thread.c @@ -205,7 +205,7 @@ * segfaults. Now we lock the whole routine. */ static struct key * -find_key(int key, void *value) +find_key(int set_value, int key, void *value) { struct key *p, *prev_p; long id = PyThread_get_thread_ident(); @@ -215,8 +215,11 @@ PyThread_acquire_lock(keymutex, 1); prev_p = NULL; for (p = keyhead; p != NULL; p = p->next) { - if (p->id == id && p->key == key) + if (p->id == id && p->key == key) { + if (set_value) + p->value = value; goto Done; + } /* Sanity check. These states should never happen but if * they do we must abort. Otherwise we'll end up spinning in * in a tight loop with the lock held. A similar check is done @@ -227,7 +230,7 @@ if (p->next == keyhead) Py_FatalError("tls find_key: circular list(!)"); } - if (value == NULL) { + if (!set_value && value == NULL) { assert(p == NULL); goto Done; } @@ -279,19 +282,12 @@ PyThread_release_lock(keymutex); } -/* Confusing: If the current thread has an association for key, - * value is ignored, and 0 is returned. Else an attempt is made to create - * an association of key to value for the current thread. 0 is returned - * if that succeeds, but -1 is returned if there's not enough memory - * to create the association. value must not be NULL. - */ int PyThread_set_key_value(int key, void *value) { struct key *p; - assert(value != NULL); - p = find_key(key, value); + p = find_key(1, key, value); if (p == NULL) return -1; else @@ -304,7 +300,7 @@ void * PyThread_get_key_value(int key) { - struct key *p = find_key(key, NULL); + struct key *p = find_key(0, key, NULL); if (p == NULL) return NULL; diff --git a/Python/thread_nt.h b/Python/thread_nt.h --- a/Python/thread_nt.h +++ b/Python/thread_nt.h @@ -389,20 +389,11 @@ TlsFree(key); } -/* We must be careful to emulate the strange semantics implemented in thread.c, - * where the value is only set if it hasn't been set before. - */ int PyThread_set_key_value(int key, void *value) { BOOL ok; - void *oldvalue; - assert(value != NULL); - oldvalue = TlsGetValue(key); - if (oldvalue != NULL) - /* ignore value if already set */ - return 0; ok = TlsSetValue(key, value); if (!ok) return -1; diff --git a/Python/thread_pthread.h b/Python/thread_pthread.h --- a/Python/thread_pthread.h +++ b/Python/thread_pthread.h @@ -627,9 +627,6 @@ PyThread_set_key_value(int key, void *value) { int fail; - void *oldValue = pthread_getspecific(key); - if (oldValue != NULL) - return 0; fail = pthread_setspecific(key, value); return fail ? -1 : 0; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 12:47:52 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 13 Dec 2013 12:47:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5OTY5?= =?utf-8?q?=3A_PyBytes=5FFromFormatV=28=29_now_raises_an_OverflowError_if_?= =?utf-8?b?IiVjIg==?= Message-ID: <3dgqr8185Sz7Ljb@mail.python.org> http://hg.python.org/cpython/rev/68e0dbc492de changeset: 87932:68e0dbc492de branch: 3.3 parent: 87928:08c95dd68cfc user: Victor Stinner date: Fri Dec 13 12:14:44 2013 +0100 summary: Issue #19969: PyBytes_FromFormatV() now raises an OverflowError if "%c" argument is not in range [0; 255]. files: Lib/test/test_bytes.py | 6 ++++++ Misc/NEWS | 3 +++ Objects/bytesobject.c | 19 ++++++++++++++++--- 3 files changed, 25 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -729,6 +729,12 @@ self.assertEqual(PyBytes_FromFormat(b's:%s', c_char_p(b'cstr')), b's:cstr') + # Issue #19969 + self.assertRaises(OverflowError, + PyBytes_FromFormat, b'%c', c_int(-1)) + self.assertRaises(OverflowError, + PyBytes_FromFormat, b'%c', c_int(256)) + class ByteArrayTest(BaseBytesTest, unittest.TestCase): type2test = bytearray diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #19969: PyBytes_FromFormatV() now raises an OverflowError if "%c" + argument is not in range [0; 255]. + - Issue #14432: Generator now clears the borrowed reference to the thread state. Fix a crash when a generator is created in a C thread that is destroyed while the generator is still used. The issue was that a generator diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -186,8 +186,17 @@ switch (*f) { case 'c': - (void)va_arg(count, int); - /* fall through... */ + { + int c = va_arg(count, int); + if (c < 0 || c > 255) { + PyErr_SetString(PyExc_OverflowError, + "PyBytes_FromFormatV(): %c format " + "expects an integer in range [0; 255]"); + return NULL; + } + n++; + break; + } case '%': n++; break; @@ -267,8 +276,12 @@ switch (*f) { case 'c': - *s++ = va_arg(vargs, int); + { + int c = va_arg(vargs, int); + /* c has been checked for overflow in the first step */ + *s++ = (unsigned char)c; break; + } case 'd': if (longflag) sprintf(s, "%ld", va_arg(vargs, long)); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 12:47:53 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 13 Dec 2013 12:47:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E3=29_Issue_=2319969=3A_PyBytes=5FFromFormat?= =?utf-8?q?V=28=29_now_raises_an_OverflowError_if?= Message-ID: <3dgqr9363zz7Ljr@mail.python.org> http://hg.python.org/cpython/rev/969e38b2f336 changeset: 87933:969e38b2f336 parent: 87931:5d078b0bae75 parent: 87932:68e0dbc492de user: Victor Stinner date: Fri Dec 13 12:15:31 2013 +0100 summary: (Merge 3.3) Issue #19969: PyBytes_FromFormatV() now raises an OverflowError if "%c" argument is not in range [0; 255]. files: Lib/test/test_bytes.py | 6 ++++++ Misc/NEWS | 3 +++ Objects/bytesobject.c | 19 ++++++++++++++++--- 3 files changed, 25 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -743,6 +743,12 @@ self.assertEqual(PyBytes_FromFormat(b's:%s', c_char_p(b'cstr')), b's:cstr') + # Issue #19969 + self.assertRaises(OverflowError, + PyBytes_FromFormat, b'%c', c_int(-1)) + self.assertRaises(OverflowError, + PyBytes_FromFormat, b'%c', c_int(256)) + class ByteArrayTest(BaseBytesTest, unittest.TestCase): type2test = bytearray diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #19969: PyBytes_FromFormatV() now raises an OverflowError if "%c" + argument is not in range [0; 255]. + - Issue #19787: PyThread_set_key_value() now always set the value. In Python 3.3, the function did nothing if the key already exists (if the current value is a non-NULL pointer). diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -195,8 +195,17 @@ switch (*f) { case 'c': - (void)va_arg(count, int); - /* fall through... */ + { + int c = va_arg(count, int); + if (c < 0 || c > 255) { + PyErr_SetString(PyExc_OverflowError, + "PyBytes_FromFormatV(): %c format " + "expects an integer in range [0; 255]"); + return NULL; + } + n++; + break; + } case '%': n++; break; @@ -276,8 +285,12 @@ switch (*f) { case 'c': - *s++ = va_arg(vargs, int); + { + int c = va_arg(vargs, int); + /* c has been checked for overflow in the first step */ + *s++ = (unsigned char)c; break; + } case 'd': if (longflag) sprintf(s, "%ld", va_arg(vargs, long)); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 12:47:54 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 13 Dec 2013 12:47:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_change_styl?= =?utf-8?q?e_of_the_sequence_diagram_to_focus_on_the_control_flow?= Message-ID: <3dgqrB4xDPz7Ljr@mail.python.org> http://hg.python.org/cpython/rev/87308d426b7c changeset: 87934:87308d426b7c user: Victor Stinner date: Fri Dec 13 12:47:39 2013 +0100 summary: asyncio doc: change style of the sequence diagram to focus on the control flow files: Doc/library/tulip_coro.dia | Bin Doc/library/tulip_coro.png | Bin 2 files changed, 0 insertions(+), 0 deletions(-) diff --git a/Doc/library/tulip_coro.dia b/Doc/library/tulip_coro.dia index c4e19a066589b568b64cbc3cc9d5d8c60c4a2a08..eccf4fa7f5a5e0775fd7ade25e3010c3a194efc1 GIT binary patch [stripped] diff --git a/Doc/library/tulip_coro.png b/Doc/library/tulip_coro.png index e083bc37de88d7fa55bf4124b077aef81b8c798a..65b6951550e1e5313bdfad671ff6fd5538f24f1a GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 12:52:43 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 13 Dec 2013 12:52:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio_doc=3A_explain_whe?= =?utf-8?q?re_does_the_task_come_from=2E?= Message-ID: <3dgqxl37VDz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/ad150dd97561 changeset: 87935:ad150dd97561 user: Victor Stinner date: Fri Dec 13 12:51:24 2013 +0100 summary: asyncio doc: explain where does the task come from. files: Doc/library/asyncio-task.rst | 10 ++++++---- 1 files changed, 6 insertions(+), 4 deletions(-) diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -108,10 +108,12 @@ .. image:: tulip_coro.png :align: center -The diagram shows the logical links between the task and the two coroutines, it -does not describe exactly how things work internally. For example, the sleep -coroutine creates an internal future which uses -:meth:`BaseEventLoop.call_later` to wake up the task in 1 second. +The "Task" is created by the :meth:`BaseEventLoop.run_until_complete` method +when it gets a coroutine instead of a task. + +The diagram shows the control flow, it does not describe exactly how things +work internally. For example, the sleep coroutine creates an internal future +which uses :meth:`BaseEventLoop.call_later` to wake up the task in 1 second. InvalidStateError -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 14:33:18 2013 From: python-checkins at python.org (victor.stinner) Date: Fri, 13 Dec 2013 14:33:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2314432=3A_Document?= =?utf-8?q?_the_removal_of_the_PyFrameObject=2Ef=5Ftstate_field?= Message-ID: <3dgt9p3HdBz7LjY@mail.python.org> http://hg.python.org/cpython/rev/278dd7eb2f2b changeset: 87936:278dd7eb2f2b user: Victor Stinner date: Fri Dec 13 14:33:01 2013 +0100 summary: Issue #14432: Document the removal of the PyFrameObject.f_tstate field files: Doc/whatsnew/3.4.rst | 30 ++++++++++++++++++++---------- 1 files changed, 20 insertions(+), 10 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1001,6 +1001,9 @@ Porting to Python 3.4 ===================== +Changes in the Python API +------------------------- + This section lists previously described changes and other bugfixes that may require changes to your code. @@ -1027,10 +1030,6 @@ name but an empty list instead. Determing if a module is a package should be done using ``hasattr(module, '__path__')``. -* :c:func:`PyErr_SetImportError` now sets :exc:`TypeError` when its **msg** - argument is not set. Previously only ``NULL`` was returned with no exception - set. - * :func:`py_compile.compile` now raises :exc:`FileExistsError` if the file path it would write to is a symlink or a non-regular file. This is to act as a warning that import will overwrite those files with a regular file regardless @@ -1055,11 +1054,6 @@ :func:`inspect.unwrap` to access the first function in the chain that has no ``__wrapped__`` attribute. -* (C API) The result of the :c:data:`PyOS_ReadlineFunctionPointer` callback must - now be a string allocated by :c:func:`PyMem_RawMalloc` or - :c:func:`PyMem_RawRealloc`, or *NULL* if an error occurred, instead of a - string allocated by :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`. - * :class:`importlib.machinery.PathFinder` now passes on the current working directory to objects in :data:`sys.path_hooks` for the empty string. This results in :data:`sys.path_importer_cache` never containing ``''``, thus @@ -1069,7 +1063,23 @@ ``-m`` with the interpreter (this does not influence when the path to a file is specified on the command-line). -* (C API) :c:func:`PyThread_set_key_value` now always set the value. In Python +Changes in the C API +-------------------- + +* :c:func:`PyErr_SetImportError` now sets :exc:`TypeError` when its **msg** + argument is not set. Previously only ``NULL`` was returned with no exception + set. + +* The result of the :c:data:`PyOS_ReadlineFunctionPointer` callback must + now be a string allocated by :c:func:`PyMem_RawMalloc` or + :c:func:`PyMem_RawRealloc`, or *NULL* if an error occurred, instead of a + string allocated by :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`. + +* :c:func:`PyThread_set_key_value` now always set the value. In Python 3.3, the function did nothing if the key already exists (if the current value is a non-NULL pointer). +* The ``f_tstate`` (thread state) field of the :c:type:`PyFrameObject` + structure has been removed to fix a bug: see :issue:`14432` for the + rationale. + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 17:43:19 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 13 Dec 2013 17:43:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319946=3A_Raise_Im?= =?utf-8?q?portError_when_the_main_module_cannot_be_found?= Message-ID: <3dgyP35VKYz7Lkf@mail.python.org> http://hg.python.org/cpython/rev/cea42629ddf5 changeset: 87937:cea42629ddf5 user: Brett Cannon date: Fri Dec 13 11:43:10 2013 -0500 summary: Issue #19946: Raise ImportError when the main module cannot be found by multiprocessing.spawn (before it was raising an AttributeError). files: Lib/multiprocessing/spawn.py | 2 ++ Misc/NEWS | 3 +++ 2 files changed, 5 insertions(+), 0 deletions(-) diff --git a/Lib/multiprocessing/spawn.py b/Lib/multiprocessing/spawn.py --- a/Lib/multiprocessing/spawn.py +++ b/Lib/multiprocessing/spawn.py @@ -248,6 +248,8 @@ main_module = types.ModuleType(main_name) # XXX Use a target of main_module? spec = importlib.find_spec(main_name, path=dirs) + if spec is None: + raise ImportError(name=main_name) methods = importlib._bootstrap._SpecMethods(spec) methods.init_module_attrs(main_module) main_module.__name__ = '__mp_main__' diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #19946: multiprocessing.spawn now raises ImportError when the module to + be used as the main module cannot be imported. + - Issue #17919: select.poll.poll() again works with poll.POLLNVAL on AIX. - Issue #19063: if a Charset's body_encoding was set to None, the email -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 18:31:10 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 13 Dec 2013 18:31:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzE4MDM2OiB1cGRh?= =?utf-8?q?te_=2Epyc_FAQ_entry_in_light_of_PEP_3147=2E?= Message-ID: <3dgzSG0ZZxz7LkZ@mail.python.org> http://hg.python.org/cpython/rev/a14f830196ec changeset: 87938:a14f830196ec branch: 3.3 parent: 87932:68e0dbc492de user: R David Murray date: Fri Dec 13 12:29:29 2013 -0500 summary: #18036: update .pyc FAQ entry in light of PEP 3147. Initial patch by Phil Connell. files: Doc/faq/programming.rst | 43 +++++++++++++++++----------- 1 files changed, 26 insertions(+), 17 deletions(-) diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -1607,26 +1607,34 @@ How do I create a .pyc file? ---------------------------- -When a module is imported for the first time (or when the source is more recent -than the current compiled file) a ``.pyc`` file containing the compiled code -should be created in the same directory as the ``.py`` file. +When a module is imported for the first time (or when the source file has +changed since the current compiled file was created) a ``.pyc`` file containing +the compiled code should be created in a ``__pycache__`` subdirectory of the +directory containing the ``.py`` file. The ``.pyc`` file will have a +filename that starts with the same name as the ``.py`` file, and ends with +``.pyc``, with a middle component that depends on the particular ``python`` +binary that created it. (See :pep:`3147` for details.) -One reason that a ``.pyc`` file may not be created is permissions problems with -the directory. This can happen, for example, if you develop as one user but run -as another, such as if you are testing with a web server. Creation of a .pyc -file is automatic if you're importing a module and Python has the ability -(permissions, free space, etc...) to write the compiled module back to the -directory. +One reason that a ``.pyc`` file may not be created is a permissions problem +with the directory containing the source file, meaning that the ``__pycache__`` +subdirectory cannot be created. This can happen, for example, if you develop as +one user but run as another, such as if you are testing with a web server. + +Unless the :envvar:`PYTHONDONTWRITEBYTECODE` environment variable is set, +creation of a .pyc file is automatic if you're importing a module and Python +has the ability (permissions, free space, etc...) to create a ``__pycache__`` +subdirectory and write the compiled module to that subdirectory. Running Python on a top level script is not considered an import and no ``.pyc`` will be created. For example, if you have a top-level module -``foo.py`` that imports another module ``xyz.py``, when you run ``foo``, -``xyz.pyc`` will be created since ``xyz`` is imported, but no ``foo.pyc`` file -will be created since ``foo.py`` isn't being imported. +``foo.py`` that imports another module ``xyz.py``, when you run ``foo`` (by +typing ``python foo.py`` as a shell command), a ``.pyc`` will be created for +``xyz`` because ``xyz`` is imported, but no ``.pyc`` file will be created for +``foo`` since ``foo.py`` isn't being imported. -If you need to create ``foo.pyc`` -- that is, to create a ``.pyc`` file for a module -that is not imported -- you can, using the :mod:`py_compile` and -:mod:`compileall` modules. +If you need to create a ``.pyc`` file for ``foo`` -- that is, to create a +``.pyc`` file for a module that is not imported -- you can, using the +:mod:`py_compile` and :mod:`compileall` modules. The :mod:`py_compile` module can manually compile any module. One way is to use the ``compile()`` function in that module interactively:: @@ -1634,8 +1642,9 @@ >>> import py_compile >>> py_compile.compile('foo.py') # doctest: +SKIP -This will write the ``.pyc`` to the same location as ``foo.py`` (or you can -override that with the optional parameter ``cfile``). +This will write the ``.pyc`` to a ``__pycache__`` subdirectory in the same +location as ``foo.py`` (or you can override that with the optional parameter +``cfile``). You can also automatically compile all files in a directory or directories using the :mod:`compileall` module. You can do it from the shell prompt by running -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 18:31:11 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 13 Dec 2013 18:31:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge=3A_=2318036=3A_update_=2Epyc_FAQ_entry_in_light_of?= =?utf-8?q?_PEP_3147=2E?= Message-ID: <3dgzSH3fKXz7Lky@mail.python.org> http://hg.python.org/cpython/rev/a757bdfce6b3 changeset: 87939:a757bdfce6b3 parent: 87937:cea42629ddf5 parent: 87938:a14f830196ec user: R David Murray date: Fri Dec 13 12:30:29 2013 -0500 summary: Merge: #18036: update .pyc FAQ entry in light of PEP 3147. files: Doc/faq/programming.rst | 43 +++++++++++++++++----------- 1 files changed, 26 insertions(+), 17 deletions(-) diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -1607,26 +1607,34 @@ How do I create a .pyc file? ---------------------------- -When a module is imported for the first time (or when the source is more recent -than the current compiled file) a ``.pyc`` file containing the compiled code -should be created in the same directory as the ``.py`` file. +When a module is imported for the first time (or when the source file has +changed since the current compiled file was created) a ``.pyc`` file containing +the compiled code should be created in a ``__pycache__`` subdirectory of the +directory containing the ``.py`` file. The ``.pyc`` file will have a +filename that starts with the same name as the ``.py`` file, and ends with +``.pyc``, with a middle component that depends on the particular ``python`` +binary that created it. (See :pep:`3147` for details.) -One reason that a ``.pyc`` file may not be created is permissions problems with -the directory. This can happen, for example, if you develop as one user but run -as another, such as if you are testing with a web server. Creation of a .pyc -file is automatic if you're importing a module and Python has the ability -(permissions, free space, etc...) to write the compiled module back to the -directory. +One reason that a ``.pyc`` file may not be created is a permissions problem +with the directory containing the source file, meaning that the ``__pycache__`` +subdirectory cannot be created. This can happen, for example, if you develop as +one user but run as another, such as if you are testing with a web server. + +Unless the :envvar:`PYTHONDONTWRITEBYTECODE` environment variable is set, +creation of a .pyc file is automatic if you're importing a module and Python +has the ability (permissions, free space, etc...) to create a ``__pycache__`` +subdirectory and write the compiled module to that subdirectory. Running Python on a top level script is not considered an import and no ``.pyc`` will be created. For example, if you have a top-level module -``foo.py`` that imports another module ``xyz.py``, when you run ``foo``, -``xyz.pyc`` will be created since ``xyz`` is imported, but no ``foo.pyc`` file -will be created since ``foo.py`` isn't being imported. +``foo.py`` that imports another module ``xyz.py``, when you run ``foo`` (by +typing ``python foo.py`` as a shell command), a ``.pyc`` will be created for +``xyz`` because ``xyz`` is imported, but no ``.pyc`` file will be created for +``foo`` since ``foo.py`` isn't being imported. -If you need to create ``foo.pyc`` -- that is, to create a ``.pyc`` file for a module -that is not imported -- you can, using the :mod:`py_compile` and -:mod:`compileall` modules. +If you need to create a ``.pyc`` file for ``foo`` -- that is, to create a +``.pyc`` file for a module that is not imported -- you can, using the +:mod:`py_compile` and :mod:`compileall` modules. The :mod:`py_compile` module can manually compile any module. One way is to use the ``compile()`` function in that module interactively:: @@ -1634,8 +1642,9 @@ >>> import py_compile >>> py_compile.compile('foo.py') # doctest: +SKIP -This will write the ``.pyc`` to the same location as ``foo.py`` (or you can -override that with the optional parameter ``cfile``). +This will write the ``.pyc`` to a ``__pycache__`` subdirectory in the same +location as ``foo.py`` (or you can override that with the optional parameter +``cfile``). You can also automatically compile all files in a directory or directories using the :mod:`compileall` module. You can do it from the shell prompt by running -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 19:58:56 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 13 Dec 2013 19:58:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5OTYz?= =?utf-8?q?=3A_Document_that_importlib=2Eimport=5Fmodule=28=29_will_import?= Message-ID: <3dh1PX6VH1z7LjY@mail.python.org> http://hg.python.org/cpython/rev/a44be62026fc changeset: 87940:a44be62026fc branch: 3.3 parent: 87938:a14f830196ec user: Brett Cannon date: Fri Dec 13 13:57:41 2013 -0500 summary: Issue #19963: Document that importlib.import_module() will import parent packages automatically. files: Doc/library/importlib.rst | 7 +++++-- Misc/NEWS | 3 +++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -82,10 +82,13 @@ derived from :func:`importlib.__import__`, including requiring the package from which an import is occurring to have been previously imported (i.e., *package* must already be imported). The most important difference - is that :func:`import_module` returns the most nested package or module - that was imported (e.g. ``pkg.mod``), while :func:`__import__` returns the + is that :func:`import_module` returns the specified package or module + (e.g. ``pkg.mod``), while :func:`__import__` returns the top-level package or module (e.g. ``pkg``). + .. versionchanged:: 3.3 + Parent packages are automatically imported. + .. function:: find_loader(name, path=None) Find the loader for a module, optionally within the specified *path*. If the diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -169,6 +169,9 @@ Documentation ------------- +- Issue #19963: Document that importlib.import_module() no longer requires + importing parent packages separately. + - Issue #18840: Introduce the json module in the tutorial, and deemphasize the pickle module. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 19:58:58 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 13 Dec 2013 19:58:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_merge_for_issue_=2319963?= Message-ID: <3dh1PZ1PX1z7LkD@mail.python.org> http://hg.python.org/cpython/rev/33938321d46f changeset: 87941:33938321d46f parent: 87939:a757bdfce6b3 parent: 87940:a44be62026fc user: Brett Cannon date: Fri Dec 13 13:58:47 2013 -0500 summary: merge for issue #19963 files: Doc/library/importlib.rst | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -82,10 +82,13 @@ derived from :func:`importlib.__import__`, including requiring the package from which an import is occurring to have been previously imported (i.e., *package* must already be imported). The most important difference - is that :func:`import_module` returns the most nested package or module - that was imported (e.g. ``pkg.mod``), while :func:`__import__` returns the + is that :func:`import_module` returns the specified package or module + (e.g. ``pkg.mod``), while :func:`__import__` returns the top-level package or module (e.g. ``pkg``). + .. versionchanged:: 3.3 + Parent packages are automatically imported. + .. function:: find_loader(name, path=None) Find the loader for a module, optionally within the specified *path*. If the -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 22:47:28 2013 From: python-checkins at python.org (brett.cannon) Date: Fri, 13 Dec 2013 22:47:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319704=3A_Port_tes?= =?utf-8?q?t=2Etest=5Fthreaded=5Fimport_to_PEP_451?= Message-ID: <3dh580127qz7LkG@mail.python.org> http://hg.python.org/cpython/rev/dbb9c23e1887 changeset: 87942:dbb9c23e1887 user: Brett Cannon date: Fri Dec 13 16:47:19 2013 -0500 summary: Issue #19704: Port test.test_threaded_import to PEP 451 files: Lib/test/test_threaded_import.py | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_threaded_import.py b/Lib/test/test_threaded_import.py --- a/Lib/test/test_threaded_import.py +++ b/Lib/test/test_threaded_import.py @@ -57,7 +57,7 @@ } class Finder: - """A dummy finder to detect concurrent access to its find_module() + """A dummy finder to detect concurrent access to its find_spec() method.""" def __init__(self): @@ -65,8 +65,8 @@ self.x = 0 self.lock = threading.Lock() - def find_module(self, name, path=None): - # Simulate some thread-unsafe behaviour. If calls to find_module() + def find_spec(self, name, path=None, target=None): + # Simulate some thread-unsafe behaviour. If calls to find_spec() # are properly serialized, `x` will end up the same as `numcalls`. # Otherwise not. assert imp.lock_held() @@ -80,7 +80,7 @@ """A dummy finder which flushes sys.path_importer_cache when it gets called.""" - def find_module(self, name, path=None): + def find_spec(self, name, path=None, target=None): sys.path_importer_cache.clear() @@ -145,13 +145,13 @@ # dedicated meta_path entry. flushing_finder = FlushingFinder() def path_hook(path): - finder.find_module('') + finder.find_spec('') raise ImportError sys.path_hooks.insert(0, path_hook) sys.meta_path.append(flushing_finder) try: # Flush the cache a first time - flushing_finder.find_module('') + flushing_finder.find_spec('') numtests = self.check_parallel_module_init() self.assertGreater(finder.numcalls, 0) self.assertEqual(finder.x, finder.numcalls) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 13 23:22:08 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 13 Dec 2013 23:22:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_=2319975=3A_remove_unused_?= =?utf-8?q?imports_from_webbrowser_module=2E?= Message-ID: <3dh5w02vYRz7Lk7@mail.python.org> http://hg.python.org/cpython/rev/f2c6b0485ce6 changeset: 87943:f2c6b0485ce6 user: R David Murray date: Fri Dec 13 17:21:42 2013 -0500 summary: #19975: remove unused imports from webbrowser module. Report and patch by Claudiu Popa. files: Lib/webbrowser.py | 3 --- 1 files changed, 0 insertions(+), 3 deletions(-) diff --git a/Lib/webbrowser.py b/Lib/webbrowser.py --- a/Lib/webbrowser.py +++ b/Lib/webbrowser.py @@ -2,14 +2,11 @@ """Interfaces for launching and remotely controlling Web browsers.""" # Maintained by Georg Brandl. -import io import os import shlex import shutil import sys -import stat import subprocess -import time __all__ = ["Error", "open", "open_new", "open_new_tab", "get", "register"] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 14 02:54:15 2013 From: python-checkins at python.org (r.david.murray) Date: Sat, 14 Dec 2013 02:54:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzE5OTcwOiBGaXgg?= =?utf-8?q?some_comment_typos=2E?= Message-ID: <3dhBcl4nMlz7LkT@mail.python.org> http://hg.python.org/cpython/rev/8e18a3b54bbe changeset: 87944:8e18a3b54bbe branch: 3.3 parent: 87940:a44be62026fc user: R David Murray date: Fri Dec 13 20:52:19 2013 -0500 summary: #19970: Fix some comment typos. Report and patch by Vajrasky Kok. files: Lib/test/test_signal.py | 2 +- Modules/faulthandler.c | 2 +- Modules/posixmodule.c | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_signal.py b/Lib/test/test_signal.py --- a/Lib/test/test_signal.py +++ b/Lib/test/test_signal.py @@ -825,7 +825,7 @@ # Unblock SIGUSR1 try: - # unblock the pending signal calls immediatly the signal handler + # unblock the pending signal calls immediately the signal handler signal.pthread_sigmask(signal.SIG_UNBLOCK, [signum]) except ZeroDivisionError: pass diff --git a/Modules/faulthandler.c b/Modules/faulthandler.c --- a/Modules/faulthandler.c +++ b/Modules/faulthandler.c @@ -301,7 +301,7 @@ return; } #endif - /* call the previous signal handler: it is called immediatly if we use + /* call the previous signal handler: it is called immediately if we use sigaction() thanks to SA_NODEFER flag, otherwise it is deferred */ raise(signum); } diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -1742,7 +1742,7 @@ /* FILE_FLAG_BACKUP_SEMANTICS is required to open a directory */ /* FILE_FLAG_OPEN_REPARSE_POINT does not follow the symlink. Because of this, calls like GetFinalPathNameByHandle will return - the symlink path agin and not the actual final path. */ + the symlink path again and not the actual final path. */ FILE_ATTRIBUTE_NORMAL|FILE_FLAG_BACKUP_SEMANTICS| FILE_FLAG_OPEN_REPARSE_POINT, NULL); @@ -1838,7 +1838,7 @@ /* FILE_FLAG_BACKUP_SEMANTICS is required to open a directory */ /* FILE_FLAG_OPEN_REPARSE_POINT does not follow the symlink. Because of this, calls like GetFinalPathNameByHandle will return - the symlink path agin and not the actual final path. */ + the symlink path again and not the actual final path. */ FILE_ATTRIBUTE_NORMAL|FILE_FLAG_BACKUP_SEMANTICS| FILE_FLAG_OPEN_REPARSE_POINT, NULL); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 14 02:54:16 2013 From: python-checkins at python.org (r.david.murray) Date: Sat, 14 Dec 2013 02:54:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge=3A_=2319970=3A_Fix_some_comment_typos=2E?= Message-ID: <3dhBcm6pNQz7Lky@mail.python.org> http://hg.python.org/cpython/rev/358a35471f9f changeset: 87945:358a35471f9f parent: 87943:f2c6b0485ce6 parent: 87944:8e18a3b54bbe user: R David Murray date: Fri Dec 13 20:53:26 2013 -0500 summary: Merge: #19970: Fix some comment typos. files: Lib/test/test_signal.py | 2 +- Modules/faulthandler.c | 2 +- Modules/posixmodule.c | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_signal.py b/Lib/test/test_signal.py --- a/Lib/test/test_signal.py +++ b/Lib/test/test_signal.py @@ -873,7 +873,7 @@ # Unblock SIGUSR1 try: - # unblock the pending signal calls immediatly the signal handler + # unblock the pending signal calls immediately the signal handler signal.pthread_sigmask(signal.SIG_UNBLOCK, [signum]) except ZeroDivisionError: pass diff --git a/Modules/faulthandler.c b/Modules/faulthandler.c --- a/Modules/faulthandler.c +++ b/Modules/faulthandler.c @@ -303,7 +303,7 @@ return; } #endif - /* call the previous signal handler: it is called immediatly if we use + /* call the previous signal handler: it is called immediately if we use sigaction() thanks to SA_NODEFER flag, otherwise it is deferred */ raise(signum); } diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -1666,7 +1666,7 @@ /* FILE_FLAG_BACKUP_SEMANTICS is required to open a directory */ /* FILE_FLAG_OPEN_REPARSE_POINT does not follow the symlink. Because of this, calls like GetFinalPathNameByHandle will return - the symlink path agin and not the actual final path. */ + the symlink path again and not the actual final path. */ FILE_ATTRIBUTE_NORMAL|FILE_FLAG_BACKUP_SEMANTICS| FILE_FLAG_OPEN_REPARSE_POINT, NULL); @@ -1762,7 +1762,7 @@ /* FILE_FLAG_BACKUP_SEMANTICS is required to open a directory */ /* FILE_FLAG_OPEN_REPARSE_POINT does not follow the symlink. Because of this, calls like GetFinalPathNameByHandle will return - the symlink path agin and not the actual final path. */ + the symlink path again and not the actual final path. */ FILE_ATTRIBUTE_NORMAL|FILE_FLAG_BACKUP_SEMANTICS| FILE_FLAG_OPEN_REPARSE_POINT, NULL); -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sat Dec 14 09:41:42 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 14 Dec 2013 09:41:42 +0100 Subject: [Python-checkins] Daily reference leaks (358a35471f9f): sum=-9 Message-ID: results for 358a35471f9f on branch "default" -------------------------------------------- test_site leaked [2, -2, 0] references, sum=0 test_site leaked [2, -2, 0] memory blocks, sum=0 test_urllib2net leaked [1587, -1598, 0] references, sum=-11 test_urllib2net leaked [1406, -1406, 2] memory blocks, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogOhIURl', '-x'] From python-checkins at python.org Sat Dec 14 11:43:36 2013 From: python-checkins at python.org (ezio.melotti) Date: Sat, 14 Dec 2013 11:43:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogIzE5OTgxOiBmaXgg?= =?utf-8?q?typo_in_email=2Emailbox_docs=2E__Patch_by_Claudiu_Popa=2E?= Message-ID: <3dhQMX2qRnz7Ljs@mail.python.org> http://hg.python.org/cpython/rev/31fd129960d5 changeset: 87946:31fd129960d5 branch: 2.7 parent: 87930:64f32a31cd49 user: Ezio Melotti date: Sat Dec 14 12:42:29 2013 +0200 summary: #19981: fix typo in email.mailbox docs. Patch by Claudiu Popa. files: Doc/library/mailbox.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/mailbox.rst b/Doc/library/mailbox.rst --- a/Doc/library/mailbox.rst +++ b/Doc/library/mailbox.rst @@ -1664,7 +1664,7 @@ due to malformed messages in the mailbox:: import mailbox - import email.Errors + import email.errors list_names = ('python-list', 'python-dev', 'python-bugs') @@ -1674,7 +1674,7 @@ for key in inbox.iterkeys(): try: message = inbox[key] - except email.Errors.MessageParseError: + except email.errors.MessageParseError: continue # The message is malformed. Just leave it. for name in list_names: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 14 11:43:37 2013 From: python-checkins at python.org (ezio.melotti) Date: Sat, 14 Dec 2013 11:43:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzE5OTgxOiBmaXgg?= =?utf-8?q?typo_in_email=2Emailbox_docs=2E__Patch_by_Claudiu_Popa=2E?= Message-ID: <3dhQMY4dnsz7Ljs@mail.python.org> http://hg.python.org/cpython/rev/364ca376956f changeset: 87947:364ca376956f branch: 3.3 parent: 87944:8e18a3b54bbe user: Ezio Melotti date: Sat Dec 14 12:42:29 2013 +0200 summary: #19981: fix typo in email.mailbox docs. Patch by Claudiu Popa. files: Doc/library/mailbox.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/mailbox.rst b/Doc/library/mailbox.rst --- a/Doc/library/mailbox.rst +++ b/Doc/library/mailbox.rst @@ -1550,7 +1550,7 @@ due to malformed messages in the mailbox:: import mailbox - import email.Errors + import email.errors list_names = ('python-list', 'python-dev', 'python-bugs') @@ -1560,7 +1560,7 @@ for key in inbox.iterkeys(): try: message = inbox[key] - except email.Errors.MessageParseError: + except email.errors.MessageParseError: continue # The message is malformed. Just leave it. for name in list_names: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 14 11:43:38 2013 From: python-checkins at python.org (ezio.melotti) Date: Sat, 14 Dec 2013 11:43:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogIzE5OTgxOiBtZXJnZSB3aXRoIDMuMy4=?= Message-ID: <3dhQMZ6HSLz7LkT@mail.python.org> http://hg.python.org/cpython/rev/0d8fe7d688a9 changeset: 87948:0d8fe7d688a9 parent: 87945:358a35471f9f parent: 87947:364ca376956f user: Ezio Melotti date: Sat Dec 14 12:43:08 2013 +0200 summary: #19981: merge with 3.3. files: Doc/library/mailbox.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/mailbox.rst b/Doc/library/mailbox.rst --- a/Doc/library/mailbox.rst +++ b/Doc/library/mailbox.rst @@ -1550,7 +1550,7 @@ due to malformed messages in the mailbox:: import mailbox - import email.Errors + import email.errors list_names = ('python-list', 'python-dev', 'python-bugs') @@ -1560,7 +1560,7 @@ for key in inbox.iterkeys(): try: message = inbox[key] - except email.Errors.MessageParseError: + except email.errors.MessageParseError: continue # The message is malformed. Just leave it. for name in list_names: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 14 12:59:22 2013 From: python-checkins at python.org (stefan.krah) Date: Sat, 14 Dec 2013 12:59:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Fix_C++_header?= =?utf-8?q?_usage=2E__This_=5F=5FSTDC=5FLIMIT=5FMACROS_scheme_can_still_be?= =?utf-8?q?_subverted?= Message-ID: <3dhS2y0b2rz7LjS@mail.python.org> http://hg.python.org/cpython/rev/b9a7b2973dd6 changeset: 87949:b9a7b2973dd6 branch: 3.3 parent: 87947:364ca376956f user: Stefan Krah date: Sat Dec 14 12:58:09 2013 +0100 summary: Fix C++ header usage. This __STDC_LIMIT_MACROS scheme can still be subverted by including stdint.h before mpdecimal.h. In that case the only option left is to compile with -D_STDC_LIMIT_MACROS. files: Modules/_decimal/libmpdec/mpdecimal.h | 20 ++++++++------ 1 files changed, 11 insertions(+), 9 deletions(-) diff --git a/Modules/_decimal/libmpdec/mpdecimal.h b/Modules/_decimal/libmpdec/mpdecimal.h --- a/Modules/_decimal/libmpdec/mpdecimal.h +++ b/Modules/_decimal/libmpdec/mpdecimal.h @@ -32,6 +32,10 @@ #ifdef __cplusplus extern "C" { + #ifndef __STDC_LIMIT_MACROS + #define __STDC_LIMIT_MACROS + #define MPD_CLEAR_STDC_LIMIT_MACROS + #endif #endif @@ -55,18 +59,12 @@ #define MPD_HIDE_SYMBOLS_END #define EXTINLINE extern inline #else + #ifdef HAVE_STDINT_H + #include + #endif #ifdef HAVE_INTTYPES_H #include #endif - #ifdef HAVE_STDINT_H - #if defined(__cplusplus) && !defined(__STDC_LIMIT_MACROS) - #define __STDC_LIMIT_MACROS - #include - #undef __STDC_LIMIT_MACROS - #else - #include - #endif - #endif #ifndef __GNUC_STDC_INLINE__ #define __GNUC_STDC_INLINE__ 1 #endif @@ -835,6 +833,10 @@ #ifdef __cplusplus + #ifdef MPD_CLEAR_STDC_LIMIT_MACROS + #undef MPD_CLEAR_STDC_LIMIT_MACROS + #undef __STDC_LIMIT_MACROS + #endif } /* END extern "C" */ #endif -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 14 12:59:23 2013 From: python-checkins at python.org (stefan.krah) Date: Sat, 14 Dec 2013 12:59:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgZnJvbSAzLjMu?= Message-ID: <3dhS2z2KlWz7Ljs@mail.python.org> http://hg.python.org/cpython/rev/1fed51da4721 changeset: 87950:1fed51da4721 parent: 87948:0d8fe7d688a9 parent: 87949:b9a7b2973dd6 user: Stefan Krah date: Sat Dec 14 12:58:59 2013 +0100 summary: Merge from 3.3. files: Modules/_decimal/libmpdec/mpdecimal.h | 20 ++++++++------ 1 files changed, 11 insertions(+), 9 deletions(-) diff --git a/Modules/_decimal/libmpdec/mpdecimal.h b/Modules/_decimal/libmpdec/mpdecimal.h --- a/Modules/_decimal/libmpdec/mpdecimal.h +++ b/Modules/_decimal/libmpdec/mpdecimal.h @@ -32,6 +32,10 @@ #ifdef __cplusplus extern "C" { + #ifndef __STDC_LIMIT_MACROS + #define __STDC_LIMIT_MACROS + #define MPD_CLEAR_STDC_LIMIT_MACROS + #endif #endif @@ -55,18 +59,12 @@ #define MPD_HIDE_SYMBOLS_END #define EXTINLINE extern inline #else + #ifdef HAVE_STDINT_H + #include + #endif #ifdef HAVE_INTTYPES_H #include #endif - #ifdef HAVE_STDINT_H - #if defined(__cplusplus) && !defined(__STDC_LIMIT_MACROS) - #define __STDC_LIMIT_MACROS - #include - #undef __STDC_LIMIT_MACROS - #else - #include - #endif - #endif #ifndef __GNUC_STDC_INLINE__ #define __GNUC_STDC_INLINE__ 1 #endif @@ -835,6 +833,10 @@ #ifdef __cplusplus + #ifdef MPD_CLEAR_STDC_LIMIT_MACROS + #undef MPD_CLEAR_STDC_LIMIT_MACROS + #undef __STDC_LIMIT_MACROS + #endif } /* END extern "C" */ #endif -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 14 13:44:00 2013 From: python-checkins at python.org (stefan.krah) Date: Sat, 14 Dec 2013 13:44:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319972=3A_Add_rare?= =?utf-8?q?ly_used_freefunc=2E__This_fixes_a_leak_if_sys=2Eexit=28=29?= Message-ID: <3dhT2S5L39z7LjZ@mail.python.org> http://hg.python.org/cpython/rev/8a78988fdb04 changeset: 87951:8a78988fdb04 user: Stefan Krah date: Sat Dec 14 13:43:10 2013 +0100 summary: Issue #19972: Add rarely used freefunc. This fixes a leak if sys.exit() is used in a program. files: Modules/_pickle.c | 22 ++++++++++++++-------- 1 files changed, 14 insertions(+), 8 deletions(-) diff --git a/Modules/_pickle.c b/Modules/_pickle.c --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -7464,6 +7464,12 @@ return 0; } +static void +pickle_free(PyObject *m) +{ + _Pickle_ClearState(_Pickle_GetState(m)); +} + static int pickle_traverse(PyObject *m, visitproc visit, void *arg) { @@ -7485,14 +7491,14 @@ static struct PyModuleDef _picklemodule = { PyModuleDef_HEAD_INIT, - "_pickle", /* m_name */ - pickle_module_doc, /* m_doc */ - sizeof(PickleState), /* m_size */ - pickle_methods, /* m_methods */ - NULL, /* m_reload */ - pickle_traverse, /* m_traverse */ - pickle_clear, /* m_clear */ - NULL /* m_free */ + "_pickle", /* m_name */ + pickle_module_doc, /* m_doc */ + sizeof(PickleState), /* m_size */ + pickle_methods, /* m_methods */ + NULL, /* m_reload */ + pickle_traverse, /* m_traverse */ + pickle_clear, /* m_clear */ + (freefunc)pickle_free /* m_free */ }; PyMODINIT_FUNC -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 14 17:26:23 2013 From: python-checkins at python.org (r.david.murray) Date: Sat, 14 Dec 2013 17:26:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_=2319970=3A_fix_additional?= =?utf-8?q?_typo_in_3=2E4_asyncio_docs=2E?= Message-ID: <3dhYz337jBz7Ljl@mail.python.org> http://hg.python.org/cpython/rev/561822250761 changeset: 87952:561822250761 user: R David Murray date: Sat Dec 14 11:26:06 2013 -0500 summary: #19970: fix additional typo in 3.4 asyncio docs. files: Doc/library/asyncio-protocol.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -649,7 +649,7 @@ server.close() loop.close() -:meth:`Transport.close` can be called immediatly after +:meth:`Transport.close` can be called immediately after :meth:`WriteTransport.write` even if data are not sent yet on the socket: both methods are asynchronous. ``yield from`` is not needed because these transport methods don't return coroutines. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 14 18:19:17 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 14 Dec 2013 18:19:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE3OTE5?= =?utf-8?q?=3A_Fixed_integer_overflow_in_the_eventmask_parameter=2E?= Message-ID: <3dhb856VTcz7Ljn@mail.python.org> http://hg.python.org/cpython/rev/9bee6cc30db7 changeset: 87953:9bee6cc30db7 branch: 2.7 parent: 87946:31fd129960d5 user: Serhiy Storchaka date: Sat Dec 14 19:11:04 2013 +0200 summary: Issue #17919: Fixed integer overflow in the eventmask parameter. files: Lib/test/test_poll.py | 13 +++++++---- Misc/NEWS | 3 +- Modules/selectmodule.c | 32 +++++++++++++++++++++++------ 3 files changed, 35 insertions(+), 13 deletions(-) diff --git a/Lib/test/test_poll.py b/Lib/test/test_poll.py --- a/Lib/test/test_poll.py +++ b/Lib/test/test_poll.py @@ -3,7 +3,7 @@ import os import random import select -import _testcapi +from _testcapi import USHRT_MAX, INT_MAX, UINT_MAX try: import threading except ImportError: @@ -159,10 +159,13 @@ if x != 5: self.fail('Overflow must have occurred') - pollster = select.poll() - # Issue 15989 - self.assertRaises(OverflowError, pollster.poll, _testcapi.INT_MAX + 1) - self.assertRaises(OverflowError, pollster.poll, _testcapi.UINT_MAX + 1) + # Issues #15989, #17919 + self.assertRaises(OverflowError, pollster.register, 0, -1) + self.assertRaises(OverflowError, pollster.register, 0, USHRT_MAX + 1) + self.assertRaises(OverflowError, pollster.modify, 1, -1) + self.assertRaises(OverflowError, pollster.modify, 1, USHRT_MAX + 1) + self.assertRaises(OverflowError, pollster.poll, INT_MAX + 1) + self.assertRaises(OverflowError, pollster.poll, UINT_MAX + 1) @unittest.skipUnless(threading, 'Threading required for this test.') @reap_threads diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,7 +23,8 @@ Library ------- -- Issue #17919: select.poll.poll() again works with poll.POLLNVAL on AIX. +- Issue #17919: select.poll.register() again works with poll.POLLNVAL on AIX. + Fixed integer overflow in the eventmask parameter. - Issue #17200: telnetlib's read_until and expect timeout was broken by the fix to Issue #14635 in Python 2.7.4 to be interpreted as milliseconds diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -347,7 +347,7 @@ assert(i < self->ufd_len); /* Never overflow */ self->ufds[i].fd = (int)PyInt_AsLong(key); - self->ufds[i].events = (short)PyInt_AsLong(value); + self->ufds[i].events = (short)(unsigned short)PyInt_AsLong(value); i++; } assert(i == self->ufd_len); @@ -355,6 +355,24 @@ return 1; } +static int +ushort_converter(PyObject *obj, void *ptr) +{ + unsigned long uval; + + uval = PyLong_AsUnsignedLong(obj); + if (uval == (unsigned long)-1 && PyErr_Occurred()) + return 0; + if (uval > USHRT_MAX) { + PyErr_SetString(PyExc_OverflowError, + "Python int too large for C unsigned short"); + return 0; + } + + *(unsigned short *)ptr = Py_SAFE_DOWNCAST(uval, unsigned long, unsigned short); + return 1; +} + PyDoc_STRVAR(poll_register_doc, "register(fd [, eventmask] ) -> None\n\n\ Register a file descriptor with the polling object.\n\ @@ -366,12 +384,12 @@ poll_register(pollObject *self, PyObject *args) { PyObject *o, *key, *value; - int fd, events = POLLIN | POLLPRI | POLLOUT; + int fd; + unsigned short events = POLLIN | POLLPRI | POLLOUT; int err; - if (!PyArg_ParseTuple(args, "O|i:register", &o, &events)) { + if (!PyArg_ParseTuple(args, "O|O&:register", &o, ushort_converter, &events)) return NULL; - } fd = PyObject_AsFileDescriptor(o); if (fd == -1) return NULL; @@ -409,12 +427,12 @@ poll_modify(pollObject *self, PyObject *args) { PyObject *o, *key, *value; - int fd, events; + int fd; + unsigned short events; int err; - if (!PyArg_ParseTuple(args, "Oi:modify", &o, &events)) { + if (!PyArg_ParseTuple(args, "OO&:modify", &o, ushort_converter, &events)) return NULL; - } fd = PyObject_AsFileDescriptor(o); if (fd == -1) return NULL; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 14 18:19:19 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 14 Dec 2013 18:19:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE3OTE5?= =?utf-8?q?=3A_Fixed_integer_overflow_in_the_eventmask_parameter=2E?= Message-ID: <3dhb872RSSz7LkT@mail.python.org> http://hg.python.org/cpython/rev/87bbe810e4e7 changeset: 87954:87bbe810e4e7 branch: 3.3 parent: 87949:b9a7b2973dd6 user: Serhiy Storchaka date: Sat Dec 14 19:12:02 2013 +0200 summary: Issue #17919: Fixed integer overflow in the eventmask parameter. files: Lib/test/test_devpoll.py | 11 +++++++ Lib/test/test_poll.py | 13 +++++--- Misc/NEWS | 3 +- Modules/selectmodule.c | 40 ++++++++++++++++++++------- 4 files changed, 50 insertions(+), 17 deletions(-) diff --git a/Lib/test/test_devpoll.py b/Lib/test/test_devpoll.py --- a/Lib/test/test_devpoll.py +++ b/Lib/test/test_devpoll.py @@ -87,6 +87,17 @@ self.assertRaises(OverflowError, pollster.poll, 1 << 63) self.assertRaises(OverflowError, pollster.poll, 1 << 64) + def test_events_mask_overflow(self): + pollster = select.devpoll() + w, r = os.pipe() + pollster.register(w) + # Issue #17919 + self.assertRaises(OverflowError, pollster.register, 0, -1) + self.assertRaises(OverflowError, pollster.register, 0, USHRT_MAX + 1) + self.assertRaises(OverflowError, pollster.modify, 1, -1) + self.assertRaises(OverflowError, pollster.modify, 1, USHRT_MAX + 1) + + def test_main(): run_unittest(DevPollTests) diff --git a/Lib/test/test_poll.py b/Lib/test/test_poll.py --- a/Lib/test/test_poll.py +++ b/Lib/test/test_poll.py @@ -3,7 +3,7 @@ import os import random import select -import _testcapi +from _testcapi import USHRT_MAX, INT_MAX, UINT_MAX try: import threading except ImportError: @@ -159,10 +159,13 @@ if x != 5: self.fail('Overflow must have occurred') - pollster = select.poll() - # Issue 15989 - self.assertRaises(OverflowError, pollster.poll, _testcapi.INT_MAX + 1) - self.assertRaises(OverflowError, pollster.poll, _testcapi.UINT_MAX + 1) + # Issues #15989, #17919 + self.assertRaises(OverflowError, pollster.register, 0, -1) + self.assertRaises(OverflowError, pollster.register, 0, USHRT_MAX + 1) + self.assertRaises(OverflowError, pollster.modify, 1, -1) + self.assertRaises(OverflowError, pollster.modify, 1, USHRT_MAX + 1) + self.assertRaises(OverflowError, pollster.poll, INT_MAX + 1) + self.assertRaises(OverflowError, pollster.poll, UINT_MAX + 1) @unittest.skipUnless(threading, 'Threading required for this test.') @reap_threads diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -32,7 +32,8 @@ Library ------- -- Issue #17919: select.poll.poll() again works with poll.POLLNVAL on AIX. +- Issue #17919: select.poll.register() again works with poll.POLLNVAL on AIX. + Fixed integer overflow in the eventmask parameter. - Issue #19063: if a Charset's body_encoding was set to None, the email package would generate a message claiming the Content-Transfer-Encoding diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -361,7 +361,7 @@ assert(i < self->ufd_len); /* Never overflow */ self->ufds[i].fd = (int)PyLong_AsLong(key); - self->ufds[i].events = (short)PyLong_AsLong(value); + self->ufds[i].events = (short)(unsigned short)PyLong_AsLong(value); i++; } assert(i == self->ufd_len); @@ -369,6 +369,24 @@ return 1; } +static int +ushort_converter(PyObject *obj, void *ptr) +{ + unsigned long uval; + + uval = PyLong_AsUnsignedLong(obj); + if (uval == (unsigned long)-1 && PyErr_Occurred()) + return 0; + if (uval > USHRT_MAX) { + PyErr_SetString(PyExc_OverflowError, + "Python int too large for C unsigned short"); + return 0; + } + + *(unsigned short *)ptr = Py_SAFE_DOWNCAST(uval, unsigned long, unsigned short); + return 1; +} + PyDoc_STRVAR(poll_register_doc, "register(fd [, eventmask] ) -> None\n\n\ Register a file descriptor with the polling object.\n\ @@ -380,12 +398,12 @@ poll_register(pollObject *self, PyObject *args) { PyObject *o, *key, *value; - int fd, events = POLLIN | POLLPRI | POLLOUT; + int fd; + unsigned short events = POLLIN | POLLPRI | POLLOUT; int err; - if (!PyArg_ParseTuple(args, "O|i:register", &o, &events)) { + if (!PyArg_ParseTuple(args, "O|O&:register", &o, ushort_converter, &events)) return NULL; - } fd = PyObject_AsFileDescriptor(o); if (fd == -1) return NULL; @@ -423,12 +441,12 @@ poll_modify(pollObject *self, PyObject *args) { PyObject *o, *key, *value; - int fd, events; + int fd; + unsigned short events; int err; - if (!PyArg_ParseTuple(args, "Oi:modify", &o, &events)) { + if (!PyArg_ParseTuple(args, "OO&:modify", &o, ushort_converter, &events)) return NULL; - } fd = PyObject_AsFileDescriptor(o); if (fd == -1) return NULL; @@ -726,11 +744,11 @@ internal_devpoll_register(devpollObject *self, PyObject *args, int remove) { PyObject *o; - int fd, events = POLLIN | POLLPRI | POLLOUT; + int fd; + unsigned short events = POLLIN | POLLPRI | POLLOUT; - if (!PyArg_ParseTuple(args, "O|i:register", &o, &events)) { + if (!PyArg_ParseTuple(args, "O|O&:register", &o, ushort_converter, &events)) return NULL; - } fd = PyObject_AsFileDescriptor(o); if (fd == -1) return NULL; @@ -746,7 +764,7 @@ } self->fds[self->n_fds].fd = fd; - self->fds[self->n_fds].events = events; + self->fds[self->n_fds].events = (signed short)events; if (++self->n_fds == self->max_n_fds) { if (devpoll_flush(self)) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 14 18:19:20 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 14 Dec 2013 18:19:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2317919=3A_Fixed_integer_overflow_in_the_eventmas?= =?utf-8?q?k_parameter=2E?= Message-ID: <3dhb885h8xz7LkZ@mail.python.org> http://hg.python.org/cpython/rev/2fbb3c77f157 changeset: 87955:2fbb3c77f157 parent: 87952:561822250761 parent: 87954:87bbe810e4e7 user: Serhiy Storchaka date: Sat Dec 14 19:18:39 2013 +0200 summary: Issue #17919: Fixed integer overflow in the eventmask parameter. files: Lib/test/test_devpoll.py | 11 +++++++ Lib/test/test_poll.py | 13 +++++--- Misc/NEWS | 3 +- Modules/selectmodule.c | 40 ++++++++++++++++++++------- 4 files changed, 50 insertions(+), 17 deletions(-) diff --git a/Lib/test/test_devpoll.py b/Lib/test/test_devpoll.py --- a/Lib/test/test_devpoll.py +++ b/Lib/test/test_devpoll.py @@ -121,6 +121,17 @@ self.assertEqual(os.get_inheritable(devpoll.fileno()), False) + def test_events_mask_overflow(self): + pollster = select.devpoll() + w, r = os.pipe() + pollster.register(w) + # Issue #17919 + self.assertRaises(OverflowError, pollster.register, 0, -1) + self.assertRaises(OverflowError, pollster.register, 0, USHRT_MAX + 1) + self.assertRaises(OverflowError, pollster.modify, 1, -1) + self.assertRaises(OverflowError, pollster.modify, 1, USHRT_MAX + 1) + + def test_main(): run_unittest(DevPollTests) diff --git a/Lib/test/test_poll.py b/Lib/test/test_poll.py --- a/Lib/test/test_poll.py +++ b/Lib/test/test_poll.py @@ -4,7 +4,7 @@ import subprocess import random import select -import _testcapi +from _testcapi import USHRT_MAX, INT_MAX, UINT_MAX try: import threading except ImportError: @@ -161,10 +161,13 @@ if x != 5: self.fail('Overflow must have occurred') - pollster = select.poll() - # Issue 15989 - self.assertRaises(OverflowError, pollster.poll, _testcapi.INT_MAX + 1) - self.assertRaises(OverflowError, pollster.poll, _testcapi.UINT_MAX + 1) + # Issues #15989, #17919 + self.assertRaises(OverflowError, pollster.register, 0, -1) + self.assertRaises(OverflowError, pollster.register, 0, USHRT_MAX + 1) + self.assertRaises(OverflowError, pollster.modify, 1, -1) + self.assertRaises(OverflowError, pollster.modify, 1, USHRT_MAX + 1) + self.assertRaises(OverflowError, pollster.poll, INT_MAX + 1) + self.assertRaises(OverflowError, pollster.poll, UINT_MAX + 1) @unittest.skipUnless(threading, 'Threading required for this test.') @reap_threads diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -47,7 +47,8 @@ - Issue #19946: multiprocessing.spawn now raises ImportError when the module to be used as the main module cannot be imported. -- Issue #17919: select.poll.poll() again works with poll.POLLNVAL on AIX. +- Issue #17919: select.poll.register() again works with poll.POLLNVAL on AIX. + Fixed integer overflow in the eventmask parameter. - Issue #19063: if a Charset's body_encoding was set to None, the email package would generate a message claiming the Content-Transfer-Encoding diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -356,7 +356,7 @@ assert(i < self->ufd_len); /* Never overflow */ self->ufds[i].fd = (int)PyLong_AsLong(key); - self->ufds[i].events = (short)PyLong_AsLong(value); + self->ufds[i].events = (short)(unsigned short)PyLong_AsLong(value); i++; } assert(i == self->ufd_len); @@ -364,6 +364,24 @@ return 1; } +static int +ushort_converter(PyObject *obj, void *ptr) +{ + unsigned long uval; + + uval = PyLong_AsUnsignedLong(obj); + if (uval == (unsigned long)-1 && PyErr_Occurred()) + return 0; + if (uval > USHRT_MAX) { + PyErr_SetString(PyExc_OverflowError, + "Python int too large for C unsigned short"); + return 0; + } + + *(unsigned short *)ptr = Py_SAFE_DOWNCAST(uval, unsigned long, unsigned short); + return 1; +} + PyDoc_STRVAR(poll_register_doc, "register(fd [, eventmask] ) -> None\n\n\ Register a file descriptor with the polling object.\n\ @@ -375,12 +393,12 @@ poll_register(pollObject *self, PyObject *args) { PyObject *o, *key, *value; - int fd, events = POLLIN | POLLPRI | POLLOUT; + int fd; + unsigned short events = POLLIN | POLLPRI | POLLOUT; int err; - if (!PyArg_ParseTuple(args, "O|i:register", &o, &events)) { + if (!PyArg_ParseTuple(args, "O|O&:register", &o, ushort_converter, &events)) return NULL; - } fd = PyObject_AsFileDescriptor(o); if (fd == -1) return NULL; @@ -418,12 +436,12 @@ poll_modify(pollObject *self, PyObject *args) { PyObject *o, *key, *value; - int fd, events; + int fd; + unsigned short events; int err; - if (!PyArg_ParseTuple(args, "Oi:modify", &o, &events)) { + if (!PyArg_ParseTuple(args, "OO&:modify", &o, ushort_converter, &events)) return NULL; - } fd = PyObject_AsFileDescriptor(o); if (fd == -1) return NULL; @@ -728,14 +746,14 @@ internal_devpoll_register(devpollObject *self, PyObject *args, int remove) { PyObject *o; - int fd, events = POLLIN | POLLPRI | POLLOUT; + int fd; + unsigned short events = POLLIN | POLLPRI | POLLOUT; if (self->fd_devpoll < 0) return devpoll_err_closed(); - if (!PyArg_ParseTuple(args, "O|i:register", &o, &events)) { + if (!PyArg_ParseTuple(args, "O|O&:register", &o, ushort_converter, &events)) return NULL; - } fd = PyObject_AsFileDescriptor(o); if (fd == -1) return NULL; @@ -751,7 +769,7 @@ } self->fds[self->n_fds].fd = fd; - self->fds[self->n_fds].events = events; + self->fds[self->n_fds].events = (signed short)events; if (++self->n_fds == self->max_n_fds) { if (devpoll_flush(self)) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 14 19:43:24 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 14 Dec 2013 19:43:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5NjIz?= =?utf-8?q?=3A_Fixed_writing_to_unseekable_files_in_the_aifc_module=2E?= Message-ID: <3dhd181BFKz7LkG@mail.python.org> http://hg.python.org/cpython/rev/c10ea224392d changeset: 87956:c10ea224392d branch: 2.7 parent: 87953:9bee6cc30db7 user: Serhiy Storchaka date: Sat Dec 14 20:34:33 2013 +0200 summary: Issue #19623: Fixed writing to unseekable files in the aifc module. Fixed writing 'ulaw' (lower case) compressed AIFC files. files: Lib/aifc.py | 15 ++++-- Lib/test/audiotests.py | 63 ++++++++++++++++++++++++++++++ Lib/test/test_aifc.py | 35 +++++----------- Lib/test/test_sunau.py | 23 +++------- Lib/test/test_wave.py | 26 +++++------- Misc/NEWS | 3 + 6 files changed, 106 insertions(+), 59 deletions(-) diff --git a/Lib/aifc.py b/Lib/aifc.py --- a/Lib/aifc.py +++ b/Lib/aifc.py @@ -778,7 +778,7 @@ def _ensure_header_written(self, datasize): if not self._nframeswritten: - if self._comptype in ('ULAW', 'ALAW'): + if self._comptype in ('ULAW', 'ulaw', 'ALAW', 'alaw'): if not self._sampwidth: self._sampwidth = 2 if self._sampwidth != 2: @@ -844,7 +844,7 @@ if self._datalength & 1: self._datalength = self._datalength + 1 if self._aifc: - if self._comptype in ('ULAW', 'ALAW'): + if self._comptype in ('ULAW', 'ulaw', 'ALAW', 'alaw'): self._datalength = self._datalength // 2 if self._datalength & 1: self._datalength = self._datalength + 1 @@ -852,7 +852,10 @@ self._datalength = (self._datalength + 3) // 4 if self._datalength & 1: self._datalength = self._datalength + 1 - self._form_length_pos = self._file.tell() + try: + self._form_length_pos = self._file.tell() + except (AttributeError, IOError): + self._form_length_pos = None commlength = self._write_form_length(self._datalength) if self._aifc: self._file.write('AIFC') @@ -864,7 +867,8 @@ self._file.write('COMM') _write_ulong(self._file, commlength) _write_short(self._file, self._nchannels) - self._nframes_pos = self._file.tell() + if self._form_length_pos is not None: + self._nframes_pos = self._file.tell() _write_ulong(self._file, self._nframes) if self._comptype in ('ULAW', 'ulaw', 'ALAW', 'alaw', 'G722'): _write_short(self._file, 8) @@ -875,7 +879,8 @@ self._file.write(self._comptype) _write_string(self._file, self._compname) self._file.write('SSND') - self._ssnd_length_pos = self._file.tell() + if self._form_length_pos is not None: + self._ssnd_length_pos = self._file.tell() _write_ulong(self._file, self._datalength + 8) _write_ulong(self._file, 0) _write_ulong(self._file, 0) diff --git a/Lib/test/audiotests.py b/Lib/test/audiotests.py --- a/Lib/test/audiotests.py +++ b/Lib/test/audiotests.py @@ -6,6 +6,13 @@ import sys import base64 +class UnseekableIO(io.FileIO): + def tell(self): + raise io.UnsupportedOperation + + def seek(self, *args, **kwargs): + raise io.UnsupportedOperation + def fromhex(s): return base64.b16decode(s.replace(' ', '')) @@ -133,6 +140,62 @@ self.assertEqual(testfile.read(13), b'ababagalamaga') self.check_file(testfile, self.nframes, self.frames) + def test_unseekable_read(self): + f = self.create_file(TESTFN) + f.setnframes(self.nframes) + f.writeframes(self.frames) + f.close() + + with UnseekableIO(TESTFN, 'rb') as testfile: + self.check_file(testfile, self.nframes, self.frames) + + def test_unseekable_write(self): + with UnseekableIO(TESTFN, 'wb') as testfile: + f = self.create_file(testfile) + f.setnframes(self.nframes) + f.writeframes(self.frames) + f.close() + self.fout = None + + self.check_file(TESTFN, self.nframes, self.frames) + + def test_unseekable_incompleted_write(self): + with UnseekableIO(TESTFN, 'wb') as testfile: + testfile.write(b'ababagalamaga') + f = self.create_file(testfile) + f.setnframes(self.nframes + 1) + try: + f.writeframes(self.frames) + except IOError: + pass + try: + f.close() + except IOError: + pass + + with open(TESTFN, 'rb') as testfile: + self.assertEqual(testfile.read(13), b'ababagalamaga') + self.check_file(testfile, self.nframes + 1, self.frames) + + def test_unseekable_overflowed_write(self): + with UnseekableIO(TESTFN, 'wb') as testfile: + testfile.write(b'ababagalamaga') + f = self.create_file(testfile) + f.setnframes(self.nframes - 1) + try: + f.writeframes(self.frames) + except IOError: + pass + try: + f.close() + except IOError: + pass + + with open(TESTFN, 'rb') as testfile: + self.assertEqual(testfile.read(13), b'ababagalamaga') + framesize = self.nchannels * self.sampwidth + self.check_file(testfile, self.nframes - 1, self.frames[:-framesize]) + class AudioTestsWithSourceFile(AudioTests): diff --git a/Lib/test/test_aifc.py b/Lib/test/test_aifc.py --- a/Lib/test/test_aifc.py +++ b/Lib/test/test_aifc.py @@ -9,10 +9,14 @@ import aifc -class AifcPCM8Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): +class AifcTest(audiotests.AudioWriteTests, + audiotests.AudioTestsWithSourceFile): module = aifc + close_fd = True + test_unseekable_read = None + + +class AifcPCM8Test(AifcTest, unittest.TestCase): sndfilename = 'pluck-pcm8.aiff' sndfilenframes = 3307 nchannels = 2 @@ -27,13 +31,9 @@ 11FA 3EFB BCFC 66FF CF04 4309 C10E 5112 EE17 8216 7F14 8012 \ 490E 520D EF0F CE0F E40C 630A 080A 2B0B 510E 8B11 B60E 440A \ """) - close_fd = True -class AifcPCM16Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = aifc +class AifcPCM16Test(AifcTest, unittest.TestCase): sndfilename = 'pluck-pcm16.aiff' sndfilenframes = 3307 nchannels = 2 @@ -50,13 +50,9 @@ EEE21753 82071665 7FFF1443 8004128F 49A20EAF 52BB0DBA EFB40F60 CE3C0FBF \ E4B30CEC 63430A5C 08C80A20 2BBB0B08 514A0E43 8BCF1139 B6F60EEB 44120A5E \ """) - close_fd = True -class AifcPCM24Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = aifc +class AifcPCM24Test(AifcTest, unittest.TestCase): sndfilename = 'pluck-pcm24.aiff' sndfilenframes = 3307 nchannels = 2 @@ -79,13 +75,9 @@ E4B49C0CEA2D 6344A80A5A7C 08C8FE0A1FFE 2BB9860B0A0E \ 51486F0E44E1 8BCC64113B05 B6F4EC0EEB36 4413170A5B48 \ """) - close_fd = True -class AifcPCM32Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = aifc +class AifcPCM32Test(AifcTest, unittest.TestCase): sndfilename = 'pluck-pcm32.aiff' sndfilenframes = 3307 nchannels = 2 @@ -108,13 +100,9 @@ E4B49CC00CEA2D90 6344A8800A5A7CA0 08C8FE800A1FFEE0 2BB986C00B0A0E00 \ 51486F800E44E190 8BCC6480113B0580 B6F4EC000EEB3630 441317800A5B48A0 \ """) - close_fd = True -class AifcULAWTest(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = aifc +class AifcULAWTest(AifcTest, unittest.TestCase): sndfilename = 'pluck-ulaw.aifc' sndfilenframes = 3307 nchannels = 2 @@ -133,7 +121,6 @@ """) if sys.byteorder != 'big': frames = audiotests.byteswap2(frames) - close_fd = True class AifcMiscTest(audiotests.AudioTests, unittest.TestCase): diff --git a/Lib/test/test_sunau.py b/Lib/test/test_sunau.py --- a/Lib/test/test_sunau.py +++ b/Lib/test/test_sunau.py @@ -5,10 +5,12 @@ import sunau -class SunauPCM8Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): +class SunauTest(audiotests.AudioWriteTests, + audiotests.AudioTestsWithSourceFile): module = sunau + + +class SunauPCM8Test(SunauTest, unittest.TestCase): sndfilename = 'pluck-pcm8.au' sndfilenframes = 3307 nchannels = 2 @@ -25,10 +27,7 @@ """) -class SunauPCM16Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = sunau +class SunauPCM16Test(SunauTest, unittest.TestCase): sndfilename = 'pluck-pcm16.au' sndfilenframes = 3307 nchannels = 2 @@ -47,10 +46,7 @@ """) -class SunauPCM32Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = sunau +class SunauPCM32Test(SunauTest, unittest.TestCase): sndfilename = 'pluck-pcm32.au' sndfilenframes = 3307 nchannels = 2 @@ -75,10 +71,7 @@ """) -class SunauULAWTest(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = sunau +class SunauULAWTest(SunauTest, unittest.TestCase): sndfilename = 'pluck-ulaw.au' sndfilenframes = 3307 nchannels = 2 diff --git a/Lib/test/test_wave.py b/Lib/test/test_wave.py --- a/Lib/test/test_wave.py +++ b/Lib/test/test_wave.py @@ -5,10 +5,15 @@ import wave -class WavePCM8Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): +class WaveTest(audiotests.AudioWriteTests, + audiotests.AudioTestsWithSourceFile): module = wave + test_unseekable_write = None + test_unseekable_overflowed_write = None + test_unseekable_incompleted_write = None + + +class WavePCM8Test(WaveTest, unittest.TestCase): sndfilename = 'pluck-pcm8.wav' sndfilenframes = 3307 nchannels = 2 @@ -25,10 +30,7 @@ """) -class WavePCM16Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = wave +class WavePCM16Test(WaveTest, unittest.TestCase): sndfilename = 'pluck-pcm16.wav' sndfilenframes = 3307 nchannels = 2 @@ -55,10 +57,7 @@ -class WavePCM24Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = wave +class WavePCM24Test(WaveTest, unittest.TestCase): sndfilename = 'pluck-pcm24.wav' sndfilenframes = 3307 nchannels = 2 @@ -85,10 +84,7 @@ frames = audiotests.byteswap3(frames) -class WavePCM32Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = wave +class WavePCM32Test(WaveTest, unittest.TestCase): sndfilename = 'pluck-pcm32.wav' sndfilenframes = 3307 nchannels = 2 diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,9 @@ Library ------- +- Issue #19623: Fixed writing to unseekable files in the aifc module. + Fixed writing 'ulaw' (lower case) compressed AIFC files. + - Issue #17919: select.poll.register() again works with poll.POLLNVAL on AIX. Fixed integer overflow in the eventmask parameter. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 14 19:43:25 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 14 Dec 2013 19:43:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5NjIz?= =?utf-8?q?=3A_Fixed_writing_to_unseekable_files_in_the_aifc_module=2E?= Message-ID: <3dhd194FnKz7Lkk@mail.python.org> http://hg.python.org/cpython/rev/35f6a5937a63 changeset: 87957:35f6a5937a63 branch: 3.3 parent: 87954:87bbe810e4e7 user: Serhiy Storchaka date: Sat Dec 14 20:35:04 2013 +0200 summary: Issue #19623: Fixed writing to unseekable files in the aifc module. files: Lib/aifc.py | 11 +++- Lib/test/audiotests.py | 62 ++++++++++++++++++++++++++++++ Lib/test/test_aifc.py | 41 +++++-------------- Lib/test/test_sunau.py | 23 +++------- Lib/test/test_wave.py | 26 +++++------- Misc/NEWS | 2 + 6 files changed, 103 insertions(+), 62 deletions(-) diff --git a/Lib/aifc.py b/Lib/aifc.py --- a/Lib/aifc.py +++ b/Lib/aifc.py @@ -771,7 +771,10 @@ self._datalength = (self._datalength + 3) // 4 if self._datalength & 1: self._datalength = self._datalength + 1 - self._form_length_pos = self._file.tell() + try: + self._form_length_pos = self._file.tell() + except (AttributeError, OSError): + self._form_length_pos = None commlength = self._write_form_length(self._datalength) if self._aifc: self._file.write(b'AIFC') @@ -783,7 +786,8 @@ self._file.write(b'COMM') _write_ulong(self._file, commlength) _write_short(self._file, self._nchannels) - self._nframes_pos = self._file.tell() + if self._form_length_pos is not None: + self._nframes_pos = self._file.tell() _write_ulong(self._file, self._nframes) if self._comptype in (b'ULAW', b'ulaw', b'ALAW', b'alaw', b'G722'): _write_short(self._file, 8) @@ -794,7 +798,8 @@ self._file.write(self._comptype) _write_string(self._file, self._compname) self._file.write(b'SSND') - self._ssnd_length_pos = self._file.tell() + if self._form_length_pos is not None: + self._ssnd_length_pos = self._file.tell() _write_ulong(self._file, self._datalength + 8) _write_ulong(self._file, 0) _write_ulong(self._file, 0) diff --git a/Lib/test/audiotests.py b/Lib/test/audiotests.py --- a/Lib/test/audiotests.py +++ b/Lib/test/audiotests.py @@ -5,6 +5,13 @@ import pickle import sys +class UnseekableIO(io.FileIO): + def tell(self): + raise io.UnsupportedOperation + + def seek(self, *args, **kwargs): + raise io.UnsupportedOperation + def byteswap2(data): a = array.array('h') a.frombytes(data) @@ -129,6 +136,61 @@ self.assertEqual(testfile.read(13), b'ababagalamaga') self.check_file(testfile, self.nframes, self.frames) + def test_unseekable_read(self): + f = self.create_file(TESTFN) + f.setnframes(self.nframes) + f.writeframes(self.frames) + f.close() + + with UnseekableIO(TESTFN, 'rb') as testfile: + self.check_file(testfile, self.nframes, self.frames) + + def test_unseekable_write(self): + with UnseekableIO(TESTFN, 'wb') as testfile: + f = self.create_file(testfile) + f.setnframes(self.nframes) + f.writeframes(self.frames) + f.close() + + self.check_file(TESTFN, self.nframes, self.frames) + + def test_unseekable_incompleted_write(self): + with UnseekableIO(TESTFN, 'wb') as testfile: + testfile.write(b'ababagalamaga') + f = self.create_file(testfile) + f.setnframes(self.nframes + 1) + try: + f.writeframes(self.frames) + except OSError: + pass + try: + f.close() + except OSError: + pass + + with open(TESTFN, 'rb') as testfile: + self.assertEqual(testfile.read(13), b'ababagalamaga') + self.check_file(testfile, self.nframes + 1, self.frames) + + def test_unseekable_overflowed_write(self): + with UnseekableIO(TESTFN, 'wb') as testfile: + testfile.write(b'ababagalamaga') + f = self.create_file(testfile) + f.setnframes(self.nframes - 1) + try: + f.writeframes(self.frames) + except OSError: + pass + try: + f.close() + except OSError: + pass + + with open(TESTFN, 'rb') as testfile: + self.assertEqual(testfile.read(13), b'ababagalamaga') + framesize = self.nchannels * self.sampwidth + self.check_file(testfile, self.nframes - 1, self.frames[:-framesize]) + class AudioTestsWithSourceFile(AudioTests): diff --git a/Lib/test/test_aifc.py b/Lib/test/test_aifc.py --- a/Lib/test/test_aifc.py +++ b/Lib/test/test_aifc.py @@ -8,10 +8,14 @@ import aifc -class AifcPCM8Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): +class AifcTest(audiotests.AudioWriteTests, + audiotests.AudioTestsWithSourceFile): module = aifc + close_fd = True + test_unseekable_read = None + + +class AifcPCM8Test(AifcTest, unittest.TestCase): sndfilename = 'pluck-pcm8.aiff' sndfilenframes = 3307 nchannels = 2 @@ -26,13 +30,9 @@ 11FA 3EFB BCFC 66FF CF04 4309 C10E 5112 EE17 8216 7F14 8012 \ 490E 520D EF0F CE0F E40C 630A 080A 2B0B 510E 8B11 B60E 440A \ """) - close_fd = True -class AifcPCM16Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = aifc +class AifcPCM16Test(AifcTest, unittest.TestCase): sndfilename = 'pluck-pcm16.aiff' sndfilenframes = 3307 nchannels = 2 @@ -49,13 +49,9 @@ EEE21753 82071665 7FFF1443 8004128F 49A20EAF 52BB0DBA EFB40F60 CE3C0FBF \ E4B30CEC 63430A5C 08C80A20 2BBB0B08 514A0E43 8BCF1139 B6F60EEB 44120A5E \ """) - close_fd = True -class AifcPCM24Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = aifc +class AifcPCM24Test(AifcTest, unittest.TestCase): sndfilename = 'pluck-pcm24.aiff' sndfilenframes = 3307 nchannels = 2 @@ -78,13 +74,9 @@ E4B49C0CEA2D 6344A80A5A7C 08C8FE0A1FFE 2BB9860B0A0E \ 51486F0E44E1 8BCC64113B05 B6F4EC0EEB36 4413170A5B48 \ """) - close_fd = True -class AifcPCM32Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = aifc +class AifcPCM32Test(AifcTest, unittest.TestCase): sndfilename = 'pluck-pcm32.aiff' sndfilenframes = 3307 nchannels = 2 @@ -107,13 +99,9 @@ E4B49CC00CEA2D90 6344A8800A5A7CA0 08C8FE800A1FFEE0 2BB986C00B0A0E00 \ 51486F800E44E190 8BCC6480113B0580 B6F4EC000EEB3630 441317800A5B48A0 \ """) - close_fd = True -class AifcULAWTest(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = aifc +class AifcULAWTest(AifcTest, unittest.TestCase): sndfilename = 'pluck-ulaw.aifc' sndfilenframes = 3307 nchannels = 2 @@ -132,13 +120,9 @@ """) if sys.byteorder != 'big': frames = audiotests.byteswap2(frames) - close_fd = True -class AifcALAWTest(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = aifc +class AifcALAWTest(AifcTest, unittest.TestCase): sndfilename = 'pluck-alaw.aifc' sndfilenframes = 3307 nchannels = 2 @@ -157,7 +141,6 @@ """) if sys.byteorder != 'big': frames = audiotests.byteswap2(frames) - close_fd = True class AifcMiscTest(audiotests.AudioTests, unittest.TestCase): diff --git a/Lib/test/test_sunau.py b/Lib/test/test_sunau.py --- a/Lib/test/test_sunau.py +++ b/Lib/test/test_sunau.py @@ -5,10 +5,12 @@ import sunau -class SunauPCM8Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): +class SunauTest(audiotests.AudioWriteTests, + audiotests.AudioTestsWithSourceFile): module = sunau + + +class SunauPCM8Test(SunauTest, unittest.TestCase): sndfilename = 'pluck-pcm8.au' sndfilenframes = 3307 nchannels = 2 @@ -25,10 +27,7 @@ """) -class SunauPCM16Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = sunau +class SunauPCM16Test(SunauTest, unittest.TestCase): sndfilename = 'pluck-pcm16.au' sndfilenframes = 3307 nchannels = 2 @@ -47,10 +46,7 @@ """) -class SunauPCM32Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = sunau +class SunauPCM32Test(SunauTest, unittest.TestCase): sndfilename = 'pluck-pcm32.au' sndfilenframes = 3307 nchannels = 2 @@ -75,10 +71,7 @@ """) -class SunauULAWTest(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = sunau +class SunauULAWTest(SunauTest, unittest.TestCase): sndfilename = 'pluck-ulaw.au' sndfilenframes = 3307 nchannels = 2 diff --git a/Lib/test/test_wave.py b/Lib/test/test_wave.py --- a/Lib/test/test_wave.py +++ b/Lib/test/test_wave.py @@ -5,10 +5,15 @@ import wave -class WavePCM8Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): +class WaveTest(audiotests.AudioWriteTests, + audiotests.AudioTestsWithSourceFile): module = wave + test_unseekable_write = None + test_unseekable_overflowed_write = None + test_unseekable_incompleted_write = None + + +class WavePCM8Test(WaveTest, unittest.TestCase): sndfilename = 'pluck-pcm8.wav' sndfilenframes = 3307 nchannels = 2 @@ -25,10 +30,7 @@ """) -class WavePCM16Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = wave +class WavePCM16Test(WaveTest, unittest.TestCase): sndfilename = 'pluck-pcm16.wav' sndfilenframes = 3307 nchannels = 2 @@ -55,10 +57,7 @@ -class WavePCM24Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = wave +class WavePCM24Test(WaveTest, unittest.TestCase): sndfilename = 'pluck-pcm24.wav' sndfilenframes = 3307 nchannels = 2 @@ -85,10 +84,7 @@ frames = audiotests.byteswap3(frames) -class WavePCM32Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = wave +class WavePCM32Test(WaveTest, unittest.TestCase): sndfilename = 'pluck-pcm32.wav' sndfilenframes = 3307 nchannels = 2 diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -32,6 +32,8 @@ Library ------- +- Issue #19623: Fixed writing to unseekable files in the aifc module. + - Issue #17919: select.poll.register() again works with poll.POLLNVAL on AIX. Fixed integer overflow in the eventmask parameter. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 14 19:43:27 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 14 Dec 2013 19:43:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319623=3A_Fixed_writing_to_unseekable_files_in_t?= =?utf-8?q?he_aifc_module=2E?= Message-ID: <3dhd1C06BSz7LlS@mail.python.org> http://hg.python.org/cpython/rev/804406d79b45 changeset: 87958:804406d79b45 parent: 87955:2fbb3c77f157 parent: 87957:35f6a5937a63 user: Serhiy Storchaka date: Sat Dec 14 20:42:22 2013 +0200 summary: Issue #19623: Fixed writing to unseekable files in the aifc module. files: Lib/aifc.py | 11 ++++++++--- Lib/test/test_aifc.py | 3 --- Lib/test/test_sunau.py | 28 +++++++++------------------- Lib/test/test_wave.py | 23 ++++++++--------------- Misc/NEWS | 2 ++ 5 files changed, 27 insertions(+), 40 deletions(-) diff --git a/Lib/aifc.py b/Lib/aifc.py --- a/Lib/aifc.py +++ b/Lib/aifc.py @@ -790,7 +790,10 @@ self._datalength = (self._datalength + 3) // 4 if self._datalength & 1: self._datalength = self._datalength + 1 - self._form_length_pos = self._file.tell() + try: + self._form_length_pos = self._file.tell() + except (AttributeError, OSError): + self._form_length_pos = None commlength = self._write_form_length(self._datalength) if self._aifc: self._file.write(b'AIFC') @@ -802,7 +805,8 @@ self._file.write(b'COMM') _write_ulong(self._file, commlength) _write_short(self._file, self._nchannels) - self._nframes_pos = self._file.tell() + if self._form_length_pos is not None: + self._nframes_pos = self._file.tell() _write_ulong(self._file, self._nframes) if self._comptype in (b'ULAW', b'ulaw', b'ALAW', b'alaw', b'G722'): _write_short(self._file, 8) @@ -813,7 +817,8 @@ self._file.write(self._comptype) _write_string(self._file, self._compname) self._file.write(b'SSND') - self._ssnd_length_pos = self._file.tell() + if self._form_length_pos is not None: + self._ssnd_length_pos = self._file.tell() _write_ulong(self._file, self._datalength + 8) _write_ulong(self._file, 0) _write_ulong(self._file, 0) diff --git a/Lib/test/test_aifc.py b/Lib/test/test_aifc.py --- a/Lib/test/test_aifc.py +++ b/Lib/test/test_aifc.py @@ -14,9 +14,6 @@ module = aifc close_fd = True test_unseekable_read = None - test_unseekable_write = None - test_unseekable_incompleted_write = None - test_unseekable_overflowed_write = None class AifcPCM8Test(AifcTest, unittest.TestCase): diff --git a/Lib/test/test_sunau.py b/Lib/test/test_sunau.py --- a/Lib/test/test_sunau.py +++ b/Lib/test/test_sunau.py @@ -6,10 +6,12 @@ import sunau -class SunauPCM8Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): +class SunauTest(audiotests.AudioWriteTests, + audiotests.AudioTestsWithSourceFile): module = sunau + + +class SunauPCM8Test(SunauTest, unittest.TestCase): sndfilename = 'pluck-pcm8.au' sndfilenframes = 3307 nchannels = 2 @@ -26,10 +28,7 @@ """) -class SunauPCM16Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = sunau +class SunauPCM16Test(SunauTest, unittest.TestCase): sndfilename = 'pluck-pcm16.au' sndfilenframes = 3307 nchannels = 2 @@ -48,10 +47,7 @@ """) -class SunauPCM24Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = sunau +class SunauPCM24Test(SunauTest, unittest.TestCase): sndfilename = 'pluck-pcm24.au' sndfilenframes = 3307 nchannels = 2 @@ -76,10 +72,7 @@ """) -class SunauPCM32Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = sunau +class SunauPCM32Test(SunauTest, unittest.TestCase): sndfilename = 'pluck-pcm32.au' sndfilenframes = 3307 nchannels = 2 @@ -104,10 +97,7 @@ """) -class SunauULAWTest(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = sunau +class SunauULAWTest(SunauTest, unittest.TestCase): sndfilename = 'pluck-ulaw.au' sndfilenframes = 3307 nchannels = 2 diff --git a/Lib/test/test_wave.py b/Lib/test/test_wave.py --- a/Lib/test/test_wave.py +++ b/Lib/test/test_wave.py @@ -6,10 +6,12 @@ import wave -class WavePCM8Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): +class WaveTest(audiotests.AudioWriteTests, + audiotests.AudioTestsWithSourceFile): module = wave + + +class WavePCM8Test(WaveTest, unittest.TestCase): sndfilename = 'pluck-pcm8.wav' sndfilenframes = 3307 nchannels = 2 @@ -26,10 +28,7 @@ """) -class WavePCM16Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = wave +class WavePCM16Test(WaveTest, unittest.TestCase): sndfilename = 'pluck-pcm16.wav' sndfilenframes = 3307 nchannels = 2 @@ -50,10 +49,7 @@ frames = byteswap(frames, 2) -class WavePCM24Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = wave +class WavePCM24Test(WaveTest, unittest.TestCase): sndfilename = 'pluck-pcm24.wav' sndfilenframes = 3307 nchannels = 2 @@ -80,10 +76,7 @@ frames = byteswap(frames, 3) -class WavePCM32Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = wave +class WavePCM32Test(WaveTest, unittest.TestCase): sndfilename = 'pluck-pcm32.wav' sndfilenframes = 3307 nchannels = 2 diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,8 @@ Library ------- +- Issue #19623: Fixed writing to unseekable files in the aifc module. + - Issue #19946: multiprocessing.spawn now raises ImportError when the module to be used as the main module cannot be imported. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 14 20:08:21 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 14 Dec 2013 20:08:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE3NTc2?= =?utf-8?q?=3A_Removed_deprecation_warnings_added_in_changeset_618cca51a27?= =?utf-8?q?e=2E?= Message-ID: <3dhdYx4JdSz7LjT@mail.python.org> http://hg.python.org/cpython/rev/a3de2b3881c1 changeset: 87959:a3de2b3881c1 branch: 3.3 parent: 87957:35f6a5937a63 user: Serhiy Storchaka date: Sat Dec 14 21:07:09 2013 +0200 summary: Issue #17576: Removed deprecation warnings added in changeset 618cca51a27e. files: Lib/test/test_getargs2.py | 24 ++++++++---------------- Lib/test/test_index.py | 3 +-- Lib/test/test_int.py | 9 +++------ Misc/NEWS | 3 --- Objects/abstract.c | 9 --------- Objects/longobject.c | 9 --------- 6 files changed, 12 insertions(+), 45 deletions(-) diff --git a/Lib/test/test_getargs2.py b/Lib/test/test_getargs2.py --- a/Lib/test/test_getargs2.py +++ b/Lib/test/test_getargs2.py @@ -75,8 +75,7 @@ self.assertEqual(99, getargs_b(Int())) self.assertEqual(0, getargs_b(IntSubclass())) self.assertRaises(TypeError, getargs_b, BadInt()) - with self.assertWarns(DeprecationWarning): - self.assertEqual(1, getargs_b(BadInt2())) + self.assertEqual(1, getargs_b(BadInt2())) self.assertEqual(0, getargs_b(BadInt3())) self.assertRaises(OverflowError, getargs_b, -1) @@ -94,8 +93,7 @@ self.assertEqual(99, getargs_B(Int())) self.assertEqual(0, getargs_B(IntSubclass())) self.assertRaises(TypeError, getargs_B, BadInt()) - with self.assertWarns(DeprecationWarning): - self.assertEqual(1, getargs_B(BadInt2())) + self.assertEqual(1, getargs_B(BadInt2())) self.assertEqual(0, getargs_B(BadInt3())) self.assertEqual(UCHAR_MAX, getargs_B(-1)) @@ -113,8 +111,7 @@ self.assertEqual(99, getargs_H(Int())) self.assertEqual(0, getargs_H(IntSubclass())) self.assertRaises(TypeError, getargs_H, BadInt()) - with self.assertWarns(DeprecationWarning): - self.assertEqual(1, getargs_H(BadInt2())) + self.assertEqual(1, getargs_H(BadInt2())) self.assertEqual(0, getargs_H(BadInt3())) self.assertEqual(USHRT_MAX, getargs_H(-1)) @@ -133,8 +130,7 @@ self.assertEqual(99, getargs_I(Int())) self.assertEqual(0, getargs_I(IntSubclass())) self.assertRaises(TypeError, getargs_I, BadInt()) - with self.assertWarns(DeprecationWarning): - self.assertEqual(1, getargs_I(BadInt2())) + self.assertEqual(1, getargs_I(BadInt2())) self.assertEqual(0, getargs_I(BadInt3())) self.assertEqual(UINT_MAX, getargs_I(-1)) @@ -174,8 +170,7 @@ self.assertEqual(99, getargs_h(Int())) self.assertEqual(0, getargs_h(IntSubclass())) self.assertRaises(TypeError, getargs_h, BadInt()) - with self.assertWarns(DeprecationWarning): - self.assertEqual(1, getargs_h(BadInt2())) + self.assertEqual(1, getargs_h(BadInt2())) self.assertEqual(0, getargs_h(BadInt3())) self.assertRaises(OverflowError, getargs_h, SHRT_MIN-1) @@ -193,8 +188,7 @@ self.assertEqual(99, getargs_i(Int())) self.assertEqual(0, getargs_i(IntSubclass())) self.assertRaises(TypeError, getargs_i, BadInt()) - with self.assertWarns(DeprecationWarning): - self.assertEqual(1, getargs_i(BadInt2())) + self.assertEqual(1, getargs_i(BadInt2())) self.assertEqual(0, getargs_i(BadInt3())) self.assertRaises(OverflowError, getargs_i, INT_MIN-1) @@ -212,8 +206,7 @@ self.assertEqual(99, getargs_l(Int())) self.assertEqual(0, getargs_l(IntSubclass())) self.assertRaises(TypeError, getargs_l, BadInt()) - with self.assertWarns(DeprecationWarning): - self.assertEqual(1, getargs_l(BadInt2())) + self.assertEqual(1, getargs_l(BadInt2())) self.assertEqual(0, getargs_l(BadInt3())) self.assertRaises(OverflowError, getargs_l, LONG_MIN-1) @@ -254,8 +247,7 @@ self.assertEqual(99, getargs_L(Int())) self.assertEqual(0, getargs_L(IntSubclass())) self.assertRaises(TypeError, getargs_L, BadInt()) - with self.assertWarns(DeprecationWarning): - self.assertEqual(1, getargs_L(BadInt2())) + self.assertEqual(1, getargs_L(BadInt2())) self.assertEqual(0, getargs_L(BadInt3())) self.assertRaises(OverflowError, getargs_L, LLONG_MIN-1) diff --git a/Lib/test/test_index.py b/Lib/test/test_index.py --- a/Lib/test/test_index.py +++ b/Lib/test/test_index.py @@ -81,8 +81,7 @@ return True bad_int = BadInt() - with self.assertWarns(DeprecationWarning): - n = operator.index(bad_int) + n = operator.index(bad_int) self.assertEqual(n, 1) bad_int = BadInt2() diff --git a/Lib/test/test_int.py b/Lib/test/test_int.py --- a/Lib/test/test_int.py +++ b/Lib/test/test_int.py @@ -359,18 +359,15 @@ return True bad_int = BadInt() - with self.assertWarns(DeprecationWarning): - n = int(bad_int) + n = int(bad_int) self.assertEqual(n, 1) bad_int = BadInt2() - with self.assertWarns(DeprecationWarning): - n = int(bad_int) + n = int(bad_int) self.assertEqual(n, 1) bad_int = TruncReturnsBadInt() - with self.assertWarns(DeprecationWarning): - n = int(bad_int) + n = int(bad_int) self.assertEqual(n, 1) good_int = TruncReturnsIntSubclass() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -19,9 +19,6 @@ contains a frame, and the frame kept a reference to the Python state of the destroyed C thread. The crash occurs when a trace function is setup. -- Issue #17576: Deprecation warning emitted now when __int__() or __index__() - return not int instance. - - Issue #19932: Fix typo in import.h, missing whitespaces in function prototypes. - Issue #19729: In str.format(), fix recursive expansion in format spec. diff --git a/Objects/abstract.c b/Objects/abstract.c --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -1162,15 +1162,6 @@ Py_DECREF(result); return NULL; } - /* Issue #17576: warn if 'result' not of exact type int. */ - if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, - "__index__ returned non-int (type %.200s). " - "The ability to return an instance of a strict subclass of int " - "is deprecated, and may be removed in a future version of Python.", - result->ob_type->tp_name)) { - Py_DECREF(result); - return NULL; - } return result; } diff --git a/Objects/longobject.c b/Objects/longobject.c --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -153,15 +153,6 @@ Py_DECREF(result); return NULL; } - /* Issue #17576: warn if 'result' not of exact type int. */ - if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, - "__int__ returned non-int (type %.200s). " - "The ability to return an instance of a strict subclass of int " - "is deprecated, and may be removed in a future version of Python.", - result->ob_type->tp_name)) { - Py_DECREF(result); - return NULL; - } return (PyLongObject *)result; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 14 20:08:22 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 14 Dec 2013 20:08:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Null_merge?= Message-ID: <3dhdYy6NFTz7Ll1@mail.python.org> http://hg.python.org/cpython/rev/953d8ec1aeb3 changeset: 87960:953d8ec1aeb3 parent: 87958:804406d79b45 parent: 87959:a3de2b3881c1 user: Serhiy Storchaka date: Sat Dec 14 21:07:51 2013 +0200 summary: Null merge files: -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sun Dec 15 09:42:38 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 15 Dec 2013 09:42:38 +0100 Subject: [Python-checkins] Daily reference leaks (953d8ec1aeb3): sum=4 Message-ID: results for 953d8ec1aeb3 on branch "default" -------------------------------------------- test_site leaked [2, 0, 0] references, sum=2 test_site leaked [2, 0, 0] memory blocks, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogH85NCd', '-x'] From python-checkins at python.org Sun Dec 15 11:33:22 2013 From: python-checkins at python.org (nick.coghlan) Date: Sun, 15 Dec 2013 11:33:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319700=3A_set_=5F?= =?utf-8?q?=5Fspec=5F=5F_appropriately_in_runpy?= Message-ID: <3dj25G1ldGz7LjS@mail.python.org> http://hg.python.org/cpython/rev/51dddfead80a changeset: 87961:51dddfead80a user: Nick Coghlan date: Sun Dec 15 20:33:02 2013 +1000 summary: Issue #19700: set __spec__ appropriately in runpy Note that __spec__.name is not currently guaranteed to be in sys.modules when the code is running, only __name__ is. The "running module is in sys.modules" invariant will be expanded to also cover __spec__.name in a subsequent patch. files: Doc/library/runpy.rst | 69 ++++-- Lib/runpy.py | 115 +++++++---- Lib/test/script_helper.py | 6 +- Lib/test/test_cmd_line_script.py | 19 +- Lib/test/test_runpy.py | 179 +++++++++++++++--- 5 files changed, 276 insertions(+), 112 deletions(-) diff --git a/Doc/library/runpy.rst b/Doc/library/runpy.rst --- a/Doc/library/runpy.rst +++ b/Doc/library/runpy.rst @@ -44,28 +44,22 @@ below are defined in the supplied dictionary, those definitions are overridden by :func:`run_module`. - The special global variables ``__name__``, ``__file__``, ``__cached__``, - ``__loader__`` - and ``__package__`` are set in the globals dictionary before the module - code is executed (Note that this is a minimal set of variables - other - variables may be set implicitly as an interpreter implementation detail). + The special global variables ``__name__``, ``__spec__``, ``__file__``, + ``__cached__``, ``__loader__`` and ``__package__`` are set in the globals + dictionary before the module code is executed (Note that this is a + minimal set of variables - other variables may be set implicitly as an + interpreter implementation detail). ``__name__`` is set to *run_name* if this optional argument is not :const:`None`, to ``mod_name + '.__main__'`` if the named module is a package and to the *mod_name* argument otherwise. - ``__file__`` is set to the name provided by the module loader. If the - loader does not make filename information available, this variable is set - to :const:`None`. + ``__spec__`` will be set appropriately for the *actually* imported + module (that is, ``__spec__.name`` will always be *mod_name* or + ``mod_name + '.__main__``, never *run_name*). - ``__cached__`` will be set to ``None``. - - ``__loader__`` is set to the :pep:`302` module loader used to retrieve the - code for the module (This loader may be a wrapper around the standard - import mechanism). - - ``__package__`` is set to *mod_name* if the named module is a package and - to ``mod_name.rpartition('.')[0]`` otherwise. + ``__file__``, ``__cached__``, ``__loader__`` and ``__package__`` are + :ref:`set as normal ` based on the module spec. If the argument *alter_sys* is supplied and evaluates to :const:`True`, then ``sys.argv[0]`` is updated with the value of ``__file__`` and @@ -83,8 +77,13 @@ Added ability to execute packages by looking for a ``__main__`` submodule. .. versionchanged:: 3.2 - Added ``__cached__`` global variable (see :PEP:`3147`). + Added ``__cached__`` global variable (see :pep:`3147`). + .. versionchanged:: 3.4 + Updated to take advantage of the module spec feature added by + :pep:`451`. This allows ``__cached__`` to be set correctly for modules + run this way, as well as ensuring the real module name is always + accessible as ``__spec__.name``. .. function:: run_path(file_path, init_globals=None, run_name=None) @@ -108,23 +107,28 @@ below are defined in the supplied dictionary, those definitions are overridden by :func:`run_path`. - The special global variables ``__name__``, ``__file__``, ``__loader__`` - and ``__package__`` are set in the globals dictionary before the module - code is executed (Note that this is a minimal set of variables - other - variables may be set implicitly as an interpreter implementation detail). + The special global variables ``__name__``, ``__spec__``, ``__file__``, + ``__cached__``, ``__loader__`` and ``__package__`` are set in the globals + dictionary before the module code is executed (Note that this is a + minimal set of variables - other variables may be set implicitly as an + interpreter implementation detail). ``__name__`` is set to *run_name* if this optional argument is not :const:`None` and to ``''`` otherwise. - ``__file__`` is set to the name provided by the module loader. If the - loader does not make filename information available, this variable is set - to :const:`None`. For a simple script, this will be set to ``file_path``. + If the supplied path directly references a script file (whether as source + or as precompiled byte code), then ``__file__`` will be set to the + supplied path, and ``__spec__``, ``__cached__``, ``__loader__`` and + ``__package__`` will all be set to :const:`None`. - ``__loader__`` is set to the :pep:`302` module loader used to retrieve the - code for the module (This loader may be a wrapper around the standard - import mechanism). For a simple script, this will be set to :const:`None`. + ``__spec__`` will be set to :const:`None` if the supplied path is a + direct path to a script (as source or as precompiled bytecode). - ``__package__`` is set to ``__name__.rpartition('.')[0]``. + If the supplied path is a reference to a valid sys.path entry, then + ``__spec__`` will be set appropriately for the imported ``__main__`` + module (that is, ``__spec__.name`` will always be ``__main__``). + ``__file__``, ``__cached__``, ``__loader__`` and ``__package__`` will be + :ref:`set as normal ` based on the module spec. A number of alterations are also made to the :mod:`sys` module. Firstly, ``sys.path`` may be altered as described above. ``sys.argv[0]`` is updated @@ -141,6 +145,12 @@ .. versionadded:: 3.2 + .. versionchanged:: 3.4 + Updated to take advantage of the module spec feature added by + :pep:`451`. This allows ``__cached__`` to be set correctly in the + case where ``__main__`` is imported from a valid sys.path entry rather + than being executed directly. + .. seealso:: :pep:`338` - Executing modules as scripts @@ -149,6 +159,9 @@ :pep:`366` - Main module explicit relative imports PEP written and implemented by Nick Coghlan. + :pep:`451` - A ModuleSpec Type for the Import System + PEP written and implemented by Eric Snow + :ref:`using-on-general` - CPython command line details The :func:`importlib.import_module` function diff --git a/Lib/runpy.py b/Lib/runpy.py --- a/Lib/runpy.py +++ b/Lib/runpy.py @@ -14,7 +14,9 @@ import sys import importlib.machinery # importlib first so we can test #15386 via -m import types -from pkgutil import read_code, get_loader, get_importer +from importlib import find_spec +from importlib.util import spec_from_loader +from pkgutil import read_code, get_importer __all__ = [ "run_module", "run_path", @@ -58,51 +60,76 @@ self.value = self._sentinel sys.argv[0] = self._saved_value +# TODO: Replace these helpers with importlib._bootstrap._SpecMethods def _run_code(code, run_globals, init_globals=None, - mod_name=None, mod_fname=None, - mod_loader=None, pkg_name=None): + mod_name=None, mod_spec=None, + pkg_name=None, script_name=None): """Helper to run code in nominated namespace""" if init_globals is not None: run_globals.update(init_globals) + if mod_spec is None: + loader = None + fname = script_name + cached = None + else: + loader = mod_spec.loader + fname = mod_spec.origin + cached = mod_spec.cached + if pkg_name is None: + pkg_name = mod_spec.parent run_globals.update(__name__ = mod_name, - __file__ = mod_fname, - __cached__ = None, + __file__ = fname, + __cached__ = cached, __doc__ = None, - __loader__ = mod_loader, - __package__ = pkg_name) + __loader__ = loader, + __package__ = pkg_name, + __spec__ = mod_spec) exec(code, run_globals) return run_globals def _run_module_code(code, init_globals=None, - mod_name=None, mod_fname=None, - mod_loader=None, pkg_name=None): + mod_name=None, mod_spec=None, + pkg_name=None, script_name=None): """Helper to run code in new namespace with sys modified""" - with _TempModule(mod_name) as temp_module, _ModifiedArgv0(mod_fname): + fname = script_name if mod_spec is None else mod_spec.origin + with _TempModule(mod_name) as temp_module, _ModifiedArgv0(fname): mod_globals = temp_module.module.__dict__ _run_code(code, mod_globals, init_globals, - mod_name, mod_fname, mod_loader, pkg_name) + mod_name, mod_spec, pkg_name, script_name) # Copy the globals of the temporary module, as they # may be cleared when the temporary module goes away return mod_globals.copy() -# This helper is needed due to a missing component in the PEP 302 -# loader protocol (specifically, "get_filename" is non-standard) -# Since we can't introduce new features in maintenance releases, -# support was added to zipimporter under the name '_get_filename' -def _get_filename(loader, mod_name): - for attr in ("get_filename", "_get_filename"): - meth = getattr(loader, attr, None) - if meth is not None: - return os.path.abspath(meth(mod_name)) - return None +def _fixed_find_spec(mod_name): + # find_spec has the same annoying behaviour as find_loader did (it + # fails to work properly for dotted names), so this is a fixed version + # ala pkgutil.get_loader + if mod_name.startswith('.'): + msg = "Relative module name {!r} not supported".format(mod_name) + raise ImportError(msg) + path = None + pkg_name = mod_name.rpartition(".")[0] + if pkg_name: + pkg = importlib.import_module(pkg_name) + path = getattr(pkg, "__path__", None) + if path is None: + return None + try: + return importlib.find_spec(mod_name, path) + except (ImportError, AttributeError, TypeError, ValueError) as ex: + # This hack fixes an impedance mismatch between pkgutil and + # importlib, where the latter raises other errors for cases where + # pkgutil previously raised ImportError + msg = "Error while finding spec for {!r} ({}: {})" + raise ImportError(msg.format(mod_name, type(ex), ex)) from ex # Helper to get the loader, code and filename for a module def _get_module_details(mod_name): - loader = get_loader(mod_name) - if loader is None: + spec = _fixed_find_spec(mod_name) + if spec is None: raise ImportError("No module named %s" % mod_name) - if loader.is_package(mod_name): + if spec.submodule_search_locations is not None: if mod_name == "__main__" or mod_name.endswith(".__main__"): raise ImportError("Cannot use package as __main__ module") try: @@ -111,11 +138,14 @@ except ImportError as e: raise ImportError(("%s; %r is a package and cannot " + "be directly executed") %(e, mod_name)) + loader = spec.loader + if loader is None: + raise ImportError("%r is a namespace package and cannot be executed" + % mod_name) code = loader.get_code(mod_name) if code is None: raise ImportError("No code object available for %s" % mod_name) - filename = _get_filename(loader, mod_name) - return mod_name, loader, code, filename + return mod_name, spec, code # XXX ncoghlan: Should this be documented and made public? # (Current thoughts: don't repeat the mistake that lead to its @@ -137,9 +167,9 @@ """ try: if alter_argv or mod_name != "__main__": # i.e. -m switch - mod_name, loader, code, fname = _get_module_details(mod_name) + mod_name, mod_spec, code = _get_module_details(mod_name) else: # i.e. directory or zipfile execution - mod_name, loader, code, fname = _get_main_module_details() + mod_name, mod_spec, code = _get_main_module_details() except ImportError as exc: # Try to provide a good error message # for directories, zip files and the -m switch @@ -152,12 +182,11 @@ info = "can't find '__main__' module in %r" % sys.argv[0] msg = "%s: %s" % (sys.executable, info) sys.exit(msg) - pkg_name = mod_name.rpartition('.')[0] main_globals = sys.modules["__main__"].__dict__ if alter_argv: - sys.argv[0] = fname + sys.argv[0] = mod_spec.origin return _run_code(code, main_globals, None, - "__main__", fname, loader, pkg_name) + "__main__", mod_spec) def run_module(mod_name, init_globals=None, run_name=None, alter_sys=False): @@ -165,17 +194,14 @@ Returns the resulting top level namespace dictionary """ - mod_name, loader, code, fname = _get_module_details(mod_name) + mod_name, mod_spec, code = _get_module_details(mod_name) if run_name is None: run_name = mod_name - pkg_name = mod_name.rpartition('.')[0] if alter_sys: - return _run_module_code(code, init_globals, run_name, - fname, loader, pkg_name) + return _run_module_code(code, init_globals, run_name, mod_spec) else: # Leave the sys module alone - return _run_code(code, {}, init_globals, run_name, - fname, loader, pkg_name) + return _run_code(code, {}, init_globals, run_name, mod_spec) def _get_main_module_details(): # Helper that gives a nicer error message when attempting to @@ -204,10 +230,7 @@ # That didn't work, so try it as normal source code with open(fname, "rb") as f: code = compile(f.read(), fname, 'exec') - loader = importlib.machinery.SourceFileLoader(run_name, fname) - else: - loader = importlib.machinery.SourcelessFileLoader(run_name, fname) - return code, loader + return code, fname def run_path(path_name, init_globals=None, run_name=None): """Execute code located at the specified filesystem location @@ -231,9 +254,9 @@ if isinstance(importer, type(None)) or is_NullImporter: # Not a valid sys.path entry, so run the code directly # execfile() doesn't help as we want to allow compiled files - code, mod_loader = _get_code_from_file(run_name, path_name) - return _run_module_code(code, init_globals, run_name, path_name, - mod_loader, pkg_name) + code, fname = _get_code_from_file(run_name, path_name) + return _run_module_code(code, init_globals, run_name, + pkg_name=pkg_name, script_name=fname) else: # Importer is defined for path, so add it to # the start of sys.path @@ -245,12 +268,12 @@ # have no choice and we have to remove it even while we read the # code. If we don't do this, a __loader__ attribute in the # existing __main__ module may prevent location of the new module. - mod_name, loader, code, fname = _get_main_module_details() + mod_name, mod_spec, code = _get_main_module_details() with _TempModule(run_name) as temp_module, \ _ModifiedArgv0(path_name): mod_globals = temp_module.module.__dict__ return _run_code(code, mod_globals, init_globals, - run_name, fname, loader, pkg_name).copy() + run_name, mod_spec, pkg_name).copy() finally: try: sys.path.remove(path_name) diff --git a/Lib/test/script_helper.py b/Lib/test/script_helper.py --- a/Lib/test/script_helper.py +++ b/Lib/test/script_helper.py @@ -101,8 +101,10 @@ subprocess._cleanup() return data -def make_script(script_dir, script_basename, source): - script_filename = script_basename+os.extsep+'py' +def make_script(script_dir, script_basename, source, omit_suffix=False): + script_filename = script_basename + if not omit_suffix: + script_filename += os.extsep + 'py' script_name = os.path.join(script_dir, script_filename) # The script should be encoded to UTF-8, the default string encoding script_file = open(script_name, 'w', encoding='utf-8') diff --git a/Lib/test/test_cmd_line_script.py b/Lib/test/test_cmd_line_script.py --- a/Lib/test/test_cmd_line_script.py +++ b/Lib/test/test_cmd_line_script.py @@ -41,11 +41,28 @@ _loader = __loader__ if __loader__ is BuiltinImporter else type(__loader__) print('__loader__==%a' % _loader) print('__file__==%a' % __file__) -assertEqual(__cached__, None) +print('__cached__==%a' % __cached__) print('__package__==%r' % __package__) +# Check PEP 451 details +import os.path +if __package__ is not None: + print('__main__ was located through the import system') + assertIdentical(__spec__.loader, __loader__) + expected_spec_name = os.path.splitext(os.path.basename(__file__))[0] + if __package__: + expected_spec_name = __package__ + "." + expected_spec_name + assertEqual(__spec__.name, expected_spec_name) + assertEqual(__spec__.parent, __package__) + assertIdentical(__spec__.submodule_search_locations, None) + assertEqual(__spec__.origin, __file__) + if __spec__.cached is not None: + assertEqual(__spec__.cached, __cached__) # Check the sys module import sys assertIdentical(globals(), sys.modules[__name__].__dict__) +if __spec__ is not None: + # XXX: We're not currently making __main__ available under its real name + pass # assertIdentical(globals(), sys.modules[__spec__.name].__dict__) from test import test_cmd_line_script example_args_list = test_cmd_line_script.example_args assertEqual(sys.argv[1:], example_args_list) diff --git a/Lib/test/test_runpy.py b/Lib/test/test_runpy.py --- a/Lib/test/test_runpy.py +++ b/Lib/test/test_runpy.py @@ -5,7 +5,7 @@ import sys import re import tempfile -import importlib +import importlib, importlib.machinery, importlib.util import py_compile from test.support import ( forget, make_legacy_pyc, run_unittest, unload, verbose, no_tracing, @@ -47,7 +47,7 @@ "__cached__": None, "__package__": None, "__doc__": None, -# "__spec__": None, # XXX Uncomment. + "__spec__": None } example_namespace = { "sys": sys, @@ -57,7 +57,7 @@ "run_name_in_sys_modules": False, "module_in_sys_modules": False, "nested": dict(implicit_namespace, - x=1, __name__="", __loader__=None, __spec__=None), + x=1, __name__="", __loader__=None), } example_namespace.update(implicit_namespace) @@ -68,11 +68,19 @@ # testing occurs at those upper layers as well, not just at the utility # layer + # Figuring out the loader details in advance is hard to do, so we skip + # checking the full details of loader and loader_state + CHECKED_SPEC_ATTRIBUTES = ["name", "parent", "origin", "cached", + "has_location", "submodule_search_locations"] + def assertNamespaceMatches(self, result_ns, expected_ns): """Check two namespaces match. Ignores any unspecified interpreter created names """ + # Avoid side effects + result_ns = result_ns.copy() + expected_ns = expected_ns.copy() # Impls are permitted to add extra names, so filter them out for k in list(result_ns): if k.startswith("__") and k.endswith("__"): @@ -80,7 +88,25 @@ result_ns.pop(k) if k not in expected_ns["nested"]: result_ns["nested"].pop(k) - # Don't use direct dict comparison - the diffs are too hard to debug + # Spec equality includes the loader, so we take the spec out of the + # result namespace and check that separately + result_spec = result_ns.pop("__spec__") + expected_spec = expected_ns.pop("__spec__") + if expected_spec is None: + self.assertIsNone(result_spec) + else: + # If an expected loader is set, we just check we got the right + # type, rather than checking for full equality + if expected_spec.loader is not None: + self.assertEqual(type(result_spec.loader), + type(expected_spec.loader)) + for attr in self.CHECKED_SPEC_ATTRIBUTES: + k = "__spec__." + attr + actual = (k, getattr(result_spec, attr)) + expected = (k, getattr(expected_spec, attr)) + self.assertEqual(actual, expected) + # For the rest, we still don't use direct dict comparison on the + # namespace, as the diffs are too hard to debug if anything breaks self.assertEqual(set(result_ns), set(expected_ns)) for k in result_ns: actual = (k, result_ns[k]) @@ -131,12 +157,16 @@ mod_fname = "Some other nonsense" mod_loader = "Now you're just being silly" mod_package = '' # Treat as a top level module + mod_spec = importlib.machinery.ModuleSpec(mod_name, + origin=mod_fname, + loader=mod_loader) expected_ns = example_namespace.copy() expected_ns.update({ "__name__": mod_name, "__file__": mod_fname, "__loader__": mod_loader, "__package__": mod_package, + "__spec__": mod_spec, "run_argv0": mod_fname, "run_name_in_sys_modules": True, "module_in_sys_modules": True, @@ -145,9 +175,7 @@ return _run_module_code(example_source, init_globals, mod_name, - mod_fname, - mod_loader, - mod_package) + mod_spec) self.check_code_execution(create_ns, expected_ns) # TODO: Use self.addCleanup to get rid of a lot of try-finally blocks @@ -177,31 +205,43 @@ def test_library_module(self): self.assertEqual(run_module("runpy")["__name__"], "runpy") - def _add_pkg_dir(self, pkg_dir): + def _add_pkg_dir(self, pkg_dir, namespace=False): os.mkdir(pkg_dir) + if namespace: + return None pkg_fname = os.path.join(pkg_dir, "__init__.py") create_empty_file(pkg_fname) return pkg_fname - def _make_pkg(self, source, depth, mod_base="runpy_test"): + def _make_pkg(self, source, depth, mod_base="runpy_test", + *, namespace=False, parent_namespaces=False): + # Enforce a couple of internal sanity checks on test cases + if (namespace or parent_namespaces) and not depth: + raise RuntimeError("Can't mark top level module as a " + "namespace package") pkg_name = "__runpy_pkg__" test_fname = mod_base+os.extsep+"py" pkg_dir = sub_dir = os.path.realpath(tempfile.mkdtemp()) if verbose > 1: print(" Package tree in:", sub_dir) sys.path.insert(0, pkg_dir) if verbose > 1: print(" Updated sys.path:", sys.path[0]) - for i in range(depth): - sub_dir = os.path.join(sub_dir, pkg_name) - pkg_fname = self._add_pkg_dir(sub_dir) - if verbose > 1: print(" Next level in:", sub_dir) - if verbose > 1: print(" Created:", pkg_fname) + if depth: + namespace_flags = [parent_namespaces] * depth + namespace_flags[-1] = namespace + for namespace_flag in namespace_flags: + sub_dir = os.path.join(sub_dir, pkg_name) + pkg_fname = self._add_pkg_dir(sub_dir, namespace_flag) + if verbose > 1: print(" Next level in:", sub_dir) + if verbose > 1: print(" Created:", pkg_fname) mod_fname = os.path.join(sub_dir, test_fname) mod_file = open(mod_fname, "w") mod_file.write(source) mod_file.close() if verbose > 1: print(" Created:", mod_fname) mod_name = (pkg_name+".")*depth + mod_base - return pkg_dir, mod_fname, mod_name + mod_spec = importlib.util.spec_from_file_location(mod_name, + mod_fname) + return pkg_dir, mod_fname, mod_name, mod_spec def _del_pkg(self, top, depth, mod_name): for entry in list(sys.modules): @@ -231,20 +271,29 @@ def _fix_ns_for_legacy_pyc(self, ns, alter_sys): char_to_add = "c" if __debug__ else "o" ns["__file__"] += char_to_add + ns["__cached__"] = ns["__file__"] + spec = ns["__spec__"] + new_spec = importlib.util.spec_from_file_location(spec.name, + ns["__file__"]) + ns["__spec__"] = new_spec if alter_sys: ns["run_argv0"] += char_to_add - def _check_module(self, depth, alter_sys=False): - pkg_dir, mod_fname, mod_name = ( - self._make_pkg(example_source, depth)) + def _check_module(self, depth, alter_sys=False, + *, namespace=False, parent_namespaces=False): + pkg_dir, mod_fname, mod_name, mod_spec = ( + self._make_pkg(example_source, depth, + namespace=namespace, + parent_namespaces=parent_namespaces)) forget(mod_name) expected_ns = example_namespace.copy() expected_ns.update({ "__name__": mod_name, "__file__": mod_fname, + "__cached__": mod_spec.cached, "__package__": mod_name.rpartition(".")[0], -# "__spec__": None, # XXX Needs to be set. + "__spec__": mod_spec, }) if alter_sys: expected_ns.update({ @@ -271,17 +320,21 @@ self._del_pkg(pkg_dir, depth, mod_name) if verbose > 1: print("Module executed successfully") - def _check_package(self, depth, alter_sys=False): - pkg_dir, mod_fname, mod_name = ( - self._make_pkg(example_source, depth, "__main__")) + def _check_package(self, depth, alter_sys=False, + *, namespace=False, parent_namespaces=False): + pkg_dir, mod_fname, mod_name, mod_spec = ( + self._make_pkg(example_source, depth, "__main__", + namespace=namespace, + parent_namespaces=parent_namespaces)) pkg_name = mod_name.rpartition(".")[0] forget(mod_name) expected_ns = example_namespace.copy() expected_ns.update({ "__name__": mod_name, "__file__": mod_fname, + "__cached__": importlib.util.cache_from_source(mod_fname), "__package__": pkg_name, -# "__spec__": None, # XXX Needs to be set. + "__spec__": mod_spec, }) if alter_sys: expected_ns.update({ @@ -337,7 +390,7 @@ from . import sibling from ..uncle.cousin import nephew """ - pkg_dir, mod_fname, mod_name = ( + pkg_dir, mod_fname, mod_name, mod_spec = ( self._make_pkg(contents, depth)) if run_name is None: expected_name = mod_name @@ -376,11 +429,31 @@ if verbose > 1: print("Testing package depth:", depth) self._check_module(depth) + def test_run_module_in_namespace_package(self): + for depth in range(1, 4): + if verbose > 1: print("Testing package depth:", depth) + self._check_module(depth, namespace=True, parent_namespaces=True) + def test_run_package(self): for depth in range(1, 4): if verbose > 1: print("Testing package depth:", depth) self._check_package(depth) + def test_run_package_in_namespace_package(self): + for depth in range(1, 4): + if verbose > 1: print("Testing package depth:", depth) + self._check_package(depth, parent_namespaces=True) + + def test_run_namespace_package(self): + for depth in range(1, 4): + if verbose > 1: print("Testing package depth:", depth) + self._check_package(depth, namespace=True) + + def test_run_namespace_package_in_namespace_package(self): + for depth in range(1, 4): + if verbose > 1: print("Testing package depth:", depth) + self._check_package(depth, namespace=True, parent_namespaces=True) + def test_run_module_alter_sys(self): for depth in range(4): if verbose > 1: print("Testing package depth:", depth) @@ -404,14 +477,16 @@ def test_run_name(self): depth = 1 run_name = "And now for something completely different" - pkg_dir, mod_fname, mod_name = ( + pkg_dir, mod_fname, mod_name, mod_spec = ( self._make_pkg(example_source, depth)) forget(mod_name) expected_ns = example_namespace.copy() expected_ns.update({ "__name__": run_name, "__file__": mod_fname, + "__cached__": importlib.util.cache_from_source(mod_fname), "__package__": mod_name.rpartition(".")[0], + "__spec__": mod_spec, }) def create_ns(init_globals): return run_module(mod_name, init_globals, run_name) @@ -440,7 +515,7 @@ pkg_name = ".".join([base_name] * max_depth) expected_packages.add(pkg_name) expected_modules.add(pkg_name + ".runpy_test") - pkg_dir, mod_fname, mod_name = ( + pkg_dir, mod_fname, mod_name, mod_spec = ( self._make_pkg("", max_depth)) self.addCleanup(self._del_pkg, pkg_dir, max_depth, mod_name) for depth in range(2, max_depth+1): @@ -458,21 +533,39 @@ class RunPathTestCase(unittest.TestCase, CodeExecutionMixin): """Unit tests for runpy.run_path""" - def _make_test_script(self, script_dir, script_basename, source=None): + def _make_test_script(self, script_dir, script_basename, + source=None, omit_suffix=False): if source is None: source = example_source - return make_script(script_dir, script_basename, source) + return make_script(script_dir, script_basename, + source, omit_suffix) def _check_script(self, script_name, expected_name, expected_file, - expected_argv0): + expected_argv0, mod_name=None, + expect_spec=True, check_loader=True): # First check is without run_name def create_ns(init_globals): return run_path(script_name, init_globals) expected_ns = example_namespace.copy() + if mod_name is None: + spec_name = expected_name + else: + spec_name = mod_name + if expect_spec: + mod_spec = importlib.util.spec_from_file_location(spec_name, + expected_file) + mod_cached = mod_spec.cached + if not check_loader: + mod_spec.loader = None + else: + mod_spec = mod_cached = None + expected_ns.update({ "__name__": expected_name, "__file__": expected_file, + "__cached__": mod_cached, "__package__": "", + "__spec__": mod_spec, "run_argv0": expected_argv0, "run_name_in_sys_modules": True, "module_in_sys_modules": True, @@ -482,6 +575,12 @@ run_name = "prove.issue15230.is.fixed" def create_ns(init_globals): return run_path(script_name, init_globals, run_name) + if expect_spec and mod_name is None: + mod_spec = importlib.util.spec_from_file_location(run_name, + expected_file) + if not check_loader: + mod_spec.loader = None + expected_ns["__spec__"] = mod_spec expected_ns["__name__"] = run_name expected_ns["__package__"] = run_name.rpartition(".")[0] self.check_code_execution(create_ns, expected_ns) @@ -495,7 +594,15 @@ mod_name = 'script' script_name = self._make_test_script(script_dir, mod_name) self._check_script(script_name, "", script_name, - script_name) + script_name, expect_spec=False) + + def test_basic_script_no_suffix(self): + with temp_dir() as script_dir: + mod_name = 'script' + script_name = self._make_test_script(script_dir, mod_name, + omit_suffix=True) + self._check_script(script_name, "", script_name, + script_name, expect_spec=False) def test_script_compiled(self): with temp_dir() as script_dir: @@ -504,14 +611,14 @@ compiled_name = py_compile.compile(script_name, doraise=True) os.remove(script_name) self._check_script(compiled_name, "", compiled_name, - compiled_name) + compiled_name, expect_spec=False) def test_directory(self): with temp_dir() as script_dir: mod_name = '__main__' script_name = self._make_test_script(script_dir, mod_name) self._check_script(script_dir, "", script_name, - script_dir) + script_dir, mod_name=mod_name) def test_directory_compiled(self): with temp_dir() as script_dir: @@ -522,7 +629,7 @@ if not sys.dont_write_bytecode: legacy_pyc = make_legacy_pyc(script_name) self._check_script(script_dir, "", legacy_pyc, - script_dir) + script_dir, mod_name=mod_name) def test_directory_error(self): with temp_dir() as script_dir: @@ -536,7 +643,8 @@ mod_name = '__main__' script_name = self._make_test_script(script_dir, mod_name) zip_name, fname = make_zip_script(script_dir, 'test_zip', script_name) - self._check_script(zip_name, "", fname, zip_name) + self._check_script(zip_name, "", fname, zip_name, + mod_name=mod_name, check_loader=False) def test_zipfile_compiled(self): with temp_dir() as script_dir: @@ -545,7 +653,8 @@ compiled_name = py_compile.compile(script_name, doraise=True) zip_name, fname = make_zip_script(script_dir, 'test_zip', compiled_name) - self._check_script(zip_name, "", fname, zip_name) + self._check_script(zip_name, "", fname, zip_name, + mod_name=mod_name, check_loader=False) def test_zipfile_error(self): with temp_dir() as script_dir: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 15 15:37:06 2013 From: python-checkins at python.org (christian.heimes) Date: Sun, 15 Dec 2013 15:37:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Import_USHRT=5FMAX_to_fix_?= =?utf-8?q?failing_test=5Fdevpool_test?= Message-ID: <3dj7VV4kHkz7LjQ@mail.python.org> http://hg.python.org/cpython/rev/039306b45230 changeset: 87962:039306b45230 user: Christian Heimes date: Sun Dec 15 15:36:20 2013 +0100 summary: Import USHRT_MAX to fix failing test_devpool test files: Lib/test/test_devpoll.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_devpoll.py b/Lib/test/test_devpoll.py --- a/Lib/test/test_devpoll.py +++ b/Lib/test/test_devpoll.py @@ -8,6 +8,7 @@ import sys import unittest from test.support import TESTFN, run_unittest +from _testcapi import USHRT_MAX try: select.devpoll -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 15 17:02:44 2013 From: python-checkins at python.org (christian.heimes) Date: Sun, 15 Dec 2013 17:02:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Skip_some_tests_that_requi?= =?utf-8?q?re_server_name_indication_=28SNI=29?= Message-ID: <3dj9PJ4c42z7Ljs@mail.python.org> http://hg.python.org/cpython/rev/7d897144c39c changeset: 87963:7d897144c39c user: Christian Heimes date: Sun Dec 15 17:02:36 2013 +0100 summary: Skip some tests that require server name indication (SNI) files: Lib/test/test_asyncio/test_events.py | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -10,6 +10,9 @@ import ssl except ImportError: ssl = None + HAS_SNI = False +else: + from ssl import HAS_SNI import subprocess import sys import threading @@ -666,6 +669,7 @@ server.close() @unittest.skipIf(ssl is None, 'No ssl module') + @unittest.skipUnless(HAS_SNI, 'No SNI support in ssl module') def test_create_server_ssl_verify_failed(self): proto = None @@ -694,6 +698,7 @@ server.close() @unittest.skipIf(ssl is None, 'No ssl module') + @unittest.skipUnless(HAS_SNI, 'No SNI support in ssl module') def test_create_server_ssl_match_failed(self): proto = None @@ -724,6 +729,7 @@ server.close() @unittest.skipIf(ssl is None, 'No ssl module') + @unittest.skipUnless(HAS_SNI, 'No SNI support in ssl module') def test_create_server_ssl_verified(self): proto = None -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 15 19:38:30 2013 From: python-checkins at python.org (christian.heimes) Date: Sun, 15 Dec 2013 19:38:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_test=5Fftplib=3A_skip_chec?= =?utf-8?q?k=5Fhostname_test_when_SNI_is_not_available?= Message-ID: <3djDs20NCKz7Lk6@mail.python.org> http://hg.python.org/cpython/rev/5d6d6ee9125a changeset: 87964:5d6d6ee9125a user: Christian Heimes date: Sun Dec 15 19:38:22 2013 +0100 summary: test_ftplib: skip check_hostname test when SNI is not available files: Lib/test/test_ftplib.py | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_ftplib.py b/Lib/test/test_ftplib.py --- a/Lib/test/test_ftplib.py +++ b/Lib/test/test_ftplib.py @@ -15,6 +15,9 @@ import ssl except ImportError: ssl = None + HAS_SNI = False +else: + from ssl import HAS_SNI from unittest import TestCase, skipUnless from test import support @@ -924,6 +927,7 @@ self.client.ccc() self.assertRaises(ValueError, self.client.sock.unwrap) + @unittest.skipUnless(HAS_SNI, 'No SNI support in ssl module') def test_check_hostname(self): self.client.quit() ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 15 19:39:39 2013 From: python-checkins at python.org (christian.heimes) Date: Sun, 15 Dec 2013 19:39:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_test=5Fimaplib=3A_skip_che?= =?utf-8?q?ck=5Fhostname_test_when_SNI_is_not_available?= Message-ID: <3djDtM1Mhhz7Lk6@mail.python.org> http://hg.python.org/cpython/rev/7670f8b4234c changeset: 87965:7670f8b4234c user: Christian Heimes date: Sun Dec 15 19:39:32 2013 +0100 summary: test_imaplib: skip check_hostname test when SNI is not available files: Lib/test/test_imaplib.py | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_imaplib.py b/Lib/test/test_imaplib.py --- a/Lib/test/test_imaplib.py +++ b/Lib/test/test_imaplib.py @@ -18,6 +18,9 @@ import ssl except ImportError: ssl = None + HAS_SNI = False +else: + from ssl import HAS_SNI CERTFILE = None CAFILE = None @@ -349,6 +352,7 @@ imap_class = IMAP4_SSL @reap_threads + @unittest.skipUnless(HAS_SNI, 'No SNI support in ssl module') def test_ssl_verified(self): ssl_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) ssl_context.verify_mode = ssl.CERT_REQUIRED -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 15 19:50:23 2013 From: python-checkins at python.org (christian.heimes) Date: Sun, 15 Dec 2013 19:50:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_fixed_test=5Fftplib?= Message-ID: <3djF6l1HkSz7LkT@mail.python.org> http://hg.python.org/cpython/rev/6cb93e45a3be changeset: 87966:6cb93e45a3be user: Christian Heimes date: Sun Dec 15 19:50:13 2013 +0100 summary: fixed test_ftplib files: Lib/test/test_ftplib.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_ftplib.py b/Lib/test/test_ftplib.py --- a/Lib/test/test_ftplib.py +++ b/Lib/test/test_ftplib.py @@ -927,7 +927,7 @@ self.client.ccc() self.assertRaises(ValueError, self.client.sock.unwrap) - @unittest.skipUnless(HAS_SNI, 'No SNI support in ssl module') + @skipUnless(HAS_SNI, 'No SNI support in ssl module') def test_check_hostname(self): self.client.quit() ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 15 20:56:21 2013 From: python-checkins at python.org (raymond.hettinger) Date: Sun, 15 Dec 2013 20:56:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Minor_code_clean-up=2E__Ke?= =?utf-8?q?ep_the_C-API_all_in_one_section=2E?= Message-ID: <3djGZs0rtRz7LkW@mail.python.org> http://hg.python.org/cpython/rev/24058a1eb022 changeset: 87967:24058a1eb022 user: Raymond Hettinger date: Sun Dec 15 11:56:14 2013 -0800 summary: Minor code clean-up. Keep the C-API all in one section. files: Objects/setobject.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -36,9 +36,6 @@ #define dummy (&_dummy_struct) -/* Exported for the gdb plugin's benefit. */ -PyObject *_PySet_Dummy = dummy; - /* ======================================================================== */ /* ======= Begin logic for probing the hash table ========================= */ @@ -2345,6 +2342,9 @@ return set_update_internal((PySetObject *)set, iterable); } +/* Exported for the gdb plugin's benefit. */ +PyObject *_PySet_Dummy = dummy; + #ifdef Py_DEBUG /* Test code to be called with any three element set. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 15 21:03:32 2013 From: python-checkins at python.org (stefan.krah) Date: Sun, 15 Dec 2013 21:03:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5OTg2?= =?utf-8?q?=3A_Avoid_an_incorrect_warning_of_older_gcc_versions=2E?= Message-ID: <3djGl85dcCz7Lk3@mail.python.org> http://hg.python.org/cpython/rev/274b293435fb changeset: 87968:274b293435fb branch: 3.3 parent: 87959:a3de2b3881c1 user: Stefan Krah date: Sun Dec 15 20:45:08 2013 +0100 summary: Issue #19986: Avoid an incorrect warning of older gcc versions. files: Modules/_decimal/libmpdec/mpdecimal.c | 11 ++++++----- 1 files changed, 6 insertions(+), 5 deletions(-) diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -4421,21 +4421,22 @@ const mpd_context_t *ctx, uint32_t *status) { uint32_t workstatus = 0; - const mpd_t *cc = c; + mpd_t *cc = NULL; if (result == c) { if ((cc = mpd_qncopy(c)) == NULL) { mpd_seterror(result, MPD_Malloc_error, status); return; } + c = cc; } _mpd_qmul(result, a, b, ctx, &workstatus); if (!(workstatus&MPD_Invalid_operation)) { - mpd_qadd(result, result, cc, ctx, &workstatus); - } - - if (cc != c) mpd_del((mpd_t *)cc); + mpd_qadd(result, result, c, ctx, &workstatus); + } + + if (cc) mpd_del(cc); *status |= workstatus; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 15 21:03:34 2013 From: python-checkins at python.org (stefan.krah) Date: Sun, 15 Dec 2013 21:03:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogTWVyZ2UgZnJvbSAzLjMu?= Message-ID: <3djGlB0PYJz7Lkr@mail.python.org> http://hg.python.org/cpython/rev/936b35acec50 changeset: 87969:936b35acec50 parent: 87966:6cb93e45a3be parent: 87968:274b293435fb user: Stefan Krah date: Sun Dec 15 20:52:08 2013 +0100 summary: Merge from 3.3. files: Modules/_decimal/libmpdec/mpdecimal.c | 11 ++++++----- 1 files changed, 6 insertions(+), 5 deletions(-) diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -4421,21 +4421,22 @@ const mpd_context_t *ctx, uint32_t *status) { uint32_t workstatus = 0; - const mpd_t *cc = c; + mpd_t *cc = NULL; if (result == c) { if ((cc = mpd_qncopy(c)) == NULL) { mpd_seterror(result, MPD_Malloc_error, status); return; } + c = cc; } _mpd_qmul(result, a, b, ctx, &workstatus); if (!(workstatus&MPD_Invalid_operation)) { - mpd_qadd(result, result, cc, ctx, &workstatus); - } - - if (cc != c) mpd_del((mpd_t *)cc); + mpd_qadd(result, result, c, ctx, &workstatus); + } + + if (cc) mpd_del(cc); *status |= workstatus; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 15 21:03:35 2013 From: python-checkins at python.org (stefan.krah) Date: Sun, 15 Dec 2013 21:03:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?b?KTogTWVyZ2Uu?= Message-ID: <3djGlC2BZSz7Lkr@mail.python.org> http://hg.python.org/cpython/rev/2dcf1482b3e6 changeset: 87970:2dcf1482b3e6 parent: 87969:936b35acec50 parent: 87967:24058a1eb022 user: Stefan Krah date: Sun Dec 15 21:02:28 2013 +0100 summary: Merge. files: Objects/setobject.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Objects/setobject.c b/Objects/setobject.c --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -36,9 +36,6 @@ #define dummy (&_dummy_struct) -/* Exported for the gdb plugin's benefit. */ -PyObject *_PySet_Dummy = dummy; - /* ======================================================================== */ /* ======= Begin logic for probing the hash table ========================= */ @@ -2345,6 +2342,9 @@ return set_update_internal((PySetObject *)set, iterable); } +/* Exported for the gdb plugin's benefit. */ +PyObject *_PySet_Dummy = dummy; + #ifdef Py_DEBUG /* Test code to be called with any three element set. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 15 21:44:52 2013 From: python-checkins at python.org (christian.heimes) Date: Sun, 15 Dec 2013 21:44:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_test=5Fpoplib=3A_skip_stls?= =?utf-8?q?=5Fcontext_test_when_SNI_is_not_available?= Message-ID: <3djHfr4Cdyz7LkJ@mail.python.org> http://hg.python.org/cpython/rev/c00c1715d599 changeset: 87971:c00c1715d599 user: Christian Heimes date: Sun Dec 15 21:44:43 2013 +0100 summary: test_poplib: skip stls_context test when SNI is not available files: Lib/test/test_poplib.py | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_poplib.py b/Lib/test/test_poplib.py --- a/Lib/test/test_poplib.py +++ b/Lib/test/test_poplib.py @@ -21,10 +21,14 @@ SUPPORTS_SSL = False if hasattr(poplib, 'POP3_SSL'): import ssl + from ssl import HAS_SNI SUPPORTS_SSL = True CERTFILE = os.path.join(os.path.dirname(__file__) or os.curdir, "keycert3.pem") CAFILE = os.path.join(os.path.dirname(__file__) or os.curdir, "pycacert.pem") +else: + HAS_SNI = False + requires_ssl = skipUnless(SUPPORTS_SSL, 'SSL not supported') # the dummy data returned by server when LIST and RETR commands are issued @@ -330,6 +334,7 @@ self.assertEqual(resp, expected) @requires_ssl + @skipUnless(HAS_SNI, 'No SNI support in ssl module') def test_stls_context(self): expected = b'+OK Begin TLS negotiation' ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 15 21:49:26 2013 From: python-checkins at python.org (christian.heimes) Date: Sun, 15 Dec 2013 21:49:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_test=5Fssl=3A_skip_tests_w?= =?utf-8?q?hen_SNI_is_not_available?= Message-ID: <3djHm63bDlz7LkL@mail.python.org> http://hg.python.org/cpython/rev/629852f6d186 changeset: 87972:629852f6d186 user: Christian Heimes date: Sun Dec 15 21:49:17 2013 +0100 summary: test_ssl: skip tests when SNI is not available files: Lib/test/test_ssl.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1419,6 +1419,7 @@ s.close() self.assertEqual(len(ctx.get_ca_certs()), 1) + @needs_sni def test_context_setget(self): # Check that the context of a connected socket can be replaced. with support.transient_internet("svn.python.org"): @@ -1970,6 +1971,7 @@ cert = s.getpeercert() self.assertTrue(cert, "Can't get peer certificate.") + @needs_sni def test_check_hostname(self): if support.verbose: sys.stdout.write("\n") -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 02:57:06 2013 From: python-checkins at python.org (r.david.murray) Date: Mon, 16 Dec 2013 02:57:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzE5NTMyOiBtYWtl?= =?utf-8?q?_compileall_with_no_file/dir_args_respect_-f_and_-q=2E?= Message-ID: <3djQb65JNFz7LkL@mail.python.org> http://hg.python.org/cpython/rev/6afad4f29249 changeset: 87973:6afad4f29249 branch: 3.3 parent: 87968:274b293435fb user: R David Murray date: Sun Dec 15 20:49:38 2013 -0500 summary: #19532: make compileall with no file/dir args respect -f and -q. Patch by Vajrasky Kok. files: Lib/compileall.py | 3 ++- Lib/test/test_compileall.py | 23 +++++++++++++++++++++++ Misc/NEWS | 3 +++ 3 files changed, 28 insertions(+), 1 deletions(-) diff --git a/Lib/compileall.py b/Lib/compileall.py --- a/Lib/compileall.py +++ b/Lib/compileall.py @@ -228,7 +228,8 @@ success = False return success else: - return compile_path(legacy=args.legacy) + return compile_path(legacy=args.legacy, force=args.force, + quiet=args.quiet) except KeyboardInterrupt: print("\n[interrupted]") return False diff --git a/Lib/test/test_compileall.py b/Lib/test/test_compileall.py --- a/Lib/test/test_compileall.py +++ b/Lib/test/test_compileall.py @@ -177,6 +177,29 @@ self.assertNotCompiled(self.initfn) self.assertNotCompiled(self.barfn) + def test_no_args_respects_force_flag(self): + bazfn = script_helper.make_script(self.directory, 'baz', '') + self.assertRunOK(PYTHONPATH=self.directory) + pycpath = imp.cache_from_source(bazfn) + # Set atime/mtime backward to avoid file timestamp resolution issues + os.utime(pycpath, (time.time()-60,)*2) + mtime = os.stat(pycpath).st_mtime + # Without force, no recompilation + self.assertRunOK(PYTHONPATH=self.directory) + mtime2 = os.stat(pycpath).st_mtime + self.assertEqual(mtime, mtime2) + # Now force it. + self.assertRunOK('-f', PYTHONPATH=self.directory) + mtime2 = os.stat(pycpath).st_mtime + self.assertNotEqual(mtime, mtime2) + + def test_no_args_respects_quiet_flag(self): + script_helper.make_script(self.directory, 'baz', '') + noisy = self.assertRunOK(PYTHONPATH=self.directory) + self.assertIn(b'Listing ', noisy) + quiet = self.assertRunOK('-q', PYTHONPATH=self.directory) + self.assertNotIn(b'Listing ', quiet) + # Ensure that the default behavior of compileall's CLI is to create # PEP 3147 pyc/pyo files. for name, ext, switch in [ diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,9 @@ Library ------- +- Issue #19532: python -m compileall with no filename/directory arguments now + respects the -f and -q flags instead of ignoring them. + - Issue #19623: Fixed writing to unseekable files in the aifc module. - Issue #17919: select.poll.register() again works with poll.POLLNVAL on AIX. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 02:57:08 2013 From: python-checkins at python.org (r.david.murray) Date: Mon, 16 Dec 2013 02:57:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge=3A_=2319532=3A_make_compileall_with_no_file/dir_ar?= =?utf-8?q?gs_respect_-f_and_-q=2E?= Message-ID: <3djQb804nLz7Lls@mail.python.org> http://hg.python.org/cpython/rev/0e07ab605e0b changeset: 87974:0e07ab605e0b parent: 87972:629852f6d186 parent: 87973:6afad4f29249 user: R David Murray date: Sun Dec 15 20:56:00 2013 -0500 summary: Merge: #19532: make compileall with no file/dir args respect -f and -q. files: Lib/compileall.py | 3 ++- Lib/test/test_compileall.py | 25 +++++++++++++++++++++++-- Misc/NEWS | 3 +++ 3 files changed, 28 insertions(+), 3 deletions(-) diff --git a/Lib/compileall.py b/Lib/compileall.py --- a/Lib/compileall.py +++ b/Lib/compileall.py @@ -229,7 +229,8 @@ success = False return success else: - return compile_path(legacy=args.legacy) + return compile_path(legacy=args.legacy, force=args.force, + quiet=args.quiet) except KeyboardInterrupt: print("\n[interrupted]") return False diff --git a/Lib/test/test_compileall.py b/Lib/test/test_compileall.py --- a/Lib/test/test_compileall.py +++ b/Lib/test/test_compileall.py @@ -5,8 +5,6 @@ import py_compile import shutil import struct -import subprocess -import sys import tempfile import time import unittest @@ -181,6 +179,29 @@ self.assertNotCompiled(self.initfn) self.assertNotCompiled(self.barfn) + def test_no_args_respects_force_flag(self): + bazfn = script_helper.make_script(self.directory, 'baz', '') + self.assertRunOK(PYTHONPATH=self.directory) + pycpath = importlib.util.cache_from_source(bazfn) + # Set atime/mtime backward to avoid file timestamp resolution issues + os.utime(pycpath, (time.time()-60,)*2) + mtime = os.stat(pycpath).st_mtime + # Without force, no recompilation + self.assertRunOK(PYTHONPATH=self.directory) + mtime2 = os.stat(pycpath).st_mtime + self.assertEqual(mtime, mtime2) + # Now force it. + self.assertRunOK('-f', PYTHONPATH=self.directory) + mtime2 = os.stat(pycpath).st_mtime + self.assertNotEqual(mtime, mtime2) + + def test_no_args_respects_quiet_flag(self): + script_helper.make_script(self.directory, 'baz', '') + noisy = self.assertRunOK(PYTHONPATH=self.directory) + self.assertIn(b'Listing ', noisy) + quiet = self.assertRunOK('-q', PYTHONPATH=self.directory) + self.assertNotIn(b'Listing ', quiet) + # Ensure that the default behavior of compileall's CLI is to create # PEP 3147 pyc/pyo files. for name, ext, switch in [ diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #19532: python -m compileall with no filename/directory arguments now + respects the -f and -q flags instead of ignoring them. + - Issue #19623: Fixed writing to unseekable files in the aifc module. - Issue #19946: multiprocessing.spawn now raises ImportError when the module to -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Mon Dec 16 09:41:10 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 16 Dec 2013 09:41:10 +0100 Subject: [Python-checkins] Daily reference leaks (0e07ab605e0b): sum=-4 Message-ID: results for 0e07ab605e0b on branch "default" -------------------------------------------- test_site leaked [0, 0, -2] references, sum=-2 test_site leaked [0, 0, -2] memory blocks, sum=-2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogvxHyaw', '-x'] From python-checkins at python.org Mon Dec 16 13:28:59 2013 From: python-checkins at python.org (christian.heimes) Date: Mon, 16 Dec 2013 13:28:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE3OTE5?= =?utf-8?q?=3A_add_missing_import_of_USHRT=5FMAX?= Message-ID: <3djhcC61F2z7Lk3@mail.python.org> http://hg.python.org/cpython/rev/c42647d76bd1 changeset: 87975:c42647d76bd1 branch: 3.3 parent: 87973:6afad4f29249 user: Christian Heimes date: Mon Dec 16 13:27:16 2013 +0100 summary: Issue #17919: add missing import of USHRT_MAX files: Lib/test/test_devpoll.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_devpoll.py b/Lib/test/test_devpoll.py --- a/Lib/test/test_devpoll.py +++ b/Lib/test/test_devpoll.py @@ -4,6 +4,7 @@ import os, select, random, unittest, sys from test.support import TESTFN, run_unittest +from _testcapi import USHRT_MAX try: select.devpoll -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 13:29:01 2013 From: python-checkins at python.org (christian.heimes) Date: Mon, 16 Dec 2013 13:29:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2317919=3A_add_missing_import_of_USHRT=5FMAX?= Message-ID: <3djhcF0nFPz7LkX@mail.python.org> http://hg.python.org/cpython/rev/1f3f4147c35e changeset: 87976:1f3f4147c35e parent: 87974:0e07ab605e0b parent: 87975:c42647d76bd1 user: Christian Heimes date: Mon Dec 16 13:28:20 2013 +0100 summary: Issue #17919: add missing import of USHRT_MAX files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 13:50:42 2013 From: python-checkins at python.org (charles-francois.natali) Date: Mon, 16 Dec 2013 13:50:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5OTY1?= =?utf-8?q?=3A_Make_sure_that_Python-ast=2Eh_is_properly_taken_into_accoun?= =?utf-8?q?t_in_the?= Message-ID: <3djj5G1yhGz7Lkm@mail.python.org> http://hg.python.org/cpython/rev/874813a3523d changeset: 87977:874813a3523d branch: 2.7 parent: 87956:c10ea224392d user: Charles-Fran?ois Natali date: Sun Dec 15 19:07:44 2013 +0100 summary: Issue #19965: Make sure that Python-ast.h is properly taken into account in the makefile. files: Makefile.pre.in | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -750,7 +750,8 @@ Include/warnings.h \ Include/weakrefobject.h \ pyconfig.h \ - $(PARSER_HEADERS) + $(PARSER_HEADERS) \ + $(AST_H) $(LIBRARY_OBJS) $(MODOBJS) Modules/python.o: $(PYTHON_HEADERS) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 13:50:43 2013 From: python-checkins at python.org (charles-francois.natali) Date: Mon, 16 Dec 2013 13:50:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5OTY1?= =?utf-8?q?=3A_Make_sure_that_Python-ast=2Eh_is_properly_taken_into_accoun?= =?utf-8?q?t_in_the?= Message-ID: <3djj5H3q40z7Lld@mail.python.org> http://hg.python.org/cpython/rev/cfe0a293551f changeset: 87978:cfe0a293551f branch: 3.3 parent: 87813:78efa2c06447 user: Charles-Fran?ois Natali date: Sun Dec 15 19:09:00 2013 +0100 summary: Issue #19965: Make sure that Python-ast.h is properly taken into account in the makefile. files: Makefile.pre.in | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -848,7 +848,8 @@ $(srcdir)/Include/warnings.h \ $(srcdir)/Include/weakrefobject.h \ pyconfig.h \ - $(PARSER_HEADERS) + $(PARSER_HEADERS) \ + $(AST_H) $(LIBRARY_OBJS) $(MODOBJS) Modules/python.o: $(PYTHON_HEADERS) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 13:50:46 2013 From: python-checkins at python.org (charles-francois.natali) Date: Mon, 16 Dec 2013 13:50:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuMyk6?= =?utf-8?q?_Merge=2E?= Message-ID: <3djj5L0P28z7Llp@mail.python.org> http://hg.python.org/cpython/rev/06122a4ecd37 changeset: 87979:06122a4ecd37 branch: 3.3 parent: 87978:cfe0a293551f parent: 87959:a3de2b3881c1 user: Charles-Fran?ois Natali date: Sun Dec 15 19:12:07 2013 +0100 summary: Merge. files: Doc/distutils/index.rst | 10 + Doc/extending/index.rst | 9 + Doc/faq/programming.rst | 43 +- Doc/install/index.rst | 18 +- Doc/library/bz2.rst | 5 + Doc/library/gzip.rst | 5 + Doc/library/importlib.rst | 7 +- Doc/library/lzma.rst | 5 + Doc/library/mailbox.rst | 4 +- Doc/license.rst | 33 + Include/import.h | 8 +- Include/longobject.h | 6 + Lib/aifc.py | 11 +- Lib/email/charset.py | 11 +- Lib/email/message.py | 26 +- Lib/idlelib/PyShell.py | 32 +- Lib/json/__init__.py | 7 +- Lib/lib2to3/fixes/fix_import.py | 2 +- Lib/platform.py | 9 +- Lib/subprocess.py | 9 +- Lib/telnetlib.py | 6 +- Lib/test/audiotests.py | 62 ++ Lib/test/datetimetester.py | 1 + Lib/test/multibytecodec_support.py | 4 +- Lib/test/string_tests.py | 4 +- Lib/test/test_abstract_numbers.py | 4 - Lib/test/test_aifc.py | 41 +- Lib/test/test_array.py | 2 +- Lib/test/test_bytes.py | 6 + Lib/test/test_codecencodings_iso2022.py | 1 + Lib/test/test_configparser.py | 3 +- Lib/test/test_decimal.py | 19 +- Lib/test/test_devpoll.py | 11 + Lib/test/test_dis.py | 14 +- Lib/test/test_docxmlrpc.py | 6 +- Lib/test/test_email/test_email.py | 32 + Lib/test/test_fileio.py | 3 +- Lib/test/test_float.py | 2 +- Lib/test/test_functools.py | 10 +- Lib/test/test_getargs2.py | 62 ++ Lib/test/test_grp.py | 4 +- Lib/test/test_imp.py | 2 +- Lib/test/test_index.py | 35 +- Lib/test/test_int.py | 73 ++- Lib/test/test_io.py | 12 +- Lib/test/test_memoryview.py | 14 +- Lib/test/test_multiprocessing.py | 22 +- Lib/test/test_nis.py | 6 +- Lib/test/test_ntpath.py | 2 +- Lib/test/test_os.py | 4 +- Lib/test/test_platform.py | 15 + Lib/test/test_poll.py | 17 +- Lib/test/test_pwd.py | 17 +- Lib/test/test_reprlib.py | 13 +- Lib/test/test_shutil.py | 4 +- Lib/test/test_signal.py | 2 +- Lib/test/test_site.py | 18 +- Lib/test/test_socket.py | 14 +- Lib/test/test_strptime.py | 6 +- Lib/test/test_sunau.py | 23 +- Lib/test/test_tempfile.py | 12 +- Lib/test/test_thread.py | 42 +- Lib/test/test_threading.py | 43 + Lib/test/test_time.py | 3 +- Lib/test/test_unicode.py | 10 +- Lib/test/test_urllibnet.py | 7 +- Lib/test/test_warnings.py | 2 +- Lib/test/test_wave.py | 26 +- Lib/test/test_xmlrpc_net.py | 2 - Lib/test/test_zipimport.py | 2 +- Misc/ACKS | 1 + Misc/NEWS | 61 ++- Modules/_decimal/libmpdec/basearith.c | 2 +- Modules/_decimal/libmpdec/basearith.h | 2 +- Modules/_decimal/libmpdec/bits.h | 2 +- Modules/_decimal/libmpdec/constants.c | 2 +- Modules/_decimal/libmpdec/constants.h | 2 +- Modules/_decimal/libmpdec/context.c | 2 +- Modules/_decimal/libmpdec/convolute.c | 2 +- Modules/_decimal/libmpdec/convolute.h | 2 +- Modules/_decimal/libmpdec/crt.c | 2 +- Modules/_decimal/libmpdec/crt.h | 2 +- Modules/_decimal/libmpdec/difradix2.c | 2 +- Modules/_decimal/libmpdec/difradix2.h | 2 +- Modules/_decimal/libmpdec/fnt.c | 2 +- Modules/_decimal/libmpdec/fnt.h | 2 +- Modules/_decimal/libmpdec/fourstep.c | 2 +- Modules/_decimal/libmpdec/fourstep.h | 2 +- Modules/_decimal/libmpdec/io.c | 2 +- Modules/_decimal/libmpdec/io.h | 2 +- Modules/_decimal/libmpdec/literature/fnt.py | 2 +- Modules/_decimal/libmpdec/literature/mulmod-64.txt | 2 +- Modules/_decimal/libmpdec/literature/umodarith.lisp | 2 +- Modules/_decimal/libmpdec/memory.c | 2 +- Modules/_decimal/libmpdec/memory.h | 2 +- Modules/_decimal/libmpdec/mpdecimal.c | 18 +- Modules/_decimal/libmpdec/mpdecimal.h | 22 +- Modules/_decimal/libmpdec/numbertheory.c | 2 +- Modules/_decimal/libmpdec/numbertheory.h | 2 +- Modules/_decimal/libmpdec/sixstep.c | 2 +- Modules/_decimal/libmpdec/sixstep.h | 2 +- Modules/_decimal/libmpdec/transpose.c | 4 +- Modules/_decimal/libmpdec/transpose.h | 2 +- Modules/_decimal/libmpdec/typearith.h | 2 +- Modules/_decimal/libmpdec/umodarith.h | 2 +- Modules/_decimal/libmpdec/vccompat.h | 2 +- Modules/_decimal/libmpdec/vcdiv64.asm | 2 +- Modules/_testcapimodule.c | 93 ++++ Modules/faulthandler.c | 2 +- Modules/posixmodule.c | 4 +- Modules/selectmodule.c | 39 +- Objects/abstract.c | 83 +-- Objects/bytesobject.c | 19 +- Objects/genobject.c | 3 + Objects/longobject.c | 220 ++++----- PCbuild/pythoncore.vcxproj.filters | 8 +- 116 files changed, 1091 insertions(+), 555 deletions(-) diff --git a/Doc/distutils/index.rst b/Doc/distutils/index.rst --- a/Doc/distutils/index.rst +++ b/Doc/distutils/index.rst @@ -12,6 +12,16 @@ make Python modules and extensions easily available to a wider audience with very little overhead for build/release/install mechanics. +.. note:: + + This guide only covers the basic tools for building and distributing + extensions that are provided as part of this version of Python. Third + party tools offer easier to use and more secure alternatives. Refer to the + `quick recommendations section + `__ + in the Python Packaging User Guide for more information. + + .. toctree:: :maxdepth: 2 :numbered: diff --git a/Doc/extending/index.rst b/Doc/extending/index.rst --- a/Doc/extending/index.rst +++ b/Doc/extending/index.rst @@ -21,6 +21,15 @@ For a detailed description of the whole Python/C API, see the separate :ref:`c-api-index`. +.. note:: + + This guide only covers the basic tools for creating extensions provided + as part of this version of CPython. Third party tools may offer simpler + alternatives. Refer to the `binary extensions section + `__ + in the Python Packaging User Guide for more information. + + .. toctree:: :maxdepth: 2 :numbered: diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -1607,26 +1607,34 @@ How do I create a .pyc file? ---------------------------- -When a module is imported for the first time (or when the source is more recent -than the current compiled file) a ``.pyc`` file containing the compiled code -should be created in the same directory as the ``.py`` file. +When a module is imported for the first time (or when the source file has +changed since the current compiled file was created) a ``.pyc`` file containing +the compiled code should be created in a ``__pycache__`` subdirectory of the +directory containing the ``.py`` file. The ``.pyc`` file will have a +filename that starts with the same name as the ``.py`` file, and ends with +``.pyc``, with a middle component that depends on the particular ``python`` +binary that created it. (See :pep:`3147` for details.) -One reason that a ``.pyc`` file may not be created is permissions problems with -the directory. This can happen, for example, if you develop as one user but run -as another, such as if you are testing with a web server. Creation of a .pyc -file is automatic if you're importing a module and Python has the ability -(permissions, free space, etc...) to write the compiled module back to the -directory. +One reason that a ``.pyc`` file may not be created is a permissions problem +with the directory containing the source file, meaning that the ``__pycache__`` +subdirectory cannot be created. This can happen, for example, if you develop as +one user but run as another, such as if you are testing with a web server. + +Unless the :envvar:`PYTHONDONTWRITEBYTECODE` environment variable is set, +creation of a .pyc file is automatic if you're importing a module and Python +has the ability (permissions, free space, etc...) to create a ``__pycache__`` +subdirectory and write the compiled module to that subdirectory. Running Python on a top level script is not considered an import and no ``.pyc`` will be created. For example, if you have a top-level module -``foo.py`` that imports another module ``xyz.py``, when you run ``foo``, -``xyz.pyc`` will be created since ``xyz`` is imported, but no ``foo.pyc`` file -will be created since ``foo.py`` isn't being imported. +``foo.py`` that imports another module ``xyz.py``, when you run ``foo`` (by +typing ``python foo.py`` as a shell command), a ``.pyc`` will be created for +``xyz`` because ``xyz`` is imported, but no ``.pyc`` file will be created for +``foo`` since ``foo.py`` isn't being imported. -If you need to create ``foo.pyc`` -- that is, to create a ``.pyc`` file for a module -that is not imported -- you can, using the :mod:`py_compile` and -:mod:`compileall` modules. +If you need to create a ``.pyc`` file for ``foo`` -- that is, to create a +``.pyc`` file for a module that is not imported -- you can, using the +:mod:`py_compile` and :mod:`compileall` modules. The :mod:`py_compile` module can manually compile any module. One way is to use the ``compile()`` function in that module interactively:: @@ -1634,8 +1642,9 @@ >>> import py_compile >>> py_compile.compile('foo.py') # doctest: +SKIP -This will write the ``.pyc`` to the same location as ``foo.py`` (or you can -override that with the optional parameter ``cfile``). +This will write the ``.pyc`` to a ``__pycache__`` subdirectory in the same +location as ``foo.py`` (or you can override that with the optional parameter +``cfile``). You can also automatically compile all files in a directory or directories using the :mod:`compileall` module. You can do it from the shell prompt by running diff --git a/Doc/install/index.rst b/Doc/install/index.rst --- a/Doc/install/index.rst +++ b/Doc/install/index.rst @@ -20,12 +20,20 @@ Finally, it might be useful to include all the material from my "Care and Feeding of a Python Installation" talk in here somewhere. Yow! -.. topic:: Abstract +This document describes the Python Distribution Utilities ("Distutils") from the +end-user's point-of-view, describing how to extend the capabilities of a +standard Python installation by building and installing third-party Python +modules and extensions. - This document describes the Python Distribution Utilities ("Distutils") from the - end-user's point-of-view, describing how to extend the capabilities of a - standard Python installation by building and installing third-party Python - modules and extensions. + +.. note:: + + This guide only covers the basic tools for installing extensions that are + provided as part of this version of Python. Third party tools offer easier + to use and more secure alternatives. Refer to the + `quick recommendations section + `__ + in the Python Packaging User Guide for more information. .. _inst-intro: diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst --- a/Doc/library/bz2.rst +++ b/Doc/library/bz2.rst @@ -91,6 +91,11 @@ byte of data will be returned (unless at EOF). The exact number of bytes returned is unspecified. + .. note:: While calling :meth:`peek` does not change the file position of + the :class:`BZ2File`, it may change the position of the underlying file + object (e.g. if the :class:`BZ2File` was constructed by passing a file + object for *filename*). + .. versionadded:: 3.3 .. versionchanged:: 3.1 diff --git a/Doc/library/gzip.rst b/Doc/library/gzip.rst --- a/Doc/library/gzip.rst +++ b/Doc/library/gzip.rst @@ -113,6 +113,11 @@ the call. The number of bytes returned may be more or less than requested. + .. note:: While calling :meth:`peek` does not change the file position of + the :class:`GzipFile`, it may change the position of the underlying + file object (e.g. if the :class:`GzipFile` was constructed with the + *fileobj* parameter). + .. versionadded:: 3.2 .. versionchanged:: 3.1 diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -82,10 +82,13 @@ derived from :func:`importlib.__import__`, including requiring the package from which an import is occurring to have been previously imported (i.e., *package* must already be imported). The most important difference - is that :func:`import_module` returns the most nested package or module - that was imported (e.g. ``pkg.mod``), while :func:`__import__` returns the + is that :func:`import_module` returns the specified package or module + (e.g. ``pkg.mod``), while :func:`__import__` returns the top-level package or module (e.g. ``pkg``). + .. versionchanged:: 3.3 + Parent packages are automatically imported. + .. function:: find_loader(name, path=None) Find the loader for a module, optionally within the specified *path*. If the diff --git a/Doc/library/lzma.rst b/Doc/library/lzma.rst --- a/Doc/library/lzma.rst +++ b/Doc/library/lzma.rst @@ -98,6 +98,11 @@ byte of data will be returned, unless EOF has been reached. The exact number of bytes returned is unspecified (the *size* argument is ignored). + .. note:: While calling :meth:`peek` does not change the file position of + the :class:`LZMAFile`, it may change the position of the underlying + file object (e.g. if the :class:`LZMAFile` was constructed by passing a + file object for *filename*). + Compressing and decompressing data in memory -------------------------------------------- diff --git a/Doc/library/mailbox.rst b/Doc/library/mailbox.rst --- a/Doc/library/mailbox.rst +++ b/Doc/library/mailbox.rst @@ -1550,7 +1550,7 @@ due to malformed messages in the mailbox:: import mailbox - import email.Errors + import email.errors list_names = ('python-list', 'python-dev', 'python-bugs') @@ -1560,7 +1560,7 @@ for key in inbox.iterkeys(): try: message = inbox[key] - except email.Errors.MessageParseError: + except email.errors.MessageParseError: continue # The message is malformed. Just leave it. for name in list_names: diff --git a/Doc/license.rst b/Doc/license.rst --- a/Doc/license.rst +++ b/Doc/license.rst @@ -845,3 +845,36 @@ Jean-loup Gailly Mark Adler jloup at gzip.org madler at alumni.caltech.edu + +libmpdec +-------- + +The :mod:`_decimal` Module is built using an included copy of the libmpdec +library unless the build is configured ``--with-system-libmpdec``:: + + Copyright (c) 2008-2016 Stefan Krah. All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + 1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + + THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS "AS IS" AND + ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + SUCH DAMAGE. + + diff --git a/Include/import.h b/Include/import.h --- a/Include/import.h +++ b/Include/import.h @@ -86,15 +86,15 @@ PyAPI_FUNC(void) _PyImport_ReInitLock(void); -PyAPI_FUNC(PyObject *)_PyImport_FindBuiltin( +PyAPI_FUNC(PyObject *) _PyImport_FindBuiltin( const char *name /* UTF-8 encoded string */ ); -PyAPI_FUNC(PyObject *)_PyImport_FindExtensionObject(PyObject *, PyObject *); -PyAPI_FUNC(int)_PyImport_FixupBuiltin( +PyAPI_FUNC(PyObject *) _PyImport_FindExtensionObject(PyObject *, PyObject *); +PyAPI_FUNC(int) _PyImport_FixupBuiltin( PyObject *mod, char *name /* UTF-8 encoded string */ ); -PyAPI_FUNC(int)_PyImport_FixupExtensionObject(PyObject*, PyObject *, PyObject *); +PyAPI_FUNC(int) _PyImport_FixupExtensionObject(PyObject*, PyObject *, PyObject *); struct _inittab { char *name; /* ASCII encoded string */ diff --git a/Include/longobject.h b/Include/longobject.h --- a/Include/longobject.h +++ b/Include/longobject.h @@ -152,6 +152,12 @@ unsigned char* bytes, size_t n, int little_endian, int is_signed); +/* _PyLong_FromNbInt: Convert the given object to a PyLongObject + using the nb_int slot, if available. Raise TypeError if either the + nb_int slot is not available or the result of the call to nb_int + returns something not of type int. +*/ +PyAPI_FUNC(PyLongObject *)_PyLong_FromNbInt(PyObject *); /* _PyLong_Format: Convert the long to a string object with given base, appending a base prefix of 0[box] if base is 2, 8 or 16. */ diff --git a/Lib/aifc.py b/Lib/aifc.py --- a/Lib/aifc.py +++ b/Lib/aifc.py @@ -771,7 +771,10 @@ self._datalength = (self._datalength + 3) // 4 if self._datalength & 1: self._datalength = self._datalength + 1 - self._form_length_pos = self._file.tell() + try: + self._form_length_pos = self._file.tell() + except (AttributeError, OSError): + self._form_length_pos = None commlength = self._write_form_length(self._datalength) if self._aifc: self._file.write(b'AIFC') @@ -783,7 +786,8 @@ self._file.write(b'COMM') _write_ulong(self._file, commlength) _write_short(self._file, self._nchannels) - self._nframes_pos = self._file.tell() + if self._form_length_pos is not None: + self._nframes_pos = self._file.tell() _write_ulong(self._file, self._nframes) if self._comptype in (b'ULAW', b'ulaw', b'ALAW', b'alaw', b'G722'): _write_short(self._file, 8) @@ -794,7 +798,8 @@ self._file.write(self._comptype) _write_string(self._file, self._compname) self._file.write(b'SSND') - self._ssnd_length_pos = self._file.tell() + if self._form_length_pos is not None: + self._ssnd_length_pos = self._file.tell() _write_ulong(self._file, self._datalength + 8) _write_ulong(self._file, 0) _write_ulong(self._file, 0) diff --git a/Lib/email/charset.py b/Lib/email/charset.py --- a/Lib/email/charset.py +++ b/Lib/email/charset.py @@ -386,7 +386,8 @@ string using the ascii codec produces the correct string version of the content. """ - # 7bit/8bit encodings return the string unchanged (module conversions) + if not string: + return string if self.body_encoding is BASE64: if isinstance(string, str): string = string.encode(self.output_charset) @@ -398,13 +399,9 @@ # character set, then, we must turn it into pseudo bytes via the # latin1 charset, which will encode any byte as a single code point # between 0 and 255, which is what body_encode is expecting. - # - # Note that this clause doesn't handle the case of a _payload that - # is already bytes. It never did, and the semantics of _payload - # being bytes has never been nailed down, so fixing that is a - # longer term TODO. if isinstance(string, str): - string = string.encode(self.output_charset).decode('latin1') + string = string.encode(self.output_charset) + string = string.decode('latin1') return email.quoprimime.body_encode(string) else: if isinstance(string, str): diff --git a/Lib/email/message.py b/Lib/email/message.py --- a/Lib/email/message.py +++ b/Lib/email/message.py @@ -275,9 +275,19 @@ Optional charset sets the message's default character set. See set_charset() for details. """ - if isinstance(payload, bytes): - payload = payload.decode('ascii', 'surrogateescape') - self._payload = payload + if hasattr(payload, 'encode'): + if charset is None: + # We should check for ASCII-only here, but we can't do that + # for backward compatibility reasons. Fixed in 3.4. + self._payload = payload + return + if not isinstance(charset, Charset): + charset = Charset(charset) + payload = payload.encode(charset.output_charset) + if hasattr(payload, 'decode'): + self._payload = payload.decode('ascii', 'surrogateescape') + else: + self._payload = payload if charset is not None: self.set_charset(charset) @@ -316,7 +326,15 @@ try: cte(self) except TypeError: - self._payload = charset.body_encode(self._payload) + # This if is for backward compatibility and will be removed + # in 3.4 when the ascii check is added to set_payload. + payload = self._payload + if payload: + try: + payload = payload.encode('ascii', 'surrogateescape') + except UnicodeError: + payload = payload.encode(charset.output_charset) + self._payload = charset.body_encode(payload) self.add_header('Content-Transfer-Encoding', cte) def get_charset(self): diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py old mode 100644 new mode 100755 --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -1331,8 +1331,11 @@ def write(self, s): if self.closed: raise ValueError("write to closed file") - if not isinstance(s, str): - raise TypeError('must be str, not ' + type(s).__name__) + if type(s) is not str: + if not isinstance(s, str): + raise TypeError('must be str, not ' + type(s).__name__) + # See issue #19481 + s = str.__str__(s) return self.shell.write(s, self.tags) @@ -1531,20 +1534,22 @@ args.remove(filename) if not args: flist.new() + if enable_shell: shell = flist.open_shell() if not shell: return # couldn't open shell - if macosxSupport.runningAsOSXApp() and flist.dict: # On OSX: when the user has double-clicked on a file that causes # IDLE to be launched the shell window will open just in front of # the file she wants to see. Lower the interpreter window when # there are open files. shell.top.lower() + else: + shell = flist.pyshell - shell = flist.pyshell - # handle remaining options: + # Handle remaining options. If any of these are set, enable_shell + # was set also, so shell must be true to reach here. if debug: shell.open_debugger() if startup: @@ -1552,7 +1557,7 @@ os.environ.get("PYTHONSTARTUP") if filename and os.path.isfile(filename): shell.interp.execfile(filename) - if shell and cmd or script: + if cmd or script: shell.interp.runcommand("""if 1: import sys as _sys _sys.argv = %r @@ -1563,13 +1568,14 @@ elif script: shell.interp.prepend_syspath(script) shell.interp.execfile(script) - - # Check for problematic OS X Tk versions and print a warning message - # in the IDLE shell window; this is less intrusive than always opening - # a separate window. - tkversionwarning = macosxSupport.tkVersionWarning(root) - if tkversionwarning: - shell.interp.runcommand(''.join(("print('", tkversionwarning, "')"))) + elif shell: + # If there is a shell window and no cmd or script in progress, + # check for problematic OS X Tk versions and print a warning + # message in the IDLE shell window; this is less intrusive + # than always opening a separate window. + tkversionwarning = macosxSupport.tkVersionWarning(root) + if tkversionwarning: + shell.interp.runcommand("print('%s')" % tkversionwarning) while flist.inversedict: # keep IDLE running while files are open. root.mainloop() diff --git a/Lib/json/__init__.py b/Lib/json/__init__.py --- a/Lib/json/__init__.py +++ b/Lib/json/__init__.py @@ -3,11 +3,8 @@ interchange format. :mod:`json` exposes an API familiar to users of the standard library -:mod:`marshal` and :mod:`pickle` modules. It is the externally maintained -version of the :mod:`json` library contained in Python 2.6, but maintains -compatibility with Python 2.4 and Python 2.5 and (currently) has -significant performance advantages, even without using the optional C -extension for speedups. +:mod:`marshal` and :mod:`pickle` modules. It is derived from a +version of the externally maintained simplejson library. Encoding basic Python object hierarchies:: diff --git a/Lib/lib2to3/fixes/fix_import.py b/Lib/lib2to3/fixes/fix_import.py --- a/Lib/lib2to3/fixes/fix_import.py +++ b/Lib/lib2to3/fixes/fix_import.py @@ -32,7 +32,7 @@ elif node.type == syms.dotted_as_names: pending.extend(node.children[::-2]) else: - raise AssertionError("unkown node type") + raise AssertionError("unknown node type") class FixImport(fixer_base.BaseFix): diff --git a/Lib/platform.py b/Lib/platform.py --- a/Lib/platform.py +++ b/Lib/platform.py @@ -129,6 +129,10 @@ # Standard Unix uses /dev/null DEV_NULL = '/dev/null' +# Directory to search for configuration information on Unix. +# Constant used by test_platform to test linux_distribution(). +_UNIXCONFDIR = '/etc' + ### Platform specific APIs _libc_search = re.compile(b'(__libc_init)' @@ -315,7 +319,7 @@ """ try: - etc = os.listdir('/etc') + etc = os.listdir(_UNIXCONFDIR) except os.error: # Probably not a Unix system return distname,version,id @@ -331,7 +335,8 @@ return _dist_try_harder(distname,version,id) # Read the first line - with open('/etc/'+file, 'r') as f: + with open(os.path.join(_UNIXCONFDIR, file), 'r', + encoding='utf-8', errors='surrogateescape') as f: firstline = f.readline() _distname, _version, _id = _parse_release_file(firstline) diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -1621,6 +1621,9 @@ self._save_input(input) + if self._input: + input_view = memoryview(self._input) + while self._fd2file: timeout = self._remaining_time(endtime) if timeout is not None and timeout < 0: @@ -1638,8 +1641,8 @@ for fd, mode in ready: if mode & select.POLLOUT: - chunk = self._input[self._input_offset : - self._input_offset + _PIPE_BUF] + chunk = input_view[self._input_offset : + self._input_offset + _PIPE_BUF] try: self._input_offset += os.write(fd, chunk) except OSError as e: @@ -1651,7 +1654,7 @@ if self._input_offset >= len(self._input): close_unregister_and_remove(fd) elif mode & select_POLLIN_POLLPRI: - data = os.read(fd, 4096) + data = os.read(fd, 32768) if not data: close_unregister_and_remove(fd) self._fd2output[fd].append(data) diff --git a/Lib/telnetlib.py b/Lib/telnetlib.py --- a/Lib/telnetlib.py +++ b/Lib/telnetlib.py @@ -315,7 +315,8 @@ poller.register(self, poll_in_or_priority_flags) while i < 0 and not self.eof: try: - ready = poller.poll(call_timeout) + ready = poller.poll(None if timeout is None + else 1000 * call_timeout) except select.error as e: if e.errno == errno.EINTR: if timeout is not None: @@ -683,7 +684,8 @@ poller.register(self, poll_in_or_priority_flags) while not m and not self.eof: try: - ready = poller.poll(call_timeout) + ready = poller.poll(None if timeout is None + else 1000 * call_timeout) except select.error as e: if e.errno == errno.EINTR: if timeout is not None: diff --git a/Lib/test/audiotests.py b/Lib/test/audiotests.py --- a/Lib/test/audiotests.py +++ b/Lib/test/audiotests.py @@ -5,6 +5,13 @@ import pickle import sys +class UnseekableIO(io.FileIO): + def tell(self): + raise io.UnsupportedOperation + + def seek(self, *args, **kwargs): + raise io.UnsupportedOperation + def byteswap2(data): a = array.array('h') a.frombytes(data) @@ -129,6 +136,61 @@ self.assertEqual(testfile.read(13), b'ababagalamaga') self.check_file(testfile, self.nframes, self.frames) + def test_unseekable_read(self): + f = self.create_file(TESTFN) + f.setnframes(self.nframes) + f.writeframes(self.frames) + f.close() + + with UnseekableIO(TESTFN, 'rb') as testfile: + self.check_file(testfile, self.nframes, self.frames) + + def test_unseekable_write(self): + with UnseekableIO(TESTFN, 'wb') as testfile: + f = self.create_file(testfile) + f.setnframes(self.nframes) + f.writeframes(self.frames) + f.close() + + self.check_file(TESTFN, self.nframes, self.frames) + + def test_unseekable_incompleted_write(self): + with UnseekableIO(TESTFN, 'wb') as testfile: + testfile.write(b'ababagalamaga') + f = self.create_file(testfile) + f.setnframes(self.nframes + 1) + try: + f.writeframes(self.frames) + except OSError: + pass + try: + f.close() + except OSError: + pass + + with open(TESTFN, 'rb') as testfile: + self.assertEqual(testfile.read(13), b'ababagalamaga') + self.check_file(testfile, self.nframes + 1, self.frames) + + def test_unseekable_overflowed_write(self): + with UnseekableIO(TESTFN, 'wb') as testfile: + testfile.write(b'ababagalamaga') + f = self.create_file(testfile) + f.setnframes(self.nframes - 1) + try: + f.writeframes(self.frames) + except OSError: + pass + try: + f.close() + except OSError: + pass + + with open(TESTFN, 'rb') as testfile: + self.assertEqual(testfile.read(13), b'ababagalamaga') + framesize = self.nchannels * self.sampwidth + self.check_file(testfile, self.nframes - 1, self.frames[:-framesize]) + class AudioTestsWithSourceFile(AudioTests): diff --git a/Lib/test/datetimetester.py b/Lib/test/datetimetester.py --- a/Lib/test/datetimetester.py +++ b/Lib/test/datetimetester.py @@ -2025,6 +2025,7 @@ class TestSubclassDateTime(TestDateTime): theclass = SubclassDatetime # Override tests not designed for subclass + @unittest.skip('not appropriate for subclasses') def test_roundtrip(self): pass diff --git a/Lib/test/multibytecodec_support.py b/Lib/test/multibytecodec_support.py --- a/Lib/test/multibytecodec_support.py +++ b/Lib/test/multibytecodec_support.py @@ -73,7 +73,7 @@ def test_xmlcharrefreplace(self): if self.has_iso10646: - return + self.skipTest('encoding contains full ISO 10646 map') s = "\u0b13\u0b23\u0b60 nd eggs" self.assertEqual( @@ -83,7 +83,7 @@ def test_customreplace_encode(self): if self.has_iso10646: - return + self.skipTest('encoding contains full ISO 10646 map') from html.entities import codepoint2name diff --git a/Lib/test/string_tests.py b/Lib/test/string_tests.py --- a/Lib/test/string_tests.py +++ b/Lib/test/string_tests.py @@ -663,10 +663,10 @@ self.checkraises(TypeError, 'hello', 'replace', 42, 'h') self.checkraises(TypeError, 'hello', 'replace', 'h', 42) + @unittest.skipIf(sys.maxsize > (1 << 32) or struct.calcsize('P') != 4, + 'only applies to 32-bit platforms') def test_replace_overflow(self): # Check for overflow checking on 32 bit machines - if sys.maxsize != 2147483647 or struct.calcsize("P") > 4: - return A2_16 = "A" * (2**16) self.checkraises(OverflowError, A2_16, "replace", "", A2_16) self.checkraises(OverflowError, A2_16, "replace", "A", A2_16) diff --git a/Lib/test/test_abstract_numbers.py b/Lib/test/test_abstract_numbers.py --- a/Lib/test/test_abstract_numbers.py +++ b/Lib/test/test_abstract_numbers.py @@ -4,7 +4,6 @@ import operator import unittest from numbers import Complex, Real, Rational, Integral -from test import support class TestNumbers(unittest.TestCase): def test_int(self): @@ -40,9 +39,6 @@ self.assertRaises(TypeError, float, c1) self.assertRaises(TypeError, int, c1) -def test_main(): - support.run_unittest(TestNumbers) - if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_aifc.py b/Lib/test/test_aifc.py --- a/Lib/test/test_aifc.py +++ b/Lib/test/test_aifc.py @@ -8,10 +8,14 @@ import aifc -class AifcPCM8Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): +class AifcTest(audiotests.AudioWriteTests, + audiotests.AudioTestsWithSourceFile): module = aifc + close_fd = True + test_unseekable_read = None + + +class AifcPCM8Test(AifcTest, unittest.TestCase): sndfilename = 'pluck-pcm8.aiff' sndfilenframes = 3307 nchannels = 2 @@ -26,13 +30,9 @@ 11FA 3EFB BCFC 66FF CF04 4309 C10E 5112 EE17 8216 7F14 8012 \ 490E 520D EF0F CE0F E40C 630A 080A 2B0B 510E 8B11 B60E 440A \ """) - close_fd = True -class AifcPCM16Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = aifc +class AifcPCM16Test(AifcTest, unittest.TestCase): sndfilename = 'pluck-pcm16.aiff' sndfilenframes = 3307 nchannels = 2 @@ -49,13 +49,9 @@ EEE21753 82071665 7FFF1443 8004128F 49A20EAF 52BB0DBA EFB40F60 CE3C0FBF \ E4B30CEC 63430A5C 08C80A20 2BBB0B08 514A0E43 8BCF1139 B6F60EEB 44120A5E \ """) - close_fd = True -class AifcPCM24Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = aifc +class AifcPCM24Test(AifcTest, unittest.TestCase): sndfilename = 'pluck-pcm24.aiff' sndfilenframes = 3307 nchannels = 2 @@ -78,13 +74,9 @@ E4B49C0CEA2D 6344A80A5A7C 08C8FE0A1FFE 2BB9860B0A0E \ 51486F0E44E1 8BCC64113B05 B6F4EC0EEB36 4413170A5B48 \ """) - close_fd = True -class AifcPCM32Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = aifc +class AifcPCM32Test(AifcTest, unittest.TestCase): sndfilename = 'pluck-pcm32.aiff' sndfilenframes = 3307 nchannels = 2 @@ -107,13 +99,9 @@ E4B49CC00CEA2D90 6344A8800A5A7CA0 08C8FE800A1FFEE0 2BB986C00B0A0E00 \ 51486F800E44E190 8BCC6480113B0580 B6F4EC000EEB3630 441317800A5B48A0 \ """) - close_fd = True -class AifcULAWTest(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = aifc +class AifcULAWTest(AifcTest, unittest.TestCase): sndfilename = 'pluck-ulaw.aifc' sndfilenframes = 3307 nchannels = 2 @@ -132,13 +120,9 @@ """) if sys.byteorder != 'big': frames = audiotests.byteswap2(frames) - close_fd = True -class AifcALAWTest(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = aifc +class AifcALAWTest(AifcTest, unittest.TestCase): sndfilename = 'pluck-alaw.aifc' sndfilenframes = 3307 nchannels = 2 @@ -157,7 +141,6 @@ """) if sys.byteorder != 'big': frames = audiotests.byteswap2(frames) - close_fd = True class AifcMiscTest(audiotests.AudioTests, unittest.TestCase): diff --git a/Lib/test/test_array.py b/Lib/test/test_array.py --- a/Lib/test/test_array.py +++ b/Lib/test/test_array.py @@ -946,7 +946,7 @@ try: import gc except ImportError: - return + self.skipTest('gc module not available') a = array.array(self.typecode) l = [iter(a)] l.append(l) diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -729,6 +729,12 @@ self.assertEqual(PyBytes_FromFormat(b's:%s', c_char_p(b'cstr')), b's:cstr') + # Issue #19969 + self.assertRaises(OverflowError, + PyBytes_FromFormat, b'%c', c_int(-1)) + self.assertRaises(OverflowError, + PyBytes_FromFormat, b'%c', c_int(256)) + class ByteArrayTest(BaseBytesTest, unittest.TestCase): type2test = bytearray diff --git a/Lib/test/test_codecencodings_iso2022.py b/Lib/test/test_codecencodings_iso2022.py --- a/Lib/test/test_codecencodings_iso2022.py +++ b/Lib/test/test_codecencodings_iso2022.py @@ -36,6 +36,7 @@ # iso2022_kr.txt cannot be used to test "chunk coding": the escape # sequence is only written on the first line + @unittest.skip('iso2022_kr.txt cannot be used to test "chunk coding"') def test_chunkcoding(self): pass diff --git a/Lib/test/test_configparser.py b/Lib/test/test_configparser.py --- a/Lib/test/test_configparser.py +++ b/Lib/test/test_configparser.py @@ -707,8 +707,7 @@ def test_read_returns_file_list(self): if self.delimiters[0] != '=': - # skip reading the file if we're using an incompatible format - return + self.skipTest('incompatible format') file1 = support.findfile("cfgparser.1") # check when we pass a mix of readable and non-readable files: cf = self.newconfig() diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -290,7 +290,6 @@ global skip_expected if skip_expected: raise unittest.SkipTest - return with open(file) as f: for line in f: line = line.replace('\r\n', '').replace('\n', '') @@ -301,7 +300,6 @@ #Exception raised where there shouldn't have been one. self.fail('Exception "'+exception.__class__.__name__ + '" raised on line '+line) - return def eval_line(self, s): if s.find(' -> ') >= 0 and s[:2] != '--' and not s.startswith(' --'): @@ -461,7 +459,6 @@ self.assertEqual(myexceptions, theirexceptions, 'Incorrect flags set in ' + s + ' -- got ' + str(myexceptions)) - return def getexceptions(self): return [e for e in Signals[self.decimal] if self.context.flags[e]] @@ -1073,7 +1070,7 @@ try: from locale import CHAR_MAX except ImportError: - return + self.skipTest('locale.CHAR_MAX not available') def make_grouping(lst): return ''.join([chr(x) for x in lst]) if self.decimal == C else lst @@ -1164,8 +1161,12 @@ decimal_point = locale.localeconv()['decimal_point'] thousands_sep = locale.localeconv()['thousands_sep'] - if decimal_point != '\u066b' or thousands_sep != '\u066c': - return + if decimal_point != '\u066b': + self.skipTest('inappropriate decimal point separator' + '({!a} not {!a})'.format(decimal_point, '\u066b')) + if thousands_sep != '\u066c': + self.skipTest('inappropriate thousands separator' + '({!a} not {!a})'.format(thousands_sep, '\u066c')) self.assertEqual(format(Decimal('100000000.123'), 'n'), '100\u066c000\u066c000\u066b123') @@ -1515,7 +1516,6 @@ cls.assertTrue(c1.flags[Inexact]) for sig in Overflow, Underflow, DivisionByZero, InvalidOperation: cls.assertFalse(c1.flags[sig]) - return def thfunc2(cls): Decimal = cls.decimal.Decimal @@ -1560,7 +1560,6 @@ cls.assertTrue(thiscontext.flags[Inexact]) for sig in Overflow, Underflow, DivisionByZero, InvalidOperation: cls.assertFalse(thiscontext.flags[sig]) - return class ThreadingTest(unittest.TestCase): '''Unit tests for thread local contexts in Decimal.''' @@ -1602,7 +1601,6 @@ DefaultContext.prec = save_prec DefaultContext.Emax = save_emax DefaultContext.Emin = save_emin - return @unittest.skipUnless(threading, 'threading required') class CThreadingTest(ThreadingTest): @@ -4525,7 +4523,6 @@ self.assertEqual(d1._sign, b1._sign) self.assertEqual(d1._int, b1._int) self.assertEqual(d1._exp, b1._exp) - return Decimal(d1) self.assertEqual(d1._sign, b1._sign) @@ -5271,7 +5268,7 @@ try: from locale import CHAR_MAX except ImportError: - return + self.skipTest('locale.CHAR_MAX not available') def make_grouping(lst): return ''.join([chr(x) for x in lst]) diff --git a/Lib/test/test_devpoll.py b/Lib/test/test_devpoll.py --- a/Lib/test/test_devpoll.py +++ b/Lib/test/test_devpoll.py @@ -87,6 +87,17 @@ self.assertRaises(OverflowError, pollster.poll, 1 << 63) self.assertRaises(OverflowError, pollster.poll, 1 << 64) + def test_events_mask_overflow(self): + pollster = select.devpoll() + w, r = os.pipe() + pollster.register(w) + # Issue #17919 + self.assertRaises(OverflowError, pollster.register, 0, -1) + self.assertRaises(OverflowError, pollster.register, 0, USHRT_MAX + 1) + self.assertRaises(OverflowError, pollster.modify, 1, -1) + self.assertRaises(OverflowError, pollster.modify, 1, USHRT_MAX + 1) + + def test_main(): run_unittest(DevPollTests) diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py --- a/Lib/test/test_dis.py +++ b/Lib/test/test_dis.py @@ -217,16 +217,18 @@ def test_bug_708901(self): self.do_disassembly_test(bug708901, dis_bug708901) + # Test has been disabled due to change in the way + # list comps are handled. The byte code now includes + # a memory address and a file location, so they change from + # run to run. + @unittest.skip('disabled due to a change in the way list comps are handled') def test_bug_1333982(self): # XXX: re-enable this test! # This one is checking bytecodes generated for an `assert` statement, # so fails if the tests are run with -O. Skip this test then. - pass # Test has been disabled due to change in the way - # list comps are handled. The byte code now includes - # a memory address and a file location, so they change from - # run to run. - # if __debug__: - # self.do_disassembly_test(bug1333982, dis_bug1333982) + + if __debug__: + self.do_disassembly_test(bug1333982, dis_bug1333982) def test_big_linenos(self): def func(count): diff --git a/Lib/test/test_docxmlrpc.py b/Lib/test/test_docxmlrpc.py --- a/Lib/test/test_docxmlrpc.py +++ b/Lib/test/test_docxmlrpc.py @@ -202,10 +202,12 @@ """ Test that annotations works as expected """ self.client.request("GET", "/") response = self.client.getresponse() + docstring = (b'' if sys.flags.optimize >= 2 else + b'
Use function annotations.
') self.assertIn( (b'
annotation' - b'(x: int)
Use function annotations.' - b'
\n
' + b'(x: int)
' + docstring + b'
\n' + b'
' b'method_annotation(x: bytes)
'), response.read()) diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -92,6 +92,38 @@ msg.set_payload('This is a string payload', charset) self.assertEqual(msg.get_charset().input_charset, 'iso-8859-1') + def test_set_payload_with_8bit_data_and_charset(self): + data = b'\xd0\x90\xd0\x91\xd0\x92' + charset = Charset('utf-8') + msg = Message() + msg.set_payload(data, charset) + self.assertEqual(msg['content-transfer-encoding'], 'base64') + self.assertEqual(msg.get_payload(decode=True), data) + self.assertEqual(msg.get_payload(), '0JDQkdCS\n') + + def test_set_payload_with_non_ascii_and_charset_body_encoding_none(self): + data = b'\xd0\x90\xd0\x91\xd0\x92' + charset = Charset('utf-8') + charset.body_encoding = None # Disable base64 encoding + msg = Message() + msg.set_payload(data.decode('utf-8'), charset) + self.assertEqual(msg['content-transfer-encoding'], '8bit') + self.assertEqual(msg.get_payload(decode=True), data) + + def test_set_payload_with_8bit_data_and_charset_body_encoding_none(self): + data = b'\xd0\x90\xd0\x91\xd0\x92' + charset = Charset('utf-8') + charset.body_encoding = None # Disable base64 encoding + msg = Message() + msg.set_payload(data, charset) + self.assertEqual(msg['content-transfer-encoding'], '8bit') + self.assertEqual(msg.get_payload(decode=True), data) + + def test_set_payload_to_list(self): + msg = Message() + msg.set_payload([]) + self.assertEqual(msg.get_payload(), []) + def test_get_charsets(self): eq = self.assertEqual diff --git a/Lib/test/test_fileio.py b/Lib/test/test_fileio.py --- a/Lib/test/test_fileio.py +++ b/Lib/test/test_fileio.py @@ -341,8 +341,7 @@ try: fn = TESTFN.encode("ascii") except UnicodeEncodeError: - # Skip test - return + self.skipTest('could not encode %r to ascii' % TESTFN) f = _FileIO(fn, "w") try: f.write(b"abc") diff --git a/Lib/test/test_float.py b/Lib/test/test_float.py --- a/Lib/test/test_float.py +++ b/Lib/test/test_float.py @@ -70,7 +70,7 @@ # it still has to accept the normal python syntax import locale if not locale.localeconv()['decimal_point'] == ',': - return + self.skipTest('decimal_point is not ","') self.assertEqual(float(" 3.14 "), 3.14) self.assertEqual(float("+3.14 "), 3.14) diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -45,8 +45,6 @@ self.assertEqual(p.args, (1, 2)) self.assertEqual(p.keywords, dict(a=10, b=20)) # attributes should not be writable - if not isinstance(self.thetype, type): - return self.assertRaises(AttributeError, setattr, p, 'func', map) self.assertRaises(AttributeError, setattr, p, 'args', (1, 2)) self.assertRaises(AttributeError, setattr, p, 'keywords', dict(a=1, b=2)) @@ -210,11 +208,13 @@ thetype = PythonPartial # the python version hasn't a nice repr - def test_repr(self): pass + test_repr = None # the python version isn't picklable - def test_pickle(self): pass - def test_setstate_refcount(self): pass + test_pickle = test_setstate_refcount = None + + # the python version isn't a type + test_attributes = None class TestUpdateWrapper(unittest.TestCase): diff --git a/Lib/test/test_getargs2.py b/Lib/test/test_getargs2.py --- a/Lib/test/test_getargs2.py +++ b/Lib/test/test_getargs2.py @@ -50,12 +50,33 @@ def __int__(self): return 99 +class IntSubclass(int): + def __int__(self): + return 99 + +class BadInt: + def __int__(self): + return 1.0 + +class BadInt2: + def __int__(self): + return True + +class BadInt3(int): + def __int__(self): + return True + + class Unsigned_TestCase(unittest.TestCase): def test_b(self): from _testcapi import getargs_b # b returns 'unsigned char', and does range checking (0 ... UCHAR_MAX) self.assertRaises(TypeError, getargs_b, 3.14) self.assertEqual(99, getargs_b(Int())) + self.assertEqual(0, getargs_b(IntSubclass())) + self.assertRaises(TypeError, getargs_b, BadInt()) + self.assertEqual(1, getargs_b(BadInt2())) + self.assertEqual(0, getargs_b(BadInt3())) self.assertRaises(OverflowError, getargs_b, -1) self.assertEqual(0, getargs_b(0)) @@ -70,6 +91,10 @@ # B returns 'unsigned char', no range checking self.assertRaises(TypeError, getargs_B, 3.14) self.assertEqual(99, getargs_B(Int())) + self.assertEqual(0, getargs_B(IntSubclass())) + self.assertRaises(TypeError, getargs_B, BadInt()) + self.assertEqual(1, getargs_B(BadInt2())) + self.assertEqual(0, getargs_B(BadInt3())) self.assertEqual(UCHAR_MAX, getargs_B(-1)) self.assertEqual(0, getargs_B(0)) @@ -84,6 +109,10 @@ # H returns 'unsigned short', no range checking self.assertRaises(TypeError, getargs_H, 3.14) self.assertEqual(99, getargs_H(Int())) + self.assertEqual(0, getargs_H(IntSubclass())) + self.assertRaises(TypeError, getargs_H, BadInt()) + self.assertEqual(1, getargs_H(BadInt2())) + self.assertEqual(0, getargs_H(BadInt3())) self.assertEqual(USHRT_MAX, getargs_H(-1)) self.assertEqual(0, getargs_H(0)) @@ -99,6 +128,10 @@ # I returns 'unsigned int', no range checking self.assertRaises(TypeError, getargs_I, 3.14) self.assertEqual(99, getargs_I(Int())) + self.assertEqual(0, getargs_I(IntSubclass())) + self.assertRaises(TypeError, getargs_I, BadInt()) + self.assertEqual(1, getargs_I(BadInt2())) + self.assertEqual(0, getargs_I(BadInt3())) self.assertEqual(UINT_MAX, getargs_I(-1)) self.assertEqual(0, getargs_I(0)) @@ -115,6 +148,10 @@ # it does not accept float, or instances with __int__ self.assertRaises(TypeError, getargs_k, 3.14) self.assertRaises(TypeError, getargs_k, Int()) + self.assertEqual(0, getargs_k(IntSubclass())) + self.assertRaises(TypeError, getargs_k, BadInt()) + self.assertRaises(TypeError, getargs_k, BadInt2()) + self.assertEqual(0, getargs_k(BadInt3())) self.assertEqual(ULONG_MAX, getargs_k(-1)) self.assertEqual(0, getargs_k(0)) @@ -131,6 +168,10 @@ # h returns 'short', and does range checking (SHRT_MIN ... SHRT_MAX) self.assertRaises(TypeError, getargs_h, 3.14) self.assertEqual(99, getargs_h(Int())) + self.assertEqual(0, getargs_h(IntSubclass())) + self.assertRaises(TypeError, getargs_h, BadInt()) + self.assertEqual(1, getargs_h(BadInt2())) + self.assertEqual(0, getargs_h(BadInt3())) self.assertRaises(OverflowError, getargs_h, SHRT_MIN-1) self.assertEqual(SHRT_MIN, getargs_h(SHRT_MIN)) @@ -145,6 +186,10 @@ # i returns 'int', and does range checking (INT_MIN ... INT_MAX) self.assertRaises(TypeError, getargs_i, 3.14) self.assertEqual(99, getargs_i(Int())) + self.assertEqual(0, getargs_i(IntSubclass())) + self.assertRaises(TypeError, getargs_i, BadInt()) + self.assertEqual(1, getargs_i(BadInt2())) + self.assertEqual(0, getargs_i(BadInt3())) self.assertRaises(OverflowError, getargs_i, INT_MIN-1) self.assertEqual(INT_MIN, getargs_i(INT_MIN)) @@ -159,6 +204,10 @@ # l returns 'long', and does range checking (LONG_MIN ... LONG_MAX) self.assertRaises(TypeError, getargs_l, 3.14) self.assertEqual(99, getargs_l(Int())) + self.assertEqual(0, getargs_l(IntSubclass())) + self.assertRaises(TypeError, getargs_l, BadInt()) + self.assertEqual(1, getargs_l(BadInt2())) + self.assertEqual(0, getargs_l(BadInt3())) self.assertRaises(OverflowError, getargs_l, LONG_MIN-1) self.assertEqual(LONG_MIN, getargs_l(LONG_MIN)) @@ -174,6 +223,10 @@ # (PY_SSIZE_T_MIN ... PY_SSIZE_T_MAX) self.assertRaises(TypeError, getargs_n, 3.14) self.assertRaises(TypeError, getargs_n, Int()) + self.assertEqual(0, getargs_n(IntSubclass())) + self.assertRaises(TypeError, getargs_n, BadInt()) + self.assertRaises(TypeError, getargs_n, BadInt2()) + self.assertEqual(0, getargs_n(BadInt3())) self.assertRaises(OverflowError, getargs_n, PY_SSIZE_T_MIN-1) self.assertEqual(PY_SSIZE_T_MIN, getargs_n(PY_SSIZE_T_MIN)) @@ -192,6 +245,10 @@ self.assertRaises(TypeError, getargs_L, 3.14) self.assertRaises(TypeError, getargs_L, "Hello") self.assertEqual(99, getargs_L(Int())) + self.assertEqual(0, getargs_L(IntSubclass())) + self.assertRaises(TypeError, getargs_L, BadInt()) + self.assertEqual(1, getargs_L(BadInt2())) + self.assertEqual(0, getargs_L(BadInt3())) self.assertRaises(OverflowError, getargs_L, LLONG_MIN-1) self.assertEqual(LLONG_MIN, getargs_L(LLONG_MIN)) @@ -206,6 +263,11 @@ # K return 'unsigned long long', no range checking self.assertRaises(TypeError, getargs_K, 3.14) self.assertRaises(TypeError, getargs_K, Int()) + self.assertEqual(0, getargs_K(IntSubclass())) + self.assertRaises(TypeError, getargs_K, BadInt()) + self.assertRaises(TypeError, getargs_K, BadInt2()) + self.assertEqual(0, getargs_K(BadInt3())) + self.assertEqual(ULLONG_MAX, getargs_K(ULLONG_MAX)) self.assertEqual(0, getargs_K(0)) self.assertEqual(0, getargs_K(ULLONG_MAX+1)) diff --git a/Lib/test/test_grp.py b/Lib/test/test_grp.py --- a/Lib/test/test_grp.py +++ b/Lib/test/test_grp.py @@ -26,8 +26,10 @@ for e in entries: self.check_value(e) + def test_values_extended(self): + entries = grp.getgrall() if len(entries) > 1000: # Huge group file (NIS?) -- skip the rest - return + self.skipTest('huge group file, extended test skipped') for e in entries: e2 = grp.getgrgid(e.gr_gid) diff --git a/Lib/test/test_imp.py b/Lib/test/test_imp.py --- a/Lib/test/test_imp.py +++ b/Lib/test/test_imp.py @@ -245,7 +245,7 @@ if found[0] is not None: found[0].close() if found[2][2] != imp.C_EXTENSION: - return + self.skipTest("found module doesn't appear to be a C extension") imp.load_module(name, None, *found[1:]) def test_multiple_calls_to_get_data(self): diff --git a/Lib/test/test_index.py b/Lib/test/test_index.py --- a/Lib/test/test_index.py +++ b/Lib/test/test_index.py @@ -9,7 +9,7 @@ class TrapInt(int): def __index__(self): - return self + return int(self) class BaseTestCase(unittest.TestCase): def setUp(self): @@ -55,6 +55,39 @@ self.assertRaises(TypeError, slice(self.o).indices, 0) self.assertRaises(TypeError, slice(self.n).indices, 0) + def test_int_subclass_with_index(self): + # __index__ should be used when computing indices, even for int + # subclasses. See issue #17576. + class MyInt(int): + def __index__(self): + return int(self) + 1 + + my_int = MyInt(7) + direct_index = my_int.__index__() + operator_index = operator.index(my_int) + self.assertEqual(direct_index, 8) + self.assertEqual(operator_index, 7) + # Both results should be of exact type int. + self.assertIs(type(direct_index), int) + #self.assertIs(type(operator_index), int) + + def test_index_returns_int_subclass(self): + class BadInt: + def __index__(self): + return True + + class BadInt2(int): + def __index__(self): + return True + + bad_int = BadInt() + n = operator.index(bad_int) + self.assertEqual(n, 1) + + bad_int = BadInt2() + n = operator.index(bad_int) + self.assertEqual(n, 0) + class SeqTestCase: # This test case isn't run directly. It just defines common tests diff --git a/Lib/test/test_int.py b/Lib/test/test_int.py --- a/Lib/test/test_int.py +++ b/Lib/test/test_int.py @@ -263,32 +263,7 @@ def __int__(self): return 42 - class Foo1(object): - def __int__(self): - return 42 - - class Foo2(int): - def __int__(self): - return 42 - - class Foo3(int): - def __int__(self): - return self - - class Foo4(int): - def __int__(self): - return 42 - - class Foo5(int): - def __int__(self): - return 42. - self.assertEqual(int(Foo0()), 42) - self.assertEqual(int(Foo1()), 42) - self.assertEqual(int(Foo2()), 42) - self.assertEqual(int(Foo3()), 0) - self.assertEqual(int(Foo4()), 42) - self.assertRaises(TypeError, int, Foo5()) class Classic: pass @@ -351,6 +326,54 @@ with self.assertRaises(TypeError): int(TruncReturnsBadInt()) + def test_int_subclass_with_int(self): + class MyInt(int): + def __int__(self): + return 42 + + class BadInt(int): + def __int__(self): + return 42.0 + + my_int = MyInt(7) + self.assertEqual(my_int, 7) + self.assertEqual(int(my_int), 42) + + self.assertRaises(TypeError, int, BadInt()) + + def test_int_returns_int_subclass(self): + class BadInt: + def __int__(self): + return True + + class BadInt2(int): + def __int__(self): + return True + + class TruncReturnsBadInt: + def __trunc__(self): + return BadInt() + + class TruncReturnsIntSubclass: + def __trunc__(self): + return True + + bad_int = BadInt() + n = int(bad_int) + self.assertEqual(n, 1) + + bad_int = BadInt2() + n = int(bad_int) + self.assertEqual(n, 1) + + bad_int = TruncReturnsBadInt() + n = int(bad_int) + self.assertEqual(n, 1) + + good_int = TruncReturnsIntSubclass() + n = int(good_int) + self.assertEqual(n, 1) + def test_error_message(self): def check(s, base=None): with self.assertRaises(ValueError, diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -421,14 +421,9 @@ # a long time to build the >2GB file and takes >2GB of disk space # therefore the resource must be enabled to run this test. if sys.platform[:3] == 'win' or sys.platform == 'darwin': - if not support.is_resource_enabled("largefile"): - print("\nTesting large file ops skipped on %s." % sys.platform, - file=sys.stderr) - print("It requires %d bytes and a long time." % self.LARGE, - file=sys.stderr) - print("Use 'regrtest.py -u largefile test_io' to run it.", - file=sys.stderr) - return + support.requires( + 'largefile', + 'test requires %s bytes and a long time to run' % self.LARGE) with self.open(support.TESTFN, "w+b", 0) as f: self.large_file_ops(f) with self.open(support.TESTFN, "w+b") as f: @@ -698,6 +693,7 @@ self.assertEqual(42, bufio.fileno()) + @unittest.skip('test having existential crisis') def test_no_fileno(self): # XXX will we always have fileno() function? If so, kill # this test. Else, write it. diff --git a/Lib/test/test_memoryview.py b/Lib/test/test_memoryview.py --- a/Lib/test/test_memoryview.py +++ b/Lib/test/test_memoryview.py @@ -57,7 +57,7 @@ def test_setitem_readonly(self): if not self.ro_type: - return + self.skipTest("no read-only type to test") b = self.ro_type(self._source) oldrefcount = sys.getrefcount(b) m = self._view(b) @@ -71,7 +71,7 @@ def test_setitem_writable(self): if not self.rw_type: - return + self.skipTest("no writable type to test") tp = self.rw_type b = self.rw_type(self._source) oldrefcount = sys.getrefcount(b) @@ -189,13 +189,13 @@ def test_attributes_readonly(self): if not self.ro_type: - return + self.skipTest("no read-only type to test") m = self.check_attributes_with_type(self.ro_type) self.assertEqual(m.readonly, True) def test_attributes_writable(self): if not self.rw_type: - return + self.skipTest("no writable type to test") m = self.check_attributes_with_type(self.rw_type) self.assertEqual(m.readonly, False) @@ -301,7 +301,7 @@ # buffer as writable causing a segfault if using mmap tp = self.ro_type if tp is None: - return + self.skipTest("no read-only type to test") b = tp(self._source) m = self._view(b) i = io.BytesIO(b'ZZZZ') @@ -370,12 +370,12 @@ itemsize = array.array('i').itemsize format = 'i' + @unittest.skip('XXX test should be adapted for non-byte buffers') def test_getbuffer(self): - # XXX Test should be adapted for non-byte buffers pass + @unittest.skip('XXX NotImplementedError: tolist() only supports byte views') def test_tolist(self): - # XXX NotImplementedError: tolist() only supports byte views pass diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -195,7 +195,7 @@ def test_current(self): if self.TYPE == 'threads': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) current = self.current_process() authkey = current.authkey @@ -209,7 +209,7 @@ def test_daemon_argument(self): if self.TYPE == "threads": - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) # By default uses the current process's daemon flag. proc0 = self.Process(target=self._test) @@ -274,7 +274,7 @@ def test_terminate(self): if self.TYPE == 'threads': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) p = self.Process(target=self._test_terminate) p.daemon = True @@ -378,7 +378,7 @@ def test_sentinel(self): if self.TYPE == "threads": - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) event = self.Event() p = self.Process(target=self._test_sentinel, args=(event,)) with self.assertRaises(ValueError): @@ -434,7 +434,7 @@ def test_stderr_flush(self): # sys.stderr is flushed at process shutdown (issue #13812) if self.TYPE == "threads": - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) testfn = test.support.TESTFN self.addCleanup(test.support.unlink, testfn) @@ -462,7 +462,7 @@ def test_sys_exit(self): # See Issue 13854 if self.TYPE == 'threads': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) testfn = test.support.TESTFN self.addCleanup(test.support.unlink, testfn) @@ -671,7 +671,7 @@ try: self.assertEqual(q.qsize(), 0) except NotImplementedError: - return + self.skipTest('qsize method not implemented') q.put(1) self.assertEqual(q.qsize(), 1) q.put(5) @@ -779,7 +779,7 @@ def test_timeout(self): if self.TYPE != 'processes': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) sem = self.Semaphore(0) acquire = TimingWrapper(sem.acquire) @@ -1399,7 +1399,7 @@ def test_thousand(self): if self.TYPE == 'manager': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) passes = 1000 lock = self.Lock() conn, child_conn = self.Pipe(False) @@ -1694,7 +1694,7 @@ def test_map_unplicklable(self): # Issue #19425 -- failure to pickle should not cause a hang if self.TYPE == 'threads': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) class A(object): def __reduce__(self): raise RuntimeError('cannot pickle') @@ -2188,7 +2188,7 @@ def test_sendbytes(self): if self.TYPE != 'processes': - return + self.skipTest('test not appropriate for {}'.format(self.TYPE)) msg = latin('abcdefghijklmnopqrstuvwxyz') a, b = self.Pipe() diff --git a/Lib/test/test_nis.py b/Lib/test/test_nis.py --- a/Lib/test/test_nis.py +++ b/Lib/test/test_nis.py @@ -12,11 +12,7 @@ maps = nis.maps() except nis.error as msg: # NIS is probably not active, so this test isn't useful - if support.verbose: - print("Test Skipped:", msg) - # Can't raise SkipTest as regrtest only recognizes the exception - # import time. - return + self.skipTest(str(msg)) try: # On some systems, this map is only accessible to the # super user diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py --- a/Lib/test/test_ntpath.py +++ b/Lib/test/test_ntpath.py @@ -218,7 +218,7 @@ import nt tester('ntpath.abspath("C:\\")', "C:\\") except ImportError: - pass + self.skipTest('nt module not available') def test_relpath(self): currentdir = os.path.split(os.getcwd())[-1] diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -263,7 +263,7 @@ except OSError as e: # On AtheOS, glibc always returns ENOSYS if e.errno == errno.ENOSYS: - return + self.skipTest('glibc always returns ENOSYS on AtheOS') # Make sure direct access works self.assertEqual(result.f_bfree, result[3]) @@ -480,7 +480,7 @@ os.stat(r"c:\pagefile.sys") except WindowsError as e: if e.errno == 2: # file does not exist; cannot run test - return + self.skipTest(r'c:\pagefile.sys does not exist') self.fail("Could not stat pagefile.sys") @unittest.skipUnless(sys.platform == "win32", "Win32 specific tests") diff --git a/Lib/test/test_platform.py b/Lib/test/test_platform.py --- a/Lib/test/test_platform.py +++ b/Lib/test/test_platform.py @@ -1,7 +1,9 @@ +from unittest import mock import os import platform import subprocess import sys +import tempfile import unittest import warnings @@ -295,6 +297,19 @@ returncode = ret >> 8 self.assertEqual(returncode, len(data)) + def test_linux_distribution_encoding(self): + # Issue #17429 + with tempfile.TemporaryDirectory() as tempdir: + filename = os.path.join(tempdir, 'fedora-release') + with open(filename, 'w', encoding='utf-8') as f: + f.write('Fedora release 19 (Schr\xf6dinger\u2019s Cat)\n') + + with mock.patch('platform._UNIXCONFDIR', tempdir): + distname, version, distid = platform.linux_distribution() + + self.assertEqual(distname, 'Fedora') + self.assertEqual(version, '19') + self.assertEqual(distid, 'Schr\xf6dinger\u2019s Cat') def test_main(): support.run_unittest( diff --git a/Lib/test/test_poll.py b/Lib/test/test_poll.py --- a/Lib/test/test_poll.py +++ b/Lib/test/test_poll.py @@ -3,7 +3,7 @@ import os import random import select -import _testcapi +from _testcapi import USHRT_MAX, INT_MAX, UINT_MAX try: import threading except ImportError: @@ -159,14 +159,13 @@ if x != 5: self.fail('Overflow must have occurred') - pollster = select.poll() - # Issue 15989 - self.assertRaises(OverflowError, pollster.register, 0, - _testcapi.SHRT_MAX + 1) - self.assertRaises(OverflowError, pollster.register, 0, - _testcapi.USHRT_MAX + 1) - self.assertRaises(OverflowError, pollster.poll, _testcapi.INT_MAX + 1) - self.assertRaises(OverflowError, pollster.poll, _testcapi.UINT_MAX + 1) + # Issues #15989, #17919 + self.assertRaises(OverflowError, pollster.register, 0, -1) + self.assertRaises(OverflowError, pollster.register, 0, USHRT_MAX + 1) + self.assertRaises(OverflowError, pollster.modify, 1, -1) + self.assertRaises(OverflowError, pollster.modify, 1, USHRT_MAX + 1) + self.assertRaises(OverflowError, pollster.poll, INT_MAX + 1) + self.assertRaises(OverflowError, pollster.poll, UINT_MAX + 1) @unittest.skipUnless(threading, 'Threading required for this test.') @reap_threads diff --git a/Lib/test/test_pwd.py b/Lib/test/test_pwd.py --- a/Lib/test/test_pwd.py +++ b/Lib/test/test_pwd.py @@ -8,8 +8,6 @@ def test_values(self): entries = pwd.getpwall() - entriesbyname = {} - entriesbyuid = {} for e in entries: self.assertEqual(len(e), 7) @@ -32,13 +30,20 @@ # for one uid # self.assertEqual(pwd.getpwuid(e.pw_uid), e) # instead of this collect all entries for one uid - # and check afterwards + # and check afterwards (done in test_values_extended) + + def test_values_extended(self): + entries = pwd.getpwall() + entriesbyname = {} + entriesbyuid = {} + + if len(entries) > 1000: # Huge passwd file (NIS?) -- skip this test + self.skipTest('passwd file is huge; extended test skipped') + + for e in entries: entriesbyname.setdefault(e.pw_name, []).append(e) entriesbyuid.setdefault(e.pw_uid, []).append(e) - if len(entries) > 1000: # Huge passwd file (NIS?) -- skip the rest - return - # check whether the entry returned by getpwuid() # for each uid is among those from getpwall() for this uid for e in entries: diff --git a/Lib/test/test_reprlib.py b/Lib/test/test_reprlib.py --- a/Lib/test/test_reprlib.py +++ b/Lib/test/test_reprlib.py @@ -167,8 +167,15 @@ eq(r([[[[[[[{}]]]]]]]), "[[[[[[[...]]]]]]]") def test_cell(self): - # XXX Hmm? How to get at a cell object? - pass + def get_cell(): + x = 42 + def inner(): + return x + return inner + x = get_cell().__closure__[0] + self.assertRegex(repr(x), r'') + self.assertRegex(r(x), r'') def test_descriptors(self): eq = self.assertEqual @@ -272,6 +279,7 @@ eq(repr(foo.foo), "" % foo.__name__) + @unittest.skip('need a suitable object') def test_object(self): # XXX Test the repr of a type with a really long tp_name but with no # tp_repr. WIBNI we had ::Inline? :) @@ -319,6 +327,7 @@ '= 0, "Error resolving host to ip.") try: hname, aliases, ipaddrs = socket.gethostbyaddr(ip) except socket.error: # Probably a similar problem as above; skip this test - return + self.skipTest('name lookup failure') all_host_names = [hostname, hname] + aliases fqhn = socket.getfqdn(ip) if not fqhn in all_host_names: @@ -932,9 +932,9 @@ try: from socket import inet_pton, AF_INET6, has_ipv6 if not has_ipv6: - return + self.skipTest('IPv6 not available') except ImportError: - return + self.skipTest('could not import needed symbols from socket') f = lambda a: inet_pton(AF_INET6, a) assertInvalid = lambda a: self.assertRaises( (socket.error, ValueError), f, a @@ -1010,9 +1010,9 @@ try: from socket import inet_ntop, AF_INET6, has_ipv6 if not has_ipv6: - return + self.skipTest('IPv6 not available') except ImportError: - return + self.skipTest('could not import needed symbols from socket') f = lambda a: inet_ntop(AF_INET6, a) assertInvalid = lambda a: self.assertRaises( (socket.error, ValueError), f, a @@ -1045,7 +1045,7 @@ my_ip_addr = socket.gethostbyname(socket.gethostname()) except socket.error: # Probably name lookup wasn't set up right; skip this test - return + self.skipTest('name lookup failure') self.assertIn(name[0], ("0.0.0.0", my_ip_addr), '%s invalid' % name[0]) self.assertEqual(name[1], port) diff --git a/Lib/test/test_strptime.py b/Lib/test/test_strptime.py --- a/Lib/test/test_strptime.py +++ b/Lib/test/test_strptime.py @@ -323,7 +323,7 @@ # when time.tzname[0] == time.tzname[1] and time.daylight tz_name = time.tzname[0] if tz_name.upper() in ("UTC", "GMT"): - return + self.skipTest('need non-UTC/GMT timezone') try: original_tzname = time.tzname original_daylight = time.daylight @@ -536,7 +536,7 @@ try: locale.setlocale(locale.LC_TIME, ('en_US', 'UTF8')) except locale.Error: - return + self.skipTest('test needs en_US.UTF8 locale') try: _strptime._strptime_time('10', '%d') # Get id of current cache object. @@ -553,7 +553,7 @@ # If this is the case just suppress the exception and fall-through # to the resetting to the original locale. except locale.Error: - pass + self.skipTest('test needs de_DE.UTF8 locale') # Make sure we don't trample on the locale setting once we leave the # test. finally: diff --git a/Lib/test/test_sunau.py b/Lib/test/test_sunau.py --- a/Lib/test/test_sunau.py +++ b/Lib/test/test_sunau.py @@ -5,10 +5,12 @@ import sunau -class SunauPCM8Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): +class SunauTest(audiotests.AudioWriteTests, + audiotests.AudioTestsWithSourceFile): module = sunau + + +class SunauPCM8Test(SunauTest, unittest.TestCase): sndfilename = 'pluck-pcm8.au' sndfilenframes = 3307 nchannels = 2 @@ -25,10 +27,7 @@ """) -class SunauPCM16Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = sunau +class SunauPCM16Test(SunauTest, unittest.TestCase): sndfilename = 'pluck-pcm16.au' sndfilenframes = 3307 nchannels = 2 @@ -47,10 +46,7 @@ """) -class SunauPCM32Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = sunau +class SunauPCM32Test(SunauTest, unittest.TestCase): sndfilename = 'pluck-pcm32.au' sndfilenframes = 3307 nchannels = 2 @@ -75,10 +71,7 @@ """) -class SunauULAWTest(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = sunau +class SunauULAWTest(SunauTest, unittest.TestCase): sndfilename = 'pluck-ulaw.au' sndfilenframes = 3307 nchannels = 2 diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -324,10 +324,9 @@ finally: os.rmdir(dir) + @unittest.skipUnless(has_stat, 'os.stat not available') def test_file_mode(self): # _mkstemp_inner creates files with the proper mode - if not has_stat: - return # ugh, can't use SkipTest. file = self.do_create() mode = stat.S_IMODE(os.stat(file.name).st_mode) @@ -339,10 +338,9 @@ expected = user * (1 + 8 + 64) self.assertEqual(mode, expected) + @unittest.skipUnless(has_spawnl, 'os.spawnl not available') def test_noinherit(self): # _mkstemp_inner file handles are not inherited by child processes - if not has_spawnl: - return # ugh, can't use SkipTest. if support.verbose: v="v" @@ -377,10 +375,9 @@ "child process caught fatal signal %d" % -retval) self.assertFalse(retval > 0, "child process reports failure %d"%retval) + @unittest.skipUnless(has_textmode, "text mode not available") def test_textmode(self): # _mkstemp_inner can create files in text mode - if not has_textmode: - return # ugh, can't use SkipTest. # A text file is truncated at the first Ctrl+Z byte f = self.do_create(bin=0) @@ -556,10 +553,9 @@ finally: os.rmdir(dir) + @unittest.skipUnless(has_stat, 'os.stat not available') def test_mode(self): # mkdtemp creates directories with the proper mode - if not has_stat: - return # ugh, can't use SkipTest. dir = self.do_create() try: diff --git a/Lib/test/test_thread.py b/Lib/test/test_thread.py --- a/Lib/test/test_thread.py +++ b/Lib/test/test_thread.py @@ -68,39 +68,35 @@ thread.stack_size(0) self.assertEqual(thread.stack_size(), 0, "stack_size not reset to default") - if os.name not in ("nt", "os2", "posix"): - return - - tss_supported = True + @unittest.skipIf(os.name not in ("nt", "os2", "posix"), 'test meant for nt, os2, and posix') + def test_nt_and_posix_stack_size(self): try: thread.stack_size(4096) except ValueError: verbose_print("caught expected ValueError setting " "stack_size(4096)") except thread.error: - tss_supported = False - verbose_print("platform does not support changing thread stack " - "size") + self.skipTest("platform does not support changing thread stack " + "size") - if tss_supported: - fail_msg = "stack_size(%d) failed - should succeed" - for tss in (262144, 0x100000, 0): - thread.stack_size(tss) - self.assertEqual(thread.stack_size(), tss, fail_msg % tss) - verbose_print("successfully set stack_size(%d)" % tss) + fail_msg = "stack_size(%d) failed - should succeed" + for tss in (262144, 0x100000, 0): + thread.stack_size(tss) + self.assertEqual(thread.stack_size(), tss, fail_msg % tss) + verbose_print("successfully set stack_size(%d)" % tss) - for tss in (262144, 0x100000): - verbose_print("trying stack_size = (%d)" % tss) - self.next_ident = 0 - self.created = 0 - for i in range(NUMTASKS): - self.newtask() + for tss in (262144, 0x100000): + verbose_print("trying stack_size = (%d)" % tss) + self.next_ident = 0 + self.created = 0 + for i in range(NUMTASKS): + self.newtask() - verbose_print("waiting for all tasks to complete") - self.done_mutex.acquire() - verbose_print("all tasks done") + verbose_print("waiting for all tasks to complete") + self.done_mutex.acquire() + verbose_print("all tasks done") - thread.stack_size(0) + thread.stack_size(0) def test__count(self): # Test the _count() function. diff --git a/Lib/test/test_threading.py b/Lib/test/test_threading.py --- a/Lib/test/test_threading.py +++ b/Lib/test/test_threading.py @@ -17,6 +17,10 @@ import os from test.script_helper import assert_python_ok, assert_python_failure import subprocess +try: + import _testcapi +except ImportError: + _testcapi = None from test import lock_tests @@ -769,6 +773,45 @@ for t in threads: t.join() + @unittest.skipIf(_testcapi is None, "need _testcapi module") + def test_frame_tstate_tracing(self): + # Issue #14432: Crash when a generator is created in a C thread that is + # destroyed while the generator is still used. The issue was that a + # generator contains a frame, and the frame kept a reference to the + # Python state of the destroyed C thread. The crash occurs when a trace + # function is setup. + + def noop_trace(frame, event, arg): + # no operation + return noop_trace + + def generator(): + while 1: + yield "genereator" + + def callback(): + if callback.gen is None: + callback.gen = generator() + return next(callback.gen) + callback.gen = None + + old_trace = sys.gettrace() + sys.settrace(noop_trace) + try: + # Install a trace function + threading.settrace(noop_trace) + + # Create a generator in a C thread which exits after the call + _testcapi.call_in_temporary_c_thread(callback) + + # Call the generator in a different Python thread, check that the + # generator didn't keep a reference to the destroyed thread state + for test in range(3): + # The trace function is still called here + callback() + finally: + sys.settrace(old_trace) + class ThreadingExceptionTests(BaseTestCase): # A RuntimeError should be raised if Thread.start() is called diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py --- a/Lib/test/test_time.py +++ b/Lib/test/test_time.py @@ -463,8 +463,7 @@ try: tmp = locale.setlocale(locale.LC_ALL, "fr_FR") except locale.Error: - # skip this test - return + self.skipTest('could not set locale.LC_ALL to fr_FR') # This should not cause an exception time.strftime("%B", (2009,2,1,0,0,0,0,0,0)) diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -1952,12 +1952,12 @@ self.assertEqual(repr('\U00010000'), "'%c'" % (0x10000,)) # printable self.assertEqual(repr('\U00014000'), "'\\U00014000'") # nonprintable + # This test only affects 32-bit platforms because expandtabs can only take + # an int as the max value, not a 64-bit C long. If expandtabs is changed + # to take a 64-bit long, this test should apply to all platforms. + @unittest.skipIf(sys.maxsize > (1 << 32) or struct.calcsize('P') != 4, + 'only applies to 32-bit platforms') def test_expandtabs_overflows_gracefully(self): - # This test only affects 32-bit platforms because expandtabs can only take - # an int as the max value, not a 64-bit C long. If expandtabs is changed - # to take a 64-bit long, this test should apply to all platforms. - if sys.maxsize > (1 << 32) or struct.calcsize('P') != 4: - return self.assertRaises(OverflowError, 't\tt\t'.expandtabs, sys.maxsize) @support.cpython_only diff --git a/Lib/test/test_urllibnet.py b/Lib/test/test_urllibnet.py --- a/Lib/test/test_urllibnet.py +++ b/Lib/test/test_urllibnet.py @@ -98,11 +98,10 @@ open_url.close() self.assertEqual(code, 404) + # On Windows, socket handles are not file descriptors; this + # test can't pass on Windows. + @unittest.skipIf(sys.platform in ('win32',), 'not appropriate for Windows') def test_fileno(self): - if sys.platform in ('win32',): - # On Windows, socket handles are not file descriptors; this - # test can't pass on Windows. - return # Make sure fd returned by fileno is valid. with self.urlopen("http://www.python.org/", timeout=None) as open_url: fd = open_url.fileno() diff --git a/Lib/test/test_warnings.py b/Lib/test/test_warnings.py --- a/Lib/test/test_warnings.py +++ b/Lib/test/test_warnings.py @@ -682,7 +682,7 @@ # Explicit tests for the test.support convenience wrapper wmod = self.module if wmod is not sys.modules['warnings']: - return + self.skipTest('module to test is not loaded warnings module') with support.check_warnings(quiet=False) as w: self.assertEqual(w.warnings, []) wmod.simplefilter("always") diff --git a/Lib/test/test_wave.py b/Lib/test/test_wave.py --- a/Lib/test/test_wave.py +++ b/Lib/test/test_wave.py @@ -5,10 +5,15 @@ import wave -class WavePCM8Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): +class WaveTest(audiotests.AudioWriteTests, + audiotests.AudioTestsWithSourceFile): module = wave + test_unseekable_write = None + test_unseekable_overflowed_write = None + test_unseekable_incompleted_write = None + + +class WavePCM8Test(WaveTest, unittest.TestCase): sndfilename = 'pluck-pcm8.wav' sndfilenframes = 3307 nchannels = 2 @@ -25,10 +30,7 @@ """) -class WavePCM16Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = wave +class WavePCM16Test(WaveTest, unittest.TestCase): sndfilename = 'pluck-pcm16.wav' sndfilenframes = 3307 nchannels = 2 @@ -55,10 +57,7 @@ -class WavePCM24Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = wave +class WavePCM24Test(WaveTest, unittest.TestCase): sndfilename = 'pluck-pcm24.wav' sndfilenframes = 3307 nchannels = 2 @@ -85,10 +84,7 @@ frames = audiotests.byteswap3(frames) -class WavePCM32Test(audiotests.AudioWriteTests, - audiotests.AudioTestsWithSourceFile, - unittest.TestCase): - module = wave +class WavePCM32Test(WaveTest, unittest.TestCase): sndfilename = 'pluck-pcm32.wav' sndfilenframes = 3307 nchannels = 2 diff --git a/Lib/test/test_xmlrpc_net.py b/Lib/test/test_xmlrpc_net.py --- a/Lib/test/test_xmlrpc_net.py +++ b/Lib/test/test_xmlrpc_net.py @@ -20,7 +20,6 @@ t0 = server.currentTime.getCurrentTime() except socket.error as e: self.skipTest("network error: %s" % e) - return # Perform a minimal sanity check on the result, just to be sure # the request means what we think it means. @@ -44,7 +43,6 @@ builders = server.getAllBuilders() except socket.error as e: self.skipTest("network error: %s" % e) - return self.addCleanup(lambda: server('close')()) # Perform a minimal sanity check on the result, just to be sure diff --git a/Lib/test/test_zipimport.py b/Lib/test/test_zipimport.py --- a/Lib/test/test_zipimport.py +++ b/Lib/test/test_zipimport.py @@ -111,7 +111,7 @@ # so we'll simply skip it then. Bug #765456. # if "zlib" in sys.builtin_module_names: - return + self.skipTest('zlib is a builtin module') if "zlib" in sys.modules: del sys.modules["zlib"] files = {"zlib.py": (NOW, test_src)} diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -689,6 +689,7 @@ Andrew Kuchling Dave Kuhlman Jon Kuhn +Toshio Kuratomi Vladimir Kushnir Erno Kuusela Ross Lagerwall diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,17 @@ Core and Builtins ----------------- +- Issue #19969: PyBytes_FromFormatV() now raises an OverflowError if "%c" + argument is not in range [0; 255]. + +- Issue #14432: Generator now clears the borrowed reference to the thread + state. Fix a crash when a generator is created in a C thread that is + destroyed while the generator is still used. The issue was that a generator + contains a frame, and the frame kept a reference to the Python state of the + destroyed C thread. The crash occurs when a trace function is setup. + +- Issue #19932: Fix typo in import.h, missing whitespaces in function prototypes. + - Issue #19729: In str.format(), fix recursive expansion in format spec. - Issue #19638: Fix possible crash / undefined behaviour from huge (more than 2 @@ -18,6 +29,34 @@ Library ------- +- Issue #19623: Fixed writing to unseekable files in the aifc module. + +- Issue #17919: select.poll.register() again works with poll.POLLNVAL on AIX. + Fixed integer overflow in the eventmask parameter. + +- Issue #19063: if a Charset's body_encoding was set to None, the email + package would generate a message claiming the Content-Transfer-Encoding + was 7bit, and produce garbage output for the content. This now works. + A couple of other set_payload mishandlings of non-ASCII are also fixed. + +- Issue #17200: telnetlib's read_until and expect timeout was broken by the + fix to Issue #14635 in Python 3.3.0 to be interpreted as milliseconds + instead of seconds when the platform supports select.poll (ie: everywhere). + It is now treated as seconds once again. + +- Issue #17429: platform.linux_distribution() now decodes files from the UTF-8 + encoding with the surrogateescape error handler, instead of decoding from the + locale encoding in strict mode. It fixes the function on Fedora 19 which is + probably the first major distribution release with a non-ASCII name. Patch + written by Toshio Kuratomi. + +- Issue #19929: Call os.read with 32768 within subprocess.Popen.communicate + rather than 4096 for efficiency. A microbenchmark shows Linux and OS X + both using ~50% less cpu time this way. + +- Issue #19506: Use a memoryview to avoid a data copy when piping data + to stdin within subprocess.Popen.communicate. 5-10% less cpu usage. + - Issue #19839: Fix regression in bz2 module's handling of non-bzip2 data at EOF, and analogous bug in lzma module. @@ -91,9 +130,26 @@ - Issue #19286: Directories in ``package_data`` are no longer added to the filelist, preventing failure outlined in the ticket. +IDLE +---- + +- Issue #19481: print() of string subclass instance in IDLE no longer hangs. + +- Issue #18270: Prevent possible IDLE AttributeError on OS X when no initial + shell window is present. + Tests ----- +- Issue #19828: Fixed test_site when the whole suite is run with -S. + +- Issue #19928: Implemented a test for repr() of cell objects. + +- Issue #19535: Fixed test_docxmlrpc when python is run with -OO. + +- Issue #19926: Removed unneeded test_main from test_abstract_numbers. + Patch by Vajrasky Kok. + - Issue #19595: Re-enabled a long-disabled test in test_winsound. - Issue #19588: Fixed tests in test_random that were silently skipped most @@ -106,13 +162,16 @@ import, converting from test_main to unittest.main, and running the _testcapi module tests within a unittest TestCase. -- Issue #18702: All skipped tests now reported as skipped. +- Issue #18702, 19572: All skipped tests now reported as skipped. - Issue #19085: Added basic tests for all tkinter widget options. Documentation ------------- +- Issue #19963: Document that importlib.import_module() no longer requires + importing parent packages separately. + - Issue #18840: Introduce the json module in the tutorial, and deemphasize the pickle module. diff --git a/Modules/_decimal/libmpdec/basearith.c b/Modules/_decimal/libmpdec/basearith.c --- a/Modules/_decimal/libmpdec/basearith.c +++ b/Modules/_decimal/libmpdec/basearith.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/basearith.h b/Modules/_decimal/libmpdec/basearith.h --- a/Modules/_decimal/libmpdec/basearith.h +++ b/Modules/_decimal/libmpdec/basearith.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/bits.h b/Modules/_decimal/libmpdec/bits.h --- a/Modules/_decimal/libmpdec/bits.h +++ b/Modules/_decimal/libmpdec/bits.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/constants.c b/Modules/_decimal/libmpdec/constants.c --- a/Modules/_decimal/libmpdec/constants.c +++ b/Modules/_decimal/libmpdec/constants.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/constants.h b/Modules/_decimal/libmpdec/constants.h --- a/Modules/_decimal/libmpdec/constants.h +++ b/Modules/_decimal/libmpdec/constants.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/context.c b/Modules/_decimal/libmpdec/context.c --- a/Modules/_decimal/libmpdec/context.c +++ b/Modules/_decimal/libmpdec/context.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/convolute.c b/Modules/_decimal/libmpdec/convolute.c --- a/Modules/_decimal/libmpdec/convolute.c +++ b/Modules/_decimal/libmpdec/convolute.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/convolute.h b/Modules/_decimal/libmpdec/convolute.h --- a/Modules/_decimal/libmpdec/convolute.h +++ b/Modules/_decimal/libmpdec/convolute.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/crt.c b/Modules/_decimal/libmpdec/crt.c --- a/Modules/_decimal/libmpdec/crt.c +++ b/Modules/_decimal/libmpdec/crt.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/crt.h b/Modules/_decimal/libmpdec/crt.h --- a/Modules/_decimal/libmpdec/crt.h +++ b/Modules/_decimal/libmpdec/crt.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/difradix2.c b/Modules/_decimal/libmpdec/difradix2.c --- a/Modules/_decimal/libmpdec/difradix2.c +++ b/Modules/_decimal/libmpdec/difradix2.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/difradix2.h b/Modules/_decimal/libmpdec/difradix2.h --- a/Modules/_decimal/libmpdec/difradix2.h +++ b/Modules/_decimal/libmpdec/difradix2.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/fnt.c b/Modules/_decimal/libmpdec/fnt.c --- a/Modules/_decimal/libmpdec/fnt.c +++ b/Modules/_decimal/libmpdec/fnt.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/fnt.h b/Modules/_decimal/libmpdec/fnt.h --- a/Modules/_decimal/libmpdec/fnt.h +++ b/Modules/_decimal/libmpdec/fnt.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/fourstep.c b/Modules/_decimal/libmpdec/fourstep.c --- a/Modules/_decimal/libmpdec/fourstep.c +++ b/Modules/_decimal/libmpdec/fourstep.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/fourstep.h b/Modules/_decimal/libmpdec/fourstep.h --- a/Modules/_decimal/libmpdec/fourstep.h +++ b/Modules/_decimal/libmpdec/fourstep.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/io.c b/Modules/_decimal/libmpdec/io.c --- a/Modules/_decimal/libmpdec/io.c +++ b/Modules/_decimal/libmpdec/io.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/io.h b/Modules/_decimal/libmpdec/io.h --- a/Modules/_decimal/libmpdec/io.h +++ b/Modules/_decimal/libmpdec/io.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/literature/fnt.py b/Modules/_decimal/libmpdec/literature/fnt.py --- a/Modules/_decimal/libmpdec/literature/fnt.py +++ b/Modules/_decimal/libmpdec/literature/fnt.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2008-2012 Stefan Krah. All rights reserved. +# Copyright (c) 2008-2016 Stefan Krah. All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/literature/mulmod-64.txt b/Modules/_decimal/libmpdec/literature/mulmod-64.txt --- a/Modules/_decimal/libmpdec/literature/mulmod-64.txt +++ b/Modules/_decimal/libmpdec/literature/mulmod-64.txt @@ -59,7 +59,7 @@ Maximum numbers of step b): --------------------------- -# To avoid unneccessary formalism, define: +# To avoid unnecessary formalism, define: def R(hi, lo, z): return divmod(hi * z - hi + lo, 2**64) diff --git a/Modules/_decimal/libmpdec/literature/umodarith.lisp b/Modules/_decimal/libmpdec/literature/umodarith.lisp --- a/Modules/_decimal/libmpdec/literature/umodarith.lisp +++ b/Modules/_decimal/libmpdec/literature/umodarith.lisp @@ -1,5 +1,5 @@ ; -; Copyright (c) 2008-2012 Stefan Krah. All rights reserved. +; Copyright (c) 2008-2016 Stefan Krah. All rights reserved. ; ; Redistribution and use in source and binary forms, with or without ; modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/memory.c b/Modules/_decimal/libmpdec/memory.c --- a/Modules/_decimal/libmpdec/memory.c +++ b/Modules/_decimal/libmpdec/memory.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/memory.h b/Modules/_decimal/libmpdec/memory.h --- a/Modules/_decimal/libmpdec/memory.h +++ b/Modules/_decimal/libmpdec/memory.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions @@ -3202,9 +3202,9 @@ } static inline void -_mpd_ptrswap(mpd_t **a, mpd_t **b) -{ - mpd_t *t = *a; +_mpd_ptrswap(const mpd_t **a, const mpd_t **b) +{ + const mpd_t *t = *a; *a = *b; *b = t; } @@ -3232,7 +3232,7 @@ _mpd_qaddsub(mpd_t *result, const mpd_t *a, const mpd_t *b, uint8_t sign_b, const mpd_context_t *ctx, uint32_t *status) { - mpd_t *big, *small; + const mpd_t *big, *small; MPD_NEW_STATIC(big_aligned,0,0,0,0); MPD_NEW_CONST(tiny,0,0,1,1,1,1); mpd_uint_t carry; @@ -3242,7 +3242,7 @@ /* compare exponents */ - big = (mpd_t *)a; small = (mpd_t *)b; + big = a; small = b; if (big->exp != small->exp) { if (small->exp > big->exp) { _mpd_ptrswap(&big, &small); @@ -4421,7 +4421,7 @@ const mpd_context_t *ctx, uint32_t *status) { uint32_t workstatus = 0; - mpd_t *cc = (mpd_t *)c; + const mpd_t *cc = c; if (result == c) { if ((cc = mpd_qncopy(c)) == NULL) { @@ -4435,7 +4435,7 @@ mpd_qadd(result, result, cc, ctx, &workstatus); } - if (cc != c) mpd_del(cc); + if (cc != c) mpd_del((mpd_t *)cc); *status |= workstatus; } @@ -5727,7 +5727,7 @@ _mpd_qmul(mpd_t *result, const mpd_t *a, const mpd_t *b, const mpd_context_t *ctx, uint32_t *status) { - mpd_t *big = (mpd_t *)a, *small = (mpd_t *)b; + const mpd_t *big = a, *small = b; mpd_uint_t *rdata = NULL; mpd_uint_t rbuf[MPD_MINALLOC_MAX]; mpd_size_t rsize, i; diff --git a/Modules/_decimal/libmpdec/mpdecimal.h b/Modules/_decimal/libmpdec/mpdecimal.h --- a/Modules/_decimal/libmpdec/mpdecimal.h +++ b/Modules/_decimal/libmpdec/mpdecimal.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions @@ -32,6 +32,10 @@ #ifdef __cplusplus extern "C" { + #ifndef __STDC_LIMIT_MACROS + #define __STDC_LIMIT_MACROS + #define MPD_CLEAR_STDC_LIMIT_MACROS + #endif #endif @@ -55,18 +59,12 @@ #define MPD_HIDE_SYMBOLS_END #define EXTINLINE extern inline #else + #ifdef HAVE_STDINT_H + #include + #endif #ifdef HAVE_INTTYPES_H #include #endif - #ifdef HAVE_STDINT_H - #if defined(__cplusplus) && !defined(__STDC_LIMIT_MACROS) - #define __STDC_LIMIT_MACROS - #include - #undef __STDC_LIMIT_MACROS - #else - #include - #endif - #endif #ifndef __GNUC_STDC_INLINE__ #define __GNUC_STDC_INLINE__ 1 #endif @@ -835,6 +833,10 @@ #ifdef __cplusplus + #ifdef MPD_CLEAR_STDC_LIMIT_MACROS + #undef MPD_CLEAR_STDC_LIMIT_MACROS + #undef __STDC_LIMIT_MACROS + #endif } /* END extern "C" */ #endif diff --git a/Modules/_decimal/libmpdec/numbertheory.c b/Modules/_decimal/libmpdec/numbertheory.c --- a/Modules/_decimal/libmpdec/numbertheory.c +++ b/Modules/_decimal/libmpdec/numbertheory.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/numbertheory.h b/Modules/_decimal/libmpdec/numbertheory.h --- a/Modules/_decimal/libmpdec/numbertheory.h +++ b/Modules/_decimal/libmpdec/numbertheory.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/sixstep.c b/Modules/_decimal/libmpdec/sixstep.c --- a/Modules/_decimal/libmpdec/sixstep.c +++ b/Modules/_decimal/libmpdec/sixstep.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/sixstep.h b/Modules/_decimal/libmpdec/sixstep.h --- a/Modules/_decimal/libmpdec/sixstep.h +++ b/Modules/_decimal/libmpdec/sixstep.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/transpose.c b/Modules/_decimal/libmpdec/transpose.c --- a/Modules/_decimal/libmpdec/transpose.c +++ b/Modules/_decimal/libmpdec/transpose.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions @@ -169,7 +169,7 @@ /* * Transpose 2^n * 2^n matrix. For cache efficiency, the matrix is split into * square blocks with side length 'SIDE'. First, the blocks are transposed, - * then a square tranposition is done on each individual block. + * then a square transposition is done on each individual block. */ static void squaretrans_pow2(mpd_uint_t *matrix, mpd_size_t size) diff --git a/Modules/_decimal/libmpdec/transpose.h b/Modules/_decimal/libmpdec/transpose.h --- a/Modules/_decimal/libmpdec/transpose.h +++ b/Modules/_decimal/libmpdec/transpose.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/typearith.h b/Modules/_decimal/libmpdec/typearith.h --- a/Modules/_decimal/libmpdec/typearith.h +++ b/Modules/_decimal/libmpdec/typearith.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/umodarith.h b/Modules/_decimal/libmpdec/umodarith.h --- a/Modules/_decimal/libmpdec/umodarith.h +++ b/Modules/_decimal/libmpdec/umodarith.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/vccompat.h b/Modules/_decimal/libmpdec/vccompat.h --- a/Modules/_decimal/libmpdec/vccompat.h +++ b/Modules/_decimal/libmpdec/vccompat.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2008-2012 Stefan Krah. All rights reserved. + * Copyright (c) 2008-2016 Stefan Krah. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions diff --git a/Modules/_decimal/libmpdec/vcdiv64.asm b/Modules/_decimal/libmpdec/vcdiv64.asm --- a/Modules/_decimal/libmpdec/vcdiv64.asm +++ b/Modules/_decimal/libmpdec/vcdiv64.asm @@ -1,5 +1,5 @@ ; -; Copyright (c) 2008-2012 Stefan Krah. All rights reserved. +; Copyright (c) 2008-2016 Stefan Krah. All rights reserved. ; ; Redistribution and use in source and binary forms, with or without ; modification, are permitted provided that the following conditions diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -2477,6 +2477,95 @@ return Py_BuildValue("Nl", _PyLong_FromTime_t(sec), nsec); } +#ifdef WITH_THREAD +typedef struct { + PyThread_type_lock start_event; + PyThread_type_lock exit_event; + PyObject *callback; +} test_c_thread_t; + +static void +temporary_c_thread(void *data) +{ + test_c_thread_t *test_c_thread = data; + PyGILState_STATE state; + PyObject *res; + + PyThread_release_lock(test_c_thread->start_event); + + /* Allocate a Python thread state for this thread */ + state = PyGILState_Ensure(); + + res = PyObject_CallFunction(test_c_thread->callback, "", NULL); + Py_CLEAR(test_c_thread->callback); + + if (res == NULL) { + PyErr_Print(); + } + else { + Py_DECREF(res); + } + + /* Destroy the Python thread state for this thread */ + PyGILState_Release(state); + + PyThread_release_lock(test_c_thread->exit_event); + + PyThread_exit_thread(); +} + +static PyObject * +call_in_temporary_c_thread(PyObject *self, PyObject *callback) +{ + PyObject *res = NULL; + test_c_thread_t test_c_thread; + long thread; + + PyEval_InitThreads(); + + test_c_thread.start_event = PyThread_allocate_lock(); + test_c_thread.exit_event = PyThread_allocate_lock(); + test_c_thread.callback = NULL; + if (!test_c_thread.start_event || !test_c_thread.exit_event) { + PyErr_SetString(PyExc_RuntimeError, "could not allocate lock"); + goto exit; + } + + Py_INCREF(callback); + test_c_thread.callback = callback; + + PyThread_acquire_lock(test_c_thread.start_event, 1); + PyThread_acquire_lock(test_c_thread.exit_event, 1); + + thread = PyThread_start_new_thread(temporary_c_thread, &test_c_thread); + if (thread == -1) { + PyErr_SetString(PyExc_RuntimeError, "unable to start the thread"); + PyThread_release_lock(test_c_thread.start_event); + PyThread_release_lock(test_c_thread.exit_event); + goto exit; + } + + PyThread_acquire_lock(test_c_thread.start_event, 1); + PyThread_release_lock(test_c_thread.start_event); + + Py_BEGIN_ALLOW_THREADS + PyThread_acquire_lock(test_c_thread.exit_event, 1); + PyThread_release_lock(test_c_thread.exit_event); + Py_END_ALLOW_THREADS + + Py_INCREF(Py_None); + res = Py_None; + +exit: + Py_CLEAR(test_c_thread.callback); + if (test_c_thread.start_event) + PyThread_free_lock(test_c_thread.start_event); + if (test_c_thread.exit_event) + PyThread_free_lock(test_c_thread.exit_event); + return res; +} +#endif /* WITH_THREAD */ + static PyMethodDef TestMethods[] = { {"raise_exception", raise_exception, METH_VARARGS}, @@ -2574,6 +2663,10 @@ {"pytime_object_to_time_t", test_pytime_object_to_time_t, METH_VARARGS}, {"pytime_object_to_timeval", test_pytime_object_to_timeval, METH_VARARGS}, {"pytime_object_to_timespec", test_pytime_object_to_timespec, METH_VARARGS}, +#ifdef WITH_THREAD + {"call_in_temporary_c_thread", call_in_temporary_c_thread, METH_O, + PyDoc_STR("set_error_class(error_class) -> None")}, +#endif {NULL, NULL} /* sentinel */ }; diff --git a/Modules/faulthandler.c b/Modules/faulthandler.c --- a/Modules/faulthandler.c +++ b/Modules/faulthandler.c @@ -301,7 +301,7 @@ return; } #endif - /* call the previous signal handler: it is called immediatly if we use + /* call the previous signal handler: it is called immediately if we use sigaction() thanks to SA_NODEFER flag, otherwise it is deferred */ raise(signum); } diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -1742,7 +1742,7 @@ /* FILE_FLAG_BACKUP_SEMANTICS is required to open a directory */ /* FILE_FLAG_OPEN_REPARSE_POINT does not follow the symlink. Because of this, calls like GetFinalPathNameByHandle will return - the symlink path agin and not the actual final path. */ + the symlink path again and not the actual final path. */ FILE_ATTRIBUTE_NORMAL|FILE_FLAG_BACKUP_SEMANTICS| FILE_FLAG_OPEN_REPARSE_POINT, NULL); @@ -1838,7 +1838,7 @@ /* FILE_FLAG_BACKUP_SEMANTICS is required to open a directory */ /* FILE_FLAG_OPEN_REPARSE_POINT does not follow the symlink. Because of this, calls like GetFinalPathNameByHandle will return - the symlink path agin and not the actual final path. */ + the symlink path again and not the actual final path. */ FILE_ATTRIBUTE_NORMAL|FILE_FLAG_BACKUP_SEMANTICS| FILE_FLAG_OPEN_REPARSE_POINT, NULL); diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -361,7 +361,7 @@ assert(i < self->ufd_len); /* Never overflow */ self->ufds[i].fd = (int)PyLong_AsLong(key); - self->ufds[i].events = (short)PyLong_AsLong(value); + self->ufds[i].events = (short)(unsigned short)PyLong_AsLong(value); i++; } assert(i == self->ufd_len); @@ -369,6 +369,24 @@ return 1; } +static int +ushort_converter(PyObject *obj, void *ptr) +{ + unsigned long uval; + + uval = PyLong_AsUnsignedLong(obj); + if (uval == (unsigned long)-1 && PyErr_Occurred()) + return 0; + if (uval > USHRT_MAX) { + PyErr_SetString(PyExc_OverflowError, + "Python int too large for C unsigned short"); + return 0; + } + + *(unsigned short *)ptr = Py_SAFE_DOWNCAST(uval, unsigned long, unsigned short); + return 1; +} + PyDoc_STRVAR(poll_register_doc, "register(fd [, eventmask] ) -> None\n\n\ Register a file descriptor with the polling object.\n\ @@ -381,12 +399,11 @@ { PyObject *o, *key, *value; int fd; - short events = POLLIN | POLLPRI | POLLOUT; + unsigned short events = POLLIN | POLLPRI | POLLOUT; int err; - if (!PyArg_ParseTuple(args, "O|h:register", &o, &events)) { + if (!PyArg_ParseTuple(args, "O|O&:register", &o, ushort_converter, &events)) return NULL; - } fd = PyObject_AsFileDescriptor(o); if (fd == -1) return NULL; @@ -424,12 +441,12 @@ poll_modify(pollObject *self, PyObject *args) { PyObject *o, *key, *value; - int fd, events; + int fd; + unsigned short events; int err; - if (!PyArg_ParseTuple(args, "Oi:modify", &o, &events)) { + if (!PyArg_ParseTuple(args, "OO&:modify", &o, ushort_converter, &events)) return NULL; - } fd = PyObject_AsFileDescriptor(o); if (fd == -1) return NULL; @@ -727,11 +744,11 @@ internal_devpoll_register(devpollObject *self, PyObject *args, int remove) { PyObject *o; - int fd, events = POLLIN | POLLPRI | POLLOUT; + int fd; + unsigned short events = POLLIN | POLLPRI | POLLOUT; - if (!PyArg_ParseTuple(args, "O|i:register", &o, &events)) { + if (!PyArg_ParseTuple(args, "O|O&:register", &o, ushort_converter, &events)) return NULL; - } fd = PyObject_AsFileDescriptor(o); if (fd == -1) return NULL; @@ -747,7 +764,7 @@ } self->fds[self->n_fds].fd = fd; - self->fds[self->n_fds].events = events; + self->fds[self->n_fds].events = (signed short)events; if (++self->n_fds == self->max_n_fds) { if (devpoll_flush(self)) diff --git a/Objects/abstract.c b/Objects/abstract.c --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -1132,7 +1132,7 @@ return type_error("bad operand type for abs(): '%.200s'", o); } -/* Return a Python int from the object item +/* Return a Python int from the object item. Raise TypeError if the result is not an int or if the object cannot be interpreted as an index. */ @@ -1146,21 +1146,21 @@ Py_INCREF(item); return item; } - if (PyIndex_Check(item)) { - result = item->ob_type->tp_as_number->nb_index(item); - if (result && !PyLong_Check(result)) { - PyErr_Format(PyExc_TypeError, - "__index__ returned non-int " - "(type %.200s)", - result->ob_type->tp_name); - Py_DECREF(result); - return NULL; - } - } - else { + if (!PyIndex_Check(item)) { PyErr_Format(PyExc_TypeError, "'%.200s' object cannot be interpreted " "as an integer", item->ob_type->tp_name); + return NULL; + } + result = item->ob_type->tp_as_number->nb_index(item); + if (!result || PyLong_CheckExact(result)) + return result; + if (!PyLong_Check(result)) { + PyErr_Format(PyExc_TypeError, + "__index__ returned non-int (type %.200s)", + result->ob_type->tp_name); + Py_DECREF(result); + return NULL; } return result; } @@ -1212,34 +1212,6 @@ } -/* - Returns the Integral instance converted to an int. The instance is expected - to be an int or have an __int__ method. Steals integral's - reference. error_format will be used to create the TypeError if integral - isn't actually an Integral instance. error_format should be a format string - that can accept a char* naming integral's type. -*/ -static PyObject * -convert_integral_to_int(PyObject *integral, const char *error_format) -{ - PyNumberMethods *nb; - if (PyLong_Check(integral)) - return integral; - nb = Py_TYPE(integral)->tp_as_number; - if (nb->nb_int) { - PyObject *as_int = nb->nb_int(integral); - if (!as_int || PyLong_Check(as_int)) { - Py_DECREF(integral); - return as_int; - } - Py_DECREF(as_int); - } - PyErr_Format(PyExc_TypeError, error_format, Py_TYPE(integral)->tp_name); - Py_DECREF(integral); - return NULL; -} - - PyObject * PyNumber_Long(PyObject *o) { @@ -1257,29 +1229,28 @@ } m = o->ob_type->tp_as_number; if (m && m->nb_int) { /* This should include subclasses of int */ - PyObject *res = m->nb_int(o); - if (res && !PyLong_Check(res)) { - PyErr_Format(PyExc_TypeError, - "__int__ returned non-int (type %.200s)", - res->ob_type->tp_name); - Py_DECREF(res); - return NULL; - } - return res; + return (PyObject *)_PyLong_FromNbInt(o); } - if (PyLong_Check(o)) /* An int subclass without nb_int */ - return _PyLong_Copy((PyLongObject *)o); trunc_func = _PyObject_LookupSpecial(o, &PyId___trunc__); if (trunc_func) { PyObject *truncated = PyEval_CallObject(trunc_func, NULL); PyObject *int_instance; Py_DECREF(trunc_func); - if (truncated == NULL) - return NULL; + if (truncated == NULL || PyLong_Check(truncated)) + return truncated; /* __trunc__ is specified to return an Integral type, but int() needs to return a int. */ - int_instance = convert_integral_to_int(truncated, - "__trunc__ returned non-Integral (type %.200s)"); + m = truncated->ob_type->tp_as_number; + if (m == NULL || m->nb_int == NULL) { + PyErr_Format( + PyExc_TypeError, + "__trunc__ returned non-Integral (type %.200s)", + truncated->ob_type->tp_name); + Py_DECREF(truncated); + return NULL; + } + int_instance = (PyObject *)_PyLong_FromNbInt(truncated); + Py_DECREF(truncated); return int_instance; } if (PyErr_Occurred()) diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -186,8 +186,17 @@ switch (*f) { case 'c': - (void)va_arg(count, int); - /* fall through... */ + { + int c = va_arg(count, int); + if (c < 0 || c > 255) { + PyErr_SetString(PyExc_OverflowError, + "PyBytes_FromFormatV(): %c format " + "expects an integer in range [0; 255]"); + return NULL; + } + n++; + break; + } case '%': n++; break; @@ -267,8 +276,12 @@ switch (*f) { case 'c': - *s++ = va_arg(vargs, int); + { + int c = va_arg(vargs, int); + /* c has been checked for overflow in the first step */ + *s++ = (unsigned char)c; break; + } case 'd': if (longflag) sprintf(s, "%ld", va_arg(vargs, long)); diff --git a/Objects/genobject.c b/Objects/genobject.c --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -76,6 +76,7 @@ /* Generators always return to their most recent caller, not * necessarily their creator. */ + f->f_tstate = tstate; Py_XINCREF(tstate->frame); assert(f->f_back == NULL); f->f_back = tstate->frame; @@ -89,6 +90,8 @@ * cycle. */ assert(f->f_back == tstate->frame); Py_CLEAR(f->f_back); + /* Clear the borrowed reference to the thread state */ + f->f_tstate = NULL; /* If the generator just returned (as opposed to yielding), signal * that the generator is exhausted. */ diff --git a/Objects/longobject.c b/Objects/longobject.c --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -116,6 +116,47 @@ return v; } +/* _PyLong_FromNbInt: Convert the given object to a PyLongObject + using the nb_int slot, if available. Raise TypeError if either the + nb_int slot is not available or the result of the call to nb_int + returns something not of type int. +*/ +PyLongObject * +_PyLong_FromNbInt(PyObject *integral) +{ + PyNumberMethods *nb; + PyObject *result; + + /* Fast path for the case that we already have an int. */ + if (PyLong_CheckExact(integral)) { + Py_INCREF(integral); + return (PyLongObject *)integral; + } + + nb = Py_TYPE(integral)->tp_as_number; + if (nb == NULL || nb->nb_int == NULL) { + PyErr_Format(PyExc_TypeError, + "an integer is required (got type %.200s)", + Py_TYPE(integral)->tp_name); + return NULL; + } + + /* Convert using the nb_int slot, which should return something + of exact type int. */ + result = nb->nb_int(integral); + if (!result || PyLong_CheckExact(result)) + return (PyLongObject *)result; + if (!PyLong_Check(result)) { + PyErr_Format(PyExc_TypeError, + "__int__ returned non-int (type %.200s)", + result->ob_type->tp_name); + Py_DECREF(result); + return NULL; + } + return (PyLongObject *)result; +} + + /* Allocate a new int object with size digits. Return NULL and set exception if we run out of memory. */ @@ -347,28 +388,17 @@ return -1; } - if (!PyLong_Check(vv)) { - PyNumberMethods *nb; - nb = vv->ob_type->tp_as_number; - if (nb == NULL || nb->nb_int == NULL) { - PyErr_SetString(PyExc_TypeError, - "an integer is required"); - return -1; - } - vv = (*nb->nb_int) (vv); - if (vv == NULL) + if (PyLong_Check(vv)) { + v = (PyLongObject *)vv; + } + else { + v = _PyLong_FromNbInt(vv); + if (v == NULL) return -1; do_decref = 1; - if (!PyLong_Check(vv)) { - Py_DECREF(vv); - PyErr_SetString(PyExc_TypeError, - "nb_int should return int object"); - return -1; - } } res = -1; - v = (PyLongObject *)vv; i = Py_SIZE(v); switch (i) { @@ -412,7 +442,7 @@ } exit: if (do_decref) { - Py_DECREF(vv); + Py_DECREF(v); } return res; } @@ -630,36 +660,25 @@ unsigned long PyLong_AsUnsignedLongMask(register PyObject *op) { - PyNumberMethods *nb; PyLongObject *lo; unsigned long val; - if (op && PyLong_Check(op)) + if (op == NULL) { + PyErr_BadInternalCall(); + return (unsigned long)-1; + } + + if (PyLong_Check(op)) { return _PyLong_AsUnsignedLongMask(op); - - if (op == NULL || (nb = op->ob_type->tp_as_number) == NULL || - nb->nb_int == NULL) { - PyErr_SetString(PyExc_TypeError, "an integer is required"); - return (unsigned long)-1; - } - - lo = (PyLongObject*) (*nb->nb_int) (op); + } + + lo = _PyLong_FromNbInt(op); if (lo == NULL) return (unsigned long)-1; - if (PyLong_Check(lo)) { - val = _PyLong_AsUnsignedLongMask((PyObject *)lo); - Py_DECREF(lo); - if (PyErr_Occurred()) - return (unsigned long)-1; - return val; - } - else - { - Py_DECREF(lo); - PyErr_SetString(PyExc_TypeError, - "nb_int should return int object"); - return (unsigned long)-1; - } + + val = _PyLong_AsUnsignedLongMask((PyObject *)lo); + Py_DECREF(lo); + return val; } int @@ -1169,40 +1188,41 @@ PY_LONG_LONG bytes; int one = 1; int res; + int do_decref = 0; /* if nb_int was called */ if (vv == NULL) { PyErr_BadInternalCall(); return -1; } - if (!PyLong_Check(vv)) { - PyNumberMethods *nb; - PyObject *io; - if ((nb = vv->ob_type->tp_as_number) == NULL || - nb->nb_int == NULL) { - PyErr_SetString(PyExc_TypeError, "an integer is required"); + + if (PyLong_Check(vv)) { + v = (PyLongObject *)vv; + } + else { + v = _PyLong_FromNbInt(vv); + if (v == NULL) return -1; - } - io = (*nb->nb_int) (vv); - if (io == NULL) - return -1; - if (PyLong_Check(io)) { - bytes = PyLong_AsLongLong(io); - Py_DECREF(io); - return bytes; - } - Py_DECREF(io); - PyErr_SetString(PyExc_TypeError, "integer conversion failed"); - return -1; - } - - v = (PyLongObject*)vv; + do_decref = 1; + } + + res = 0; switch(Py_SIZE(v)) { - case -1: return -(sdigit)v->ob_digit[0]; - case 0: return 0; - case 1: return v->ob_digit[0]; - } - res = _PyLong_AsByteArray((PyLongObject *)vv, (unsigned char *)&bytes, - SIZEOF_LONG_LONG, IS_LITTLE_ENDIAN, 1); + case -1: + bytes = -(sdigit)v->ob_digit[0]; + break; + case 0: + bytes = 0; + break; + case 1: + bytes = v->ob_digit[0]; + break; + default: + res = _PyLong_AsByteArray((PyLongObject *)v, (unsigned char *)&bytes, + SIZEOF_LONG_LONG, IS_LITTLE_ENDIAN, 1); + } + if (do_decref) { + Py_DECREF(v); + } /* Plan 9 can't handle PY_LONG_LONG in ? : expressions */ if (res < 0) @@ -1283,36 +1303,25 @@ unsigned PY_LONG_LONG PyLong_AsUnsignedLongLongMask(register PyObject *op) { - PyNumberMethods *nb; PyLongObject *lo; unsigned PY_LONG_LONG val; - if (op && PyLong_Check(op)) + if (op == NULL) { + PyErr_BadInternalCall(); + return (unsigned long)-1; + } + + if (PyLong_Check(op)) { return _PyLong_AsUnsignedLongLongMask(op); - - if (op == NULL || (nb = op->ob_type->tp_as_number) == NULL || - nb->nb_int == NULL) { - PyErr_SetString(PyExc_TypeError, "an integer is required"); - return (unsigned PY_LONG_LONG)-1; - } - - lo = (PyLongObject*) (*nb->nb_int) (op); + } + + lo = _PyLong_FromNbInt(op); if (lo == NULL) return (unsigned PY_LONG_LONG)-1; - if (PyLong_Check(lo)) { - val = _PyLong_AsUnsignedLongLongMask((PyObject *)lo); - Py_DECREF(lo); - if (PyErr_Occurred()) - return (unsigned PY_LONG_LONG)-1; - return val; - } - else - { - Py_DECREF(lo); - PyErr_SetString(PyExc_TypeError, - "nb_int should return int object"); - return (unsigned PY_LONG_LONG)-1; - } + + val = _PyLong_AsUnsignedLongLongMask((PyObject *)lo); + Py_DECREF(lo); + return val; } #undef IS_LITTLE_ENDIAN @@ -1343,28 +1352,17 @@ return -1; } - if (!PyLong_Check(vv)) { - PyNumberMethods *nb; - nb = vv->ob_type->tp_as_number; - if (nb == NULL || nb->nb_int == NULL) { - PyErr_SetString(PyExc_TypeError, - "an integer is required"); - return -1; - } - vv = (*nb->nb_int) (vv); - if (vv == NULL) + if (PyLong_Check(vv)) { + v = (PyLongObject *)vv; + } + else { + v = _PyLong_FromNbInt(vv); + if (v == NULL) return -1; do_decref = 1; - if (!PyLong_Check(vv)) { - Py_DECREF(vv); - PyErr_SetString(PyExc_TypeError, - "nb_int should return int object"); - return -1; - } } res = -1; - v = (PyLongObject *)vv; i = Py_SIZE(v); switch (i) { @@ -1408,7 +1406,7 @@ } exit: if (do_decref) { - Py_DECREF(vv); + Py_DECREF(v); } return res; } diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -402,13 +402,15 @@ Python - Python Python + + Include + @@ -915,7 +917,9 @@ PC - + + Objects + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 13:50:47 2013 From: python-checkins at python.org (charles-francois.natali) Date: Mon, 16 Dec 2013 13:50:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319965=3A_Make_sure_that_Python-ast=2Eh_is_prope?= =?utf-8?q?rly_taken_into_account_in_the?= Message-ID: <3djj5M2XwQz7Ljw@mail.python.org> http://hg.python.org/cpython/rev/ad42ea70668e changeset: 87980:ad42ea70668e parent: 87963:7d897144c39c parent: 87979:06122a4ecd37 user: Charles-Fran?ois Natali date: Sun Dec 15 19:12:52 2013 +0100 summary: Issue #19965: Make sure that Python-ast.h is properly taken into account in the makefile. files: Makefile.pre.in | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -910,7 +910,8 @@ $(srcdir)/Include/warnings.h \ $(srcdir)/Include/weakrefobject.h \ pyconfig.h \ - $(PARSER_HEADERS) + $(PARSER_HEADERS) \ + $(AST_H) $(LIBRARY_OBJS) $(MODOBJS) Modules/python.o: $(PYTHON_HEADERS) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 13:50:48 2013 From: python-checkins at python.org (charles-francois.natali) Date: Mon, 16 Dec 2013 13:50:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuMyk6?= =?utf-8?q?_Merge=2E?= Message-ID: <3djj5N4ZQNz7Lml@mail.python.org> http://hg.python.org/cpython/rev/6f19eda86277 changeset: 87981:6f19eda86277 branch: 3.3 parent: 87979:06122a4ecd37 parent: 87973:6afad4f29249 user: Charles-Fran?ois Natali date: Mon Dec 16 12:02:42 2013 +0100 summary: Merge. files: Lib/compileall.py | 3 +- Lib/test/test_compileall.py | 23 +++++++++++++++ Misc/NEWS | 3 + Modules/_decimal/libmpdec/mpdecimal.c | 11 +++--- 4 files changed, 34 insertions(+), 6 deletions(-) diff --git a/Lib/compileall.py b/Lib/compileall.py --- a/Lib/compileall.py +++ b/Lib/compileall.py @@ -228,7 +228,8 @@ success = False return success else: - return compile_path(legacy=args.legacy) + return compile_path(legacy=args.legacy, force=args.force, + quiet=args.quiet) except KeyboardInterrupt: print("\n[interrupted]") return False diff --git a/Lib/test/test_compileall.py b/Lib/test/test_compileall.py --- a/Lib/test/test_compileall.py +++ b/Lib/test/test_compileall.py @@ -177,6 +177,29 @@ self.assertNotCompiled(self.initfn) self.assertNotCompiled(self.barfn) + def test_no_args_respects_force_flag(self): + bazfn = script_helper.make_script(self.directory, 'baz', '') + self.assertRunOK(PYTHONPATH=self.directory) + pycpath = imp.cache_from_source(bazfn) + # Set atime/mtime backward to avoid file timestamp resolution issues + os.utime(pycpath, (time.time()-60,)*2) + mtime = os.stat(pycpath).st_mtime + # Without force, no recompilation + self.assertRunOK(PYTHONPATH=self.directory) + mtime2 = os.stat(pycpath).st_mtime + self.assertEqual(mtime, mtime2) + # Now force it. + self.assertRunOK('-f', PYTHONPATH=self.directory) + mtime2 = os.stat(pycpath).st_mtime + self.assertNotEqual(mtime, mtime2) + + def test_no_args_respects_quiet_flag(self): + script_helper.make_script(self.directory, 'baz', '') + noisy = self.assertRunOK(PYTHONPATH=self.directory) + self.assertIn(b'Listing ', noisy) + quiet = self.assertRunOK('-q', PYTHONPATH=self.directory) + self.assertNotIn(b'Listing ', quiet) + # Ensure that the default behavior of compileall's CLI is to create # PEP 3147 pyc/pyo files. for name, ext, switch in [ diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,9 @@ Library ------- +- Issue #19532: python -m compileall with no filename/directory arguments now + respects the -f and -q flags instead of ignoring them. + - Issue #19623: Fixed writing to unseekable files in the aifc module. - Issue #17919: select.poll.register() again works with poll.POLLNVAL on AIX. diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -4421,21 +4421,22 @@ const mpd_context_t *ctx, uint32_t *status) { uint32_t workstatus = 0; - const mpd_t *cc = c; + mpd_t *cc = NULL; if (result == c) { if ((cc = mpd_qncopy(c)) == NULL) { mpd_seterror(result, MPD_Malloc_error, status); return; } + c = cc; } _mpd_qmul(result, a, b, ctx, &workstatus); if (!(workstatus&MPD_Invalid_operation)) { - mpd_qadd(result, result, cc, ctx, &workstatus); - } - - if (cc != c) mpd_del((mpd_t *)cc); + mpd_qadd(result, result, c, ctx, &workstatus); + } + + if (cc) mpd_del(cc); *status |= workstatus; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 13:50:49 2013 From: python-checkins at python.org (charles-francois.natali) Date: Mon, 16 Dec 2013 13:50:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?b?KTogTWVyZ2Uu?= Message-ID: <3djj5P6GK3z7LmX@mail.python.org> http://hg.python.org/cpython/rev/fff1854f4423 changeset: 87982:fff1854f4423 parent: 87974:0e07ab605e0b parent: 87980:ad42ea70668e user: Charles-Fran?ois Natali date: Mon Dec 16 12:03:05 2013 +0100 summary: Merge. files: Makefile.pre.in | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -910,7 +910,8 @@ $(srcdir)/Include/warnings.h \ $(srcdir)/Include/weakrefobject.h \ pyconfig.h \ - $(PARSER_HEADERS) + $(PARSER_HEADERS) \ + $(AST_H) $(LIBRARY_OBJS) $(MODOBJS) Modules/python.o: $(PYTHON_HEADERS) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 13:50:51 2013 From: python-checkins at python.org (charles-francois.natali) Date: Mon, 16 Dec 2013 13:50:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuMyk6?= =?utf-8?q?_Merge=2E?= Message-ID: <3djj5R0zkbz7Ln2@mail.python.org> http://hg.python.org/cpython/rev/e696c1057c98 changeset: 87983:e696c1057c98 branch: 3.3 parent: 87981:6f19eda86277 parent: 87975:c42647d76bd1 user: Charles-Fran?ois Natali date: Mon Dec 16 13:49:19 2013 +0100 summary: Merge. files: Lib/test/test_devpoll.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_devpoll.py b/Lib/test/test_devpoll.py --- a/Lib/test/test_devpoll.py +++ b/Lib/test/test_devpoll.py @@ -4,6 +4,7 @@ import os, select, random, unittest, sys from test.support import TESTFN, run_unittest +from _testcapi import USHRT_MAX try: select.devpoll -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 13:50:52 2013 From: python-checkins at python.org (charles-francois.natali) Date: Mon, 16 Dec 2013 13:50:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogTWVyZ2Uu?= Message-ID: <3djj5S2jsCz7LkY@mail.python.org> http://hg.python.org/cpython/rev/0943778ee188 changeset: 87984:0943778ee188 parent: 87982:fff1854f4423 parent: 87975:c42647d76bd1 user: Charles-Fran?ois Natali date: Mon Dec 16 13:49:30 2013 +0100 summary: Merge. files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 13:50:53 2013 From: python-checkins at python.org (charles-francois.natali) Date: Mon, 16 Dec 2013 13:50:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogTWVyZ2Uu?= Message-ID: <3djj5T4Rhbz7LmX@mail.python.org> http://hg.python.org/cpython/rev/48218c3ea952 changeset: 87985:48218c3ea952 parent: 87984:0943778ee188 parent: 87983:e696c1057c98 user: Charles-Fran?ois Natali date: Mon Dec 16 13:49:38 2013 +0100 summary: Merge. files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 13:50:54 2013 From: python-checkins at python.org (charles-francois.natali) Date: Mon, 16 Dec 2013 13:50:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?b?KTogTWVyZS4=?= Message-ID: <3djj5V66ynz7LmN@mail.python.org> http://hg.python.org/cpython/rev/c9daa98d4733 changeset: 87986:c9daa98d4733 parent: 87985:48218c3ea952 parent: 87976:1f3f4147c35e user: Charles-Fran?ois Natali date: Mon Dec 16 13:49:56 2013 +0100 summary: Mere. files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 14:22:50 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 16 Dec 2013 14:22:50 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5OTEx?= =?utf-8?q?=3A_ntpath=2Esplitdrive=28=29_now_correctly_processes_the_=27?= =?utf-8?q?=C4=B0=27_character?= Message-ID: <3djjpL1xnbz7LnP@mail.python.org> http://hg.python.org/cpython/rev/7b0d083082ea changeset: 87987:7b0d083082ea branch: 3.3 parent: 87973:6afad4f29249 user: Serhiy Storchaka date: Mon Dec 16 14:34:55 2013 +0200 summary: Issue #19911: ntpath.splitdrive() now correctly processes the '?' character (U+0130, LATIN CAPITAL LETTER I WITH DOT ABOVE). files: Lib/ntpath.py | 2 +- Lib/test/test_ntpath.py | 3 +++ Misc/NEWS | 3 +++ 3 files changed, 7 insertions(+), 1 deletions(-) diff --git a/Lib/ntpath.py b/Lib/ntpath.py --- a/Lib/ntpath.py +++ b/Lib/ntpath.py @@ -207,7 +207,7 @@ empty = _get_empty(p) if len(p) > 1: sep = _get_sep(p) - normp = normcase(p) + normp = p.replace(_get_altsep(p), sep) if (normp[0:2] == sep*2) and (normp[2:3] != sep): # is a UNC path: # vvvvvvvvvvvvvvvvvvvv drive letter or UNC path diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py --- a/Lib/test/test_ntpath.py +++ b/Lib/test/test_ntpath.py @@ -66,6 +66,9 @@ ('', '\\\\conky\\\\mountpoint\\foo\\bar')) tester('ntpath.splitdrive("//conky//mountpoint/foo/bar")', ('', '//conky//mountpoint/foo/bar')) + # Issue #19911: UNC part containing U+0130 + self.assertEqual(ntpath.splitdrive('//conky/MOUNTPO?NT/foo/bar'), + ('//conky/MOUNTPO?NT', '/foo/bar')) def test_split(self): tester('ntpath.split("c:\\foo\\bar")', ('c:\\foo', 'bar')) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,9 @@ Library ------- +- Issue #19911: ntpath.splitdrive() now correctly processes the '?' character + (U+0130, LATIN CAPITAL LETTER I WITH DOT ABOVE). + - Issue #19532: python -m compileall with no filename/directory arguments now respects the -f and -q flags instead of ignoring them. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 14:22:51 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 16 Dec 2013 14:22:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319911=3A_ntpath=2Esplitdrive=28=29_now_correctl?= =?utf-8?q?y_processes_the_=27=C4=B0=27_character?= Message-ID: <3djjpM3xDpz7Lmp@mail.python.org> http://hg.python.org/cpython/rev/63d769dfa4ef changeset: 87988:63d769dfa4ef parent: 87974:0e07ab605e0b parent: 87987:7b0d083082ea user: Serhiy Storchaka date: Mon Dec 16 14:36:10 2013 +0200 summary: Issue #19911: ntpath.splitdrive() now correctly processes the '?' character (U+0130, LATIN CAPITAL LETTER I WITH DOT ABOVE). files: Lib/ntpath.py | 2 +- Lib/test/test_ntpath.py | 3 +++ Misc/NEWS | 3 +++ 3 files changed, 7 insertions(+), 1 deletions(-) diff --git a/Lib/ntpath.py b/Lib/ntpath.py --- a/Lib/ntpath.py +++ b/Lib/ntpath.py @@ -204,7 +204,7 @@ empty = _get_empty(p) if len(p) > 1: sep = _get_sep(p) - normp = normcase(p) + normp = p.replace(_get_altsep(p), sep) if (normp[0:2] == sep*2) and (normp[2:3] != sep): # is a UNC path: # vvvvvvvvvvvvvvvvvvvv drive letter or UNC path diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py --- a/Lib/test/test_ntpath.py +++ b/Lib/test/test_ntpath.py @@ -66,6 +66,9 @@ ('', '\\\\conky\\\\mountpoint\\foo\\bar')) tester('ntpath.splitdrive("//conky//mountpoint/foo/bar")', ('', '//conky//mountpoint/foo/bar')) + # Issue #19911: UNC part containing U+0130 + self.assertEqual(ntpath.splitdrive('//conky/MOUNTPO?NT/foo/bar'), + ('//conky/MOUNTPO?NT', '/foo/bar')) def test_split(self): tester('ntpath.split("c:\\foo\\bar")', ('c:\\foo', 'bar')) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #19911: ntpath.splitdrive() now correctly processes the '?' character + (U+0130, LATIN CAPITAL LETTER I WITH DOT ABOVE). + - Issue #19532: python -m compileall with no filename/directory arguments now respects the -f and -q flags instead of ignoring them. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 14:22:52 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 16 Dec 2013 14:22:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5OTEy?= =?utf-8?q?=3A_Fixed_numerous_bugs_in_ntpath=2Esplitunc=28=29=2E?= Message-ID: <3djjpN703dz7Lmv@mail.python.org> http://hg.python.org/cpython/rev/129105f8457d changeset: 87989:129105f8457d branch: 3.3 parent: 87987:7b0d083082ea user: Serhiy Storchaka date: Mon Dec 16 15:13:28 2013 +0200 summary: Issue #19912: Fixed numerous bugs in ntpath.splitunc(). * splitunc() no more return illegal result for paths with redundant slashes. * splitunc() now correctly processes the '?' character (U+0130, LATIN CAPITAL LETTER I WITH DOT ABOVE). * Deprecation warnings now emitted for every use of splitunc(). * Added tests for splitunc(). files: Lib/ntpath.py | 26 ++++++-------------------- Lib/test/test_ntpath.py | 23 +++++++++++++++++++++++ Misc/NEWS | 4 ++++ 3 files changed, 33 insertions(+), 20 deletions(-) diff --git a/Lib/ntpath.py b/Lib/ntpath.py --- a/Lib/ntpath.py +++ b/Lib/ntpath.py @@ -243,26 +243,12 @@ """ import warnings warnings.warn("ntpath.splitunc is deprecated, use ntpath.splitdrive instead", - DeprecationWarning) - sep = _get_sep(p) - if not p[1:2]: - return p[:0], p # Drive letter present - firstTwo = p[0:2] - if normcase(firstTwo) == sep + sep: - # is a UNC path: - # vvvvvvvvvvvvvvvvvvvv equivalent to drive letter - # \\machine\mountpoint\directories... - # directory ^^^^^^^^^^^^^^^ - normp = normcase(p) - index = normp.find(sep, 2) - if index == -1: - ##raise RuntimeError, 'illegal UNC path: "' + p + '"' - return (p[:0], p) - index = normp.find(sep, index + 1) - if index == -1: - index = len(p) - return p[:index], p[index:] - return p[:0], p + DeprecationWarning, 2) + drive, path = splitdrive(p) + if len(drive) == 2: + # Drive letter present + return p[:0], p + return drive, path # Split a path in head (everything up to the last '/') and tail (the diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py --- a/Lib/test/test_ntpath.py +++ b/Lib/test/test_ntpath.py @@ -70,6 +70,29 @@ self.assertEqual(ntpath.splitdrive('//conky/MOUNTPO?NT/foo/bar'), ('//conky/MOUNTPO?NT', '/foo/bar')) + def test_splitunc(self): + with self.assertWarns(DeprecationWarning): + ntpath.splitunc('') + with support.check_warnings(('', DeprecationWarning)): + tester('ntpath.splitunc("c:\\foo\\bar")', + ('', 'c:\\foo\\bar')) + tester('ntpath.splitunc("c:/foo/bar")', + ('', 'c:/foo/bar')) + tester('ntpath.splitunc("\\\\conky\\mountpoint\\foo\\bar")', + ('\\\\conky\\mountpoint', '\\foo\\bar')) + tester('ntpath.splitunc("//conky/mountpoint/foo/bar")', + ('//conky/mountpoint', '/foo/bar')) + tester('ntpath.splitunc("\\\\\\conky\\mountpoint\\foo\\bar")', + ('', '\\\\\\conky\\mountpoint\\foo\\bar')) + tester('ntpath.splitunc("///conky/mountpoint/foo/bar")', + ('', '///conky/mountpoint/foo/bar')) + tester('ntpath.splitunc("\\\\conky\\\\mountpoint\\foo\\bar")', + ('', '\\\\conky\\\\mountpoint\\foo\\bar')) + tester('ntpath.splitunc("//conky//mountpoint/foo/bar")', + ('', '//conky//mountpoint/foo/bar')) + self.assertEqual(ntpath.splitunc('//conky/MOUNTPO?NT/foo/bar'), + ('//conky/MOUNTPO?NT', '/foo/bar')) + def test_split(self): tester('ntpath.split("c:\\foo\\bar")', ('c:\\foo', 'bar')) tester('ntpath.split("\\\\conky\\mountpoint\\foo\\bar")', diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,8 @@ Library ------- +- Issue #19912: Fixed numerous bugs in ntpath.splitunc(). + - Issue #19911: ntpath.splitdrive() now correctly processes the '?' character (U+0130, LATIN CAPITAL LETTER I WITH DOT ABOVE). @@ -147,6 +149,8 @@ Tests ----- +- Issue #19912: Added tests for ntpath.splitunc(). + - Issue #19828: Fixed test_site when the whole suite is run with -S. - Issue #19928: Implemented a test for repr() of cell objects. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 14:22:54 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 16 Dec 2013 14:22:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319912=3A_Fixed_numerous_bugs_in_ntpath=2Esplitu?= =?utf-8?b?bmMoKS4=?= Message-ID: <3djjpQ2s1vz7Lmv@mail.python.org> http://hg.python.org/cpython/rev/5e39c69bad21 changeset: 87990:5e39c69bad21 parent: 87988:63d769dfa4ef parent: 87989:129105f8457d user: Serhiy Storchaka date: Mon Dec 16 15:14:19 2013 +0200 summary: Issue #19912: Fixed numerous bugs in ntpath.splitunc(). * splitunc() no more return illegal result for paths with redundant slashes. * splitunc() now correctly processes the '?' character (U+0130, LATIN CAPITAL LETTER I WITH DOT ABOVE). * Deprecation warnings now emitted for every use of splitunc(). * Added tests for splitunc(). files: Lib/ntpath.py | 26 ++++++-------------------- Lib/test/test_ntpath.py | 23 +++++++++++++++++++++++ Misc/NEWS | 4 ++++ 3 files changed, 33 insertions(+), 20 deletions(-) diff --git a/Lib/ntpath.py b/Lib/ntpath.py --- a/Lib/ntpath.py +++ b/Lib/ntpath.py @@ -240,26 +240,12 @@ """ import warnings warnings.warn("ntpath.splitunc is deprecated, use ntpath.splitdrive instead", - DeprecationWarning) - sep = _get_sep(p) - if not p[1:2]: - return p[:0], p # Drive letter present - firstTwo = p[0:2] - if normcase(firstTwo) == sep + sep: - # is a UNC path: - # vvvvvvvvvvvvvvvvvvvv equivalent to drive letter - # \\machine\mountpoint\directories... - # directory ^^^^^^^^^^^^^^^ - normp = normcase(p) - index = normp.find(sep, 2) - if index == -1: - ##raise RuntimeError, 'illegal UNC path: "' + p + '"' - return (p[:0], p) - index = normp.find(sep, index + 1) - if index == -1: - index = len(p) - return p[:index], p[index:] - return p[:0], p + DeprecationWarning, 2) + drive, path = splitdrive(p) + if len(drive) == 2: + # Drive letter present + return p[:0], p + return drive, path # Split a path in head (everything up to the last '/') and tail (the diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py --- a/Lib/test/test_ntpath.py +++ b/Lib/test/test_ntpath.py @@ -70,6 +70,29 @@ self.assertEqual(ntpath.splitdrive('//conky/MOUNTPO?NT/foo/bar'), ('//conky/MOUNTPO?NT', '/foo/bar')) + def test_splitunc(self): + with self.assertWarns(DeprecationWarning): + ntpath.splitunc('') + with support.check_warnings(('', DeprecationWarning)): + tester('ntpath.splitunc("c:\\foo\\bar")', + ('', 'c:\\foo\\bar')) + tester('ntpath.splitunc("c:/foo/bar")', + ('', 'c:/foo/bar')) + tester('ntpath.splitunc("\\\\conky\\mountpoint\\foo\\bar")', + ('\\\\conky\\mountpoint', '\\foo\\bar')) + tester('ntpath.splitunc("//conky/mountpoint/foo/bar")', + ('//conky/mountpoint', '/foo/bar')) + tester('ntpath.splitunc("\\\\\\conky\\mountpoint\\foo\\bar")', + ('', '\\\\\\conky\\mountpoint\\foo\\bar')) + tester('ntpath.splitunc("///conky/mountpoint/foo/bar")', + ('', '///conky/mountpoint/foo/bar')) + tester('ntpath.splitunc("\\\\conky\\\\mountpoint\\foo\\bar")', + ('', '\\\\conky\\\\mountpoint\\foo\\bar')) + tester('ntpath.splitunc("//conky//mountpoint/foo/bar")', + ('', '//conky//mountpoint/foo/bar')) + self.assertEqual(ntpath.splitunc('//conky/MOUNTPO?NT/foo/bar'), + ('//conky/MOUNTPO?NT', '/foo/bar')) + def test_split(self): tester('ntpath.split("c:\\foo\\bar")', ('c:\\foo', 'bar')) tester('ntpath.split("\\\\conky\\mountpoint\\foo\\bar")', diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,8 @@ Library ------- +- Issue #19912: Fixed numerous bugs in ntpath.splitunc(). + - Issue #19911: ntpath.splitdrive() now correctly processes the '?' character (U+0130, LATIN CAPITAL LETTER I WITH DOT ABOVE). @@ -183,6 +185,8 @@ Tests ----- +- Issue #19912: Added tests for ntpath.splitunc(). + - Issue #19828: Fixed test_site when the whole suite is run with -S. - Issue #19928: Implemented a test for repr() of cell objects. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 14:22:55 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 16 Dec 2013 14:22:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5OTEy?= =?utf-8?q?=3A_Fixed_numerous_bugs_in_ntpath=2Esplitunc=28=29=2E?= Message-ID: <3djjpR644Zz7Lkf@mail.python.org> http://hg.python.org/cpython/rev/e4beb183a674 changeset: 87991:e4beb183a674 branch: 2.7 parent: 87956:c10ea224392d user: Serhiy Storchaka date: Mon Dec 16 15:15:29 2013 +0200 summary: Issue #19912: Fixed numerous bugs in ntpath.splitunc(). * splitunc() no more returns illegal result for paths with redundant slashes. * splitunc() now correctly processes the u'?' character (U+0130, LATIN CAPITAL LETTER I WITH DOT ABOVE). * Added new tests for splitunc(). files: Lib/ntpath.py | 40 ++++++++++++++++++++++------ Lib/test/test_ntpath.py | 14 ++++++++++ Misc/NEWS | 2 + 3 files changed, 47 insertions(+), 9 deletions(-) diff --git a/Lib/ntpath.py b/Lib/ntpath.py --- a/Lib/ntpath.py +++ b/Lib/ntpath.py @@ -136,6 +136,25 @@ using backslashes). unc+rest is always the input path. Paths containing drive letters never have an UNC part. """ + #if p[1:2] == ':': + #return '', p # Drive letter present + #firstTwo = p[0:2] + #if firstTwo == '//' or firstTwo == '\\\\': + ## is a UNC path: + ## vvvvvvvvvvvvvvvvvvvv equivalent to drive letter + ## \\machine\mountpoint\directories... + ## directory ^^^^^^^^^^^^^^^ + #normp = normcase(p) + #index = normp.find('\\', 2) + #if index == -1: + ###raise RuntimeError, 'illegal UNC path: "' + p + '"' + #return ("", p) + #index = normp.find('\\', index + 1) + #if index == -1: + #index = len(p) + #return p[:index], p[index:] + #return '', p + if p[1:2] == ':': return '', p # Drive letter present firstTwo = p[0:2] @@ -144,15 +163,18 @@ # vvvvvvvvvvvvvvvvvvvv equivalent to drive letter # \\machine\mountpoint\directories... # directory ^^^^^^^^^^^^^^^ - normp = normcase(p) - index = normp.find('\\', 2) - if index == -1: - ##raise RuntimeError, 'illegal UNC path: "' + p + '"' - return ("", p) - index = normp.find('\\', index + 1) - if index == -1: - index = len(p) - return p[:index], p[index:] + normp = p.replace('\\', '/') + index = normp.find('/', 2) + if index <= 2: + return '', p + index2 = normp.find('/', index + 1) + # a UNC path can't have two slashes in a row + # (after the initial two) + if index2 == index + 1: + return '', p + if index2 == -1: + index2 = len(p) + return p[:index2], p[index2:] return '', p diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py --- a/Lib/test/test_ntpath.py +++ b/Lib/test/test_ntpath.py @@ -33,10 +33,24 @@ ('c:', '/foo/bar')) def test_splitunc(self): + tester('ntpath.splitunc("c:\\foo\\bar")', + ('', 'c:\\foo\\bar')) + tester('ntpath.splitunc("c:/foo/bar")', + ('', 'c:/foo/bar')) tester('ntpath.splitunc("\\\\conky\\mountpoint\\foo\\bar")', ('\\\\conky\\mountpoint', '\\foo\\bar')) tester('ntpath.splitunc("//conky/mountpoint/foo/bar")', ('//conky/mountpoint', '/foo/bar')) + tester('ntpath.splitunc("\\\\\\conky\\mountpoint\\foo\\bar")', + ('', '\\\\\\conky\\mountpoint\\foo\\bar')) + tester('ntpath.splitunc("///conky/mountpoint/foo/bar")', + ('', '///conky/mountpoint/foo/bar')) + tester('ntpath.splitunc("\\\\conky\\\\mountpoint\\foo\\bar")', + ('', '\\\\conky\\\\mountpoint\\foo\\bar')) + tester('ntpath.splitunc("//conky//mountpoint/foo/bar")', + ('', '//conky//mountpoint/foo/bar')) + self.assertEqual(ntpath.splitunc(u'//conky/MOUNTPO\u0130NT/foo/bar'), + (u'//conky/MOUNTPO\u0130NT', u'/foo/bar')) def test_split(self): tester('ntpath.split("c:\\foo\\bar")', ('c:\\foo', 'bar')) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,8 @@ Library ------- +- Issue #19912: Fixed numerous bugs in ntpath.splitunc(). + - Issue #19623: Fixed writing to unseekable files in the aifc module. Fixed writing 'ulaw' (lower case) compressed AIFC files. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 14:22:57 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 16 Dec 2013 14:22:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf-8?q?_Merge_heads?= Message-ID: <3djjpT0cd4z7LlR@mail.python.org> http://hg.python.org/cpython/rev/80013e1d16ee changeset: 87992:80013e1d16ee branch: 2.7 parent: 87991:e4beb183a674 parent: 87977:874813a3523d user: Serhiy Storchaka date: Mon Dec 16 15:16:11 2013 +0200 summary: Merge heads files: Makefile.pre.in | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -750,7 +750,8 @@ Include/warnings.h \ Include/weakrefobject.h \ pyconfig.h \ - $(PARSER_HEADERS) + $(PARSER_HEADERS) \ + $(AST_H) $(LIBRARY_OBJS) $(MODOBJS) Modules/python.o: $(PYTHON_HEADERS) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 14:22:58 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 16 Dec 2013 14:22:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuMyk6?= =?utf-8?q?_Merge_heads?= Message-ID: <3djjpV2K2mzRF7@mail.python.org> http://hg.python.org/cpython/rev/46c5adfeba9d changeset: 87993:46c5adfeba9d branch: 3.3 parent: 87989:129105f8457d parent: 87983:e696c1057c98 user: Serhiy Storchaka date: Mon Dec 16 15:16:35 2013 +0200 summary: Merge heads files: Lib/test/test_devpoll.py | 1 + Makefile.pre.in | 3 ++- 2 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_devpoll.py b/Lib/test/test_devpoll.py --- a/Lib/test/test_devpoll.py +++ b/Lib/test/test_devpoll.py @@ -4,6 +4,7 @@ import os, select, random, unittest, sys from test.support import TESTFN, run_unittest +from _testcapi import USHRT_MAX try: select.devpoll diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -848,7 +848,8 @@ $(srcdir)/Include/warnings.h \ $(srcdir)/Include/weakrefobject.h \ pyconfig.h \ - $(PARSER_HEADERS) + $(PARSER_HEADERS) \ + $(AST_H) $(LIBRARY_OBJS) $(MODOBJS) Modules/python.o: $(PYTHON_HEADERS) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 14:22:59 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 16 Dec 2013 14:22:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?q?=29=3A_Merge_heads?= Message-ID: <3djjpW44syzRF7@mail.python.org> http://hg.python.org/cpython/rev/8c0635156267 changeset: 87994:8c0635156267 parent: 87990:5e39c69bad21 parent: 87986:c9daa98d4733 user: Serhiy Storchaka date: Mon Dec 16 15:17:13 2013 +0200 summary: Merge heads files: Makefile.pre.in | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -910,7 +910,8 @@ $(srcdir)/Include/warnings.h \ $(srcdir)/Include/weakrefobject.h \ pyconfig.h \ - $(PARSER_HEADERS) + $(PARSER_HEADERS) \ + $(AST_H) $(LIBRARY_OBJS) $(MODOBJS) Modules/python.o: $(PYTHON_HEADERS) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 14:23:00 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 16 Dec 2013 14:23:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Null_merge?= Message-ID: <3djjpX5gxNz7LlR@mail.python.org> http://hg.python.org/cpython/rev/d6b7c7d07e7c changeset: 87995:d6b7c7d07e7c parent: 87994:8c0635156267 parent: 87993:46c5adfeba9d user: Serhiy Storchaka date: Mon Dec 16 15:17:33 2013 +0200 summary: Null merge files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 14:36:09 2013 From: python-checkins at python.org (christian.heimes) Date: Mon, 16 Dec 2013 14:36:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2318215=3A_Add_scri?= =?utf-8?q?pt_Tools/ssl/test=5Fmultiple=5Fversions=2Epy_to_compile_and?= Message-ID: <3djk5j09Tpz7Ljf@mail.python.org> http://hg.python.org/cpython/rev/7719efb182e3 changeset: 87996:7719efb182e3 parent: 87976:1f3f4147c35e user: Christian Heimes date: Mon Dec 16 14:35:39 2013 +0100 summary: Issue #18215: Add script Tools/ssl/test_multiple_versions.py to compile and run Python's unit tests with multiple versions of OpenSSL. files: Misc/NEWS | 3 + Tools/ssl/test_multiple_versions.py | 241 ++++++++++++++++ 2 files changed, 244 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -200,6 +200,9 @@ Build ----- +- Issue #18215: Add script Tools/ssl/test_multiple_versions.py to compile and + run Python's unit tests with multiple versions of OpenSSL. + - Issue #19922: define _INCLUDE__STDC_A1_SOURCE in HP-UX to include mbstate_t for mbrtowc(). diff --git a/Tools/ssl/test_multiple_versions.py b/Tools/ssl/test_multiple_versions.py new file mode 100644 --- /dev/null +++ b/Tools/ssl/test_multiple_versions.py @@ -0,0 +1,241 @@ +#./python +"""Run Python tests with multiple installations of OpenSSL + +The script + + (1) downloads OpenSSL tar bundle + (2) extracts it to ../openssl/src/openssl-VERSION/ + (3) compiles OpenSSL + (4) installs OpenSSL into ../openssl/VERSION/ + (5) forces a recompilation of Python modules using the + header and library files from ../openssl/VERSION/ + (6) runs Python's test suite + +The script must be run with Python's build directory as current working +directory: + + ./python Tools/ssl/test_multiple_versions.py + +The script uses LD_RUN_PATH, LD_LIBRARY_PATH, CPPFLAGS and LDFLAGS to bend +search paths for header files and shared libraries. It's known to work on +Linux with GCC 4.x. + +(c) 2013 Christian Heimes +""" +import logging +import os +import tarfile +import shutil +import subprocess +import sys +from urllib.request import urlopen + +log = logging.getLogger("multissl") + +OPENSSL_VERSIONS = [ + "0.9.7m", "0.9.8i", "0.9.8l", "0.9.8m", "0.9.8y", "1.0.0k", "1.0.1e" +] +FULL_TESTS = [ + "test_asyncio", "test_ftplib", "test_hashlib", "test_httplib", + "test_imaplib", "test_nntplib", "test_poplib", "test_smtplib", + "test_smtpnet", "test_urllib2_localnet", "test_venv" +] +MINIMAL_TESTS = ["test_ssl", "test_hashlib"] +CADEFAULT = True +HERE = os.path.abspath(os.getcwd()) +DEST_DIR = os.path.abspath(os.path.join(HERE, os.pardir, "openssl")) + + +class BuildSSL: + url_template = "https://www.openssl.org/source/openssl-{}.tar.gz" + + module_files = ["Modules/_ssl.c", + "Modules/socketmodule.c", + "Modules/_hashopenssl.c"] + + def __init__(self, version, openssl_compile_args=(), destdir=DEST_DIR): + self._check_python_builddir() + self.version = version + self.openssl_compile_args = openssl_compile_args + # installation directory + self.install_dir = os.path.join(destdir, version) + # source file + self.src_file = os.path.join(destdir, "src", + "openssl-{}.tar.gz".format(version)) + # build directory (removed after install) + self.build_dir = os.path.join(destdir, "src", + "openssl-{}".format(version)) + + @property + def openssl_cli(self): + """openssl CLI binary""" + return os.path.join(self.install_dir, "bin", "openssl") + + @property + def openssl_version(self): + """output of 'bin/openssl version'""" + env = os.environ.copy() + env["LD_LIBRARY_PATH"] = self.lib_dir + cmd = [self.openssl_cli, "version"] + return self._subprocess_output(cmd, env=env) + + @property + def pyssl_version(self): + """Value of ssl.OPENSSL_VERSION""" + env = os.environ.copy() + env["LD_LIBRARY_PATH"] = self.lib_dir + cmd = ["./python", "-c", "import ssl; print(ssl.OPENSSL_VERSION)"] + return self._subprocess_output(cmd, env=env) + + @property + def include_dir(self): + return os.path.join(self.install_dir, "include") + + @property + def lib_dir(self): + return os.path.join(self.install_dir, "lib") + + @property + def has_openssl(self): + return os.path.isfile(self.openssl_cli) + + @property + def has_src(self): + return os.path.isfile(self.src_file) + + def _subprocess_call(self, cmd, stdout=subprocess.DEVNULL, env=None, + **kwargs): + log.debug("Call '{}'".format(" ".join(cmd))) + return subprocess.check_call(cmd, stdout=stdout, env=env, **kwargs) + + def _subprocess_output(self, cmd, env=None, **kwargs): + log.debug("Call '{}'".format(" ".join(cmd))) + out = subprocess.check_output(cmd, env=env) + return out.strip().decode("utf-8") + + def _check_python_builddir(self): + if not os.path.isfile("python") or not os.path.isfile("setup.py"): + raise ValueError("Script must be run in Python build directory") + + def _download_openssl(self): + """Download OpenSSL source dist""" + src_dir = os.path.dirname(self.src_file) + if not os.path.isdir(src_dir): + os.makedirs(src_dir) + url = self.url_template.format(self.version) + log.info("Downloading OpenSSL from {}".format(url)) + req = urlopen(url, cadefault=CADEFAULT) + # KISS, read all, write all + data = req.read() + log.info("Storing {}".format(self.src_file)) + with open(self.src_file, "wb") as f: + f.write(data) + + def _unpack_openssl(self): + """Unpack tar.gz bundle""" + # cleanup + if os.path.isdir(self.build_dir): + shutil.rmtree(self.build_dir) + os.makedirs(self.build_dir) + + tf = tarfile.open(self.src_file) + base = "openssl-{}/".format(self.version) + # force extraction into build dir + members = tf.getmembers() + for member in members: + if not member.name.startswith(base): + raise ValueError(member.name) + member.name = member.name[len(base):] + log.info("Unpacking files to {}".format(self.build_dir)) + tf.extractall(self.build_dir, members) + + def _build_openssl(self): + """Now build openssl""" + log.info("Running build in {}".format(self.install_dir)) + cwd = self.build_dir + cmd = ["./config", "shared", "--prefix={}".format(self.install_dir)] + cmd.extend(self.openssl_compile_args) + self._subprocess_call(cmd, cwd=cwd) + self._subprocess_call(["make"], cwd=cwd) + + def _install_openssl(self, remove=True): + self._subprocess_call(["make", "install"], cwd=self.build_dir) + if remove: + shutil.rmtree(self.build_dir) + + def install_openssl(self): + if not self.has_openssl: + if not self.has_src: + self._download_openssl() + else: + log.debug("Already has src {}".format(self.src_file)) + self._unpack_openssl() + self._build_openssl() + self._install_openssl() + else: + log.info("Already has installation {}".format(self.install_dir)) + # validate installation + version = self.openssl_version + if self.version not in version: + raise ValueError(version) + + def touch_pymods(self): + # force a rebuild of all modules that use OpenSSL APIs + for fname in self.module_files: + os.utime(fname) + + def recompile_pymods(self): + log.info("Using OpenSSL build from {}".format(self.build_dir)) + # overwrite header and library search paths + env = os.environ.copy() + env["CPPFLAGS"] = "-I{}".format(self.include_dir) + env["LDFLAGS"] = "-L{}".format(self.lib_dir) + # set rpath + env["LD_RUN_PATH"] = self.lib_dir + + log.info("Rebuilding Python modules") + self.touch_pymods() + cmd = ["./python", "setup.py", "build"] + self._subprocess_call(cmd, env=env) + + def check_pyssl(self): + version = self.pyssl_version + if self.version not in version: + raise ValueError(version) + + def run_pytests(self, *args): + cmd = ["./python", "-m", "test"] + cmd.extend(args) + self._subprocess_call(cmd, stdout=None) + + def run_python_tests(self, *args): + self.recompile_pymods() + self.check_pyssl() + self.run_pytests(*args) + + +def main(*args): + builders = [] + for version in OPENSSL_VERSIONS: + if version in ("0.9.8i", "0.9.8l"): + openssl_compile_args = ("no-asm",) + else: + openssl_compile_args = () + builder = BuildSSL(version, openssl_compile_args) + builder.install_openssl() + builders.append(builder) + + for builder in builders: + builder.run_python_tests(*args) + # final touch + builder.touch_pymods() + + +if __name__ == "__main__": + logging.basicConfig(level=logging.INFO, + format="*** %(levelname)s %(message)s") + args = sys.argv[1:] + if not args: + args = ["-unetwork", "-v"] + args.extend(FULL_TESTS) + main(*args) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 14:36:10 2013 From: python-checkins at python.org (christian.heimes) Date: Mon, 16 Dec 2013 14:36:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?q?=29=3A_merge?= Message-ID: <3djk5k39DKz7LkM@mail.python.org> http://hg.python.org/cpython/rev/0971a3f0a146 changeset: 87997:0971a3f0a146 parent: 87996:7719efb182e3 parent: 87995:d6b7c7d07e7c user: Christian Heimes date: Mon Dec 16 14:35:58 2013 +0100 summary: merge files: Lib/ntpath.py | 28 +++++++--------------------- Lib/test/test_ntpath.py | 26 ++++++++++++++++++++++++++ Makefile.pre.in | 3 ++- Misc/NEWS | 7 +++++++ 4 files changed, 42 insertions(+), 22 deletions(-) diff --git a/Lib/ntpath.py b/Lib/ntpath.py --- a/Lib/ntpath.py +++ b/Lib/ntpath.py @@ -204,7 +204,7 @@ empty = _get_empty(p) if len(p) > 1: sep = _get_sep(p) - normp = normcase(p) + normp = p.replace(_get_altsep(p), sep) if (normp[0:2] == sep*2) and (normp[2:3] != sep): # is a UNC path: # vvvvvvvvvvvvvvvvvvvv drive letter or UNC path @@ -240,26 +240,12 @@ """ import warnings warnings.warn("ntpath.splitunc is deprecated, use ntpath.splitdrive instead", - DeprecationWarning) - sep = _get_sep(p) - if not p[1:2]: - return p[:0], p # Drive letter present - firstTwo = p[0:2] - if normcase(firstTwo) == sep + sep: - # is a UNC path: - # vvvvvvvvvvvvvvvvvvvv equivalent to drive letter - # \\machine\mountpoint\directories... - # directory ^^^^^^^^^^^^^^^ - normp = normcase(p) - index = normp.find(sep, 2) - if index == -1: - ##raise RuntimeError, 'illegal UNC path: "' + p + '"' - return (p[:0], p) - index = normp.find(sep, index + 1) - if index == -1: - index = len(p) - return p[:index], p[index:] - return p[:0], p + DeprecationWarning, 2) + drive, path = splitdrive(p) + if len(drive) == 2: + # Drive letter present + return p[:0], p + return drive, path # Split a path in head (everything up to the last '/') and tail (the diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py --- a/Lib/test/test_ntpath.py +++ b/Lib/test/test_ntpath.py @@ -66,6 +66,32 @@ ('', '\\\\conky\\\\mountpoint\\foo\\bar')) tester('ntpath.splitdrive("//conky//mountpoint/foo/bar")', ('', '//conky//mountpoint/foo/bar')) + # Issue #19911: UNC part containing U+0130 + self.assertEqual(ntpath.splitdrive('//conky/MOUNTPO?NT/foo/bar'), + ('//conky/MOUNTPO?NT', '/foo/bar')) + + def test_splitunc(self): + with self.assertWarns(DeprecationWarning): + ntpath.splitunc('') + with support.check_warnings(('', DeprecationWarning)): + tester('ntpath.splitunc("c:\\foo\\bar")', + ('', 'c:\\foo\\bar')) + tester('ntpath.splitunc("c:/foo/bar")', + ('', 'c:/foo/bar')) + tester('ntpath.splitunc("\\\\conky\\mountpoint\\foo\\bar")', + ('\\\\conky\\mountpoint', '\\foo\\bar')) + tester('ntpath.splitunc("//conky/mountpoint/foo/bar")', + ('//conky/mountpoint', '/foo/bar')) + tester('ntpath.splitunc("\\\\\\conky\\mountpoint\\foo\\bar")', + ('', '\\\\\\conky\\mountpoint\\foo\\bar')) + tester('ntpath.splitunc("///conky/mountpoint/foo/bar")', + ('', '///conky/mountpoint/foo/bar')) + tester('ntpath.splitunc("\\\\conky\\\\mountpoint\\foo\\bar")', + ('', '\\\\conky\\\\mountpoint\\foo\\bar')) + tester('ntpath.splitunc("//conky//mountpoint/foo/bar")', + ('', '//conky//mountpoint/foo/bar')) + self.assertEqual(ntpath.splitunc('//conky/MOUNTPO?NT/foo/bar'), + ('//conky/MOUNTPO?NT', '/foo/bar')) def test_split(self): tester('ntpath.split("c:\\foo\\bar")', ('c:\\foo', 'bar')) diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -910,7 +910,8 @@ $(srcdir)/Include/warnings.h \ $(srcdir)/Include/weakrefobject.h \ pyconfig.h \ - $(PARSER_HEADERS) + $(PARSER_HEADERS) \ + $(AST_H) $(LIBRARY_OBJS) $(MODOBJS) Modules/python.o: $(PYTHON_HEADERS) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,11 @@ Library ------- +- Issue #19912: Fixed numerous bugs in ntpath.splitunc(). + +- Issue #19911: ntpath.splitdrive() now correctly processes the '?' character + (U+0130, LATIN CAPITAL LETTER I WITH DOT ABOVE). + - Issue #19532: python -m compileall with no filename/directory arguments now respects the -f and -q flags instead of ignoring them. @@ -180,6 +185,8 @@ Tests ----- +- Issue #19912: Added tests for ntpath.splitunc(). + - Issue #19828: Fixed test_site when the whole suite is run with -S. - Issue #19928: Implemented a test for repr() of cell objects. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 14:54:34 2013 From: python-checkins at python.org (christian.heimes) Date: Mon, 16 Dec 2013 14:54:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319987=3A_disable_?= =?utf-8?q?test=5Fwinsound=27s_test=5Falias=5Ffallback_test_when_no_sound?= Message-ID: <3djkVy1d85z7Ljp@mail.python.org> http://hg.python.org/cpython/rev/1824fa874f08 changeset: 87998:1824fa874f08 user: Christian Heimes date: Mon Dec 16 14:54:24 2013 +0100 summary: Issue #19987: disable test_winsound's test_alias_fallback test when no sound card is available. The test fails on Windows Server 2008. files: Lib/test/test_winsound.py | 13 +++++++------ 1 files changed, 7 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_winsound.py b/Lib/test/test_winsound.py --- a/Lib/test/test_winsound.py +++ b/Lib/test/test_winsound.py @@ -160,12 +160,13 @@ def test_alias_fallback(self): if _have_soundcard(): winsound.PlaySound('!"$%&/(#+*', winsound.SND_ALIAS) - else: - self.assertRaises( - RuntimeError, - winsound.PlaySound, - '!"$%&/(#+*', winsound.SND_ALIAS - ) + # see http://bugs.python.org/issue19987 + #else: + # self.assertRaises( + # RuntimeError, + # winsound.PlaySound, + # '!"$%&/(#+*', winsound.SND_ALIAS + # ) def test_alias_nofallback(self): if _have_soundcard(): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 16:05:54 2013 From: python-checkins at python.org (zach.ware) Date: Mon, 16 Dec 2013 16:05:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5OTg3?= =?utf-8?q?=3A_Re-write_test=5Falias=5Ffallback_in_test=5Fwinsound_to_have?= =?utf-8?q?_two?= Message-ID: <3djm5G2L7tz7Lk5@mail.python.org> http://hg.python.org/cpython/rev/f590e9aeb990 changeset: 87999:f590e9aeb990 branch: 2.7 parent: 87992:80013e1d16ee user: Zachary Ware date: Mon Dec 16 08:58:10 2013 -0600 summary: Issue #19987: Re-write test_alias_fallback in test_winsound to have two acceptable outcomes: success or RuntimeError. Without being able to actually hear whether a sound was played, either one could be right, but any other error would be a failure. files: Lib/test/test_winsound.py | 15 ++++++++------- Misc/NEWS | 2 +- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/Lib/test/test_winsound.py b/Lib/test/test_winsound.py --- a/Lib/test/test_winsound.py +++ b/Lib/test/test_winsound.py @@ -159,14 +159,15 @@ ) def test_alias_fallback(self): - if _have_soundcard(): + # In the absense of the ability to tell if a sound was actually + # played, this test has two acceptable outcomes: success (no error, + # sound was theoretically played; although as issue #19987 shows + # a box without a soundcard can "succeed") or RuntimeError. Any + # other error is a failure. + try: winsound.PlaySound('!"$%&/(#+*', winsound.SND_ALIAS) - else: - self.assertRaises( - RuntimeError, - winsound.PlaySound, - '!"$%&/(#+*', winsound.SND_ALIAS - ) + except RuntimeError: + pass def test_alias_nofallback(self): if _have_soundcard(): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -101,7 +101,7 @@ - Issue #19928: Implemented a test for repr() of cell objects. -- Issue #19595: Re-enabled a long-disabled test in test_winsound. +- Issue #19595, #19987: Re-enabled a long-disabled test in test_winsound. - Issue #19588: Fixed tests in test_random that were silently skipped most of the time. Patch by Julian Gindi. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 16:05:55 2013 From: python-checkins at python.org (zach.ware) Date: Mon, 16 Dec 2013 16:05:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5OTg3?= =?utf-8?q?=3A_Re-write_test=5Falias=5Ffallback_in_test=5Fwinsound_to_have?= =?utf-8?q?_two?= Message-ID: <3djm5H47cPz7LkG@mail.python.org> http://hg.python.org/cpython/rev/5455456945d4 changeset: 88000:5455456945d4 branch: 3.3 parent: 87993:46c5adfeba9d user: Zachary Ware date: Mon Dec 16 09:02:41 2013 -0600 summary: Issue #19987: Re-write test_alias_fallback in test_winsound to have two acceptable outcomes: success or RuntimeError. Without being able to actually hear whether a sound was played, either one could be right, but any other error would be a failure. files: Lib/test/test_winsound.py | 15 ++++++++------- Misc/NEWS | 2 +- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/Lib/test/test_winsound.py b/Lib/test/test_winsound.py --- a/Lib/test/test_winsound.py +++ b/Lib/test/test_winsound.py @@ -158,14 +158,15 @@ ) def test_alias_fallback(self): - if _have_soundcard(): + # In the absense of the ability to tell if a sound was actually + # played, this test has two acceptable outcomes: success (no error, + # sound was theoretically played; although as issue #19987 shows + # a box without a soundcard can "succeed") or RuntimeError. Any + # other error is a failure. + try: winsound.PlaySound('!"$%&/(#+*', winsound.SND_ALIAS) - else: - self.assertRaises( - RuntimeError, - winsound.PlaySound, - '!"$%&/(#+*', winsound.SND_ALIAS - ) + except RuntimeError: + pass def test_alias_nofallback(self): if _have_soundcard(): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -160,7 +160,7 @@ - Issue #19926: Removed unneeded test_main from test_abstract_numbers. Patch by Vajrasky Kok. -- Issue #19595: Re-enabled a long-disabled test in test_winsound. +- Issue #19595, #19987: Re-enabled a long-disabled test in test_winsound. - Issue #19588: Fixed tests in test_random that were silently skipped most of the time. Patch by Julian Gindi. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 16:05:56 2013 From: python-checkins at python.org (zach.ware) Date: Mon, 16 Dec 2013 16:05:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319987=3A_Merge_with_3=2E3?= Message-ID: <3djm5J5wl8z7Lkf@mail.python.org> http://hg.python.org/cpython/rev/1aa6751b298f changeset: 88001:1aa6751b298f parent: 87998:1824fa874f08 parent: 88000:5455456945d4 user: Zachary Ware date: Mon Dec 16 09:05:44 2013 -0600 summary: Issue #19987: Merge with 3.3 files: Lib/test/test_winsound.py | 16 ++++++++-------- Misc/NEWS | 2 +- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/Lib/test/test_winsound.py b/Lib/test/test_winsound.py --- a/Lib/test/test_winsound.py +++ b/Lib/test/test_winsound.py @@ -158,15 +158,15 @@ ) def test_alias_fallback(self): - if _have_soundcard(): + # In the absense of the ability to tell if a sound was actually + # played, this test has two acceptable outcomes: success (no error, + # sound was theoretically played; although as issue #19987 shows + # a box without a soundcard can "succeed") or RuntimeError. Any + # other error is a failure. + try: winsound.PlaySound('!"$%&/(#+*', winsound.SND_ALIAS) - # see http://bugs.python.org/issue19987 - #else: - # self.assertRaises( - # RuntimeError, - # winsound.PlaySound, - # '!"$%&/(#+*', winsound.SND_ALIAS - # ) + except RuntimeError: + pass def test_alias_nofallback(self): if _have_soundcard(): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -199,7 +199,7 @@ - Issue #19572: More skipped tests explicitly marked as skipped. -- Issue #19595: Re-enabled a long-disabled test in test_winsound. +- Issue #19595, #19987: Re-enabled a long-disabled test in test_winsound. - Issue #19588: Fixed tests in test_random that were silently skipped most of the time. Patch by Julian Gindi. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 17:43:52 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 16 Dec 2013 17:43:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Removed_old_im?= =?utf-8?q?plementation_of_ntpath=2Esplitunc=28=29_=28issue_=2319912=29=2E?= Message-ID: <3djpGJ0Bhqz7LkJ@mail.python.org> http://hg.python.org/cpython/rev/4de09cbd3b97 changeset: 88002:4de09cbd3b97 branch: 2.7 parent: 87999:f590e9aeb990 user: Serhiy Storchaka date: Mon Dec 16 18:43:00 2013 +0200 summary: Removed old implementation of ntpath.splitunc() (issue #19912). files: Lib/ntpath.py | 19 ------------------- 1 files changed, 0 insertions(+), 19 deletions(-) diff --git a/Lib/ntpath.py b/Lib/ntpath.py --- a/Lib/ntpath.py +++ b/Lib/ntpath.py @@ -136,25 +136,6 @@ using backslashes). unc+rest is always the input path. Paths containing drive letters never have an UNC part. """ - #if p[1:2] == ':': - #return '', p # Drive letter present - #firstTwo = p[0:2] - #if firstTwo == '//' or firstTwo == '\\\\': - ## is a UNC path: - ## vvvvvvvvvvvvvvvvvvvv equivalent to drive letter - ## \\machine\mountpoint\directories... - ## directory ^^^^^^^^^^^^^^^ - #normp = normcase(p) - #index = normp.find('\\', 2) - #if index == -1: - ###raise RuntimeError, 'illegal UNC path: "' + p + '"' - #return ("", p) - #index = normp.find('\\', index + 1) - #if index == -1: - #index = len(p) - #return p[:index], p[index:] - #return '', p - if p[1:2] == ':': return '', p # Drive letter present firstTwo = p[0:2] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 19:57:50 2013 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 16 Dec 2013 19:57:50 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319887=3A_Improve_?= =?utf-8?q?the_Path=2Eresolve=28=29_algorithm_to_support_certain_symlink?= Message-ID: <3djsDt1pY1z7LjP@mail.python.org> http://hg.python.org/cpython/rev/12a52186b4fd changeset: 88003:12a52186b4fd parent: 88001:1aa6751b298f user: Antoine Pitrou date: Mon Dec 16 19:57:41 2013 +0100 summary: Issue #19887: Improve the Path.resolve() algorithm to support certain symlink chains. Original patch by Serhiy. files: Lib/pathlib.py | 75 ++++++++++++++------------- Lib/test/test_pathlib.py | 53 +++++++++++++++++++ Misc/NEWS | 3 + 3 files changed, 96 insertions(+), 35 deletions(-) diff --git a/Lib/pathlib.py b/Lib/pathlib.py --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -254,42 +254,47 @@ def resolve(self, path): sep = self.sep - def split(p): - return [x for x in p.split(sep) if x] - def absparts(p): - # Our own abspath(), since the posixpath one makes - # the mistake of "normalizing" the path without resolving the - # symlinks first. - if not p.startswith(sep): - return split(os.getcwd()) + split(p) - else: - return split(p) - parts = absparts(str(path))[::-1] accessor = path._accessor - resolved = cur = "" - symlinks = {} - while parts: - part = parts.pop() - cur = resolved + sep + part - if cur in symlinks and symlinks[cur] <= len(parts): - # We've already seen the symlink and there's not less - # work to do than the last time. - raise RuntimeError("Symlink loop from %r" % cur) - try: - target = accessor.readlink(cur) - except OSError as e: - if e.errno != EINVAL: - raise - # Not a symlink - resolved = cur - else: - # Take note of remaining work from this symlink - symlinks[cur] = len(parts) - if target.startswith(sep): - # Symlink points to absolute path - resolved = "" - parts.extend(split(target)[::-1]) - return resolved or sep + seen = {} + def _resolve(path, rest): + if rest.startswith(sep): + path = '' + + for name in rest.split(sep): + if not name or name == '.': + # current dir + continue + if name == '..': + # parent dir + path, _, _ = path.rpartition(sep) + continue + newpath = path + sep + name + if newpath in seen: + # Already seen this path + path = seen[newpath] + if path is not None: + # use cached value + continue + # The symlink is not resolved, so we must have a symlink loop. + raise RuntimeError("Symlink loop from %r" % newpath) + # Resolve the symbolic link + try: + target = accessor.readlink(newpath) + except OSError as e: + if e.errno != EINVAL: + raise + # Not a symlink + path = newpath + else: + seen[newpath] = None # not resolved symlink + path = _resolve(path, target) + seen[newpath] = path # resolved symlink + + return path + # NOTE: according to POSIX, getcwd() cannot contain path components + # which are symlinks. + base = '' if path.is_absolute() else os.getcwd() + return _resolve(base, str(path)) or sep def is_reserved(self, parts): return False diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -1620,6 +1620,59 @@ # 'bin' self.assertIs(p.parts[2], q.parts[3]) + def _check_complex_symlinks(self, link0_target): + # Test solving a non-looping chain of symlinks (issue #19887) + P = self.cls(BASE) + self.dirlink(os.path.join('link0', 'link0'), join('link1')) + self.dirlink(os.path.join('link1', 'link1'), join('link2')) + self.dirlink(os.path.join('link2', 'link2'), join('link3')) + self.dirlink(link0_target, join('link0')) + + # Resolve absolute paths + p = (P / 'link0').resolve() + self.assertEqual(p, P) + self.assertEqual(str(p), BASE) + p = (P / 'link1').resolve() + self.assertEqual(p, P) + self.assertEqual(str(p), BASE) + p = (P / 'link2').resolve() + self.assertEqual(p, P) + self.assertEqual(str(p), BASE) + p = (P / 'link3').resolve() + self.assertEqual(p, P) + self.assertEqual(str(p), BASE) + + # Resolve relative paths + old_path = os.getcwd() + os.chdir(BASE) + try: + p = self.cls('link0').resolve() + self.assertEqual(p, P) + self.assertEqual(str(p), BASE) + p = self.cls('link1').resolve() + self.assertEqual(p, P) + self.assertEqual(str(p), BASE) + p = self.cls('link2').resolve() + self.assertEqual(p, P) + self.assertEqual(str(p), BASE) + p = self.cls('link3').resolve() + self.assertEqual(p, P) + self.assertEqual(str(p), BASE) + finally: + os.chdir(old_path) + + @with_symlinks + def test_complex_symlinks_absolute(self): + self._check_complex_symlinks(BASE) + + @with_symlinks + def test_complex_symlinks_relative(self): + self._check_complex_symlinks('.') + + @with_symlinks + def test_complex_symlinks_relative_dot_dot(self): + self._check_complex_symlinks(os.path.join('dirA', '..')) + class PathTest(_BasePathTest, unittest.TestCase): cls = pathlib.Path diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #19887: Improve the Path.resolve() algorithm to support certain + symlink chains. + - Issue #19912: Fixed numerous bugs in ntpath.splitunc(). - Issue #19911: ntpath.splitdrive() now correctly processes the '?' character -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 20:22:44 2013 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 16 Dec 2013 20:22:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319921=3A_When_Pat?= =?utf-8?q?h=2Emkdir=28=29_is_called_with_parents=3DTrue=2C_any_missing_pa?= =?utf-8?q?rent?= Message-ID: <3djsnc2RJ7z7Lk5@mail.python.org> http://hg.python.org/cpython/rev/87b81b7df7f0 changeset: 88004:87b81b7df7f0 user: Antoine Pitrou date: Mon Dec 16 20:22:37 2013 +0100 summary: Issue #19921: When Path.mkdir() is called with parents=True, any missing parent is created with the default permissions, ignoring the mode argument (mimicking the POSIX "mkdir -p" command). Patch by Serhiy. files: Doc/library/pathlib.rst | 5 ++++- Lib/pathlib.py | 2 +- Lib/test/test_pathlib.py | 13 +++++++++++-- Misc/NEWS | 4 ++++ 4 files changed, 20 insertions(+), 4 deletions(-) diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -768,7 +768,10 @@ and access flags. If the path already exists, :exc:`OSError` is raised. If *parents* is true, any missing parents of this path are created - as needed. If *parents* is false (the default), a missing parent raises + as needed; they are created with the default permissions without taking + *mode* into account (mimicking the POSIX ``mkdir -p`` command). + + If *parents* is false (the default), a missing parent raises :exc:`OSError`. diff --git a/Lib/pathlib.py b/Lib/pathlib.py --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -1101,7 +1101,7 @@ except OSError as e: if e.errno != ENOENT: raise - self.parent.mkdir(mode, True) + self.parent.mkdir(parents=True) self._accessor.mkdir(self, mode) def chmod(self, mode): diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -1478,7 +1478,6 @@ with self.assertRaises(OSError) as cm: p.mkdir() self.assertEqual(cm.exception.errno, errno.EEXIST) - # XXX test `mode` arg def test_mkdir_parents(self): # Creating a chain of directories @@ -1493,7 +1492,17 @@ with self.assertRaises(OSError) as cm: p.mkdir(parents=True) self.assertEqual(cm.exception.errno, errno.EEXIST) - # XXX test `mode` arg + # test `mode` arg + mode = stat.S_IMODE(p.stat().st_mode) # default mode + p = self.cls(BASE, 'newdirD', 'newdirE') + p.mkdir(0o555, parents=True) + self.assertTrue(p.exists()) + self.assertTrue(p.is_dir()) + if os.name != 'nt': + # the directory's permissions follow the mode argument + self.assertEqual(stat.S_IMODE(p.stat().st_mode), 0o555 & mode) + # the parent's permissions follow the default process settings + self.assertEqual(stat.S_IMODE(p.parent.stat().st_mode), mode) @with_symlinks def test_symlink_to(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,10 @@ Library ------- +- Issue #19921: When Path.mkdir() is called with parents=True, any missing + parent is created with the default permissions, ignoring the mode argument + (mimicking the POSIX "mkdir -p" command). + - Issue #19887: Improve the Path.resolve() algorithm to support certain symlink chains. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 21:00:59 2013 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 16 Dec 2013 21:00:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Mention_specific_exception?= =?utf-8?q?s_instead_of_OSError_where_appropriate=2E?= Message-ID: <3djtdl6GJsz7Ljd@mail.python.org> http://hg.python.org/cpython/rev/6fb4227dba6a changeset: 88005:6fb4227dba6a user: Antoine Pitrou date: Mon Dec 16 21:00:53 2013 +0100 summary: Mention specific exceptions instead of OSError where appropriate. files: Doc/library/pathlib.rst | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -765,14 +765,15 @@ Create a new directory at this given path. If *mode* is given, it is combined with the process' ``umask`` value to determine the file mode - and access flags. If the path already exists, :exc:`OSError` is raised. + and access flags. If the path already exists, :exc:`FileExistsError` + is raised. If *parents* is true, any missing parents of this path are created as needed; they are created with the default permissions without taking *mode* into account (mimicking the POSIX ``mkdir -p`` command). If *parents* is false (the default), a missing parent raises - :exc:`OSError`. + :exc:`FileNotFoundError`. .. method:: Path.open(mode='r', buffering=-1, encoding=None, errors=None, newline=None) @@ -879,7 +880,7 @@ with the process' ``umask`` value to determine the file mode and access flags. If the file already exists, the function succeeds if *exist_ok* is true (and its modification time is updated to the current time), - otherwise :exc:`OSError` is raised. + otherwise :exc:`FileExistsError` is raised. .. method:: Path.unlink() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 21:17:14 2013 From: python-checkins at python.org (christian.heimes) Date: Mon, 16 Dec 2013 21:17:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5OTE5?= =?utf-8?q?=3A_Fix_flacky_SSL_test=2E_connect=5Fex=28=29_sometimes_returns?= Message-ID: <3djv0V0Xylz7LjN@mail.python.org> http://hg.python.org/cpython/rev/40955ae17472 changeset: 88006:40955ae17472 branch: 3.3 parent: 88000:5455456945d4 user: Christian Heimes date: Mon Dec 16 21:15:44 2013 +0100 summary: Issue #19919: Fix flacky SSL test. connect_ex() sometimes returns EWOULDBLOCK on Windows or VMs hosted on Windows. files: Lib/test/test_ssl.py | 6 ++++-- Misc/NEWS | 3 +++ 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -835,8 +835,10 @@ cert_reqs=ssl.CERT_REQUIRED, ca_certs=SVN_PYTHON_ORG_ROOT_CERT) try: - self.assertEqual(errno.ECONNREFUSED, - s.connect_ex(("svn.python.org", 444))) + rc = s.connect_ex(("svn.python.org", 444)) + # Issue #19919: Windows machines or VMs hosted on Windows + # machines sometimes return EWOULDBLOCK. + self.assertIn(rc, (errno.ECONNREFUSED, errno.EWOULDBLOCK)) finally: s.close() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -149,6 +149,9 @@ Tests ----- +- Issue #19919: Fix flacky SSL test. connect_ex() sometimes returns + EWOULDBLOCK on Windows or VMs hosted on Windows. + - Issue #19912: Added tests for ntpath.splitunc(). - Issue #19828: Fixed test_site when the whole suite is run with -S. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 21:17:15 2013 From: python-checkins at python.org (christian.heimes) Date: Mon, 16 Dec 2013 21:17:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319919=3A_Fix_flacky_SSL_test=2E_connect=5Fex=28?= =?utf-8?q?=29_sometimes_returns?= Message-ID: <3djv0W31gdz7LjN@mail.python.org> http://hg.python.org/cpython/rev/593c3fa7aa2c changeset: 88007:593c3fa7aa2c parent: 88004:87b81b7df7f0 parent: 88006:40955ae17472 user: Christian Heimes date: Mon Dec 16 21:16:45 2013 +0100 summary: Issue #19919: Fix flacky SSL test. connect_ex() sometimes returns EWOULDBLOCK on Windows or VMs hosted on Windows. files: Lib/test/test_ssl.py | 6 ++++-- Misc/NEWS | 3 +++ 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -1201,8 +1201,10 @@ cert_reqs=ssl.CERT_REQUIRED, ca_certs=SVN_PYTHON_ORG_ROOT_CERT) try: - self.assertEqual(errno.ECONNREFUSED, - s.connect_ex(("svn.python.org", 444))) + rc = s.connect_ex(("svn.python.org", 444)) + # Issue #19919: Windows machines or VMs hosted on Windows + # machines sometimes return EWOULDBLOCK. + self.assertIn(rc, (errno.ECONNREFUSED, errno.EWOULDBLOCK)) finally: s.close() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -192,6 +192,9 @@ Tests ----- +- Issue #19919: Fix flacky SSL test. connect_ex() sometimes returns + EWOULDBLOCK on Windows or VMs hosted on Windows. + - Issue #19912: Added tests for ntpath.splitunc(). - Issue #19828: Fixed test_site when the whole suite is run with -S. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 21:17:16 2013 From: python-checkins at python.org (christian.heimes) Date: Mon, 16 Dec 2013 21:17:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?q?=29=3A_merge?= Message-ID: <3djv0X5KfLz7Ljg@mail.python.org> http://hg.python.org/cpython/rev/07869026b13b changeset: 88008:07869026b13b parent: 88007:593c3fa7aa2c parent: 88005:6fb4227dba6a user: Christian Heimes date: Mon Dec 16 21:17:02 2013 +0100 summary: merge files: Doc/library/pathlib.rst | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -765,14 +765,15 @@ Create a new directory at this given path. If *mode* is given, it is combined with the process' ``umask`` value to determine the file mode - and access flags. If the path already exists, :exc:`OSError` is raised. + and access flags. If the path already exists, :exc:`FileExistsError` + is raised. If *parents* is true, any missing parents of this path are created as needed; they are created with the default permissions without taking *mode* into account (mimicking the POSIX ``mkdir -p`` command). If *parents* is false (the default), a missing parent raises - :exc:`OSError`. + :exc:`FileNotFoundError`. .. method:: Path.open(mode='r', buffering=-1, encoding=None, errors=None, newline=None) @@ -879,7 +880,7 @@ with the process' ``umask`` value to determine the file mode and access flags. If the file already exists, the function succeeds if *exist_ok* is true (and its modification time is updated to the current time), - otherwise :exc:`OSError` is raised. + otherwise :exc:`FileExistsError` is raised. .. method:: Path.unlink() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 22:39:24 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 16 Dec 2013 22:39:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogQ2xvc2UgIzE5OTk5?= =?utf-8?q?=3A_tolerate_coarse_time_when_testing_time=2Emonotonic=28=29_on?= =?utf-8?q?_very?= Message-ID: <3djwqJ4kZxz7Ljn@mail.python.org> http://hg.python.org/cpython/rev/4864c0b914ae changeset: 88009:4864c0b914ae branch: 3.3 parent: 88006:40955ae17472 user: Victor Stinner date: Mon Dec 16 22:36:50 2013 +0100 summary: Close #19999: tolerate coarse time when testing time.monotonic() on very busy/slow buildbot files: Lib/test/test_time.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py --- a/Lib/test/test_time.py +++ b/Lib/test/test_time.py @@ -376,7 +376,7 @@ t2 = time.monotonic() dt = t2 - t1 self.assertGreater(t2, t1) - self.assertAlmostEqual(dt, 0.5, delta=0.2) + self.assertTrue(0.5 <= dt <= 1.0, dt) info = time.get_clock_info('monotonic') self.assertTrue(info.monotonic) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 22:39:26 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 16 Dec 2013 22:39:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E3=29_Close_=2319999=3A_tolerate_coarse_time?= =?utf-8?q?_when_testing_time=2Emonotonic=28=29_on?= Message-ID: <3djwqL0hJbz7Ljp@mail.python.org> http://hg.python.org/cpython/rev/a34582c53911 changeset: 88010:a34582c53911 parent: 88008:07869026b13b parent: 88009:4864c0b914ae user: Victor Stinner date: Mon Dec 16 22:38:46 2013 +0100 summary: (Merge 3.3) Close #19999: tolerate coarse time when testing time.monotonic() on very busy/slow buildbot files: Lib/test/test_time.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py --- a/Lib/test/test_time.py +++ b/Lib/test/test_time.py @@ -384,7 +384,7 @@ t2 = time.monotonic() dt = t2 - t1 self.assertGreater(t2, t1) - self.assertAlmostEqual(dt, 0.5, delta=0.2) + self.assertTrue(0.5 <= dt <= 1.0, dt) # monotonic() is a monotonic but non adjustable clock info = time.get_clock_info('monotonic') -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 22:49:10 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 16 Dec 2013 22:49:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2318283=3A_shutil?= =?utf-8?q?=2Ewhich=28=29_now_supports_bytes_argument=2C_not_only_text?= Message-ID: <3djx2Z4njHz7LjP@mail.python.org> http://hg.python.org/cpython/rev/a1a05e2724dd changeset: 88011:a1a05e2724dd user: Victor Stinner date: Mon Dec 16 22:48:48 2013 +0100 summary: Issue #18283: shutil.which() now supports bytes argument, not only text argument. files: Doc/library/shutil.rst | 4 ++++ Lib/shutil.py | 23 +++++++++++++++-------- Lib/test/test_shutil.py | 13 +++++++++++-- Misc/NEWS | 2 ++ 4 files changed, 32 insertions(+), 10 deletions(-) diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -352,6 +352,10 @@ .. versionadded:: 3.3 + .. versionchanged:: 3.4 + The :class:`bytes` type is now accepted. If *cmd* type is :class:`bytes`, + the result type is also :class:`bytes`. + .. exception:: Error diff --git a/Lib/shutil.py b/Lib/shutil.py --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -1065,6 +1065,13 @@ return os.terminal_size((columns, lines)) +# Check that a given file can be accessed with the correct mode. +# Additionally check that `file` is not a directory, as on Windows +# directories pass the os.access check. +def _access_check(fn, mode): + return (os.path.exists(fn) and os.access(fn, mode) + and not os.path.isdir(fn)) + def which(cmd, mode=os.F_OK | os.X_OK, path=None): """Given a command, mode, and a PATH string, return the path which conforms to the given mode on the PATH, or None if there is no such @@ -1075,13 +1082,6 @@ path. """ - # Check that a given file can be accessed with the correct mode. - # Additionally check that `file` is not a directory, as on Windows - # directories pass the os.access check. - def _access_check(fn, mode): - return (os.path.exists(fn) and os.access(fn, mode) - and not os.path.isdir(fn)) - # If we're given a path with a directory part, look it up directly rather # than referring to PATH directories. This includes checking relative to the # current directory, e.g. ./script @@ -1094,7 +1094,12 @@ path = os.environ.get("PATH", os.defpath) if not path: return None - path = path.split(os.pathsep) + if isinstance(cmd, bytes): + path = os.fsencode(path) + path = path.split(os.fsencode(os.pathsep)) + else: + path = os.fsdecode(path) + path = path.split(os.pathsep) if sys.platform == "win32": # The current directory takes precedence on Windows. @@ -1103,6 +1108,8 @@ # PATHEXT is necessary to check on Windows. pathext = os.environ.get("PATHEXT", "").split(os.pathsep) + if isinstance(cmd, bytes): + pathext = map(os.fsencode, pathext) # See if the given file matches any of the expected path extensions. # This will allow us to short circuit when given "python.exe". # If it does match, only test that one, otherwise we have to try diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -1326,6 +1326,7 @@ os.chmod(self.temp_file.name, stat.S_IXUSR) self.addCleanup(self.temp_file.close) self.dir, self.file = os.path.split(self.temp_file.name) + self.env_path = self.dir def test_basic(self): # Given an EXE in a directory, it should be returned. @@ -1394,7 +1395,7 @@ def test_environ_path(self): with support.EnvironmentVarGuard() as env: - env['PATH'] = self.dir + env['PATH'] = self.env_path rv = shutil.which(self.file) self.assertEqual(rv, self.temp_file.name) @@ -1402,7 +1403,7 @@ base_dir = os.path.dirname(self.dir) with support.change_cwd(path=self.dir), \ support.EnvironmentVarGuard() as env: - env['PATH'] = self.dir + env['PATH'] = self.env_path rv = shutil.which(self.file, path='') self.assertIsNone(rv) @@ -1413,6 +1414,14 @@ self.assertIsNone(rv) +class TestWhichBytes(TestWhich): + def setUp(self): + TestWhich.setUp(self) + self.dir = os.fsencode(self.dir) + self.file = os.fsencode(self.file) + self.temp_file.name = os.fsencode(self.temp_file.name) + + class TestMove(unittest.TestCase): def setUp(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,8 @@ Library ------- +- Issue #18283: shutil.which() now supports bytes argument, not only text argument. + - Issue #19921: When Path.mkdir() is called with parents=True, any missing parent is created with the default permissions, ignoring the mode argument (mimicking the POSIX "mkdir -p" command). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 23:07:01 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 16 Dec 2013 23:07:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_tracemalloc=3A_only_use_un?= =?utf-8?q?signed_types_to_compute_hash?= Message-ID: <3djxR93QfRz7Lk5@mail.python.org> http://hg.python.org/cpython/rev/f1766b7457e7 changeset: 88012:f1766b7457e7 user: Victor Stinner date: Mon Dec 16 23:05:13 2013 +0100 summary: tracemalloc: only use unsigned types to compute hash Commit to simplify the backport to python 2.7 and to make the code more consistent. files: Modules/_tracemalloc.c | 9 ++++----- 1 files changed, 4 insertions(+), 5 deletions(-) diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -336,8 +336,7 @@ traceback_hash(traceback_t *traceback) { /* code based on tuplehash() of Objects/tupleobject.c */ - Py_uhash_t x; /* Unsigned for defined overflow behavior. */ - Py_hash_t y; + Py_uhash_t x, y; /* Unsigned for defined overflow behavior. */ int len = traceback->nframe; Py_uhash_t mult = _PyHASH_MULTIPLIER; frame_t *frame; @@ -345,13 +344,13 @@ x = 0x345678UL; frame = traceback->frames; while (--len >= 0) { - y = PyObject_Hash(frame->filename); - y ^= frame->lineno; + y = (Py_uhash_t)PyObject_Hash(frame->filename); + y ^= (Py_uhash_t)frame->lineno; frame++; x = (x ^ y) * mult; /* the cast might truncate len; that doesn't change hash stability */ - mult += (Py_hash_t)(82520UL + len + len); + mult += (Py_uhash_t)(82520UL + len + len); } x += 97531UL; return x; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 23:07:03 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 16 Dec 2013 23:07:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_tracemalloc=3A_fix_usage_o?= =?utf-8?q?f_strtol=28=29=2C_value_can_be_LONG=5FMIN_or_LONG=5FMAX_on_ERAN?= =?utf-8?q?GE?= Message-ID: <3djxRC58VLz7Ljd@mail.python.org> http://hg.python.org/cpython/rev/9885e0e85c7c changeset: 88013:9885e0e85c7c user: Victor Stinner date: Mon Dec 16 23:06:19 2013 +0100 summary: tracemalloc: fix usage of strtol(), value can be LONG_MIN or LONG_MAX on ERANGE files: Modules/_tracemalloc.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -1373,11 +1373,12 @@ char *endptr = p; long value; + errno = 0; value = strtol(p, &endptr, 10); if (*endptr != '\0' || value < 1 || value > MAX_NFRAME - || (errno == ERANGE && value == ULONG_MAX)) + || errno == ERANGE) { Py_FatalError("PYTHONTRACEMALLOC: invalid number of frames"); return -1; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 16 23:39:51 2013 From: python-checkins at python.org (victor.stinner) Date: Mon, 16 Dec 2013 23:39:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Backout_a1a05e2724dd=3A_sh?= =?utf-8?q?util=2Ewhich=28bytes=29_is_a_new_feature_and_my_patch_does?= Message-ID: <3djy9305Srz7Lk5@mail.python.org> http://hg.python.org/cpython/rev/177da1b8a7b7 changeset: 88014:177da1b8a7b7 user: Victor Stinner date: Mon Dec 16 23:39:40 2013 +0100 summary: Backout a1a05e2724dd: shutil.which(bytes) is a new feature and my patch does not work on Windows files: Doc/library/shutil.rst | 4 ---- Lib/shutil.py | 23 ++++++++--------------- Lib/test/test_shutil.py | 13 ++----------- Misc/NEWS | 2 -- 4 files changed, 10 insertions(+), 32 deletions(-) diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -352,10 +352,6 @@ .. versionadded:: 3.3 - .. versionchanged:: 3.4 - The :class:`bytes` type is now accepted. If *cmd* type is :class:`bytes`, - the result type is also :class:`bytes`. - .. exception:: Error diff --git a/Lib/shutil.py b/Lib/shutil.py --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -1065,13 +1065,6 @@ return os.terminal_size((columns, lines)) -# Check that a given file can be accessed with the correct mode. -# Additionally check that `file` is not a directory, as on Windows -# directories pass the os.access check. -def _access_check(fn, mode): - return (os.path.exists(fn) and os.access(fn, mode) - and not os.path.isdir(fn)) - def which(cmd, mode=os.F_OK | os.X_OK, path=None): """Given a command, mode, and a PATH string, return the path which conforms to the given mode on the PATH, or None if there is no such @@ -1082,6 +1075,13 @@ path. """ + # Check that a given file can be accessed with the correct mode. + # Additionally check that `file` is not a directory, as on Windows + # directories pass the os.access check. + def _access_check(fn, mode): + return (os.path.exists(fn) and os.access(fn, mode) + and not os.path.isdir(fn)) + # If we're given a path with a directory part, look it up directly rather # than referring to PATH directories. This includes checking relative to the # current directory, e.g. ./script @@ -1094,12 +1094,7 @@ path = os.environ.get("PATH", os.defpath) if not path: return None - if isinstance(cmd, bytes): - path = os.fsencode(path) - path = path.split(os.fsencode(os.pathsep)) - else: - path = os.fsdecode(path) - path = path.split(os.pathsep) + path = path.split(os.pathsep) if sys.platform == "win32": # The current directory takes precedence on Windows. @@ -1108,8 +1103,6 @@ # PATHEXT is necessary to check on Windows. pathext = os.environ.get("PATHEXT", "").split(os.pathsep) - if isinstance(cmd, bytes): - pathext = map(os.fsencode, pathext) # See if the given file matches any of the expected path extensions. # This will allow us to short circuit when given "python.exe". # If it does match, only test that one, otherwise we have to try diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -1326,7 +1326,6 @@ os.chmod(self.temp_file.name, stat.S_IXUSR) self.addCleanup(self.temp_file.close) self.dir, self.file = os.path.split(self.temp_file.name) - self.env_path = self.dir def test_basic(self): # Given an EXE in a directory, it should be returned. @@ -1395,7 +1394,7 @@ def test_environ_path(self): with support.EnvironmentVarGuard() as env: - env['PATH'] = self.env_path + env['PATH'] = self.dir rv = shutil.which(self.file) self.assertEqual(rv, self.temp_file.name) @@ -1403,7 +1402,7 @@ base_dir = os.path.dirname(self.dir) with support.change_cwd(path=self.dir), \ support.EnvironmentVarGuard() as env: - env['PATH'] = self.env_path + env['PATH'] = self.dir rv = shutil.which(self.file, path='') self.assertIsNone(rv) @@ -1414,14 +1413,6 @@ self.assertIsNone(rv) -class TestWhichBytes(TestWhich): - def setUp(self): - TestWhich.setUp(self) - self.dir = os.fsencode(self.dir) - self.file = os.fsencode(self.file) - self.temp_file.name = os.fsencode(self.temp_file.name) - - class TestMove(unittest.TestCase): def setUp(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,8 +44,6 @@ Library ------- -- Issue #18283: shutil.which() now supports bytes argument, not only text argument. - - Issue #19921: When Path.mkdir() is called with parents=True, any missing parent is created with the default permissions, ignoring the mode argument (mimicking the POSIX "mkdir -p" command). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 17 06:13:42 2013 From: python-checkins at python.org (eli.bendersky) Date: Tue, 17 Dec 2013 06:13:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320001=3A_Add_the_?= =?utf-8?q?SVG_source_of_the_pathlib-inheritance_diagram_to_Hg?= Message-ID: <3dk6vV02Zbz7Lk3@mail.python.org> http://hg.python.org/cpython/rev/fe49ed665190 changeset: 88015:fe49ed665190 user: Eli Bendersky date: Mon Dec 16 21:13:40 2013 -0800 summary: Issue #20001: Add the SVG source of the pathlib-inheritance diagram to Hg files: Doc/library/pathlib-inheritance.svg | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Doc/library/pathlib-inheritance.svg b/Doc/library/pathlib-inheritance.svg new file mode 100644 --- /dev/null +++ b/Doc/library/pathlib-inheritance.svg @@ -0,0 +1,4 @@ + + + + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 17 07:11:56 2013 From: python-checkins at python.org (eric.snow) Date: Tue, 17 Dec 2013 07:11:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319713=3A_Update_i?= =?utf-8?q?mportlib_docs_for_module_spec_changes=2C_including?= Message-ID: <3dk8Bh0qpFz7LjS@mail.python.org> http://hg.python.org/cpython/rev/2c27c0e5bc50 changeset: 88016:2c27c0e5bc50 user: Eric Snow date: Mon Dec 16 23:06:52 2013 -0700 summary: Issue #19713: Update importlib docs for module spec changes, including deprecations. files: Doc/glossary.rst | 4 + Doc/library/importlib.rst | 172 ++++++++++++++++++++++--- 2 files changed, 154 insertions(+), 22 deletions(-) diff --git a/Doc/glossary.rst b/Doc/glossary.rst --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -539,6 +539,10 @@ See also :term:`package`. + module spec + A namespace containing the import-related information used to load a + module. + MRO See :term:`method resolution order`. diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -89,6 +89,22 @@ .. versionchanged:: 3.3 Parent packages are automatically imported. +.. function:: find_spec(name, path=None) + + Find the :term:`spec ` for a module, optionally within the + specified *path*. If the module is in :attr:`sys.modules`, then + ``sys.modules[name].__spec__`` is returned (unless the spec would be + ``None`` or is not set, in which case :exc:`ValueError` is raised). + Otherwise a search using :attr:`sys.meta_path` is done. ``None`` is + returned if no spec is found. + + A dotted name does not have its parent implicitly imported as that requires + loading them and that may not be desired. To properly import a submodule you + will need to import all parent packages of the submodule and use the correct + argument to *path*. + + .. versionadded:: 3.4 + .. function:: find_loader(name, path=None) Find the loader for a module, optionally within the specified *path*. If the @@ -108,6 +124,9 @@ If ``__loader__`` is not set, raise :exc:`ValueError`, just like when the attribute is set to ``None``. + .. deprecated:: 3.4 + Use :func:`find_spec` instead. + .. function:: invalidate_caches() Invalidate the internal caches of finders stored at @@ -236,9 +255,21 @@ .. versionadded:: 3.3 + .. method:: find_spec(fullname, path, target=None) + + An abstract method for finding a :term:`spec ` for + the specified module. If this is a top-level import, *path* will + be ``None``. Otherwise, this is a search for a subpackage or + module and *path* will be the value of :attr:`__path__` from the + parent package. If a spec cannot be found, ``None`` is returned. + When passed in, ``target`` is a module object that the finder may + use to make a more educated about what spec to return. + + .. versionadded:: 3.4 + .. method:: find_module(fullname, path) - An abstract method for finding a :term:`loader` for the specified + A legacy method for finding a :term:`loader` for the specified module. If this is a top-level import, *path* will be ``None``. Otherwise, this is a search for a subpackage or module and *path* will be the value of :attr:`__path__` from the parent @@ -248,6 +279,9 @@ Returns ``None`` when called instead of raising :exc:`NotImplementedError`. + .. deprecated:: 3.4 + Use :meth:`find_spec` instead. + .. method:: invalidate_caches() An optional method which, when called, should invalidate any internal @@ -268,9 +302,20 @@ .. versionadded:: 3.3 + .. method:: find_spec(fullname, target=None) + + An abstract method for finding a :term:`spec ` for + the specified module. The finder will search for the module only + within the :term:`path entry` to which it is assigned. If a spec + cannot be found, ``None`` is returned. When passed in, ``target`` + is a module object that the finder may use to make a more educated + about what spec to return. + + .. versionadded:: 3.4 + .. method:: find_loader(fullname) - An abstract method for finding a :term:`loader` for the specified + A legacy method for finding a :term:`loader` for the specified module. Returns a 2-tuple of ``(loader, portion)`` where ``portion`` is a sequence of file system locations contributing to part of a namespace package. The loader may be ``None`` while specifying ``portion`` to @@ -283,11 +328,17 @@ .. versionchanged:: 3.4 Returns ``(None, [])`` instead of raising :exc:`NotImplementedError`. + .. deprecated:: 3.4 + Use :meth:`find_spec` instead. + .. method:: find_module(fullname) A concrete implementation of :meth:`Finder.find_module` which is equivalent to ``self.find_loader(fullname)[0]``. + .. deprecated:: 3.4 + Use :meth:`find_spec` instead. + .. method:: invalidate_caches() An optional method which, when called, should invalidate any internal @@ -300,9 +351,26 @@ An abstract base class for a :term:`loader`. See :pep:`302` for the exact definition for a loader. + .. method:: create_module(spec) + + An optional method that returns the module object to use when + importing a module. create_module() may also return ``None``, + indicating that the default module creation should take place + instead. + + .. versionadded:: 3.4 + + .. method:: exec_module(module) + + An abstract method that executes the module in its own namespace + when a module is imported or reloaded. The module should already + be initialized when exec_module() is called. + + .. versionadded:: 3.4 + .. method:: load_module(fullname) - An abstract method for loading a module. If the module cannot be + A legacy method for loading a module. If the module cannot be loaded, :exc:`ImportError` is raised, otherwise the loaded module is returned. @@ -349,9 +417,16 @@ Raise :exc:`ImportError` when called instead of :exc:`NotImplementedError`. + .. deprecated:: 3.4 + The recommended API for loading a module is :meth:`exec_module` + (and optionally :meth:`create_module`). Loaders should implement + it instead of load_module(). The import machinery takes care of + all the other responsibilities of load_module() when exec_module() + is implemented. + .. method:: module_repr(module) - An optional method which when implemented calculates and returns the + A legacy method which when implemented calculates and returns the given module's repr, as a string. The module type's default repr() will use the result of this method as appropriate. @@ -360,6 +435,9 @@ .. versionchanged:: 3.4 Made optional instead of an abstractmethod. + .. deprecated:: 3.4 + The import machinery now takes care of this automatically. + .. class:: ResourceLoader @@ -434,9 +512,18 @@ .. versionadded:: 3.4 + .. method:: exec_module(module) + + Implementation of :meth:`Loader.exec_module`. + + .. versionadded:: 3.4 + .. method:: load_module(fullname) - Implementation of :meth:`Loader.load_module`. + Implementation of :meth:`Loader.load_module`. + + .. deprecated:: 3.4 + use :meth:`exec_module` instead. .. class:: ExecutionLoader @@ -482,6 +569,9 @@ Calls super's ``load_module()``. + .. deprecated:: 3.4 + Use :meth:`Loader.exec_module` instead. + .. method:: get_filename(fullname) Returns :attr:`path`. @@ -558,9 +648,18 @@ Concrete implementation of :meth:`InspectLoader.get_code`. + .. method:: exec_module(module) + + Concrete implementation of :meth:`Loader.exec_module`. + + .. versionadded:: 3.4 + .. method:: load_module(fullname) - Concrete implementation of :meth:`Loader.load_module`. + Concrete implementation of :meth:`Loader.load_module`. + + .. deprecated:: 3.4 + Use :meth:`exec_module` instead. .. method:: get_source(fullname) @@ -641,6 +740,10 @@ Only class methods are defined by this class to alleviate the need for instantiation. + .. note:: + Due to limitations in the extension module C-API, for now + BuiltinImporter does not implement :meth:`Loader.exec_module`. + .. class:: FrozenImporter @@ -671,28 +774,37 @@ Only class methods are defined by this class to alleviate the need for instantiation. + .. classmethod:: find_spec(fullname, path=None, target=None) + + Class method that attempts to find a :term:`spec ` + for the module specified by *fullname* on :data:`sys.path` or, if + defined, on *path*. For each path entry that is searched, + :data:`sys.path_importer_cache` is checked. If a non-false object + is found then it is used as the :term:`path entry finder` to look + for the module being searched for. If no entry is found in + :data:`sys.path_importer_cache`, then :data:`sys.path_hooks` is + searched for a finder for the path entry and, if found, is stored + in :data:`sys.path_importer_cache` along with being queried about + the module. If no finder is ever found then ``None`` is both + stored in the cache and returned. + + .. versionadded:: 3.4 + .. classmethod:: find_module(fullname, path=None) - Class method that attempts to find a :term:`loader` for the module - specified by *fullname* on :data:`sys.path` or, if defined, on - *path*. For each path entry that is searched, - :data:`sys.path_importer_cache` is checked. If a non-false object is - found then it is used as the :term:`path entry finder` to look for the - module being searched for. If no entry is found in - :data:`sys.path_importer_cache`, then :data:`sys.path_hooks` is - searched for a finder for the path entry and, if found, is stored in - :data:`sys.path_importer_cache` along with being queried about the - module. If no finder is ever found then ``None`` is both stored in - the cache and returned. + A legacy wrapper around :meth:`find_spec`. + + .. deprecated:: 3.4 + Use :meth:`find_spec` instead. .. classmethod:: invalidate_caches() - Calls :meth:`importlib.abc.PathEntryFinder.invalidate_caches` on all - finders stored in :attr:`sys.path_importer_cache`. + Calls :meth:`importlib.abc.PathEntryFinder.invalidate_caches` on all + finders stored in :attr:`sys.path_importer_cache`. - .. versionchanged:: 3.4 - Calls objects in :data:`sys.path_hooks` with the current working directory - for ``''`` (i.e. the empty string). + .. versionchanged:: 3.4 + Calls objects in :data:`sys.path_hooks` with the current working + directory for ``''`` (i.e. the empty string). .. class:: FileFinder(path, \*loader_details) @@ -724,6 +836,12 @@ The path the finder will search in. + .. method:: find_spec(fullname, target=None) + + Attempt to find the spec to handle *fullname* within :attr:`path`. + + .. versionadded:: 3.4 + .. method:: find_loader(fullname) Attempt to find the loader to handle *fullname* within :attr:`path`. @@ -829,6 +947,10 @@ Loads the extension module if and only if *fullname* is the same as :attr:`name` or is ``None``. + .. note:: + Due to limitations in the extension module C-API, for now + ExtensionFileLoader does not implement :meth:`Loader.exec_module`. + .. method:: is_package(fullname) Returns ``True`` if the file path points to a package's ``__init__`` @@ -1026,11 +1148,17 @@ Set ``__loader__`` if set to ``None``, as if the attribute does not exist. + .. deprecated:: 3.4 + The import machinery takes care of this automatically. + .. decorator:: set_package A :term:`decorator` for :meth:`importlib.abc.Loader.load_module` to set the :attr:`__package__` attribute on the returned module. If :attr:`__package__` is set and has a value other than ``None`` it will not be changed. + .. deprecated:: 3.4 + The import machinery takes care of this automatically. + .. function:: spec_from_loader(name, loader, *, origin=None, is_package=None) A factory function for creating a :class:`ModuleSpec` instance based -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 17 07:11:57 2013 From: python-checkins at python.org (eric.snow) Date: Tue, 17 Dec 2013 07:11:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319713=3A_Fix_mist?= =?utf-8?q?akes_in_the_import_page_of_language_reference=2E?= Message-ID: <3dk8Bj640wz7Lkh@mail.python.org> http://hg.python.org/cpython/rev/b78de8029606 changeset: 88017:b78de8029606 user: Eric Snow date: Mon Dec 16 23:10:50 2013 -0700 summary: Issue #19713: Fix mistakes in the import page of language reference. These mistakes were introduced by the initial PEP 451 merge. files: Doc/reference/import.rst | 216 ++++++++++++++------------ 1 files changed, 116 insertions(+), 100 deletions(-) diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -281,9 +281,10 @@ searches :data:`sys.meta_path`, which contains a list of meta path finder objects. These finders are queried in order to see if they know how to handle the named module. Meta path finders must implement a method called -:meth:`find_spec()` which takes two arguments, a name and an import path. -The meta path finder can use any strategy it wants to determine whether it can -handle the named module or not. +:meth:`~importlib.abc.MetaPathFinder.find_spec()` which takes three arguments: +a name, an import path, and (optionally) a target module. The meta path +finder can use any strategy it wants to determine whether it can handle +the named module or not. If the meta path finder knows how to handle the named module, it returns a spec object. If it cannot handle the named module, it returns ``None``. If @@ -291,24 +292,26 @@ a spec, then an :exc:`ImportError` is raised. Any other exceptions raised are simply propagated up, aborting the import process. -The :meth:`find_spec()` method of meta path finders is called with two -arguments. The first is the fully qualified name of the module being -imported, for example ``foo.bar.baz``. The second argument is the path -entries to use for the module search. For top-level modules, the second -argument is ``None``, but for submodules or subpackages, the second -argument is the value of the parent package's ``__path__`` attribute. If -the appropriate ``__path__`` attribute cannot be accessed, an -:exc:`ImportError` is raised. +The :meth:`~importlib.abc.MetaPathFinder.find_spec()` method of meta path +finders is called with two or three arguments. The first is the fully +qualified name of the module being imported, for example ``foo.bar.baz``. +The second argument is the path entries to use for the module search. For +top-level modules, the second argument is ``None``, but for submodules or +subpackages, the second argument is the value of the parent package's +``__path__`` attribute. If the appropriate ``__path__`` attribute cannot +be accessed, an :exc:`ImportError` is raised. The third argument is an +existing module object that will be the target of loading later. The +import system passes in a target module only during reload. The meta path may be traversed multiple times for a single import request. For example, assuming none of the modules involved has already been cached, importing ``foo.bar.baz`` will first perform a top level import, calling -``mpf.find_spec("foo", None)`` on each meta path finder (``mpf``). After +``mpf.find_spec("foo", None, None)`` on each meta path finder (``mpf``). After ``foo`` has been imported, ``foo.bar`` will be imported by traversing the meta path a second time, calling -``mpf.find_spec("foo.bar", foo.__path__)``. Once ``foo.bar`` has been +``mpf.find_spec("foo.bar", foo.__path__, None)``. Once ``foo.bar`` has been imported, the final traversal will call -``mpf.find_spec("foo.bar.baz", foo.bar.__path__)``. +``mpf.find_spec("foo.bar.baz", foo.bar.__path__, None)``. Some meta path finders only support top level imports. These importers will always return ``None`` when anything other than ``None`` is passed as the @@ -320,10 +323,11 @@ (i.e. the :term:`path based finder`). .. versionchanged:: 3.4 - The find_spec() method of meta path finders replaced :meth:`find_module()`. - which is now deprecated. While it will continue to work without change, - the import machinery will try it only if the finder does not implement - find_spec(). + The :meth:`~importlib.abc.MetaPathFinder.find_spec` method of meta path + finders replaced :meth:`~importlib.abc.MetaPathFinder.find_module`, which + is now deprecated. While it will continue to work without change, the + import machinery will try it only if the finder does not implement + ``find_spec()``. Loading @@ -350,6 +354,7 @@ raise ImportError elif not hasattr(spec.loader, 'exec_module'): module = spec.loader.load_module(spec.name) + # Set __loader__ and __package__ if missing. else: sys.modules[spec.name] = module try: @@ -360,7 +365,7 @@ except KeyError: pass raise - module_to_return = sys.modules[spec.name] + return sys.modules[spec.name] Note the following details: @@ -380,8 +385,9 @@ reloading where even the failing module is left in :data:`sys.modules`. * After the module is created but before execution, the import machinery - sets the import-related module attributes ("init_module_attrs"), as - summarized in a :ref:`later section `. + sets the import-related module attributes ("_init_module_attrs" in + the pseudo-code example above), as summarized in a + :ref:`later section `. * Module execution is the key moment of loading in which the module's namespace gets populated. Execution is entirely delegated to the @@ -392,16 +398,16 @@ .. versionchanged:: 3.4 The import system has taken over the boilerplate responsibilities of - loaders. These were previously performed by the :meth:`load_module()` - method. + loaders. These were previously performed by the + :meth:`importlib.abc.Loader.load_module` method. Loaders ------- Module loaders provide the critical function of loading: module execution. -The import machinery calls the :meth:`~importlib.abc.Loader.exec_module()` +The import machinery calls the :meth:`importlib.abc.Loader.exec_module` method with a single argument, the module object to execute. Any value -returned from exec_module() is ignored. +returned from :meth:`~importlib.abc.Loader.exec_module` is ignored. Loaders must satisfy the following requirements: @@ -409,41 +415,41 @@ dynamically loaded extension), the loader should execute the module's code in the module's global name space (``module.__dict__``). - * If loader cannot execute the module, it should raise an + * If the loader cannot execute the module, it should raise an :exc:`ImportError`, although any other exception raised during - :meth:`exec_module()` will be propagated. + :meth:`~importlib.abc.Loader.exec_module` will be propagated. In many cases, the finder and loader can be the same object; in such cases the -:meth:`finder.find_spec()` would just return a spec with the loader set -to ``self``. +:meth:`~importlib.abc.MetaPathFinder.find_spec` method would just return a +spec with the loader set to ``self``. Module loaders may opt in to creating the module object during loading -by implementing a :meth:`create_module()` method. It takes one argument, -the module spec, and returns the new module object to use during loading. -create_module() does not need to set any attributes on the module object. -If the loader does not define create_module(), the import machinery will -create the new module itself. +by implementing a :meth:`~importlib.abc.Loader.create_module` method. +It takes one argument, the module spec, and returns the new module object +to use during loading. ``create_module()`` does not need to set any attributes +on the module object. If the loader does not define ``create_module()``, the +import machinery will create the new module itself. .. versionadded:: 3.4 The create_module() method of loaders. .. versionchanged:: 3.4 - The load_module() method was replaced by exec_module() and the import + The :meth:`~importlib.abc.Loader.load_module` method was replaced by + :meth:`~importlib.abc.Loader.exec_module` and the import machinery assumed all the boilerplate responsibilities of loading. For compatibility with existing loaders, the import machinery will use - the :meth:`~importlib.abc.Loader.load_module()` method of loaders if it - exists and the loader does not also implement exec_module(). However, - load_module() has been deprecated and loaders should implement - exec_module() instead. + the ``load_module()`` method of loaders if it exists and the loader does + not also implement ``exec_module(). However, ``load_module()`` has been + deprecated and loaders should implement ``exec_module()`` instead. - The load_module() method must implement all the boilerplate loading + The ``load_module()`` method must implement all the boilerplate loading functionality described above in addition to executing the module. All the same constraints apply, with some additional clarification: * If there is an existing module object with the given name in :data:`sys.modules`, the loader must use that existing module. - (Otherwise, :func:`imp.reload` will not work correctly.) If the + (Otherwise, :func:`importlib.reload` will not work correctly.) If the named module does not exist in :data:`sys.modules`, the loader must create a new module object and add it to :data:`sys.modules`. @@ -515,6 +521,8 @@ used when importing the module. This is used primarily for introspection and during reloading. + .. versionadded:: 3.4 + .. attribute:: __path__ If the module is a package (either regular or namespace), the module @@ -600,13 +608,14 @@ * Otherwise, just use the module's ``__name__`` in the repr. .. versionchanged:: 3.4 - Use of loader.module_repr() has been deprecated and the module spec - is now used by the import machinery to generate a module repr. + Use of :meth:`loader.module_repr() ` + has been deprecated and the module spec is now used by the import + machinery to generate a module repr. For backward compatibility with Python 3.3, the module repr will be - generated by calling the loader's :meth:`module_repr()` method, if - defined, before trying either approach described above. However, the - method is deprecated. + generated by calling the loader's + :meth:`~importlib.abc.Loader.module_repr` method, if defined, before + trying either approach described above. However, the method is deprecated. The Path Based Finder @@ -616,8 +625,9 @@ single: path based finder As mentioned previously, Python comes with several default meta path finders. -One of these, called the :term:`path based finder`, searches an :term:`import -path`, which contains a list of :term:`path entries `. Each path +One of these, called the :term:`path based finder` +(:class:`~importlib.machinery.PathFinder`) , searches an :term:`import path`, +which contains a list of :term:`path entries `. Each path entry names a location to search for modules. The path based finder itself doesn't know how to import anything. Instead, it @@ -666,15 +676,15 @@ single: sys.path_importer_cache single: PYTHONPATH -The :term:`path based finder` is responsible for finding and loading Python -modules and packages whose location is specified with a string :term:`path -entry`. Most path entries name locations in the file system, but they need -not be limited to this. +The :term:`path based finder` is responsible for finding and loading +Python modules and packages whose location is specified with a string +:term:`path entry`. Most path entries name locations in the file system, +but they need not be limited to this. As a meta path finder, the :term:`path based finder` implements the -:meth:`find_spec()` protocol previously described, however it exposes -additional hooks that can be used to customize how modules are found and -loaded from the :term:`import path`. +:meth:`~importlib.abc.MetaPathFinder.find_spec` protocol previously +described, however it exposes additional hooks that can be used to +customize how modules are found and loaded from the :term:`import path`. Three variables are used by the :term:`path based finder`, :data:`sys.path`, :data:`sys.path_hooks` and :data:`sys.path_importer_cache`. The ``__path__`` @@ -694,14 +704,16 @@ The :term:`path based finder` is a :term:`meta path finder`, so the import machinery begins the :term:`import path` search by calling the path -based finder's :meth:`find_spec()` method as described previously. When -the ``path`` argument to :meth:`find_spec()` is given, it will be a +based finder's :meth:`~importlib.machinery.PathFinder.find_spec` method as +described previously. When the ``path`` argument to +:meth:`~importlib.machinery.PathFinder.find_spec` is given, it will be a list of string paths to traverse - typically a package's ``__path__`` -attribute for an import within that package. If the ``path`` argument -is ``None``, this indicates a top level import and :data:`sys.path` is used. +attribute for an import within that package. If the ``path`` argument is +``None``, this indicates a top level import and :data:`sys.path` is used. The path based finder iterates over every entry in the search path, and -for each of these, looks for an appropriate :term:`path entry finder` for the +for each of these, looks for an appropriate :term:`path entry finder` +(:class:`~importlib.abc.PathEntryFinder`) for the path entry. Because this can be an expensive operation (e.g. there may be `stat()` call overheads for this search), the path based finder maintains a cache mapping path entries to path entry finders. This cache is maintained @@ -718,18 +730,20 @@ path entry to be searched. This callable may either return a :term:`path entry finder` that can handle the path entry, or it may raise :exc:`ImportError`. An :exc:`ImportError` is used by the path based finder to -signal that the hook cannot find a :term:`path entry finder` for that -:term:`path entry`. The exception is ignored and :term:`import path` -iteration continues. The hook should expect either a string or bytes object; -the encoding of bytes objects is up to the hook (e.g. it may be a file system -encoding, UTF-8, or something else), and if the hook cannot decode the -argument, it should raise :exc:`ImportError`. +signal that the hook cannot find a :term:`path entry finder`. +for that :term:`path entry`. The +exception is ignored and :term:`import path` iteration continues. The hook +should expect either a string or bytes object; the encoding of bytes objects +is up to the hook (e.g. it may be a file system encoding, UTF-8, or something +else), and if the hook cannot decode the argument, it should raise +:exc:`ImportError`. If :data:`sys.path_hooks` iteration ends with no :term:`path entry finder` -being returned, then the path based finder's :meth:`find_spec()` method -will store ``None`` in :data:`sys.path_importer_cache` (to indicate that -there is no finder for this path entry) and return ``None``, indicating that -this :term:`meta path finder` could not find the module. +being returned, then the path based finder's +:meth:`~importlib.machinery.PathFinder.find_spec` method will store ``None`` +in :data:`sys.path_importer_cache` (to indicate that there is no finder for +this path entry) and return ``None``, indicating that this +:term:`meta path finder` could not find the module. If a :term:`path entry finder` *is* returned by one of the :term:`path entry hook` callables on :data:`sys.path_hooks`, then the following protocol is used @@ -741,12 +755,12 @@ In order to support imports of modules and initialized packages and also to contribute portions to namespace packages, path entry finders must implement -the :meth:`find_spec()` method. +the :meth:`~importlib.abc.PathEntryFinder.find_spec` method. -:meth:`find_spec()` takes one argument, the fully qualified name of the -module being imported. :meth:`find_spec()` returns a fully populated -spec for the module. This spec will always have "loader" set (with one -exception). +:meth:`~importlib.abc.PathEntryFinder.find_spec` takes two argument, the +fully qualified name of the module being imported, and the (optional) target +module. ``find_spec()`` returns a fully populated spec for the module. +This spec will always have "loader" set (with one exception). To indicate to the import machinery that the spec represents a namespace :term:`portion`. the path entry finder sets "loader" on the spec to @@ -754,42 +768,44 @@ portion. .. versionchanged:: 3.4 - find_spec() replaced find_loader() and find_module(), but of which - are now deprecated, but will be used if find_spec() is not defined. + :meth:`~importlib.abc.PathEntryFinder.find_spec` replaced + :meth:`~importlib.abc.PathEntryFinder.find_loader` and + :meth:`~importlib.abc.PathEntryFinder.find_module`, both of which + are now deprecated, but will be used if ``find_spec()`` is not defined. Older path entry finders may implement one of these two deprecated methods - instead of :meth:`find_spec()`. The methods are still respected for the - sake of backward compatibility. Howevever, if find_spec() is implemented - on the path entry finder, the legacy methods are ignored. + instead of ``find_spec()``. The methods are still respected for the + sake of backward compatibility. Howevever, if ``find_spec()`` is + implemented on the path entry finder, the legacy methods are ignored. - :meth:`find_loader()` takes one argument, the fully qualified name of the - module being imported. :meth:`find_loader()` returns a 2-tuple where the - first item is the loader and the second item is a namespace :term:`portion`. - When the first item (i.e. the loader) is ``None``, this means that while the - path entry finder does not have a loader for the named module, it knows that - the path entry contributes to a namespace portion for the named module. - This will almost always be the case where Python is asked to import a - namespace package that has no physical presence on the file system. - When a path entry finder returns ``None`` for the loader, the second - item of the 2-tuple return value must be a sequence, although it can be - empty. + :meth:`~importlib.abc.PathEntryFinder.find_loader` takes one argument, the + fully qualified name of the module being imported. ``find_loader()`` + returns a 2-tuple where the first item is the loader and the second item + is a namespace :term:`portion`. When the first item (i.e. the loader) is + ``None``, this means that while the path entry finder does not have a + loader for the named module, it knows that the path entry contributes to + a namespace portion for the named module. This will almost always be the + case where Python is asked to import a namespace package that has no + physical presence on the file system. When a path entry finder returns + ``None`` for the loader, the second item of the 2-tuple return value must + be a sequence, although it can be empty. - If :meth:`find_loader()` returns a non-``None`` loader value, the portion is + If ``find_loader()`` returns a non-``None`` loader value, the portion is ignored and the loader is returned from the path based finder, terminating the search through the path entries. For backwards compatibility with other implementations of the import protocol, many path entry finders also support the same, - traditional :meth:`find_module()` method that meta path finders support. - However path entry finder :meth:`find_module()` methods are never called + traditional ``find_module()`` method that meta path finders support. + However path entry finder ``find_module()`` methods are never called with a ``path`` argument (they are expected to record the appropriate path information from the initial call to the path hook). - The :meth:`find_module()` method on path entry finders is deprecated, + The ``find_module()`` method on path entry finders is deprecated, as it does not allow the path entry finder to contribute portions to - namespace packages. If both :meth:`find_loader()` and :meth:`find_module()` + namespace packages. If both ``find_loader()`` and ``find_module()`` exist on a path entry finder, the import system will always call - :meth:`find_loader()` in preference to :meth:`find_module()`. + ``find_loader()`` in preference to ``find_module()``. Replacing the standard import system @@ -808,9 +824,9 @@ To selectively prevent import of some modules from a hook early on the meta path (rather than disabling the standard import system entirely), it is sufficient to raise :exc:`ImportError` directly from -:meth:`find_spec` instead of returning ``None``. The latter indicates -that the meta path search should continue. while raising an exception -terminates it immediately. +:meth:`~importlib.abc.MetaPathFinder.find_spec` instead of returning +``None``. The latter indicates that the meta path search should continue, +while raising an exception terminates it immediately. Open issues -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Tue Dec 17 09:47:54 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 17 Dec 2013 09:47:54 +0100 Subject: [Python-checkins] Daily reference leaks (177da1b8a7b7): sum=0 Message-ID: results for 177da1b8a7b7 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/refloggglw03', '-x'] From python-checkins at python.org Tue Dec 17 13:17:43 2013 From: python-checkins at python.org (nick.coghlan) Date: Tue, 17 Dec 2013 13:17:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Close_=2319946=3A_use_runp?= =?utf-8?q?y_as_needed_in_multiprocessing?= Message-ID: <3dkJJl0NYFzNN5@mail.python.org> http://hg.python.org/cpython/rev/b6d6f3b4b100 changeset: 88018:b6d6f3b4b100 user: Nick Coghlan date: Tue Dec 17 22:17:26 2013 +1000 summary: Close #19946: use runpy as needed in multiprocessing - handles main files without a suffix - handles main submodules properly - adds test cases for the various kinds of __main__ files: Doc/whatsnew/3.4.rst | 15 +- Lib/multiprocessing/spawn.py | 121 ++- Lib/test/test_multiprocessing_main_handling.py | 287 ++++++++++ Misc/NEWS | 6 + 4 files changed, 375 insertions(+), 54 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -624,13 +624,22 @@ multiprocessing --------------- -On Unix two new *start methods* have been added for starting processes -using :mod:`multiprocessing`. These make the mixing of processes with -threads more robust. See :issue:`8713`. +On Unix, two new *start methods* (``spawn`` and ``forkserver``) have been +added for starting processes using :mod:`multiprocessing`. These make +the mixing of processes with threads more robust, and the ``spawn`` +method matches the semantics that multiprocessing has always used on +Windows. (Contributed by Richard Oudkerk in :issue:`8713`). Also, except when using the old *fork* start method, child processes will no longer inherit unneeded handles/file descriptors from their parents. +:mod:`multiprocessing` now relies on :mod:`runpy` (which implements the +``-m`` switch) to initialise ``__main__`` appropriately in child processes +when using the ``spawn`` or ``forkserver`` start methods. This resolves some +edge cases where combining multiprocessing, the ``-m`` command line switch +and explicit relative imports could cause obscure failures in child +processes. (Contributed by Nick Coghlan in :issue:`19946`) + os -- diff --git a/Lib/multiprocessing/spawn.py b/Lib/multiprocessing/spawn.py --- a/Lib/multiprocessing/spawn.py +++ b/Lib/multiprocessing/spawn.py @@ -11,6 +11,8 @@ import os import pickle import sys +import runpy +import types from . import get_start_method, set_start_method from . import process @@ -157,15 +159,19 @@ start_method=get_start_method(), ) - if sys.platform != 'win32' or (not WINEXE and not WINSERVICE): - main_path = getattr(sys.modules['__main__'], '__file__', None) - if not main_path and sys.argv[0] not in ('', '-c'): - main_path = sys.argv[0] + # Figure out whether to initialise main in the subprocess as a module + # or through direct execution (or to leave it alone entirely) + main_module = sys.modules['__main__'] + main_mod_name = getattr(main_module.__spec__, "name", None) + if main_mod_name is not None: + d['init_main_from_name'] = main_mod_name + elif sys.platform != 'win32' or (not WINEXE and not WINSERVICE): + main_path = getattr(main_module, '__file__', None) if main_path is not None: if (not os.path.isabs(main_path) and process.ORIGINAL_DIR is not None): main_path = os.path.join(process.ORIGINAL_DIR, main_path) - d['main_path'] = os.path.normpath(main_path) + d['init_main_from_path'] = os.path.normpath(main_path) return d @@ -206,55 +212,68 @@ if 'start_method' in data: set_start_method(data['start_method']) - if 'main_path' in data: - import_main_path(data['main_path']) + if 'init_main_from_name' in data: + _fixup_main_from_name(data['init_main_from_name']) + elif 'init_main_from_path' in data: + _fixup_main_from_path(data['init_main_from_path']) + +# Multiprocessing module helpers to fix up the main module in +# spawned subprocesses +def _fixup_main_from_name(mod_name): + # __main__.py files for packages, directories, zip archives, etc, run + # their "main only" code unconditionally, so we don't even try to + # populate anything in __main__, nor do we make any changes to + # __main__ attributes + current_main = sys.modules['__main__'] + if mod_name == "__main__" or mod_name.endswith(".__main__"): + return + + # If this process was forked, __main__ may already be populated + if getattr(current_main.__spec__, "name", None) == mod_name: + return + + # Otherwise, __main__ may contain some non-main code where we need to + # support unpickling it properly. We rerun it as __mp_main__ and make + # the normal __main__ an alias to that + old_main_modules.append(current_main) + main_module = types.ModuleType("__mp_main__") + main_content = runpy.run_module(mod_name, + run_name="__mp_main__", + alter_sys=True) + main_module.__dict__.update(main_content) + sys.modules['__main__'] = sys.modules['__mp_main__'] = main_module + + +def _fixup_main_from_path(main_path): + # If this process was forked, __main__ may already be populated + current_main = sys.modules['__main__'] + + # Unfortunately, the main ipython launch script historically had no + # "if __name__ == '__main__'" guard, so we work around that + # by treating it like a __main__.py file + # See https://github.com/ipython/ipython/issues/4698 + main_name = os.path.splitext(os.path.basename(main_path))[0] + if main_name == 'ipython': + return + + # Otherwise, if __file__ already has the setting we expect, + # there's nothing more to do + if getattr(current_main, '__file__', None) == main_path: + return + + # If the parent process has sent a path through rather than a module + # name we assume it is an executable script that may contain + # non-main code that needs to be executed + old_main_modules.append(current_main) + main_module = types.ModuleType("__mp_main__") + main_content = runpy.run_path(main_path, + run_name="__mp_main__") + main_module.__dict__.update(main_content) + sys.modules['__main__'] = sys.modules['__mp_main__'] = main_module def import_main_path(main_path): ''' Set sys.modules['__main__'] to module at main_path ''' - # XXX (ncoghlan): The following code makes several bogus - # assumptions regarding the relationship between __file__ - # and a module's real name. See PEP 302 and issue #10845 - if getattr(sys.modules['__main__'], '__file__', None) == main_path: - return - - main_name = os.path.splitext(os.path.basename(main_path))[0] - if main_name == '__init__': - main_name = os.path.basename(os.path.dirname(main_path)) - - if main_name == '__main__': - main_module = sys.modules['__main__'] - main_module.__file__ = main_path - elif main_name != 'ipython': - # Main modules not actually called __main__.py may - # contain additional code that should still be executed - import importlib - import types - - if main_path is None: - dirs = None - elif os.path.basename(main_path).startswith('__init__.py'): - dirs = [os.path.dirname(os.path.dirname(main_path))] - else: - dirs = [os.path.dirname(main_path)] - - assert main_name not in sys.modules, main_name - sys.modules.pop('__mp_main__', None) - # We should not try to load __main__ - # since that would execute 'if __name__ == "__main__"' - # clauses, potentially causing a psuedo fork bomb. - main_module = types.ModuleType(main_name) - # XXX Use a target of main_module? - spec = importlib.find_spec(main_name, path=dirs) - if spec is None: - raise ImportError(name=main_name) - methods = importlib._bootstrap._SpecMethods(spec) - methods.init_module_attrs(main_module) - main_module.__name__ = '__mp_main__' - code = spec.loader.get_code(main_name) - exec(code, main_module.__dict__) - - old_main_modules.append(sys.modules['__main__']) - sys.modules['__main__'] = sys.modules['__mp_main__'] = main_module + _fixup_main_from_path(main_path) diff --git a/Lib/test/test_multiprocessing_main_handling.py b/Lib/test/test_multiprocessing_main_handling.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_multiprocessing_main_handling.py @@ -0,0 +1,287 @@ +# tests __main__ module handling in multiprocessing + +import importlib +import importlib.machinery +import zipimport +import unittest +import sys +import os +import os.path +import py_compile + +from test import support +from test.script_helper import ( + make_pkg, make_script, make_zip_pkg, make_zip_script, + assert_python_ok, assert_python_failure, temp_dir, + spawn_python, kill_python) + +# We look inside the context module to find out which +# start methods we can check +from multiprocessing.context import _concrete_contexts + +verbose = support.verbose + +test_source = """\ +# multiprocessing includes all sorts of shenanigans to make __main__ +# attributes accessible in the subprocess in a pickle compatible way. + +# We run the "doesn't work in the interactive interpreter" example from +# the docs to make sure it *does* work from an executed __main__, +# regardless of the invocation mechanism + +import sys +import time +from multiprocessing import Pool, set_start_method + +# We use this __main__ defined function in the map call below in order to +# check that multiprocessing in correctly running the unguarded +# code in child processes and then making it available as __main__ +def f(x): + return x*x + +# Check explicit relative imports +if "check_sibling" in __file__: + # We're inside a package and not in a __main__.py file + # so make sure explicit relative imports work correctly + from . import sibling + +if __name__ == '__main__': + start_method = sys.argv[1] + set_start_method(start_method) + p = Pool(5) + results = [] + p.map_async(f, [1, 2, 3], callback=results.extend) + deadline = time.time() + 2 # up to 2 s to report the results + while not results: + time.sleep(0.05) + if time.time() > deadline: + raise RuntimeError("Timed out waiting for results") + results.sort() + print(start_method, "->", results) +""" + +test_source_main_skipped_in_children = """\ +# __main__.py files have an implied "if __name__ == '__main__'" so +# multiprocessing should always skip running them in child processes + +# This means we can't use __main__ defined functions in child processes, +# so we just use "int" as a passthrough operation below + +if __name__ != "__main__": + raise RuntimeError("Should only be called as __main__!") + +import sys +import time +from multiprocessing import Pool, set_start_method + +start_method = sys.argv[1] +set_start_method(start_method) +p = Pool(5) +results = [] +p.map_async(int, [1, 4, 9], callback=results.extend) +deadline = time.time() + 2 # up to 2 s to report the results +while not results: + time.sleep(0.05) + if time.time() > deadline: + raise RuntimeError("Timed out waiting for results") +results.sort() +print(start_method, "->", results) +""" + +# These helpers were copied from test_cmd_line_script & tweaked a bit... + +def _make_test_script(script_dir, script_basename, + source=test_source, omit_suffix=False): + to_return = make_script(script_dir, script_basename, + source, omit_suffix) + # Hack to check explicit relative imports + if script_basename == "check_sibling": + make_script(script_dir, "sibling", "") + importlib.invalidate_caches() + return to_return + +def _make_test_zip_pkg(zip_dir, zip_basename, pkg_name, script_basename, + source=test_source, depth=1): + to_return = make_zip_pkg(zip_dir, zip_basename, pkg_name, script_basename, + source, depth) + importlib.invalidate_caches() + return to_return + +# There's no easy way to pass the script directory in to get +# -m to work (avoiding that is the whole point of making +# directories and zipfiles executable!) +# So we fake it for testing purposes with a custom launch script +launch_source = """\ +import sys, os.path, runpy +sys.path.insert(0, %s) +runpy._run_module_as_main(%r) +""" + +def _make_launch_script(script_dir, script_basename, module_name, path=None): + if path is None: + path = "os.path.dirname(__file__)" + else: + path = repr(path) + source = launch_source % (path, module_name) + to_return = make_script(script_dir, script_basename, source) + importlib.invalidate_caches() + return to_return + +class MultiProcessingCmdLineMixin(): + maxDiff = None # Show full tracebacks on subprocess failure + + def setupClass(cls): + if cls.start_method not in _concrete_contexts: + raise unittest.SkipTest("%r start method not available" % + cls.start_method) + + def _check_output(self, script_name, exit_code, out, err): + if verbose > 1: + print("Output from test script %r:" % script_name) + print(out) + self.assertEqual(exit_code, 0) + self.assertEqual(err.decode('utf-8'), '') + expected_results = "%s -> [1, 4, 9]" % self.start_method + self.assertEqual(out.decode('utf-8').strip(), expected_results) + + def _check_script(self, script_name, *cmd_line_switches): + if not __debug__: + cmd_line_switches += ('-' + 'O' * sys.flags.optimize,) + run_args = cmd_line_switches + (script_name, self.start_method) + rc, out, err = assert_python_ok(*run_args, __isolated=False) + self._check_output(script_name, rc, out, err) + + def test_basic_script(self): + with temp_dir() as script_dir: + script_name = _make_test_script(script_dir, 'script') + self._check_script(script_name) + + def test_basic_script_no_suffix(self): + with temp_dir() as script_dir: + script_name = _make_test_script(script_dir, 'script', + omit_suffix=True) + self._check_script(script_name) + + def test_ipython_workaround(self): + # Some versions of the IPython launch script are missing the + # __name__ = "__main__" guard, and multiprocessing has long had + # a workaround for that case + # See https://github.com/ipython/ipython/issues/4698 + source = test_source_main_skipped_in_children + with temp_dir() as script_dir: + script_name = _make_test_script(script_dir, 'ipython', + source=source) + self._check_script(script_name) + script_no_suffix = _make_test_script(script_dir, 'ipython', + source=source, + omit_suffix=True) + self._check_script(script_no_suffix) + + def test_script_compiled(self): + with temp_dir() as script_dir: + script_name = _make_test_script(script_dir, 'script') + py_compile.compile(script_name, doraise=True) + os.remove(script_name) + pyc_file = support.make_legacy_pyc(script_name) + self._check_script(pyc_file) + + def test_directory(self): + source = self.main_in_children_source + with temp_dir() as script_dir: + script_name = _make_test_script(script_dir, '__main__', + source=source) + self._check_script(script_dir) + + def test_directory_compiled(self): + source = self.main_in_children_source + with temp_dir() as script_dir: + script_name = _make_test_script(script_dir, '__main__', + source=source) + py_compile.compile(script_name, doraise=True) + os.remove(script_name) + pyc_file = support.make_legacy_pyc(script_name) + self._check_script(script_dir) + + def test_zipfile(self): + source = self.main_in_children_source + with temp_dir() as script_dir: + script_name = _make_test_script(script_dir, '__main__', + source=source) + zip_name, run_name = make_zip_script(script_dir, 'test_zip', script_name) + self._check_script(zip_name) + + def test_zipfile_compiled(self): + source = self.main_in_children_source + with temp_dir() as script_dir: + script_name = _make_test_script(script_dir, '__main__', + source=source) + compiled_name = py_compile.compile(script_name, doraise=True) + zip_name, run_name = make_zip_script(script_dir, 'test_zip', compiled_name) + self._check_script(zip_name) + + def test_module_in_package(self): + with temp_dir() as script_dir: + pkg_dir = os.path.join(script_dir, 'test_pkg') + make_pkg(pkg_dir) + script_name = _make_test_script(pkg_dir, 'check_sibling') + launch_name = _make_launch_script(script_dir, 'launch', + 'test_pkg.check_sibling') + self._check_script(launch_name) + + def test_module_in_package_in_zipfile(self): + with temp_dir() as script_dir: + zip_name, run_name = _make_test_zip_pkg(script_dir, 'test_zip', 'test_pkg', 'script') + launch_name = _make_launch_script(script_dir, 'launch', 'test_pkg.script', zip_name) + self._check_script(launch_name) + + def test_module_in_subpackage_in_zipfile(self): + with temp_dir() as script_dir: + zip_name, run_name = _make_test_zip_pkg(script_dir, 'test_zip', 'test_pkg', 'script', depth=2) + launch_name = _make_launch_script(script_dir, 'launch', 'test_pkg.test_pkg.script', zip_name) + self._check_script(launch_name) + + def test_package(self): + source = self.main_in_children_source + with temp_dir() as script_dir: + pkg_dir = os.path.join(script_dir, 'test_pkg') + make_pkg(pkg_dir) + script_name = _make_test_script(pkg_dir, '__main__', + source=source) + launch_name = _make_launch_script(script_dir, 'launch', 'test_pkg') + self._check_script(launch_name) + + def test_package_compiled(self): + source = self.main_in_children_source + with temp_dir() as script_dir: + pkg_dir = os.path.join(script_dir, 'test_pkg') + make_pkg(pkg_dir) + script_name = _make_test_script(pkg_dir, '__main__', + source=source) + compiled_name = py_compile.compile(script_name, doraise=True) + os.remove(script_name) + pyc_file = support.make_legacy_pyc(script_name) + launch_name = _make_launch_script(script_dir, 'launch', 'test_pkg') + self._check_script(launch_name) + +# Test all supported start methods (setupClass skips as appropriate) + +class SpawnCmdLineTest(MultiProcessingCmdLineMixin, unittest.TestCase): + start_method = 'spawn' + main_in_children_source = test_source_main_skipped_in_children + +class ForkCmdLineTest(MultiProcessingCmdLineMixin, unittest.TestCase): + start_method = 'fork' + main_in_children_source = test_source + +class ForkServerCmdLineTest(MultiProcessingCmdLineMixin, unittest.TestCase): + start_method = 'forkserver' + main_in_children_source = test_source_main_skipped_in_children + +def test_main(): + support.run_unittest(SpawnCmdLineTest, + ForkCmdLineTest, + ForkServerCmdLineTest) + support.reap_children() + +if __name__ == '__main__': + test_main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,12 @@ Library ------- +- Issue #19946: multiprocessing now uses runpy to initialize __main__ in + child processes when necessary, allowing it to correctly handle scripts + without suffixes and submodules that use explicit relative imports or + otherwise rely on parent modules being correctly imported prior to + execution. + - Issue #19921: When Path.mkdir() is called with parents=True, any missing parent is created with the default permissions, ignoring the mode argument (mimicking the POSIX "mkdir -p" command). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 17 13:40:53 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 17 Dec 2013 13:40:53 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE3OTc2?= =?utf-8?q?=3A_Fixed_potential_problem_with_file=2Ewrite=28=29_not_detecti?= =?utf-8?q?ng_IO_error?= Message-ID: <3dkJqT3wyBz7LjY@mail.python.org> http://hg.python.org/cpython/rev/33c27b76a4d0 changeset: 88019:33c27b76a4d0 branch: 2.7 parent: 88002:4de09cbd3b97 user: Serhiy Storchaka date: Tue Dec 17 14:40:06 2013 +0200 summary: Issue #17976: Fixed potential problem with file.write() not detecting IO error by inspecting the return value of fwrite(). Based on patches by Jaakko Moisio and test by Victor Stinner. files: Lib/test/test_file2k.py | 8 ++++++++ Misc/ACKS | 1 + Misc/NEWS | 4 ++++ Objects/fileobject.c | 6 +++++- 4 files changed, 18 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_file2k.py b/Lib/test/test_file2k.py --- a/Lib/test/test_file2k.py +++ b/Lib/test/test_file2k.py @@ -415,6 +415,14 @@ finally: os.unlink(TESTFN) + @unittest.skipUnless(os.name == 'posix', 'test requires a posix system.') + def test_write_full(self): + # Issue #17976 + with open('/dev/full', 'w', 1) as f: + with self.assertRaises(IOError): + f.write('hello') + f.write('\n') + class FileSubclassTests(unittest.TestCase): def testExit(self): diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -710,6 +710,7 @@ Dom Mitchell Florian Mladitsch Doug Moen +Jaakko Moisio The Dragon De Monsyne Skip Montanaro Paul Moore diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -9,6 +9,10 @@ Core and Builtins ----------------- +- Issue #17976: Fixed potential problem with file.write() not detecting IO error + by inspecting the return value of fwrite(). Based on patches by Jaakko Moisio + and Victor Stinner. + - Issue #14432: Generator now clears the borrowed reference to the thread state. Fix a crash when a generator is created in a C thread that is destroyed while the generator is still used. The issue was that a generator diff --git a/Objects/fileobject.c b/Objects/fileobject.c --- a/Objects/fileobject.c +++ b/Objects/fileobject.c @@ -1804,6 +1804,7 @@ const char *s; Py_ssize_t n, n2; PyObject *encoded = NULL; + int err = 0; if (f->f_fp == NULL) return err_closed(); @@ -1849,11 +1850,14 @@ FILE_BEGIN_ALLOW_THREADS(f) errno = 0; n2 = fwrite(s, 1, n, f->f_fp); + if (n2 != n || ferror(f->f_fp)) + err = errno; FILE_END_ALLOW_THREADS(f) Py_XDECREF(encoded); if (f->f_binary) PyBuffer_Release(&pbuf); - if (n2 != n) { + if (err) { + errno = err; PyErr_SetFromErrno(PyExc_IOError); clearerr(f->f_fp); return NULL; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 17 13:54:18 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 17 Dec 2013 13:54:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Skip_test_for_?= =?utf-8?q?issue_=2317976_if_/dev/null_is_not_available=2E?= Message-ID: <3dkK6y2bGTz7Ljc@mail.python.org> http://hg.python.org/cpython/rev/237deaf9ba64 changeset: 88020:237deaf9ba64 branch: 2.7 user: Serhiy Storchaka date: Tue Dec 17 14:53:32 2013 +0200 summary: Skip test for issue #17976 if /dev/null is not available. files: Lib/test/test_file2k.py | 8 +++++++- 1 files changed, 7 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_file2k.py b/Lib/test/test_file2k.py --- a/Lib/test/test_file2k.py +++ b/Lib/test/test_file2k.py @@ -418,10 +418,16 @@ @unittest.skipUnless(os.name == 'posix', 'test requires a posix system.') def test_write_full(self): # Issue #17976 - with open('/dev/full', 'w', 1) as f: + try: + f = open('/dev/full', 'w', 1) + except IOError: + self.skipTest("requires '/dev/full'") + try: with self.assertRaises(IOError): f.write('hello') f.write('\n') + finally: + f.close() class FileSubclassTests(unittest.TestCase): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 17 14:02:39 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 17 Dec 2013 14:02:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Fixed_leak_in_?= =?utf-8?q?sys=2Eflags_initialization=2E?= Message-ID: <3dkKJb4w2nz7LjS@mail.python.org> http://hg.python.org/cpython/rev/c3c3dd78309d changeset: 88021:c3c3dd78309d branch: 2.7 user: Serhiy Storchaka date: Tue Dec 17 14:59:29 2013 +0200 summary: Fixed leak in sys.flags initialization. files: Python/sysmodule.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Python/sysmodule.c b/Python/sysmodule.c --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -1261,6 +1261,7 @@ #undef SetFlag if (PyErr_Occurred()) { + Py_DECREF(seq); return NULL; } return seq; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 17 14:02:40 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 17 Dec 2013 14:02:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Fixed_leak_in_?= =?utf-8?q?sys=2Eflags_initialization=2E?= Message-ID: <3dkKJc6rwmz7Ljg@mail.python.org> http://hg.python.org/cpython/rev/90a7277a3187 changeset: 88022:90a7277a3187 branch: 3.3 parent: 88009:4864c0b914ae user: Serhiy Storchaka date: Tue Dec 17 14:59:42 2013 +0200 summary: Fixed leak in sys.flags initialization. files: Python/sysmodule.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Python/sysmodule.c b/Python/sysmodule.c --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -1404,6 +1404,7 @@ #undef SetFlag if (PyErr_Occurred()) { + Py_DECREF(seq); return NULL; } return seq; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 17 14:02:42 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 17 Dec 2013 14:02:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Fixed_leak_in_sys=2Eflags_initialization=2E?= Message-ID: <3dkKJf1P7tz7Lk5@mail.python.org> http://hg.python.org/cpython/rev/cf939f70d596 changeset: 88023:cf939f70d596 parent: 88018:b6d6f3b4b100 parent: 88022:90a7277a3187 user: Serhiy Storchaka date: Tue Dec 17 15:00:53 2013 +0200 summary: Fixed leak in sys.flags initialization. files: Python/sysmodule.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Python/sysmodule.c b/Python/sysmodule.c --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -1465,6 +1465,7 @@ #undef SetFlag if (PyErr_Occurred()) { + Py_DECREF(seq); return NULL; } return seq; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 17 14:15:19 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 17 Dec 2013 14:15:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE2NDA0?= =?utf-8?q?=3A_Add_checks_for_return_value_of_PyInt=5FFromLong=28=29_in?= Message-ID: <3dkKbC2rdrz7LjS@mail.python.org> http://hg.python.org/cpython/rev/debdfa44f020 changeset: 88024:debdfa44f020 branch: 2.7 parent: 88021:c3c3dd78309d user: Serhiy Storchaka date: Tue Dec 17 15:09:45 2013 +0200 summary: Issue #16404: Add checks for return value of PyInt_FromLong() in sys.getwindowsversion() and ossaudiodev.setparameters(). Reported by Ned Batchelder. files: Modules/ossaudiodev.c | 9 +-------- Python/sysmodule.c | 4 ++++ 2 files changed, 5 insertions(+), 8 deletions(-) diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c --- a/Modules/ossaudiodev.c +++ b/Modules/ossaudiodev.c @@ -490,7 +490,6 @@ { int wanted_fmt, wanted_channels, wanted_rate, strict=0; int fmt, channels, rate; - PyObject * rv; /* return tuple (fmt, channels, rate) */ if (!PyArg_ParseTuple(args, "iii|i:setparameters", &wanted_fmt, &wanted_channels, &wanted_rate, @@ -532,13 +531,7 @@ /* Construct the return value: a (fmt, channels, rate) tuple that tells what the audio hardware was actually set to. */ - rv = PyTuple_New(3); - if (rv == NULL) - return NULL; - PyTuple_SET_ITEM(rv, 0, PyInt_FromLong(fmt)); - PyTuple_SET_ITEM(rv, 1, PyInt_FromLong(channels)); - PyTuple_SET_ITEM(rv, 2, PyInt_FromLong(rate)); - return rv; + return Py_BuildValue("(iii)", fmt, channels, rate); } static int diff --git a/Python/sysmodule.c b/Python/sysmodule.c --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -616,6 +616,10 @@ PyStructSequence_SET_ITEM(version, pos++, PyInt_FromLong(ver.wSuiteMask)); PyStructSequence_SET_ITEM(version, pos++, PyInt_FromLong(ver.wProductType)); + if (PyErr_Occurred()) { + Py_DECREF(version); + return NULL; + } return version; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 17 14:15:20 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 17 Dec 2013 14:15:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE2NDA0?= =?utf-8?q?=3A_Add_checks_for_return_value_of_PyLong=5FFromLong=28=29_in?= Message-ID: <3dkKbD4dvQz7LjS@mail.python.org> http://hg.python.org/cpython/rev/928c0acf7c09 changeset: 88025:928c0acf7c09 branch: 3.3 parent: 88022:90a7277a3187 user: Serhiy Storchaka date: Tue Dec 17 15:11:24 2013 +0200 summary: Issue #16404: Add checks for return value of PyLong_FromLong() in sys.getwindowsversion() and ossaudiodev.setparameters(). Reported by Ned Batchelder. files: Modules/ossaudiodev.c | 9 +-------- Python/sysmodule.c | 4 ++++ 2 files changed, 5 insertions(+), 8 deletions(-) diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c --- a/Modules/ossaudiodev.c +++ b/Modules/ossaudiodev.c @@ -561,7 +561,6 @@ { int wanted_fmt, wanted_channels, wanted_rate, strict=0; int fmt, channels, rate; - PyObject * rv; /* return tuple (fmt, channels, rate) */ if (!_is_fd_valid(self->fd)) return NULL; @@ -606,13 +605,7 @@ /* Construct the return value: a (fmt, channels, rate) tuple that tells what the audio hardware was actually set to. */ - rv = PyTuple_New(3); - if (rv == NULL) - return NULL; - PyTuple_SET_ITEM(rv, 0, PyLong_FromLong(fmt)); - PyTuple_SET_ITEM(rv, 1, PyLong_FromLong(channels)); - PyTuple_SET_ITEM(rv, 2, PyLong_FromLong(rate)); - return rv; + return Py_BuildValue("(iii)", fmt, channels, rate); } static int diff --git a/Python/sysmodule.c b/Python/sysmodule.c --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -747,6 +747,10 @@ PyStructSequence_SET_ITEM(version, pos++, PyLong_FromLong(ver.wSuiteMask)); PyStructSequence_SET_ITEM(version, pos++, PyLong_FromLong(ver.wProductType)); + if (PyErr_Occurred()) { + Py_DECREF(version); + return NULL; + } return version; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 17 14:15:22 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 17 Dec 2013 14:15:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2316404=3A_Add_checks_for_return_value_of_PyLong?= =?utf-8?q?=5FFromLong=28=29_in?= Message-ID: <3dkKbG0RG8z7Lk1@mail.python.org> http://hg.python.org/cpython/rev/d489394a73de changeset: 88026:d489394a73de parent: 88023:cf939f70d596 parent: 88025:928c0acf7c09 user: Serhiy Storchaka date: Tue Dec 17 15:12:46 2013 +0200 summary: Issue #16404: Add checks for return value of PyLong_FromLong() in sys.getwindowsversion() and ossaudiodev.setparameters(). Reported by Ned Batchelder. files: Modules/ossaudiodev.c | 9 +-------- Python/sysmodule.c | 4 ++++ 2 files changed, 5 insertions(+), 8 deletions(-) diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c --- a/Modules/ossaudiodev.c +++ b/Modules/ossaudiodev.c @@ -564,7 +564,6 @@ { int wanted_fmt, wanted_channels, wanted_rate, strict=0; int fmt, channels, rate; - PyObject * rv; /* return tuple (fmt, channels, rate) */ if (!_is_fd_valid(self->fd)) return NULL; @@ -609,13 +608,7 @@ /* Construct the return value: a (fmt, channels, rate) tuple that tells what the audio hardware was actually set to. */ - rv = PyTuple_New(3); - if (rv == NULL) - return NULL; - PyTuple_SET_ITEM(rv, 0, PyLong_FromLong(fmt)); - PyTuple_SET_ITEM(rv, 1, PyLong_FromLong(channels)); - PyTuple_SET_ITEM(rv, 2, PyLong_FromLong(rate)); - return rv; + return Py_BuildValue("(iii)", fmt, channels, rate); } static int diff --git a/Python/sysmodule.c b/Python/sysmodule.c --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -801,6 +801,10 @@ PyStructSequence_SET_ITEM(version, pos++, PyLong_FromLong(ver.wSuiteMask)); PyStructSequence_SET_ITEM(version, pos++, PyLong_FromLong(ver.wProductType)); + if (PyErr_Occurred()) { + Py_DECREF(version); + return NULL; + } return version; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 17 16:32:49 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 17 Dec 2013 16:32:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Circumventing_?= =?utf-8?q?a_bug_in_glibc_=28issue_=2317976=29=2E?= Message-ID: <3dkNds2p1fz7LkK@mail.python.org> http://hg.python.org/cpython/rev/24a043355050 changeset: 88027:24a043355050 branch: 2.7 parent: 88024:debdfa44f020 user: Serhiy Storchaka date: Tue Dec 17 17:32:20 2013 +0200 summary: Circumventing a bug in glibc (issue #17976). Patch by Jaakko Moisio. files: Objects/fileobject.c | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Objects/fileobject.c b/Objects/fileobject.c --- a/Objects/fileobject.c +++ b/Objects/fileobject.c @@ -1804,7 +1804,7 @@ const char *s; Py_ssize_t n, n2; PyObject *encoded = NULL; - int err = 0; + int err_flag = 0, err; if (f->f_fp == NULL) return err_closed(); @@ -1850,13 +1850,15 @@ FILE_BEGIN_ALLOW_THREADS(f) errno = 0; n2 = fwrite(s, 1, n, f->f_fp); - if (n2 != n || ferror(f->f_fp)) + if (n2 != n || ferror(f->f_fp)) { + err_flag = 1; err = errno; + } FILE_END_ALLOW_THREADS(f) Py_XDECREF(encoded); if (f->f_binary) PyBuffer_Release(&pbuf); - if (err) { + if (err_flag) { errno = err; PyErr_SetFromErrno(PyExc_IOError); clearerr(f->f_fp); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 17 18:09:58 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 17 Dec 2013 18:09:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogMjAwMDQ6IE5vdGUg?= =?utf-8?q?that_the_setter_in_csv=2EDictReader_is_broken=2E?= Message-ID: <3dkQny6v9Pz7Lkt@mail.python.org> http://hg.python.org/cpython/rev/75c0b65591fd changeset: 88028:75c0b65591fd branch: 2.7 user: R David Murray date: Tue Dec 17 12:09:46 2013 -0500 summary: 20004: Note that the setter in csv.DictReader is broken. This is a comment in the code because only someone reading the code would try setting fieldnames to None in the first place... files: Lib/csv.py | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Lib/csv.py b/Lib/csv.py --- a/Lib/csv.py +++ b/Lib/csv.py @@ -93,6 +93,10 @@ self.line_num = self.reader.line_num return self._fieldnames + # Issue 20004: Because DictReader is a classic class, this setter is + # ignored. At this point in 2.7's lifecycle, it is too late to change the + # base class for fear of breaking working code. If you want to change + # fieldnames without overwriting the getter, set _fieldnames directly. @fieldnames.setter def fieldnames(self, value): self._fieldnames = value -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 17 20:54:14 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 17 Dec 2013 20:54:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIwMDA3?= =?utf-8?q?=3A_HTTPResponse=2Eread=280=29_no_more_prematurely_closes_conne?= =?utf-8?q?ction=2E?= Message-ID: <3dkVRV2WVrz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/ebace0a5a33e changeset: 88029:ebace0a5a33e branch: 2.7 user: Serhiy Storchaka date: Tue Dec 17 21:49:48 2013 +0200 summary: Issue #20007: HTTPResponse.read(0) no more prematurely closes connection. Original patch by Simon Sapin. files: Lib/httplib.py | 2 +- Lib/test/test_httplib.py | 2 ++ Misc/ACKS | 1 + Misc/NEWS | 3 +++ 4 files changed, 7 insertions(+), 1 deletions(-) diff --git a/Lib/httplib.py b/Lib/httplib.py --- a/Lib/httplib.py +++ b/Lib/httplib.py @@ -565,7 +565,7 @@ # connection, and the user is reading more bytes than will be provided # (for example, reading in 1k chunks) s = self.fp.read(amt) - if not s: + if not s and amt: # Ideally, we would raise IncompleteRead if the content-length # wasn't satisfied, but it might break compatibility. self.close() diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -153,6 +153,8 @@ sock = FakeSocket(body) resp = httplib.HTTPResponse(sock) resp.begin() + self.assertEqual(resp.read(0), '') # Issue #20007 + self.assertFalse(resp.isclosed()) self.assertEqual(resp.read(), 'Text') self.assertTrue(resp.isclosed()) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -904,6 +904,7 @@ Rich Salz Kevin Samborn Ilya Sandler +Simon Sapin Mark Sapiro Ty Sarna Hugh Sasse diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -27,6 +27,9 @@ Library ------- +- Issue #20007: HTTPResponse.read(0) no more prematurely closes connection. + Original patch by Simon Sapin. + - Issue #19912: Fixed numerous bugs in ntpath.splitunc(). - Issue #19623: Fixed writing to unseekable files in the aifc module. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 17 20:54:15 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 17 Dec 2013 20:54:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzIwMDA3?= =?utf-8?q?=3A_HTTPResponse=2Eread=280=29_no_more_prematurely_closes_conne?= =?utf-8?q?ction=2E?= Message-ID: <3dkVRW6kMcz7Ljx@mail.python.org> http://hg.python.org/cpython/rev/47ae858cd661 changeset: 88030:47ae858cd661 branch: 3.3 parent: 88025:928c0acf7c09 user: Serhiy Storchaka date: Tue Dec 17 21:50:02 2013 +0200 summary: Issue #20007: HTTPResponse.read(0) no more prematurely closes connection. Original patch by Simon Sapin. files: Lib/http/client.py | 2 +- Lib/test/test_httplib.py | 3 +++ Misc/ACKS | 1 + Misc/NEWS | 3 +++ 4 files changed, 8 insertions(+), 1 deletions(-) diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -544,7 +544,7 @@ # connection, and the user is reading more bytes than will be provided # (for example, reading in 1k chunks) n = self.fp.readinto(b) - if not n: + if not n and b: # Ideally, we would raise IncompleteRead if the content-length # wasn't satisfied, but it might break compatibility. self._close_conn() diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -162,6 +162,9 @@ sock = FakeSocket(body) resp = client.HTTPResponse(sock) resp.begin() + self.assertEqual(resp.read(0), b'') # Issue #20007 + self.assertFalse(resp.isclosed()) + self.assertFalse(resp.closed) self.assertEqual(resp.read(), b"Text") self.assertTrue(resp.isclosed()) self.assertFalse(resp.closed) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1086,6 +1086,7 @@ James Sanders Ilya Sandler Rafael Santos +Simon Sapin Mark Sapiro Ty Sarna Hugh Sasse diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,9 @@ Library ------- +- Issue #20007: HTTPResponse.read(0) no more prematurely closes connection. + Original patch by Simon Sapin. + - Issue #19912: Fixed numerous bugs in ntpath.splitunc(). - Issue #19911: ntpath.splitdrive() now correctly processes the '?' character -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 17 20:54:17 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 17 Dec 2013 20:54:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2320007=3A_HTTPResponse=2Eread=280=29_no_more_pre?= =?utf-8?q?maturely_closes_connection=2E?= Message-ID: <3dkVRY3d68z7Ll8@mail.python.org> http://hg.python.org/cpython/rev/d032245a122c changeset: 88031:d032245a122c parent: 88026:d489394a73de parent: 88030:47ae858cd661 user: Serhiy Storchaka date: Tue Dec 17 21:51:40 2013 +0200 summary: Issue #20007: HTTPResponse.read(0) no more prematurely closes connection. Original patch by Simon Sapin. files: Lib/http/client.py | 2 +- Lib/test/test_httplib.py | 3 +++ Misc/ACKS | 1 + Misc/NEWS | 3 +++ 4 files changed, 8 insertions(+), 1 deletions(-) diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -538,7 +538,7 @@ # connection, and the user is reading more bytes than will be provided # (for example, reading in 1k chunks) n = self.fp.readinto(b) - if not n: + if not n and b: # Ideally, we would raise IncompleteRead if the content-length # wasn't satisfied, but it might break compatibility. self._close_conn() diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -164,6 +164,9 @@ sock = FakeSocket(body) resp = client.HTTPResponse(sock) resp.begin() + self.assertEqual(resp.read(0), b'') # Issue #20007 + self.assertFalse(resp.isclosed()) + self.assertFalse(resp.closed) self.assertEqual(resp.read(), b"Text") self.assertTrue(resp.isclosed()) self.assertFalse(resp.closed) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1136,6 +1136,7 @@ James Sanders Ilya Sandler Rafael Santos +Simon Sapin Mark Sapiro Ty Sarna Hugh Sasse diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #20007: HTTPResponse.read(0) no more prematurely closes connection. + Original patch by Simon Sapin. + - Issue #19946: multiprocessing now uses runpy to initialize __main__ in child processes when necessary, allowing it to correctly handle scripts without suffixes and submodules that use explicit relative imports or -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 18 00:28:44 2013 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 18 Dec 2013 00:28:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIwMDA2?= =?utf-8?q?=3A_Fix_sporadic_failures_in_test=5Fweakset=2E?= Message-ID: <3dkbC04Q6hz7LjW@mail.python.org> http://hg.python.org/cpython/rev/226c37c209fc changeset: 88032:226c37c209fc branch: 2.7 parent: 88029:ebace0a5a33e user: Antoine Pitrou date: Wed Dec 18 00:28:36 2013 +0100 summary: Issue #20006: Fix sporadic failures in test_weakset. files: Lib/_weakrefset.py | 2 ++ Lib/test/test_weakset.py | 7 ++++++- 2 files changed, 8 insertions(+), 1 deletions(-) diff --git a/Lib/_weakrefset.py b/Lib/_weakrefset.py --- a/Lib/_weakrefset.py +++ b/Lib/_weakrefset.py @@ -60,6 +60,8 @@ for itemref in self.data: item = itemref() if item is not None: + # Caveat: the iterator will keep a strong reference to + # `item` until it is resumed or closed. yield item def __len__(self): diff --git a/Lib/test/test_weakset.py b/Lib/test/test_weakset.py --- a/Lib/test/test_weakset.py +++ b/Lib/test/test_weakset.py @@ -473,9 +473,14 @@ def testcontext(): try: it = iter(s) - next(it) + # Start iterator + yielded = ustr(str(next(it))) # Schedule an item for removal and recreate it u = ustr(str(items.pop())) + if yielded == u: + # The iterator still has a reference to the removed item, + # advance it (issue #20006). + next(it) gc.collect() # just in case yield u finally: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 18 00:32:10 2013 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 18 Dec 2013 00:32:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Revert_misled_?= =?utf-8?q?test_change_in_f189da5bda26=2E?= Message-ID: <3dkbGy5pw1z7LkC@mail.python.org> http://hg.python.org/cpython/rev/f6109aceaf73 changeset: 88033:f6109aceaf73 branch: 3.3 parent: 88030:47ae858cd661 user: Antoine Pitrou date: Wed Dec 18 00:29:30 2013 +0100 summary: Revert misled test change in f189da5bda26. files: Lib/test/test_weakset.py | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_weakset.py b/Lib/test/test_weakset.py --- a/Lib/test/test_weakset.py +++ b/Lib/test/test_weakset.py @@ -371,7 +371,6 @@ try: it = iter(s) next(it) - del it # Schedule an item for removal and recreate it u = ustr(str(items.pop())) gc.collect() # just in case -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 18 00:32:12 2013 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 18 Dec 2013 00:32:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzIwMDA2?= =?utf-8?q?=3A_Fix_sporadic_failures_in_test=5Fweakset=2E?= Message-ID: <3dkbH00R7Hz7Ll8@mail.python.org> http://hg.python.org/cpython/rev/a3d86f80c899 changeset: 88034:a3d86f80c899 branch: 3.3 user: Antoine Pitrou date: Wed Dec 18 00:28:36 2013 +0100 summary: Issue #20006: Fix sporadic failures in test_weakset. files: Lib/_weakrefset.py | 2 ++ Lib/test/test_weakset.py | 7 ++++++- 2 files changed, 8 insertions(+), 1 deletions(-) diff --git a/Lib/_weakrefset.py b/Lib/_weakrefset.py --- a/Lib/_weakrefset.py +++ b/Lib/_weakrefset.py @@ -60,6 +60,8 @@ for itemref in self.data: item = itemref() if item is not None: + # Caveat: the iterator will keep a strong reference to + # `item` until it is resumed or closed. yield item def __len__(self): diff --git a/Lib/test/test_weakset.py b/Lib/test/test_weakset.py --- a/Lib/test/test_weakset.py +++ b/Lib/test/test_weakset.py @@ -370,9 +370,14 @@ def testcontext(): try: it = iter(s) - next(it) + # Start iterator + yielded = ustr(str(next(it))) # Schedule an item for removal and recreate it u = ustr(str(items.pop())) + if yielded == u: + # The iterator still has a reference to the removed item, + # advance it (issue #20006). + next(it) gc.collect() # just in case yield u finally: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 18 00:32:13 2013 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 18 Dec 2013 00:32:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2320006=3A_Fix_sporadic_failures_in_test=5Fweakse?= =?utf-8?q?t=2E?= Message-ID: <3dkbH129Xjz7LlJ@mail.python.org> http://hg.python.org/cpython/rev/26d92a21f6cf changeset: 88035:26d92a21f6cf parent: 88031:d032245a122c parent: 88034:a3d86f80c899 user: Antoine Pitrou date: Wed Dec 18 00:32:02 2013 +0100 summary: Issue #20006: Fix sporadic failures in test_weakset. files: Lib/_weakrefset.py | 2 ++ Lib/test/test_weakset.py | 8 ++++++-- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/Lib/_weakrefset.py b/Lib/_weakrefset.py --- a/Lib/_weakrefset.py +++ b/Lib/_weakrefset.py @@ -60,6 +60,8 @@ for itemref in self.data: item = itemref() if item is not None: + # Caveat: the iterator will keep a strong reference to + # `item` until it is resumed or closed. yield item def __len__(self): diff --git a/Lib/test/test_weakset.py b/Lib/test/test_weakset.py --- a/Lib/test/test_weakset.py +++ b/Lib/test/test_weakset.py @@ -370,10 +370,14 @@ def testcontext(): try: it = iter(s) - next(it) - del it + # Start iterator + yielded = ustr(str(next(it))) # Schedule an item for removal and recreate it u = ustr(str(items.pop())) + if yielded == u: + # The iterator still has a reference to the removed item, + # advance it (issue #20006). + next(it) gc.collect() # just in case yield u finally: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 18 03:33:10 2013 From: python-checkins at python.org (r.david.murray) Date: Wed, 18 Dec 2013 03:33:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzE5ODU1OiB1dWlk?= =?utf-8?q?=2Eget=5Fnode_now_looks_on_the_PATH_for_executables_on_unix=2E?= Message-ID: <3dkgHp6L4Zz7Ljd@mail.python.org> http://hg.python.org/cpython/rev/b0fbaed45956 changeset: 88036:b0fbaed45956 branch: 3.3 parent: 88034:a3d86f80c899 user: R David Murray date: Tue Dec 17 21:13:16 2013 -0500 summary: #19855: uuid.get_node now looks on the PATH for executables on unix. Patch by Serhiy Storchaka. files: Lib/uuid.py | 55 ++++++++++++++++++++-------------------- Misc/NEWS | 4 ++ 2 files changed, 32 insertions(+), 27 deletions(-) diff --git a/Lib/uuid.py b/Lib/uuid.py --- a/Lib/uuid.py +++ b/Lib/uuid.py @@ -312,34 +312,35 @@ return int((self.int >> 76) & 0xf) def _find_mac(command, args, hw_identifiers, get_index): - import os - for dir in ['', '/sbin/', '/usr/sbin']: - executable = os.path.join(dir, command) - if not os.path.exists(executable): - continue + import os, shutil + executable = shutil.which(command) + if executable is None: + path = os.pathsep.join(('/sbin', '/usr/sbin')) + executable = shutil.which(command, path=path) + if executable is None: + return None - try: - # LC_ALL to get English output, 2>/dev/null to - # prevent output on stderr - cmd = 'LC_ALL=C %s %s 2>/dev/null' % (executable, args) - with os.popen(cmd) as pipe: - for line in pipe: - words = line.lower().split() - for i in range(len(words)): - if words[i] in hw_identifiers: - try: - return int( - words[get_index(i)].replace(':', ''), 16) - except (ValueError, IndexError): - # Virtual interfaces, such as those provided by - # VPNs, do not have a colon-delimited MAC address - # as expected, but a 16-byte HWAddr separated by - # dashes. These should be ignored in favor of a - # real MAC address - pass - except IOError: - continue - return None + try: + # LC_MESSAGES to get English output, 2>/dev/null to + # prevent output on stderr + cmd = 'LC_MESSAGES=C %s %s 2>/dev/null' % (executable, args) + with os.popen(cmd) as pipe: + for line in pipe: + words = line.lower().split() + for i in range(len(words)): + if words[i] in hw_identifiers: + try: + return int( + words[get_index(i)].replace(':', ''), 16) + except (ValueError, IndexError): + # Virtual interfaces, such as those provided by + # VPNs, do not have a colon-delimited MAC address + # as expected, but a 16-byte HWAddr separated by + # dashes. These should be ignored in favor of a + # real MAC address + pass + except IOError: + pass def _ifconfig_getnode(): """Get the hardware address on Unix by running ifconfig.""" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,10 @@ Library ------- +- Issue #19855: uuid.getnode() on Unix now looks on the PATH for the + executables used to find the mac address, with /sbin and /usr/sbin as + fallbacks. + - Issue #20007: HTTPResponse.read(0) no more prematurely closes connection. Original patch by Simon Sapin. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 18 03:33:12 2013 From: python-checkins at python.org (r.david.murray) Date: Wed, 18 Dec 2013 03:33:12 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge=3A_=2319855=3A_uuid=2Eget=5Fnode_now_looks_on_the_?= =?utf-8?q?PATH_for_executables_on_unix=2E?= Message-ID: <3dkgHr1G7zz7Ljg@mail.python.org> http://hg.python.org/cpython/rev/2e856fcb9084 changeset: 88037:2e856fcb9084 parent: 88035:26d92a21f6cf parent: 88036:b0fbaed45956 user: R David Murray date: Tue Dec 17 21:14:41 2013 -0500 summary: Merge: #19855: uuid.get_node now looks on the PATH for executables on unix. files: Lib/uuid.py | 55 ++++++++++++++++++++-------------------- Misc/NEWS | 4 ++ 2 files changed, 32 insertions(+), 27 deletions(-) diff --git a/Lib/uuid.py b/Lib/uuid.py --- a/Lib/uuid.py +++ b/Lib/uuid.py @@ -312,34 +312,35 @@ return int((self.int >> 76) & 0xf) def _find_mac(command, args, hw_identifiers, get_index): - import os - for dir in ['', '/sbin/', '/usr/sbin']: - executable = os.path.join(dir, command) - if not os.path.exists(executable): - continue + import os, shutil + executable = shutil.which(command) + if executable is None: + path = os.pathsep.join(('/sbin', '/usr/sbin')) + executable = shutil.which(command, path=path) + if executable is None: + return None - try: - # LC_ALL to get English output, 2>/dev/null to - # prevent output on stderr - cmd = 'LC_ALL=C %s %s 2>/dev/null' % (executable, args) - with os.popen(cmd) as pipe: - for line in pipe: - words = line.lower().split() - for i in range(len(words)): - if words[i] in hw_identifiers: - try: - return int( - words[get_index(i)].replace(':', ''), 16) - except (ValueError, IndexError): - # Virtual interfaces, such as those provided by - # VPNs, do not have a colon-delimited MAC address - # as expected, but a 16-byte HWAddr separated by - # dashes. These should be ignored in favor of a - # real MAC address - pass - except OSError: - continue - return None + try: + # LC_MESSAGES to get English output, 2>/dev/null to + # prevent output on stderr + cmd = 'LC_MESSAGES=C %s %s 2>/dev/null' % (executable, args) + with os.popen(cmd) as pipe: + for line in pipe: + words = line.lower().split() + for i in range(len(words)): + if words[i] in hw_identifiers: + try: + return int( + words[get_index(i)].replace(':', ''), 16) + except (ValueError, IndexError): + # Virtual interfaces, such as those provided by + # VPNs, do not have a colon-delimited MAC address + # as expected, but a 16-byte HWAddr separated by + # dashes. These should be ignored in favor of a + # real MAC address + pass + except OSError: + pass def _ifconfig_getnode(): """Get the hardware address on Unix by running ifconfig.""" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,10 @@ Library ------- +- Issue #19855: uuid.getnode() on Unix now looks on the PATH for the + executables used to find the mac address, with /sbin and /usr/sbin as + fallbacks. + - Issue #20007: HTTPResponse.read(0) no more prematurely closes connection. Original patch by Simon Sapin. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 18 03:33:13 2013 From: python-checkins at python.org (r.david.murray) Date: Wed, 18 Dec 2013 03:33:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogIzE5ODU1OiB1dWlk?= =?utf-8?q?=2Eget=5Fnode_now_looks_on_the_PATH_for_executables_on_unix=2E?= Message-ID: <3dkgHs39VTz7Lk8@mail.python.org> http://hg.python.org/cpython/rev/9f9ae5f7c4ae changeset: 88038:9f9ae5f7c4ae branch: 2.7 parent: 88032:226c37c209fc user: R David Murray date: Tue Dec 17 21:27:56 2013 -0500 summary: #19855: uuid.get_node now looks on the PATH for executables on unix. Patch by Serhiy Storchaka. files: Lib/uuid.py | 55 ++++++++++++++++++++++------------------ Misc/NEWS | 4 ++ 2 files changed, 34 insertions(+), 25 deletions(-) diff --git a/Lib/uuid.py b/Lib/uuid.py --- a/Lib/uuid.py +++ b/Lib/uuid.py @@ -293,33 +293,38 @@ def _find_mac(command, args, hw_identifiers, get_index): import os - for dir in ['', '/sbin/', '/usr/sbin']: + path = os.environ.get("PATH", os.defpath).split(os.pathsep) + path.extend(('/sbin', '/usr/sbin')) + for dir in path: executable = os.path.join(dir, command) - if not os.path.exists(executable): - continue + if (os.path.exists(executable) and + os.access(executable, os.F_OK | os.X_OK) and + not os.path.isdir(executable)): + break + else: + return None - try: - # LC_ALL to get English output, 2>/dev/null to - # prevent output on stderr - cmd = 'LC_ALL=C %s %s 2>/dev/null' % (executable, args) - with os.popen(cmd) as pipe: - for line in pipe: - words = line.lower().split() - for i in range(len(words)): - if words[i] in hw_identifiers: - try: - return int( - words[get_index(i)].replace(':', ''), 16) - except (ValueError, IndexError): - # Virtual interfaces, such as those provided by - # VPNs, do not have a colon-delimited MAC address - # as expected, but a 16-byte HWAddr separated by - # dashes. These should be ignored in favor of a - # real MAC address - pass - except IOError: - continue - return None + try: + # LC_MESSAGES to get English output, 2>/dev/null to + # prevent output on stderr + cmd = 'LC_MESSAGES=C %s %s 2>/dev/null' % (executable, args) + with os.popen(cmd) as pipe: + for line in pipe: + words = line.lower().split() + for i in range(len(words)): + if words[i] in hw_identifiers: + try: + return int( + words[get_index(i)].replace(':', ''), 16) + except (ValueError, IndexError): + # Virtual interfaces, such as those provided by + # VPNs, do not have a colon-delimited MAC address + # as expected, but a 16-byte HWAddr separated by + # dashes. These should be ignored in favor of a + # real MAC address + pass + except IOError: + pass def _ifconfig_getnode(): """Get the hardware address on Unix by running ifconfig.""" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -27,6 +27,10 @@ Library ------- +- Issue #19855: uuid.getnode() on Unix now looks on the PATH for the + executables used to find the mac address, with /sbin and /usr/sbin as + fallbacks. + - Issue #20007: HTTPResponse.read(0) no more prematurely closes connection. Original patch by Simon Sapin. -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Wed Dec 18 09:45:01 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 18 Dec 2013 09:45:01 +0100 Subject: [Python-checkins] Daily reference leaks (26d92a21f6cf): sum=0 Message-ID: results for 26d92a21f6cf on branch "default" -------------------------------------------- test_site leaked [0, 2, -2] references, sum=0 test_site leaked [0, 2, -2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogkfJdgd', '-x'] From python-checkins at python.org Wed Dec 18 15:48:42 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 18 Dec 2013 15:48:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5NDky?= =?utf-8?q?=3A_Silently_skipped_distutils_tests_now_reported_as_skipped=2E?= Message-ID: <3dkzcV582Bz7LjW@mail.python.org> http://hg.python.org/cpython/rev/b3f3e6afe966 changeset: 88039:b3f3e6afe966 branch: 3.3 parent: 88036:b0fbaed45956 user: Serhiy Storchaka date: Wed Dec 18 16:41:01 2013 +0200 summary: Issue #19492: Silently skipped distutils tests now reported as skipped. files: Lib/distutils/tests/test_bdist_rpm.py | 40 ++++------ Lib/distutils/tests/test_build_clib.py | 7 +- Lib/distutils/tests/test_build_ext.py | 8 +- Lib/distutils/tests/test_check.py | 6 +- Lib/distutils/tests/test_config_cmd.py | 3 +- Lib/distutils/tests/test_sdist.py | 10 +- Lib/distutils/tests/test_sysconfig.py | 9 +- Lib/distutils/tests/test_unixccompiler.py | 6 +- 8 files changed, 32 insertions(+), 57 deletions(-) diff --git a/Lib/distutils/tests/test_bdist_rpm.py b/Lib/distutils/tests/test_bdist_rpm.py --- a/Lib/distutils/tests/test_bdist_rpm.py +++ b/Lib/distutils/tests/test_bdist_rpm.py @@ -43,18 +43,15 @@ sys.argv[:] = self.old_sys_argv[1] super(BuildRpmTestCase, self).tearDown() + # XXX I am unable yet to make this test work without + # spurious sdtout/stderr output under Mac OS X + @unittest.skipUnless(sys.platform.startswith('linux'), + 'spurious sdtout/stderr output under Mac OS X') + @unittest.skipIf(find_executable('rpm') is None, + 'the rpm command is not found') + @unittest.skipIf(find_executable('rpmbuild') is None, + 'the rpmbuild command is not found') def test_quiet(self): - - # XXX I am unable yet to make this test work without - # spurious sdtout/stderr output under Mac OS X - if not sys.platform.startswith('linux'): - return - - # this test will run only if the rpm commands are found - if (find_executable('rpm') is None or - find_executable('rpmbuild') is None): - return - # let's create a package tmp_dir = self.mkdtemp() pkg_dir = os.path.join(tmp_dir, 'foo') @@ -87,19 +84,16 @@ self.assertIn(('bdist_rpm', 'any', 'dist/foo-0.1-1.src.rpm'), dist.dist_files) self.assertIn(('bdist_rpm', 'any', 'dist/foo-0.1-1.noarch.rpm'), dist.dist_files) + # XXX I am unable yet to make this test work without + # spurious sdtout/stderr output under Mac OS X + @unittest.skipUnless(sys.platform.startswith('linux'), + 'spurious sdtout/stderr output under Mac OS X') + # http://bugs.python.org/issue1533164 + @unittest.skipIf(find_executable('rpm') is None, + 'the rpm command is not found') + @unittest.skipIf(find_executable('rpmbuild') is None, + 'the rpmbuild command is not found') def test_no_optimize_flag(self): - - # XXX I am unable yet to make this test work without - # spurious sdtout/stderr output under Mac OS X - if not sys.platform.startswith('linux'): - return - - # http://bugs.python.org/issue1533164 - # this test will run only if the rpm command is found - if (find_executable('rpm') is None or - find_executable('rpmbuild') is None): - return - # let's create a package that brakes bdist_rpm tmp_dir = self.mkdtemp() pkg_dir = os.path.join(tmp_dir, 'foo') diff --git a/Lib/distutils/tests/test_build_clib.py b/Lib/distutils/tests/test_build_clib.py --- a/Lib/distutils/tests/test_build_clib.py +++ b/Lib/distutils/tests/test_build_clib.py @@ -102,11 +102,8 @@ cmd.distribution.libraries = 'WONTWORK' self.assertRaises(DistutilsSetupError, cmd.finalize_options) + @unittest.skipIf(sys.platform == 'win32', "can't test on Windows") def test_run(self): - # can't test on windows - if sys.platform == 'win32': - return - pkg_dir, dist = self.create_dist() cmd = build_clib(dist) @@ -131,7 +128,7 @@ if ccmd is None: continue if find_executable(ccmd[0]) is None: - return # can't test + self.skipTest('The %r command is not found' % ccmd[0]) # this should work cmd.run() diff --git a/Lib/distutils/tests/test_build_ext.py b/Lib/distutils/tests/test_build_ext.py --- a/Lib/distutils/tests/test_build_ext.py +++ b/Lib/distutils/tests/test_build_ext.py @@ -61,9 +61,9 @@ sys.stdout = old_stdout if ALREADY_TESTED: - return + self.skipTest('Already tested in %s' % ALREADY_TESTED) else: - ALREADY_TESTED = True + ALREADY_TESTED = type(self).__name__ import xx @@ -113,10 +113,6 @@ self.assertGreater(len(cmd.library_dirs), 0) def test_user_site(self): - # site.USER_SITE was introduced in 2.6 - if sys.version < '2.6': - return - import site dist = Distribution({'name': 'xx'}) cmd = build_ext(dist) diff --git a/Lib/distutils/tests/test_check.py b/Lib/distutils/tests/test_check.py --- a/Lib/distutils/tests/test_check.py +++ b/Lib/distutils/tests/test_check.py @@ -55,9 +55,8 @@ cmd = self._run(metadata) self.assertEqual(cmd._warnings, 0) + @unittest.skipUnless(HAS_DOCUTILS, "won't test without docutils") def test_check_document(self): - if not HAS_DOCUTILS: # won't test without docutils - return pkg_info, dist = self.create_dist() cmd = check(dist) @@ -71,9 +70,8 @@ msgs = cmd._check_rst_data(rest) self.assertEqual(len(msgs), 0) + @unittest.skipUnless(HAS_DOCUTILS, "won't test without docutils") def test_check_restructuredtext(self): - if not HAS_DOCUTILS: # won't test without docutils - return # let's see if it detects broken rest in long_description broken_rest = 'title\n===\n\ntest' pkg_info, dist = self.create_dist(long_description=broken_rest) diff --git a/Lib/distutils/tests/test_config_cmd.py b/Lib/distutils/tests/test_config_cmd.py --- a/Lib/distutils/tests/test_config_cmd.py +++ b/Lib/distutils/tests/test_config_cmd.py @@ -37,9 +37,8 @@ dump_file(this_file, 'I am the header') self.assertEqual(len(self._logs), numlines+1) + @unittest.skipIf(sys.platform == 'win32', "can't test on Windows") def test_search_cpp(self): - if sys.platform == 'win32': - return pkg_dir, dist = self.create_dist() cmd = config(dist) diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py --- a/Lib/distutils/tests/test_sdist.py +++ b/Lib/distutils/tests/test_sdist.py @@ -125,13 +125,11 @@ self.assertEqual(len(content), 4) @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run') + @unittest.skipIf(find_executable('tar') is None, + "The tar command is not found") + @unittest.skipIf(find_executable('gzip') is None, + "The gzip command is not found") def test_make_distribution(self): - - # check if tar and gzip are installed - if (find_executable('tar') is None or - find_executable('gzip') is None): - return - # now building a sdist dist, cmd = self.get_cmd() diff --git a/Lib/distutils/tests/test_sysconfig.py b/Lib/distutils/tests/test_sysconfig.py --- a/Lib/distutils/tests/test_sysconfig.py +++ b/Lib/distutils/tests/test_sysconfig.py @@ -82,12 +82,9 @@ os.chdir(cwd) self.assertEqual(srcdir, srcdir2) + @unittest.skipUnless(get_default_compiler() == 'unix', + 'not testing if default compiler is not unix') def test_customize_compiler(self): - - # not testing if default compiler is not unix - if get_default_compiler() != 'unix': - return - os.environ['AR'] = 'my_ar' os.environ['ARFLAGS'] = '-arflags' @@ -150,7 +147,7 @@ import sysconfig as global_sysconfig if sysconfig.get_config_var('CUSTOMIZED_OSX_COMPILER'): - return + self.skipTest('compiler flags customized') self.assertEqual(global_sysconfig.get_config_var('LDSHARED'), sysconfig.get_config_var('LDSHARED')) self.assertEqual(global_sysconfig.get_config_var('CC'), sysconfig.get_config_var('CC')) diff --git a/Lib/distutils/tests/test_unixccompiler.py b/Lib/distutils/tests/test_unixccompiler.py --- a/Lib/distutils/tests/test_unixccompiler.py +++ b/Lib/distutils/tests/test_unixccompiler.py @@ -21,12 +21,8 @@ sys.platform = self._backup_platform sysconfig.get_config_var = self._backup_get_config_var + @unittest.skipIf(sys.platform == 'win32', "can't test on Windows") def test_runtime_libdir_option(self): - - # not tested under windows - if sys.platform == 'win32': - return - # Issue#5900 # # Ensure RUNPATH is added to extension modules with RPATH if -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 18 15:48:44 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 18 Dec 2013 15:48:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319492=3A_Silently_skipped_distutils_tests_now_r?= =?utf-8?q?eported_as_skipped=2E?= Message-ID: <3dkzcX1QWvz7LkK@mail.python.org> http://hg.python.org/cpython/rev/da3472687566 changeset: 88040:da3472687566 parent: 88037:2e856fcb9084 parent: 88039:b3f3e6afe966 user: Serhiy Storchaka date: Wed Dec 18 16:45:24 2013 +0200 summary: Issue #19492: Silently skipped distutils tests now reported as skipped. files: Lib/distutils/tests/test_bdist_rpm.py | 40 ++++------ Lib/distutils/tests/test_build_clib.py | 7 +- Lib/distutils/tests/test_build_ext.py | 8 +- Lib/distutils/tests/test_check.py | 6 +- Lib/distutils/tests/test_config_cmd.py | 3 +- Lib/distutils/tests/test_sdist.py | 20 ++--- Lib/distutils/tests/test_sysconfig.py | 9 +- Lib/distutils/tests/test_unixccompiler.py | 6 +- 8 files changed, 36 insertions(+), 63 deletions(-) diff --git a/Lib/distutils/tests/test_bdist_rpm.py b/Lib/distutils/tests/test_bdist_rpm.py --- a/Lib/distutils/tests/test_bdist_rpm.py +++ b/Lib/distutils/tests/test_bdist_rpm.py @@ -43,18 +43,15 @@ sys.argv[:] = self.old_sys_argv[1] super(BuildRpmTestCase, self).tearDown() + # XXX I am unable yet to make this test work without + # spurious sdtout/stderr output under Mac OS X + @unittest.skipUnless(sys.platform.startswith('linux'), + 'spurious sdtout/stderr output under Mac OS X') + @unittest.skipIf(find_executable('rpm') is None, + 'the rpm command is not found') + @unittest.skipIf(find_executable('rpmbuild') is None, + 'the rpmbuild command is not found') def test_quiet(self): - - # XXX I am unable yet to make this test work without - # spurious sdtout/stderr output under Mac OS X - if not sys.platform.startswith('linux'): - return - - # this test will run only if the rpm commands are found - if (find_executable('rpm') is None or - find_executable('rpmbuild') is None): - return - # let's create a package tmp_dir = self.mkdtemp() pkg_dir = os.path.join(tmp_dir, 'foo') @@ -87,19 +84,16 @@ self.assertIn(('bdist_rpm', 'any', 'dist/foo-0.1-1.src.rpm'), dist.dist_files) self.assertIn(('bdist_rpm', 'any', 'dist/foo-0.1-1.noarch.rpm'), dist.dist_files) + # XXX I am unable yet to make this test work without + # spurious sdtout/stderr output under Mac OS X + @unittest.skipUnless(sys.platform.startswith('linux'), + 'spurious sdtout/stderr output under Mac OS X') + # http://bugs.python.org/issue1533164 + @unittest.skipIf(find_executable('rpm') is None, + 'the rpm command is not found') + @unittest.skipIf(find_executable('rpmbuild') is None, + 'the rpmbuild command is not found') def test_no_optimize_flag(self): - - # XXX I am unable yet to make this test work without - # spurious sdtout/stderr output under Mac OS X - if not sys.platform.startswith('linux'): - return - - # http://bugs.python.org/issue1533164 - # this test will run only if the rpm command is found - if (find_executable('rpm') is None or - find_executable('rpmbuild') is None): - return - # let's create a package that brakes bdist_rpm tmp_dir = self.mkdtemp() pkg_dir = os.path.join(tmp_dir, 'foo') diff --git a/Lib/distutils/tests/test_build_clib.py b/Lib/distutils/tests/test_build_clib.py --- a/Lib/distutils/tests/test_build_clib.py +++ b/Lib/distutils/tests/test_build_clib.py @@ -102,11 +102,8 @@ cmd.distribution.libraries = 'WONTWORK' self.assertRaises(DistutilsSetupError, cmd.finalize_options) + @unittest.skipIf(sys.platform == 'win32', "can't test on Windows") def test_run(self): - # can't test on windows - if sys.platform == 'win32': - return - pkg_dir, dist = self.create_dist() cmd = build_clib(dist) @@ -131,7 +128,7 @@ if ccmd is None: continue if find_executable(ccmd[0]) is None: - return # can't test + self.skipTest('The %r command is not found' % ccmd[0]) # this should work cmd.run() diff --git a/Lib/distutils/tests/test_build_ext.py b/Lib/distutils/tests/test_build_ext.py --- a/Lib/distutils/tests/test_build_ext.py +++ b/Lib/distutils/tests/test_build_ext.py @@ -61,9 +61,9 @@ sys.stdout = old_stdout if ALREADY_TESTED: - return + self.skipTest('Already tested in %s' % ALREADY_TESTED) else: - ALREADY_TESTED = True + ALREADY_TESTED = type(self).__name__ import xx @@ -113,10 +113,6 @@ self.assertGreater(len(cmd.library_dirs), 0) def test_user_site(self): - # site.USER_SITE was introduced in 2.6 - if sys.version < '2.6': - return - import site dist = Distribution({'name': 'xx'}) cmd = build_ext(dist) diff --git a/Lib/distutils/tests/test_check.py b/Lib/distutils/tests/test_check.py --- a/Lib/distutils/tests/test_check.py +++ b/Lib/distutils/tests/test_check.py @@ -55,9 +55,8 @@ cmd = self._run(metadata) self.assertEqual(cmd._warnings, 0) + @unittest.skipUnless(HAS_DOCUTILS, "won't test without docutils") def test_check_document(self): - if not HAS_DOCUTILS: # won't test without docutils - return pkg_info, dist = self.create_dist() cmd = check(dist) @@ -71,9 +70,8 @@ msgs = cmd._check_rst_data(rest) self.assertEqual(len(msgs), 0) + @unittest.skipUnless(HAS_DOCUTILS, "won't test without docutils") def test_check_restructuredtext(self): - if not HAS_DOCUTILS: # won't test without docutils - return # let's see if it detects broken rest in long_description broken_rest = 'title\n===\n\ntest' pkg_info, dist = self.create_dist(long_description=broken_rest) diff --git a/Lib/distutils/tests/test_config_cmd.py b/Lib/distutils/tests/test_config_cmd.py --- a/Lib/distutils/tests/test_config_cmd.py +++ b/Lib/distutils/tests/test_config_cmd.py @@ -37,9 +37,8 @@ dump_file(this_file, 'I am the header') self.assertEqual(len(self._logs), numlines+1) + @unittest.skipIf(sys.platform == 'win32', "can't test on Windows") def test_search_cpp(self): - if sys.platform == 'win32': - return pkg_dir, dist = self.create_dist() cmd = config(dist) diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py --- a/Lib/distutils/tests/test_sdist.py +++ b/Lib/distutils/tests/test_sdist.py @@ -131,13 +131,11 @@ self.assertEqual(len(content), 4) @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run') + @unittest.skipIf(find_executable('tar') is None, + "The tar command is not found") + @unittest.skipIf(find_executable('gzip') is None, + "The gzip command is not found") def test_make_distribution(self): - - # check if tar and gzip are installed - if (find_executable('tar') is None or - find_executable('gzip') is None): - return - # now building a sdist dist, cmd = self.get_cmd() @@ -433,13 +431,11 @@ @unittest.skipUnless(zlib, "requires zlib") @unittest.skipUnless(UID_GID_SUPPORT, "Requires grp and pwd support") + @unittest.skipIf(find_executable('tar') is None, + "The tar command is not found") + @unittest.skipIf(find_executable('gzip') is None, + "The gzip command is not found") def test_make_distribution_owner_group(self): - - # check if tar and gzip are installed - if (find_executable('tar') is None or - find_executable('gzip') is None): - return - # now building a sdist dist, cmd = self.get_cmd() diff --git a/Lib/distutils/tests/test_sysconfig.py b/Lib/distutils/tests/test_sysconfig.py --- a/Lib/distutils/tests/test_sysconfig.py +++ b/Lib/distutils/tests/test_sysconfig.py @@ -80,12 +80,9 @@ os.chdir(cwd) self.assertEqual(srcdir, srcdir2) + @unittest.skipUnless(get_default_compiler() == 'unix', + 'not testing if default compiler is not unix') def test_customize_compiler(self): - - # not testing if default compiler is not unix - if get_default_compiler() != 'unix': - return - os.environ['AR'] = 'my_ar' os.environ['ARFLAGS'] = '-arflags' @@ -151,7 +148,7 @@ import sysconfig as global_sysconfig if sysconfig.get_config_var('CUSTOMIZED_OSX_COMPILER'): - return + self.skipTest('compiler flags customized') self.assertEqual(global_sysconfig.get_config_var('LDSHARED'), sysconfig.get_config_var('LDSHARED')) self.assertEqual(global_sysconfig.get_config_var('CC'), diff --git a/Lib/distutils/tests/test_unixccompiler.py b/Lib/distutils/tests/test_unixccompiler.py --- a/Lib/distutils/tests/test_unixccompiler.py +++ b/Lib/distutils/tests/test_unixccompiler.py @@ -21,12 +21,8 @@ sys.platform = self._backup_platform sysconfig.get_config_var = self._backup_get_config_var + @unittest.skipIf(sys.platform == 'win32', "can't test on Windows") def test_runtime_libdir_option(self): - - # not tested under windows - if sys.platform == 'win32': - return - # Issue#5900 # # Ensure RUNPATH is added to extension modules with RPATH if -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 18 15:48:45 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 18 Dec 2013 15:48:45 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5NDky?= =?utf-8?q?=3A_Silently_skipped_distutils_tests_now_reported_as_skipped=2E?= Message-ID: <3dkzcY4hltz7LkC@mail.python.org> http://hg.python.org/cpython/rev/d5b0bb2a1790 changeset: 88041:d5b0bb2a1790 branch: 2.7 parent: 88038:9f9ae5f7c4ae user: Serhiy Storchaka date: Wed Dec 18 16:45:37 2013 +0200 summary: Issue #19492: Silently skipped distutils tests now reported as skipped. files: Lib/distutils/tests/test_bdist_rpm.py | 40 ++++------ Lib/distutils/tests/test_build_clib.py | 7 +- Lib/distutils/tests/test_build_ext.py | 13 +-- Lib/distutils/tests/test_ccompiler.py | 7 +- Lib/distutils/tests/test_check.py | 6 +- Lib/distutils/tests/test_config_cmd.py | 3 +- Lib/distutils/tests/test_install.py | 6 +- Lib/distutils/tests/test_sdist.py | 16 +--- Lib/distutils/tests/test_sysconfig.py | 2 +- Lib/distutils/tests/test_unixccompiler.py | 6 +- 10 files changed, 37 insertions(+), 69 deletions(-) diff --git a/Lib/distutils/tests/test_bdist_rpm.py b/Lib/distutils/tests/test_bdist_rpm.py --- a/Lib/distutils/tests/test_bdist_rpm.py +++ b/Lib/distutils/tests/test_bdist_rpm.py @@ -39,18 +39,15 @@ sys.argv[:] = self.old_sys_argv[1] super(BuildRpmTestCase, self).tearDown() + # XXX I am unable yet to make this test work without + # spurious sdtout/stderr output under Mac OS X + @unittest.skipUnless(sys.platform.startswith('linux'), + 'spurious sdtout/stderr output under Mac OS X') + @unittest.skipIf(find_executable('rpm') is None, + 'the rpm command is not found') + @unittest.skipIf(find_executable('rpmbuild') is None, + 'the rpmbuild command is not found') def test_quiet(self): - - # XXX I am unable yet to make this test work without - # spurious sdtout/stderr output under Mac OS X - if sys.platform != 'linux2': - return - - # this test will run only if the rpm commands are found - if (find_executable('rpm') is None or - find_executable('rpmbuild') is None): - return - # let's create a package tmp_dir = self.mkdtemp() pkg_dir = os.path.join(tmp_dir, 'foo') @@ -83,19 +80,16 @@ self.assertIn(('bdist_rpm', 'any', 'dist/foo-0.1-1.src.rpm'), dist.dist_files) self.assertIn(('bdist_rpm', 'any', 'dist/foo-0.1-1.noarch.rpm'), dist.dist_files) + # XXX I am unable yet to make this test work without + # spurious sdtout/stderr output under Mac OS X + @unittest.skipUnless(sys.platform.startswith('linux'), + 'spurious sdtout/stderr output under Mac OS X') + # http://bugs.python.org/issue1533164 + @unittest.skipIf(find_executable('rpm') is None, + 'the rpm command is not found') + @unittest.skipIf(find_executable('rpmbuild') is None, + 'the rpmbuild command is not found') def test_no_optimize_flag(self): - - # XXX I am unable yet to make this test work without - # spurious sdtout/stderr output under Mac OS X - if sys.platform != 'linux2': - return - - # http://bugs.python.org/issue1533164 - # this test will run only if the rpm command is found - if (find_executable('rpm') is None or - find_executable('rpmbuild') is None): - return - # let's create a package that brakes bdist_rpm tmp_dir = self.mkdtemp() pkg_dir = os.path.join(tmp_dir, 'foo') diff --git a/Lib/distutils/tests/test_build_clib.py b/Lib/distutils/tests/test_build_clib.py --- a/Lib/distutils/tests/test_build_clib.py +++ b/Lib/distutils/tests/test_build_clib.py @@ -102,11 +102,8 @@ cmd.distribution.libraries = 'WONTWORK' self.assertRaises(DistutilsSetupError, cmd.finalize_options) + @unittest.skipIf(sys.platform == 'win32', "can't test on Windows") def test_run(self): - # can't test on windows - if sys.platform == 'win32': - return - pkg_dir, dist = self.create_dist() cmd = build_clib(dist) @@ -131,7 +128,7 @@ if ccmd is None: continue if find_executable(ccmd[0]) is None: - return # can't test + self.skipTest('The %r command is not found' % ccmd[0]) # this should work cmd.run() diff --git a/Lib/distutils/tests/test_build_ext.py b/Lib/distutils/tests/test_build_ext.py --- a/Lib/distutils/tests/test_build_ext.py +++ b/Lib/distutils/tests/test_build_ext.py @@ -65,9 +65,9 @@ sys.stdout = old_stdout if ALREADY_TESTED: - return + self.skipTest('Already tested in %s' % ALREADY_TESTED) else: - ALREADY_TESTED = True + ALREADY_TESTED = type(self).__name__ import xx @@ -104,11 +104,9 @@ # make sure we get some library dirs under solaris self.assertGreater(len(cmd.library_dirs), 0) + @unittest.skipIf(sys.version < '2.6', + 'site.USER_SITE was introduced in 2.6') def test_user_site(self): - # site.USER_SITE was introduced in 2.6 - if sys.version < '2.6': - return - import site dist = Distribution({'name': 'xx'}) cmd = build_ext(dist) @@ -414,9 +412,8 @@ wanted = os.path.join(cmd.build_lib, 'UpdateManager', 'fdsend' + ext) self.assertEqual(ext_path, wanted) + @unittest.skipUnless(sys.platform == 'win32', 'these tests require Windows') def test_build_ext_path_cross_platform(self): - if sys.platform != 'win32': - return dist = Distribution({'name': 'UpdateManager'}) cmd = build_ext(dist) cmd.ensure_finalized() diff --git a/Lib/distutils/tests/test_ccompiler.py b/Lib/distutils/tests/test_ccompiler.py --- a/Lib/distutils/tests/test_ccompiler.py +++ b/Lib/distutils/tests/test_ccompiler.py @@ -55,12 +55,9 @@ finally: debug.DEBUG = False + @unittest.skipUnless(get_default_compiler() == 'unix', + 'not testing if default compiler is not unix') def test_customize_compiler(self): - - # not testing if default compiler is not unix - if get_default_compiler() != 'unix': - return - os.environ['AR'] = 'my_ar' os.environ['ARFLAGS'] = '-arflags' diff --git a/Lib/distutils/tests/test_check.py b/Lib/distutils/tests/test_check.py --- a/Lib/distutils/tests/test_check.py +++ b/Lib/distutils/tests/test_check.py @@ -56,9 +56,8 @@ cmd = self._run(metadata) self.assertEqual(cmd._warnings, 0) + @unittest.skipUnless(HAS_DOCUTILS, "won't test without docutils") def test_check_document(self): - if not HAS_DOCUTILS: # won't test without docutils - return pkg_info, dist = self.create_dist() cmd = check(dist) @@ -72,9 +71,8 @@ msgs = cmd._check_rst_data(rest) self.assertEqual(len(msgs), 0) + @unittest.skipUnless(HAS_DOCUTILS, "won't test without docutils") def test_check_restructuredtext(self): - if not HAS_DOCUTILS: # won't test without docutils - return # let's see if it detects broken rest in long_description broken_rest = 'title\n===\n\ntest' pkg_info, dist = self.create_dist(long_description=broken_rest) diff --git a/Lib/distutils/tests/test_config_cmd.py b/Lib/distutils/tests/test_config_cmd.py --- a/Lib/distutils/tests/test_config_cmd.py +++ b/Lib/distutils/tests/test_config_cmd.py @@ -37,9 +37,8 @@ dump_file(this_file, 'I am the header') self.assertEqual(len(self._logs), numlines+1) + @unittest.skipIf(sys.platform == 'win32', "can't test on Windows") def test_search_cpp(self): - if sys.platform == 'win32': - return pkg_dir, dist = self.create_dist() cmd = config(dist) diff --git a/Lib/distutils/tests/test_install.py b/Lib/distutils/tests/test_install.py --- a/Lib/distutils/tests/test_install.py +++ b/Lib/distutils/tests/test_install.py @@ -65,11 +65,9 @@ check_path(cmd.install_scripts, os.path.join(destination, "bin")) check_path(cmd.install_data, destination) + @unittest.skipIf(sys.version < '2.6', + 'site.USER_SITE was introduced in 2.6') def test_user_site(self): - # site.USER_SITE was introduced in 2.6 - if sys.version < '2.6': - return - # preparing the environment for the test self.old_user_base = site.USER_BASE self.old_user_site = site.USER_SITE diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py --- a/Lib/distutils/tests/test_sdist.py +++ b/Lib/distutils/tests/test_sdist.py @@ -134,12 +134,6 @@ @unittest.skipUnless(zlib, "requires zlib") def test_make_distribution(self): - - # check if tar and gzip are installed - if (find_executable('tar') is None or - find_executable('gzip') is None): - return - # now building a sdist dist, cmd = self.get_cmd() @@ -325,13 +319,11 @@ @unittest.skipUnless(zlib, "requires zlib") @unittest.skipUnless(UID_GID_SUPPORT, "Requires grp and pwd support") + @unittest.skipIf(find_executable('tar') is None, + "The tar command is not found") + @unittest.skipIf(find_executable('gzip') is None, + "The gzip command is not found") def test_make_distribution_owner_group(self): - - # check if tar and gzip are installed - if (find_executable('tar') is None or - find_executable('gzip') is None): - return - # now building a sdist dist, cmd = self.get_cmd() diff --git a/Lib/distutils/tests/test_sysconfig.py b/Lib/distutils/tests/test_sysconfig.py --- a/Lib/distutils/tests/test_sysconfig.py +++ b/Lib/distutils/tests/test_sysconfig.py @@ -95,7 +95,7 @@ import sysconfig as global_sysconfig if sysconfig.get_config_var('CUSTOMIZED_OSX_COMPILER'): - return + self.skipTest('compiler flags customized') self.assertEqual(global_sysconfig.get_config_var('LDSHARED'), sysconfig.get_config_var('LDSHARED')) self.assertEqual(global_sysconfig.get_config_var('CC'), sysconfig.get_config_var('CC')) diff --git a/Lib/distutils/tests/test_unixccompiler.py b/Lib/distutils/tests/test_unixccompiler.py --- a/Lib/distutils/tests/test_unixccompiler.py +++ b/Lib/distutils/tests/test_unixccompiler.py @@ -21,12 +21,8 @@ sys.platform = self._backup_platform sysconfig.get_config_var = self._backup_get_config_var + @unittest.skipIf(sys.platform == 'win32', "can't test on Windows") def test_runtime_libdir_option(self): - - # not tested under windows - if sys.platform == 'win32': - return - # Issue#5900 # # Ensure RUNPATH is added to extension modules with RPATH if -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 18 19:22:49 2013 From: python-checkins at python.org (zach.ware) Date: Wed, 18 Dec 2013 19:22:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIwMDA1?= =?utf-8?q?=3A_Fix_typo_in_operator_docs=2E__Patch_by_Claudiu_Popa=2E?= Message-ID: <3dl4MY5gBRz7Ljs@mail.python.org> http://hg.python.org/cpython/rev/f0eed36bab4c changeset: 88042:f0eed36bab4c branch: 2.7 user: Zachary Ware date: Wed Dec 18 12:18:36 2013 -0600 summary: Issue #20005: Fix typo in operator docs. Patch by Claudiu Popa. files: Doc/library/operator.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/operator.rst b/Doc/library/operator.rst --- a/Doc/library/operator.rst +++ b/Doc/library/operator.rst @@ -503,7 +503,7 @@ ``(b.name, b.date)``. * After ``f = attrgetter('name.first', 'name.last')``, the call ``f(b)`` - returns ``(r.name.first, r.name.last)``. + returns ``(b.name.first, b.name.last)``. Equivalent to:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 18 19:22:51 2013 From: python-checkins at python.org (zach.ware) Date: Wed, 18 Dec 2013 19:22:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzIwMDA1?= =?utf-8?q?=3A_Fix_typo_in_operator_docs=2E__Patch_by_Claudiu_Popa=2E?= Message-ID: <3dl4Mb0LXpz7Lk8@mail.python.org> http://hg.python.org/cpython/rev/17244b5df8b6 changeset: 88043:17244b5df8b6 branch: 3.3 parent: 88039:b3f3e6afe966 user: Zachary Ware date: Wed Dec 18 12:21:49 2013 -0600 summary: Issue #20005: Fix typo in operator docs. Patch by Claudiu Popa. files: Doc/library/operator.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/operator.rst b/Doc/library/operator.rst --- a/Doc/library/operator.rst +++ b/Doc/library/operator.rst @@ -254,7 +254,7 @@ ``(b.name, b.date)``. * After ``f = attrgetter('name.first', 'name.last')``, the call ``f(b)`` - returns ``(r.name.first, r.name.last)``. + returns ``(b.name.first, b.name.last)``. Equivalent to:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 18 19:22:52 2013 From: python-checkins at python.org (zach.ware) Date: Wed, 18 Dec 2013 19:22:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Closes_=2320005=3A_Fix_typo_in_operator_docs=2E__Patch_b?= =?utf-8?q?y_Claudiu_Popa=2E?= Message-ID: <3dl4Mc2FThz7Lk4@mail.python.org> http://hg.python.org/cpython/rev/a4a00e220e08 changeset: 88044:a4a00e220e08 parent: 88040:da3472687566 parent: 88043:17244b5df8b6 user: Zachary Ware date: Wed Dec 18 12:22:35 2013 -0600 summary: Closes #20005: Fix typo in operator docs. Patch by Claudiu Popa. files: Doc/library/operator.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/operator.rst b/Doc/library/operator.rst --- a/Doc/library/operator.rst +++ b/Doc/library/operator.rst @@ -265,7 +265,7 @@ ``(b.name, b.date)``. * After ``f = attrgetter('name.first', 'name.last')``, the call ``f(b)`` - returns ``(r.name.first, r.name.last)``. + returns ``(b.name.first, b.name.last)``. Equivalent to:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 18 20:25:51 2013 From: python-checkins at python.org (gregory.p.smith) Date: Wed, 18 Dec 2013 20:25:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_remove_trailin?= =?utf-8?q?g_spaces=2E?= Message-ID: <3dl5mH6RkSz7Lly@mail.python.org> http://hg.python.org/cpython/rev/0e7c4203bd79 changeset: 88045:0e7c4203bd79 branch: 2.7 parent: 88042:f0eed36bab4c user: Gregory P. Smith date: Wed Dec 18 11:25:26 2013 -0800 summary: remove trailing spaces. files: Modules/gcmodule.c | 14 +++++++------- 1 files changed, 7 insertions(+), 7 deletions(-) diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c --- a/Modules/gcmodule.c +++ b/Modules/gcmodule.c @@ -113,7 +113,7 @@ /* NOTE: about untracking of mutable objects. - + Certain types of container cannot participate in a reference cycle, and so do not need to be tracked by the garbage collector. Untracking these objects reduces the cost of garbage collections. However, determining @@ -131,10 +131,10 @@ not survive until garbage collection. It is therefore not worthwhile to untrack eligible tuples at creation time. - Instead, all tuples except the empty tuple are tracked when created. - During garbage collection it is determined whether any surviving tuples - can be untracked. A tuple can be untracked if all of its contents are - already not tracked. Tuples are examined for untracking in all garbage + Instead, all tuples except the empty tuple are tracked when created. + During garbage collection it is determined whether any surviving tuples + can be untracked. A tuple can be untracked if all of its contents are + already not tracked. Tuples are examined for untracking in all garbage collection cycles. It may take more than one cycle to untrack a tuple. Dictionaries containing only immutable objects also do not need to be @@ -147,8 +147,8 @@ The module provides the python function is_tracked(obj), which returns the CURRENT tracking status of the object. Subsequent garbage collections may change the tracking status of the object. - - Untracking of certain containers was introduced in issue #4688, and + + Untracking of certain containers was introduced in issue #4688, and the algorithm was refined in response to issue #14775. */ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 18 20:27:31 2013 From: python-checkins at python.org (gregory.p.smith) Date: Wed, 18 Dec 2013 20:27:31 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_remove_trailin?= =?utf-8?q?g_spaces=2E?= Message-ID: <3dl5pC4Lz3z7Lkq@mail.python.org> http://hg.python.org/cpython/rev/7c53d5511b0d changeset: 88046:7c53d5511b0d branch: 3.3 parent: 88043:17244b5df8b6 user: Gregory P. Smith date: Wed Dec 18 11:27:05 2013 -0800 summary: remove trailing spaces. files: Modules/gcmodule.c | 14 +++++++------- 1 files changed, 7 insertions(+), 7 deletions(-) diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c --- a/Modules/gcmodule.c +++ b/Modules/gcmodule.c @@ -118,7 +118,7 @@ /* NOTE: about untracking of mutable objects. - + Certain types of container cannot participate in a reference cycle, and so do not need to be tracked by the garbage collector. Untracking these objects reduces the cost of garbage collections. However, determining @@ -136,10 +136,10 @@ not survive until garbage collection. It is therefore not worthwhile to untrack eligible tuples at creation time. - Instead, all tuples except the empty tuple are tracked when created. - During garbage collection it is determined whether any surviving tuples - can be untracked. A tuple can be untracked if all of its contents are - already not tracked. Tuples are examined for untracking in all garbage + Instead, all tuples except the empty tuple are tracked when created. + During garbage collection it is determined whether any surviving tuples + can be untracked. A tuple can be untracked if all of its contents are + already not tracked. Tuples are examined for untracking in all garbage collection cycles. It may take more than one cycle to untrack a tuple. Dictionaries containing only immutable objects also do not need to be @@ -152,8 +152,8 @@ The module provides the python function is_tracked(obj), which returns the CURRENT tracking status of the object. Subsequent garbage collections may change the tracking status of the object. - - Untracking of certain containers was introduced in issue #4688, and + + Untracking of certain containers was introduced in issue #4688, and the algorithm was refined in response to issue #14775. */ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 18 20:27:32 2013 From: python-checkins at python.org (gregory.p.smith) Date: Wed, 18 Dec 2013 20:27:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_remove_trailing_spaces?= Message-ID: <3dl5pD6lPYz7LkC@mail.python.org> http://hg.python.org/cpython/rev/ce89c5c3c786 changeset: 88047:ce89c5c3c786 parent: 88044:a4a00e220e08 parent: 88046:7c53d5511b0d user: Gregory P. Smith date: Wed Dec 18 11:27:21 2013 -0800 summary: remove trailing spaces files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 18 22:37:20 2013 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 18 Dec 2013 22:37:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_update_url_to_?= =?utf-8?q?spec_=28closes_=2320018=29?= Message-ID: <3dl8h00jnMz7Lk8@mail.python.org> http://hg.python.org/cpython/rev/656a40666937 changeset: 88048:656a40666937 branch: 3.3 parent: 88046:7c53d5511b0d user: Benjamin Peterson date: Wed Dec 18 15:35:18 2013 -0600 summary: update url to spec (closes #20018) files: Lib/http/cookiejar.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/http/cookiejar.py b/Lib/http/cookiejar.py --- a/Lib/http/cookiejar.py +++ b/Lib/http/cookiejar.py @@ -1981,7 +1981,7 @@ magic_re = re.compile("#( Netscape)? HTTP Cookie File") header = """\ # Netscape HTTP Cookie File -# http://www.netscape.com/newsref/std/cookie_spec.html +# http://curl.haxx.se/rfc/cookie_spec.html # This is a generated file! Do not edit. """ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 18 22:37:21 2013 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 18 Dec 2013 22:37:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_update_url_to_?= =?utf-8?q?spec_=28closes_=2320018=29?= Message-ID: <3dl8h12SMsz7LkV@mail.python.org> http://hg.python.org/cpython/rev/c0dc1400866a changeset: 88049:c0dc1400866a branch: 2.7 parent: 88045:0e7c4203bd79 user: Benjamin Peterson date: Wed Dec 18 15:36:34 2013 -0600 summary: update url to spec (closes #20018) files: Lib/_MozillaCookieJar.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/_MozillaCookieJar.py b/Lib/_MozillaCookieJar.py --- a/Lib/_MozillaCookieJar.py +++ b/Lib/_MozillaCookieJar.py @@ -39,7 +39,7 @@ magic_re = "#( Netscape)? HTTP Cookie File" header = """\ # Netscape HTTP Cookie File -# http://www.netscape.com/newsref/std/cookie_spec.html +# http://curl.haxx.se/rfc/cookie_spec.html # This is a generated file! Do not edit. """ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 18 22:37:22 2013 From: python-checkins at python.org (benjamin.peterson) Date: Wed, 18 Dec 2013 22:37:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4zICgjMjAwMTgp?= Message-ID: <3dl8h24Bk1z7Lkh@mail.python.org> http://hg.python.org/cpython/rev/5f5e1d408c0c changeset: 88050:5f5e1d408c0c parent: 88047:ce89c5c3c786 parent: 88048:656a40666937 user: Benjamin Peterson date: Wed Dec 18 15:37:03 2013 -0600 summary: merge 3.3 (#20018) files: Lib/http/cookiejar.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/http/cookiejar.py b/Lib/http/cookiejar.py --- a/Lib/http/cookiejar.py +++ b/Lib/http/cookiejar.py @@ -1973,7 +1973,7 @@ magic_re = re.compile("#( Netscape)? HTTP Cookie File") header = """\ # Netscape HTTP Cookie File -# http://www.netscape.com/newsref/std/cookie_spec.html +# http://curl.haxx.se/rfc/cookie_spec.html # This is a generated file! Do not edit. """ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 07:36:13 2013 From: python-checkins at python.org (eric.snow) Date: Thu, 19 Dec 2013 07:36:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Remove_some_dead_code_in_i?= =?utf-8?q?mportlib=2C_introduced_with_the_PEP_451_patch=2E?= Message-ID: <3dlNdn2WTrz7Llq@mail.python.org> http://hg.python.org/cpython/rev/6f41495e9336 changeset: 88051:6f41495e9336 user: Eric Snow date: Wed Dec 18 23:35:15 2013 -0700 summary: Remove some dead code in importlib, introduced with the PEP 451 patch. Early in the PEP 451 implementation some of the importlib loaders had their own _get_spec() methods to simplify accommodating them. However, later implementations removed the need. They simply failed to remove this code at the same time. :) files: Lib/importlib/_bootstrap.py | 11 +- Python/importlib.h | 1966 +++++++++++----------- 2 files changed, 983 insertions(+), 994 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -1934,15 +1934,10 @@ return None, [] return spec.loader, spec.submodule_search_locations or [] - def _get_spec(self, loader_class, fullname, path, submodule_search_locations, target): + def _get_spec(self, loader_class, fullname, path, smsl, target): loader = loader_class(fullname, path) - try: - get_spec = loader._get_spec - except AttributeError: - return spec_from_file_location(fullname, path, loader=loader, - submodule_search_locations=submodule_search_locations) - else: - return get_spec(fullname, path, submodule_search_locations, target) + return spec_from_file_location(fullname, path, loader=loader, + submodule_search_locations=smsl) def find_spec(self, fullname, target=None): """Try to find a loader for the specified module, or the namespace diff --git a/Python/importlib.h b/Python/importlib.h --- a/Python/importlib.h +++ b/Python/importlib.h [stripped] -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Thu Dec 19 09:47:10 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 19 Dec 2013 09:47:10 +0100 Subject: [Python-checkins] Daily reference leaks (5f5e1d408c0c): sum=-12 Message-ID: results for 5f5e1d408c0c on branch "default" -------------------------------------------- test_site leaked [-2, 2, 0] references, sum=0 test_site leaked [-2, 2, 0] memory blocks, sum=0 test_urllib2net leaked [-11, 0, 0] references, sum=-11 test_urllib2net leaked [-2, 1, 0] memory blocks, sum=-1 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/refloglmVwBj', '-x'] From python-checkins at python.org Thu Dec 19 12:51:35 2013 From: python-checkins at python.org (vinay.sajip) Date: Thu, 19 Dec 2013 12:51:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5OTAy?= =?utf-8?q?=3A_Added_list_of_logging_levels=2E?= Message-ID: <3dlWdg1mlNz7LkT@mail.python.org> http://hg.python.org/cpython/rev/16bfddf5a091 changeset: 88052:16bfddf5a091 branch: 2.7 parent: 88049:c0dc1400866a user: Vinay Sajip date: Thu Dec 19 11:42:18 2013 +0000 summary: Issue #19902: Added list of logging levels. files: Doc/library/logging.rst | 32 +++++++++++++++++++++++++++++ 1 files changed, 32 insertions(+), 0 deletions(-) diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -114,6 +114,8 @@ If the root is reached, and it has a level of NOTSET, then all messages will be processed. Otherwise, the root's level will be used as the effective level. + See :ref:`levels` for a list of levels. + .. method:: Logger.isEnabledFor(lvl) @@ -281,6 +283,35 @@ .. versionchanged:: 2.5 *func* and *extra* were added. + +.. _levels: + +Logging Levels +-------------- + +The numeric values of logging levels are given in the following table. These are +primarily of interest if you want to define your own levels, and need them to +have specific values relative to the predefined levels. If you define a level +with the same numeric value, it overwrites the predefined value; the predefined +name is lost. + ++--------------+---------------+ +| Level | Numeric value | ++==============+===============+ +| ``CRITICAL`` | 50 | ++--------------+---------------+ +| ``ERROR`` | 40 | ++--------------+---------------+ +| ``WARNING`` | 30 | ++--------------+---------------+ +| ``INFO`` | 20 | ++--------------+---------------+ +| ``DEBUG`` | 10 | ++--------------+---------------+ +| ``NOTSET`` | 0 | ++--------------+---------------+ + + .. _handler: Handler Objects @@ -321,6 +352,7 @@ severe than *lvl* will be ignored. When a handler is created, the level is set to :const:`NOTSET` (which causes all messages to be processed). + See :ref:`levels` for a list of levels. .. method:: Handler.setFormatter(form) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 12:51:36 2013 From: python-checkins at python.org (vinay.sajip) Date: Thu, 19 Dec 2013 12:51:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5OTAy?= =?utf-8?q?=3A_Added_list_of_logging_levels=2E?= Message-ID: <3dlWdh3YkZz7Ll4@mail.python.org> http://hg.python.org/cpython/rev/e812094d42f9 changeset: 88053:e812094d42f9 branch: 3.3 parent: 88048:656a40666937 user: Vinay Sajip date: Thu Dec 19 11:50:24 2013 +0000 summary: Issue #19902: Added list of logging levels. files: Doc/library/logging.rst | 32 +++++++++++++++++++++++++++++ 1 files changed, 32 insertions(+), 0 deletions(-) diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -113,6 +113,8 @@ If the root is reached, and it has a level of NOTSET, then all messages will be processed. Otherwise, the root's level will be used as the effective level. + See :ref:`levels` for a list of levels. + .. versionchanged:: 3.2 The *lvl* parameter now accepts a string representation of the level such as 'INFO' as an alternative to the integer constants @@ -316,6 +318,34 @@ .. versionadded:: 3.2 +.. _levels: + +Logging Levels +-------------- + +The numeric values of logging levels are given in the following table. These are +primarily of interest if you want to define your own levels, and need them to +have specific values relative to the predefined levels. If you define a level +with the same numeric value, it overwrites the predefined value; the predefined +name is lost. + ++--------------+---------------+ +| Level | Numeric value | ++==============+===============+ +| ``CRITICAL`` | 50 | ++--------------+---------------+ +| ``ERROR`` | 40 | ++--------------+---------------+ +| ``WARNING`` | 30 | ++--------------+---------------+ +| ``INFO`` | 20 | ++--------------+---------------+ +| ``DEBUG`` | 10 | ++--------------+---------------+ +| ``NOTSET`` | 0 | ++--------------+---------------+ + + .. _handler: Handler Objects @@ -356,6 +386,8 @@ severe than *lvl* will be ignored. When a handler is created, the level is set to :const:`NOTSET` (which causes all messages to be processed). + See :ref:`levels` for a list of levels. + .. versionchanged:: 3.2 The *lvl* parameter now accepts a string representation of the level such as 'INFO' as an alternative to the integer constants -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 12:51:37 2013 From: python-checkins at python.org (vinay.sajip) Date: Thu, 19 Dec 2013 12:51:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Closes_=2319902=3A_Merged_update_from_3=2E3=2E?= Message-ID: <3dlWdj5KYDz7Lkv@mail.python.org> http://hg.python.org/cpython/rev/45bd58a15bb9 changeset: 88054:45bd58a15bb9 parent: 88051:6f41495e9336 parent: 88053:e812094d42f9 user: Vinay Sajip date: Thu Dec 19 11:51:19 2013 +0000 summary: Closes #19902: Merged update from 3.3. files: Doc/library/logging.rst | 32 +++++++++++++++++++++++++++++ 1 files changed, 32 insertions(+), 0 deletions(-) diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -113,6 +113,8 @@ If the root is reached, and it has a level of NOTSET, then all messages will be processed. Otherwise, the root's level will be used as the effective level. + See :ref:`levels` for a list of levels. + .. versionchanged:: 3.2 The *lvl* parameter now accepts a string representation of the level such as 'INFO' as an alternative to the integer constants @@ -316,6 +318,34 @@ .. versionadded:: 3.2 +.. _levels: + +Logging Levels +-------------- + +The numeric values of logging levels are given in the following table. These are +primarily of interest if you want to define your own levels, and need them to +have specific values relative to the predefined levels. If you define a level +with the same numeric value, it overwrites the predefined value; the predefined +name is lost. + ++--------------+---------------+ +| Level | Numeric value | ++==============+===============+ +| ``CRITICAL`` | 50 | ++--------------+---------------+ +| ``ERROR`` | 40 | ++--------------+---------------+ +| ``WARNING`` | 30 | ++--------------+---------------+ +| ``INFO`` | 20 | ++--------------+---------------+ +| ``DEBUG`` | 10 | ++--------------+---------------+ +| ``NOTSET`` | 0 | ++--------------+---------------+ + + .. _handler: Handler Objects @@ -356,6 +386,8 @@ severe than *lvl* will be ignored. When a handler is created, the level is set to :const:`NOTSET` (which causes all messages to be processed). + See :ref:`levels` for a list of levels. + .. versionchanged:: 3.2 The *lvl* parameter now accepts a string representation of the level such as 'INFO' as an alternative to the integer constants -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 12:54:34 2013 From: python-checkins at python.org (nick.coghlan) Date: Thu, 19 Dec 2013 12:54:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319946=3A_appropri?= =?utf-8?q?ately_skip_new_multiprocessing_tests?= Message-ID: <3dlWj65zN6z7LkT@mail.python.org> http://hg.python.org/cpython/rev/460961e80e31 changeset: 88055:460961e80e31 user: Nick Coghlan date: Thu Dec 19 21:53:31 2013 +1000 summary: Issue #19946: appropriately skip new multiprocessing tests Thanks to Christian Heimes for noting the buildbot failures and to Zachary Ware for providing the patch to make the new tests play nice with both other platforms and unittest test discovery files: Lib/test/test_multiprocessing_main_handling.py | 14 +++------ 1 files changed, 5 insertions(+), 9 deletions(-) diff --git a/Lib/test/test_multiprocessing_main_handling.py b/Lib/test/test_multiprocessing_main_handling.py --- a/Lib/test/test_multiprocessing_main_handling.py +++ b/Lib/test/test_multiprocessing_main_handling.py @@ -130,10 +130,9 @@ class MultiProcessingCmdLineMixin(): maxDiff = None # Show full tracebacks on subprocess failure - def setupClass(cls): - if cls.start_method not in _concrete_contexts: - raise unittest.SkipTest("%r start method not available" % - cls.start_method) + def setUp(self): + if self.start_method not in _concrete_contexts: + self.skipTest("%r start method not available" % self.start_method) def _check_output(self, script_name, exit_code, out, err): if verbose > 1: @@ -277,11 +276,8 @@ start_method = 'forkserver' main_in_children_source = test_source_main_skipped_in_children -def test_main(): - support.run_unittest(SpawnCmdLineTest, - ForkCmdLineTest, - ForkServerCmdLineTest) +def tearDownModule(): support.reap_children() if __name__ == '__main__': - test_main() + unittest.main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 13:25:51 2013 From: python-checkins at python.org (victor.stinner) Date: Thu, 19 Dec 2013 13:25:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbjogRml4IG9zLmxpc3RkaXIoKTog?= =?utf-8?q?=5FPy=5Fdup=28=29_already_raises_an_exception_on_error=2C_no_ne?= =?utf-8?q?ed_to?= Message-ID: <3dlXPC0bmgz7Lk8@mail.python.org> http://hg.python.org/cpython/rev/a454ca39b10a changeset: 88056:a454ca39b10a user: Victor Stinner date: Thu Dec 19 13:24:49 2013 +0100 summary: Fix os.listdir(): _Py_dup() already raises an exception on error, no need to raise a new exception files: Modules/posixmodule.c | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-) diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -3746,10 +3746,8 @@ if (path->fd != -1) { /* closedir() closes the FD, so we duplicate it */ fd = _Py_dup(path->fd); - if (fd == -1) { - list = posix_error(); - goto exit; - } + if (fd == -1) + return NULL; return_str = 1; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 13:40:00 2013 From: python-checkins at python.org (victor.stinner) Date: Thu, 19 Dec 2013 13:40:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_=5FPy=5FDisplaySourceL?= =?utf-8?q?ine=28=29=2C_if_PyTokenizer=5FFindEncodingFilename=28=29_fails?= =?utf-8?q?=2C_clear?= Message-ID: <3dlXjX63rYz7Lkv@mail.python.org> http://hg.python.org/cpython/rev/298d98486794 changeset: 88057:298d98486794 user: Victor Stinner date: Thu Dec 19 13:39:32 2013 +0100 summary: Fix _Py_DisplaySourceLine(), if PyTokenizer_FindEncodingFilename() fails, clear the exception to not call open() with an exception set. files: Python/traceback.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Python/traceback.c b/Python/traceback.c --- a/Python/traceback.c +++ b/Python/traceback.c @@ -264,6 +264,8 @@ return 0; } found_encoding = PyTokenizer_FindEncodingFilename(fd, filename); + if (found_encoding == NULL) + PyErr_Clear(); encoding = (found_encoding != NULL) ? found_encoding : "utf-8"; /* Reset position */ if (lseek(fd, 0, SEEK_SET) == (off_t)-1) { -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 15:30:26 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 19 Dec 2013 15:30:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE4ODI5?= =?utf-8?q?=3A_csv=2EDialect=28=29_now_checks_type_for_delimiter=2C_escape?= =?utf-8?q?char_and?= Message-ID: <3dlb8y4x30z7Ljm@mail.python.org> http://hg.python.org/cpython/rev/5ed75e36be8e changeset: 88058:5ed75e36be8e branch: 2.7 parent: 88052:16bfddf5a091 user: Serhiy Storchaka date: Thu Dec 19 16:26:56 2013 +0200 summary: Issue #18829: csv.Dialect() now checks type for delimiter, escapechar and quotechar fields. Original patch by Vajrasky Kok. files: Lib/test/test_csv.py | 44 ++++++++++++++++++++++++++++--- Misc/NEWS | 3 ++ Modules/_csv.c | 34 ++++++++++++++---------- 3 files changed, 62 insertions(+), 19 deletions(-) diff --git a/Lib/test/test_csv.py b/Lib/test/test_csv.py --- a/Lib/test/test_csv.py +++ b/Lib/test/test_csv.py @@ -870,6 +870,7 @@ lineterminator = '\r\n' quoting = csv.QUOTE_NONE d = mydialect() + self.assertEqual(d.quoting, csv.QUOTE_NONE) mydialect.quoting = None self.assertRaises(csv.Error, mydialect) @@ -878,12 +879,21 @@ mydialect.quoting = csv.QUOTE_ALL mydialect.quotechar = '"' d = mydialect() + self.assertEqual(d.quoting, csv.QUOTE_ALL) + self.assertEqual(d.quotechar, '"') + self.assertTrue(d.doublequote) mydialect.quotechar = "''" - self.assertRaises(csv.Error, mydialect) + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"quotechar" must be an 1-character string') mydialect.quotechar = 4 - self.assertRaises(csv.Error, mydialect) + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"quotechar" must be string, not int') def test_delimiter(self): class mydialect(csv.Dialect): @@ -894,12 +904,31 @@ lineterminator = '\r\n' quoting = csv.QUOTE_NONE d = mydialect() + self.assertEqual(d.delimiter, ";") mydialect.delimiter = ":::" - self.assertRaises(csv.Error, mydialect) + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"delimiter" must be an 1-character string') + + mydialect.delimiter = "" + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"delimiter" must be an 1-character string') + + mydialect.delimiter = u"," + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"delimiter" must be string, not unicode') mydialect.delimiter = 4 - self.assertRaises(csv.Error, mydialect) + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"delimiter" must be string, not int') def test_lineterminator(self): class mydialect(csv.Dialect): @@ -910,12 +939,17 @@ lineterminator = '\r\n' quoting = csv.QUOTE_NONE d = mydialect() + self.assertEqual(d.lineterminator, '\r\n') mydialect.lineterminator = ":::" d = mydialect() + self.assertEqual(d.lineterminator, ":::") mydialect.lineterminator = 4 - self.assertRaises(csv.Error, mydialect) + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"lineterminator" must be a string') class TestSniffer(unittest.TestCase): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -27,6 +27,9 @@ Library ------- +- Issue #18829: csv.Dialect() now checks type for delimiter, escapechar and + quotechar fields. Original patch by Vajrasky Kok. + - Issue #19855: uuid.getnode() on Unix now looks on the PATH for the executables used to find the mac address, with /sbin and /usr/sbin as fallbacks. diff --git a/Modules/_csv.c b/Modules/_csv.c --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -239,19 +239,24 @@ if (src == NULL) *target = dflt; else { - if (src == Py_None || PyString_Size(src) == 0) - *target = '\0'; - else if (!PyString_Check(src) || PyString_Size(src) != 1) { - PyErr_Format(PyExc_TypeError, - "\"%s\" must be an 1-character string", - name); - return -1; - } - else { - char *s = PyString_AsString(src); - if (s == NULL) + *target = '\0'; + if (src != Py_None) { + Py_ssize_t len; + if (!PyString_Check(src)) { + PyErr_Format(PyExc_TypeError, + "\"%s\" must be string, not %.200s", name, + src->ob_type->tp_name); return -1; - *target = s[0]; + } + len = PyString_GET_SIZE(src); + if (len > 1) { + PyErr_Format(PyExc_TypeError, + "\"%s\" must be an 1-character string", + name); + return -1; + } + if (len > 0) + *target = *PyString_AS_STRING(src); } } return 0; @@ -267,7 +272,7 @@ *target = NULL; else if (!IS_BASESTRING(src)) { PyErr_Format(PyExc_TypeError, - "\"%s\" must be an string", name); + "\"%s\" must be a string", name); return -1; } else { @@ -426,7 +431,8 @@ if (dialect_check_quoting(self->quoting)) goto err; if (self->delimiter == 0) { - PyErr_SetString(PyExc_TypeError, "delimiter must be set"); + PyErr_SetString(PyExc_TypeError, + "\"delimiter\" must be an 1-character string"); goto err; } if (quotechar == Py_None && quoting == NULL) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 15:30:28 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 19 Dec 2013 15:30:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE4ODI5?= =?utf-8?q?=3A_csv=2EDialect=28=29_now_checks_type_for_delimiter=2C_escape?= =?utf-8?q?char_and?= Message-ID: <3dlb900qjfz7Lkc@mail.python.org> http://hg.python.org/cpython/rev/52d03fbdf67a changeset: 88059:52d03fbdf67a branch: 3.3 parent: 88053:e812094d42f9 user: Serhiy Storchaka date: Thu Dec 19 16:27:18 2013 +0200 summary: Issue #18829: csv.Dialect() now checks type for delimiter, escapechar and quotechar fields. Original patch by Vajrasky Kok. files: Lib/test/test_csv.py | 44 ++++++++++++++++++++++++++++--- Misc/NEWS | 3 ++ Modules/_csv.c | 9 +++++- 3 files changed, 50 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_csv.py b/Lib/test/test_csv.py --- a/Lib/test/test_csv.py +++ b/Lib/test/test_csv.py @@ -747,6 +747,7 @@ lineterminator = '\r\n' quoting = csv.QUOTE_NONE d = mydialect() + self.assertEqual(d.quoting, csv.QUOTE_NONE) mydialect.quoting = None self.assertRaises(csv.Error, mydialect) @@ -755,12 +756,21 @@ mydialect.quoting = csv.QUOTE_ALL mydialect.quotechar = '"' d = mydialect() + self.assertEqual(d.quoting, csv.QUOTE_ALL) + self.assertEqual(d.quotechar, '"') + self.assertTrue(d.doublequote) mydialect.quotechar = "''" - self.assertRaises(csv.Error, mydialect) + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"quotechar" must be an 1-character string') mydialect.quotechar = 4 - self.assertRaises(csv.Error, mydialect) + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"quotechar" must be string, not int') def test_delimiter(self): class mydialect(csv.Dialect): @@ -771,12 +781,31 @@ lineterminator = '\r\n' quoting = csv.QUOTE_NONE d = mydialect() + self.assertEqual(d.delimiter, ";") mydialect.delimiter = ":::" - self.assertRaises(csv.Error, mydialect) + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"delimiter" must be an 1-character string') + + mydialect.delimiter = "" + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"delimiter" must be an 1-character string') + + mydialect.delimiter = b"," + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"delimiter" must be string, not bytes') mydialect.delimiter = 4 - self.assertRaises(csv.Error, mydialect) + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"delimiter" must be string, not int') def test_lineterminator(self): class mydialect(csv.Dialect): @@ -787,12 +816,17 @@ lineterminator = '\r\n' quoting = csv.QUOTE_NONE d = mydialect() + self.assertEqual(d.lineterminator, '\r\n') mydialect.lineterminator = ":::" d = mydialect() + self.assertEqual(d.lineterminator, ":::") mydialect.lineterminator = 4 - self.assertRaises(csv.Error, mydialect) + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"lineterminator" must be a string') class TestSniffer(unittest.TestCase): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,9 @@ Library ------- +- Issue #18829: csv.Dialect() now checks type for delimiter, escapechar and + quotechar fields. Original patch by Vajrasky Kok. + - Issue #19855: uuid.getnode() on Unix now looks on the PATH for the executables used to find the mac address, with /sbin and /usr/sbin as fallbacks. diff --git a/Modules/_csv.c b/Modules/_csv.c --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -239,6 +239,12 @@ *target = '\0'; if (src != Py_None) { Py_ssize_t len; + if (!PyUnicode_Check(src)) { + PyErr_Format(PyExc_TypeError, + "\"%s\" must be string, not %.200s", name, + src->ob_type->tp_name); + return -1; + } len = PyUnicode_GetLength(src); if (len > 1) { PyErr_Format(PyExc_TypeError, @@ -425,7 +431,8 @@ if (dialect_check_quoting(self->quoting)) goto err; if (self->delimiter == 0) { - PyErr_SetString(PyExc_TypeError, "delimiter must be set"); + PyErr_SetString(PyExc_TypeError, + "\"delimiter\" must be an 1-character string"); goto err; } if (quotechar == Py_None && quoting == NULL) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 15:30:29 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 19 Dec 2013 15:30:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2318829=3A_csv=2EDialect=28=29_now_checks_type_fo?= =?utf-8?q?r_delimiter=2C_escapechar_and?= Message-ID: <3dlb913sH4z7LmT@mail.python.org> http://hg.python.org/cpython/rev/6b17803bfddd changeset: 88060:6b17803bfddd parent: 88057:298d98486794 parent: 88059:52d03fbdf67a user: Serhiy Storchaka date: Thu Dec 19 16:28:04 2013 +0200 summary: Issue #18829: csv.Dialect() now checks type for delimiter, escapechar and quotechar fields. Original patch by Vajrasky Kok. files: Lib/test/test_csv.py | 44 ++++++++++++++++++++++++++++--- Misc/NEWS | 3 ++ Modules/_csv.c | 9 +++++- 3 files changed, 50 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_csv.py b/Lib/test/test_csv.py --- a/Lib/test/test_csv.py +++ b/Lib/test/test_csv.py @@ -756,6 +756,7 @@ lineterminator = '\r\n' quoting = csv.QUOTE_NONE d = mydialect() + self.assertEqual(d.quoting, csv.QUOTE_NONE) mydialect.quoting = None self.assertRaises(csv.Error, mydialect) @@ -764,12 +765,21 @@ mydialect.quoting = csv.QUOTE_ALL mydialect.quotechar = '"' d = mydialect() + self.assertEqual(d.quoting, csv.QUOTE_ALL) + self.assertEqual(d.quotechar, '"') + self.assertTrue(d.doublequote) mydialect.quotechar = "''" - self.assertRaises(csv.Error, mydialect) + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"quotechar" must be an 1-character string') mydialect.quotechar = 4 - self.assertRaises(csv.Error, mydialect) + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"quotechar" must be string, not int') def test_delimiter(self): class mydialect(csv.Dialect): @@ -780,12 +790,31 @@ lineterminator = '\r\n' quoting = csv.QUOTE_NONE d = mydialect() + self.assertEqual(d.delimiter, ";") mydialect.delimiter = ":::" - self.assertRaises(csv.Error, mydialect) + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"delimiter" must be an 1-character string') + + mydialect.delimiter = "" + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"delimiter" must be an 1-character string') + + mydialect.delimiter = b"," + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"delimiter" must be string, not bytes') mydialect.delimiter = 4 - self.assertRaises(csv.Error, mydialect) + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"delimiter" must be string, not int') def test_lineterminator(self): class mydialect(csv.Dialect): @@ -796,12 +825,17 @@ lineterminator = '\r\n' quoting = csv.QUOTE_NONE d = mydialect() + self.assertEqual(d.lineterminator, '\r\n') mydialect.lineterminator = ":::" d = mydialect() + self.assertEqual(d.lineterminator, ":::") mydialect.lineterminator = 4 - self.assertRaises(csv.Error, mydialect) + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"lineterminator" must be a string') class TestSniffer(unittest.TestCase): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #18829: csv.Dialect() now checks type for delimiter, escapechar and + quotechar fields. Original patch by Vajrasky Kok. + - Issue #19855: uuid.getnode() on Unix now looks on the PATH for the executables used to find the mac address, with /sbin and /usr/sbin as fallbacks. diff --git a/Modules/_csv.c b/Modules/_csv.c --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -239,6 +239,12 @@ *target = '\0'; if (src != Py_None) { Py_ssize_t len; + if (!PyUnicode_Check(src)) { + PyErr_Format(PyExc_TypeError, + "\"%s\" must be string, not %.200s", name, + src->ob_type->tp_name); + return -1; + } len = PyUnicode_GetLength(src); if (len > 1) { PyErr_Format(PyExc_TypeError, @@ -425,7 +431,8 @@ if (dialect_check_quoting(self->quoting)) goto err; if (self->delimiter == 0) { - PyErr_SetString(PyExc_TypeError, "delimiter must be set"); + PyErr_SetString(PyExc_TypeError, + "\"delimiter\" must be an 1-character string"); goto err; } if (quotechar == Py_None && quoting == NULL) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 16:41:56 2013 From: python-checkins at python.org (victor.stinner) Date: Thu, 19 Dec 2013 16:41:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Better_assertion_in_PyObje?= =?utf-8?q?ct=5FCall=28=29_to_detect_functions_returning_a_result_with?= Message-ID: <3dlclS5qs2z7Lkc@mail.python.org> http://hg.python.org/cpython/rev/07331638ac85 changeset: 88061:07331638ac85 parent: 88057:298d98486794 user: Victor Stinner date: Thu Dec 19 13:47:35 2013 +0100 summary: Better assertion in PyObject_Call() to detect functions returning a result with an exception set (invalid state). files: Objects/abstract.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Objects/abstract.c b/Objects/abstract.c --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -2073,7 +2073,8 @@ "NULL result without error in PyObject_Call"); } #else - assert(result != NULL || PyErr_Occurred()); + assert((result != NULL && !PyErr_Occurred()) + || (result == NULL && PyErr_Occurred())); #endif return result; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 16:41:58 2013 From: python-checkins at python.org (victor.stinner) Date: Thu, 19 Dec 2013 16:41:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzIwMDI2?= =?utf-8?q?=3A_Fix_the_sqlite_module_to_handle_correctly_invalid_isolation?= =?utf-8?q?_level?= Message-ID: <3dlclV0mpfz7Llq@mail.python.org> http://hg.python.org/cpython/rev/11a161cf0e5d changeset: 88062:11a161cf0e5d branch: 3.3 parent: 88059:52d03fbdf67a user: Victor Stinner date: Thu Dec 19 16:38:03 2013 +0100 summary: Issue #20026: Fix the sqlite module to handle correctly invalid isolation level (wrong type). files: Lib/sqlite3/test/regression.py | 5 +++++ Misc/NEWS | 3 +++ Modules/_sqlite/connection.c | 5 ++++- 3 files changed, 12 insertions(+), 1 deletions(-) diff --git a/Lib/sqlite3/test/regression.py b/Lib/sqlite3/test/regression.py --- a/Lib/sqlite3/test/regression.py +++ b/Lib/sqlite3/test/regression.py @@ -330,6 +330,11 @@ datetime.datetime(2012, 4, 4, 15, 6, 0, 123456), ]) + def CheckInvalidIsolationLevelType(self): + # isolation level is a string, not an integer + self.assertRaises(TypeError, + sqlite.connect, ":memory:", isolation_level=123) + def suite(): regression_suite = unittest.makeSuite(RegressionTests, "Check") diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,9 @@ Library ------- +- Issue #20026: Fix the sqlite module to handle correctly invalid isolation + level (wrong type). + - Issue #18829: csv.Dialect() now checks type for delimiter, escapechar and quotechar fields. Original patch by Vajrasky Kok. diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -109,7 +109,10 @@ Py_INCREF(isolation_level); } self->isolation_level = NULL; - pysqlite_connection_set_isolation_level(self, isolation_level); + if (pysqlite_connection_set_isolation_level(self, isolation_level) < 0) { + Py_DECREF(isolation_level); + return -1; + } Py_DECREF(isolation_level); self->statement_cache = (pysqlite_Cache*)PyObject_CallFunction((PyObject*)&pysqlite_CacheType, "Oi", self, cached_statements); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 16:41:59 2013 From: python-checkins at python.org (victor.stinner) Date: Thu, 19 Dec 2013 16:41:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_=28Merge_3=2E3=29_Issue_=2320026=3A_Fix_the_sqlite_modul?= =?utf-8?q?e_to_handle_correctly_invalid?= Message-ID: <3dlclW2mSqz7LkL@mail.python.org> http://hg.python.org/cpython/rev/f9b6c8ef55b6 changeset: 88063:f9b6c8ef55b6 parent: 88060:6b17803bfddd parent: 88062:11a161cf0e5d user: Victor Stinner date: Thu Dec 19 16:39:00 2013 +0100 summary: (Merge 3.3) Issue #20026: Fix the sqlite module to handle correctly invalid isolation level (wrong type). files: Lib/sqlite3/test/regression.py | 5 +++++ Misc/NEWS | 3 +++ Modules/_sqlite/connection.c | 5 ++++- 3 files changed, 12 insertions(+), 1 deletions(-) diff --git a/Lib/sqlite3/test/regression.py b/Lib/sqlite3/test/regression.py --- a/Lib/sqlite3/test/regression.py +++ b/Lib/sqlite3/test/regression.py @@ -330,6 +330,11 @@ datetime.datetime(2012, 4, 4, 15, 6, 0, 123456), ]) + def CheckInvalidIsolationLevelType(self): + # isolation level is a string, not an integer + self.assertRaises(TypeError, + sqlite.connect, ":memory:", isolation_level=123) + def suite(): regression_suite = unittest.makeSuite(RegressionTests, "Check") diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #20026: Fix the sqlite module to handle correctly invalid isolation + level (wrong type). + - Issue #18829: csv.Dialect() now checks type for delimiter, escapechar and quotechar fields. Original patch by Vajrasky Kok. diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -128,7 +128,10 @@ Py_INCREF(isolation_level); } self->isolation_level = NULL; - pysqlite_connection_set_isolation_level(self, isolation_level); + if (pysqlite_connection_set_isolation_level(self, isolation_level) < 0) { + Py_DECREF(isolation_level); + return -1; + } Py_DECREF(isolation_level); self->statement_cache = (pysqlite_Cache*)PyObject_CallFunction((PyObject*)&pysqlite_CacheType, "Oi", self, cached_statements); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 16:42:00 2013 From: python-checkins at python.org (victor.stinner) Date: Thu, 19 Dec 2013 16:42:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?q?=29=3A_Merge_heads?= Message-ID: <3dlclX4b4Cz7LmT@mail.python.org> http://hg.python.org/cpython/rev/2b6802d8bee1 changeset: 88064:2b6802d8bee1 parent: 88063:f9b6c8ef55b6 parent: 88061:07331638ac85 user: Victor Stinner date: Thu Dec 19 16:41:22 2013 +0100 summary: Merge heads files: Objects/abstract.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Objects/abstract.c b/Objects/abstract.c --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -2073,7 +2073,8 @@ "NULL result without error in PyObject_Call"); } #else - assert(result != NULL || PyErr_Occurred()); + assert((result != NULL && !PyErr_Occurred()) + || (result == NULL && PyErr_Occurred())); #endif return result; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 16:45:06 2013 From: python-checkins at python.org (victor.stinner) Date: Thu, 19 Dec 2013 16:45:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIwMDI2?= =?utf-8?q?=3A_Fix_the_sqlite_module_to_handle_correctly_invalid_isolation?= =?utf-8?q?_level?= Message-ID: <3dlcq610y3z7LjY@mail.python.org> http://hg.python.org/cpython/rev/572e4b054899 changeset: 88065:572e4b054899 branch: 2.7 parent: 88058:5ed75e36be8e user: Victor Stinner date: Thu Dec 19 16:44:48 2013 +0100 summary: Issue #20026: Fix the sqlite module to handle correctly invalid isolation level (wrong type). files: Lib/sqlite3/test/regression.py | 5 +++++ Misc/NEWS | 3 +++ Modules/_sqlite/connection.c | 5 ++++- 3 files changed, 12 insertions(+), 1 deletions(-) diff --git a/Lib/sqlite3/test/regression.py b/Lib/sqlite3/test/regression.py --- a/Lib/sqlite3/test/regression.py +++ b/Lib/sqlite3/test/regression.py @@ -313,6 +313,11 @@ datetime.datetime(2012, 4, 4, 15, 6, 0, 123456), ]) + def CheckInvalidIsolationLevelType(self): + # isolation level is a string, not an integer + self.assertRaises(TypeError, + sqlite.connect, ":memory:", isolation_level=123) + def suite(): regression_suite = unittest.makeSuite(RegressionTests, "Check") diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -27,6 +27,9 @@ Library ------- +- Issue #20026: Fix the sqlite module to handle correctly invalid isolation + level (wrong type). + - Issue #18829: csv.Dialect() now checks type for delimiter, escapechar and quotechar fields. Original patch by Vajrasky Kok. diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -152,7 +152,10 @@ Py_INCREF(isolation_level); } self->isolation_level = NULL; - pysqlite_connection_set_isolation_level(self, isolation_level); + if (pysqlite_connection_set_isolation_level(self, isolation_level) < 0) { + Py_DECREF(isolation_level); + return -1; + } Py_DECREF(isolation_level); self->statement_cache = (pysqlite_Cache*)PyObject_CallFunction((PyObject*)&pysqlite_CacheType, "Oi", self, cached_statements); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 16:47:51 2013 From: python-checkins at python.org (victor.stinner) Date: Thu, 19 Dec 2013 16:47:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzIwMDI1?= =?utf-8?b?OiBzc2wuUkFORF9ieXRlcygpIGFuZCBzc2wuUkFORF9wc2V1ZG9fYnl0ZXMo?= =?utf-8?q?=29_now_raise_a?= Message-ID: <3dlctH3Xhpz7LkL@mail.python.org> http://hg.python.org/cpython/rev/68ec8949dbf1 changeset: 88066:68ec8949dbf1 branch: 3.3 parent: 88062:11a161cf0e5d user: Victor Stinner date: Thu Dec 19 16:47:04 2013 +0100 summary: Issue #20025: ssl.RAND_bytes() and ssl.RAND_pseudo_bytes() now raise a ValueError if num is negative (instead of raising a SystemError). files: Lib/test/test_ssl.py | 4 ++++ Modules/_ssl.c | 5 +++++ 2 files changed, 9 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -126,6 +126,10 @@ else: self.assertRaises(ssl.SSLError, ssl.RAND_bytes, 16) + # negative num is invalid + self.assertRaises(ValueError, ssl.RAND_bytes, -5) + self.assertRaises(ValueError, ssl.RAND_pseudo_bytes, -5) + self.assertRaises(TypeError, ssl.RAND_egd, 1) self.assertRaises(TypeError, ssl.RAND_egd, 'foo', 1) ssl.RAND_add("this is a random string", 75.0) diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -2486,6 +2486,11 @@ const char *errstr; PyObject *v; + if (len < 0) { + PyErr_SetString(PyExc_ValueError, "num must be positive"); + return NULL; + } + bytes = PyBytes_FromStringAndSize(NULL, len); if (bytes == NULL) return NULL; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 16:47:52 2013 From: python-checkins at python.org (victor.stinner) Date: Thu, 19 Dec 2013 16:47:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogKE1lcmdlIDMuMykgSXNzdWUgIzIwMDI1OiBzc2wuUkFORF9ieXRlcygp?= =?utf-8?q?_and_ssl=2ERAND=5Fpseudo=5Fbytes=28=29_now?= Message-ID: <3dlctJ6Njnz7LlV@mail.python.org> http://hg.python.org/cpython/rev/c1d2c90ece99 changeset: 88067:c1d2c90ece99 parent: 88064:2b6802d8bee1 parent: 88066:68ec8949dbf1 user: Victor Stinner date: Thu Dec 19 16:47:25 2013 +0100 summary: (Merge 3.3) Issue #20025: ssl.RAND_bytes() and ssl.RAND_pseudo_bytes() now raise a ValueError if num is negative (instead of raising a SystemError). files: Lib/test/test_ssl.py | 4 ++++ Modules/_ssl.c | 5 +++++ 2 files changed, 9 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -150,6 +150,10 @@ else: self.assertRaises(ssl.SSLError, ssl.RAND_bytes, 16) + # negative num is invalid + self.assertRaises(ValueError, ssl.RAND_bytes, -5) + self.assertRaises(ValueError, ssl.RAND_pseudo_bytes, -5) + self.assertRaises(TypeError, ssl.RAND_egd, 1) self.assertRaises(TypeError, ssl.RAND_egd, 'foo', 1) ssl.RAND_add("this is a random string", 75.0) diff --git a/Modules/_ssl.c b/Modules/_ssl.c --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -3244,6 +3244,11 @@ const char *errstr; PyObject *v; + if (len < 0) { + PyErr_SetString(PyExc_ValueError, "num must be positive"); + return NULL; + } + bytes = PyBytes_FromStringAndSize(NULL, len); if (bytes == NULL) return NULL; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 17:17:04 2013 From: python-checkins at python.org (victor.stinner) Date: Thu, 19 Dec 2013 17:17:04 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_the_C_definition_of_th?= =?utf-8?q?e_sys=2E=5Fdebugmallocstats=28=29_function=3A_the_function_has?= Message-ID: <3dldX026hTz7LjY@mail.python.org> http://hg.python.org/cpython/rev/2b31b7eff99c changeset: 88068:2b31b7eff99c user: Victor Stinner date: Thu Dec 19 17:16:42 2013 +0100 summary: Fix the C definition of the sys._debugmallocstats() function: the function has no parameter files: Lib/test/test_sys.py | 3 +++ Python/sysmodule.c | 2 +- 2 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -665,6 +665,9 @@ ret, out, err = assert_python_ok(*args) self.assertIn(b"free PyDictObjects", err) + # The function has no parameter + self.assertRaises(TypeError, sys._debugmallocstats, True) + @unittest.skipUnless(hasattr(sys, "getallocatedblocks"), "sys.getallocatedblocks unavailable on this build") def test_getallocatedblocks(self): diff --git a/Python/sysmodule.c b/Python/sysmodule.c --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -1186,7 +1186,7 @@ {"settrace", sys_settrace, METH_O, settrace_doc}, {"gettrace", sys_gettrace, METH_NOARGS, gettrace_doc}, {"call_tracing", sys_call_tracing, METH_VARARGS, call_tracing_doc}, - {"_debugmallocstats", sys_debugmallocstats, METH_VARARGS, + {"_debugmallocstats", sys_debugmallocstats, METH_NOARGS, debugmallocstats_doc}, {NULL, NULL} /* sentinel */ }; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 20:21:55 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 19 Dec 2013 20:21:55 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzU4MTU6?= =?utf-8?q?_Fixed_support_for_locales_with_modifiers=2E__Fixed_support_for?= Message-ID: <3dljdH4QFBz7Ljh@mail.python.org> http://hg.python.org/cpython/rev/3d805bee06e2 changeset: 88069:3d805bee06e2 branch: 2.7 parent: 88065:572e4b054899 user: Serhiy Storchaka date: Thu Dec 19 21:21:06 2013 +0200 summary: Issue #5815: Fixed support for locales with modifiers. Fixed support for locale encodings with hyphens. files: Lib/locale.py | 114 +++++++++++++++++---------- Lib/test/test_locale.py | 58 ++++++++++++++ Misc/NEWS | 3 + 3 files changed, 134 insertions(+), 41 deletions(-) diff --git a/Lib/locale.py b/Lib/locale.py --- a/Lib/locale.py +++ b/Lib/locale.py @@ -344,6 +344,22 @@ for x in range(256) ) +def _replace_encoding(code, encoding): + if '.' in code: + langname = code[:code.index('.')] + else: + langname = code + # Convert the encoding to a C lib compatible encoding string + norm_encoding = encodings.normalize_encoding(encoding) + #print('norm encoding: %r' % norm_encoding) + norm_encoding = encodings.aliases.aliases.get(norm_encoding, + norm_encoding) + #print('aliased encoding: %r' % norm_encoding) + encoding = locale_encoding_alias.get(norm_encoding, + norm_encoding) + #print('found encoding %r' % encoding) + return langname + '.' + encoding + def normalize(localename): """ Returns a normalized locale code for the given locale @@ -360,57 +376,73 @@ does. """ - # Normalize the locale name and extract the encoding + # Normalize the locale name and extract the encoding and modifier if isinstance(localename, _unicode): localename = localename.encode('ascii') - fullname = localename.translate(_ascii_lower_map) - if ':' in fullname: + code = localename.lower() + if ':' in code: # ':' is sometimes used as encoding delimiter. - fullname = fullname.replace(':', '.') - if '.' in fullname: - langname, encoding = fullname.split('.')[:2] - fullname = langname + '.' + encoding + code = code.replace(':', '.') + if '@' in code: + code, modifier = code.split('@', 1) else: - langname = fullname + modifier = '' + if '.' in code: + langname, encoding = code.split('.')[:2] + else: + langname = code encoding = '' - # First lookup: fullname (possibly with encoding) - norm_encoding = encoding.replace('-', '') - norm_encoding = norm_encoding.replace('_', '') - lookup_name = langname + '.' + encoding + # First lookup: fullname (possibly with encoding and modifier) + lang_enc = langname + if encoding: + norm_encoding = encoding.replace('-', '') + norm_encoding = norm_encoding.replace('_', '') + lang_enc += '.' + norm_encoding + lookup_name = lang_enc + if modifier: + lookup_name += '@' + modifier code = locale_alias.get(lookup_name, None) if code is not None: return code - #print 'first lookup failed' + #print('first lookup failed') - # Second try: langname (without encoding) - code = locale_alias.get(langname, None) - if code is not None: - #print 'langname lookup succeeded' - if '.' in code: - langname, defenc = code.split('.') - else: - langname = code - defenc = '' - if encoding: - # Convert the encoding to a C lib compatible encoding string - norm_encoding = encodings.normalize_encoding(encoding) - #print 'norm encoding: %r' % norm_encoding - norm_encoding = encodings.aliases.aliases.get(norm_encoding, - norm_encoding) - #print 'aliased encoding: %r' % norm_encoding - encoding = locale_encoding_alias.get(norm_encoding, - norm_encoding) - else: - encoding = defenc - #print 'found encoding %r' % encoding - if encoding: - return langname + '.' + encoding - else: - return langname + if modifier: + # Second try: fullname without modifier (possibly with encoding) + code = locale_alias.get(lang_enc, None) + if code is not None: + #print('lookup without modifier succeeded') + if '@' not in code: + return code + '@' + modifier + if code.split('@', 1)[1].lower() == modifier: + return code + #print('second lookup failed') - else: - return localename + if encoding: + # Third try: langname (without encoding, possibly with modifier) + lookup_name = langname + if modifier: + lookup_name += '@' + modifier + code = locale_alias.get(lookup_name, None) + if code is not None: + #print('lookup without encoding succeeded') + if '@' not in code: + return _replace_encoding(code, encoding) + code, modifier = code.split('@', 1) + return _replace_encoding(code, encoding) + '@' + modifier + + if modifier: + # Fourth try: langname (without encoding and modifier) + code = locale_alias.get(langname, None) + if code is not None: + #print('lookup without modifier and encoding succeeded') + if '@' not in code: + return _replace_encoding(code, encoding) + '@' + modifier + code, defmod = code.split('@', 1) + if defmod.lower() == modifier: + return _replace_encoding(code, encoding) + '@' + defmod + + return localename def _parse_localename(localename): @@ -429,7 +461,7 @@ code = normalize(localename) if '@' in code: # Deal with locale modifiers - code, modifier = code.split('@') + code, modifier = code.split('@', 1) if modifier == 'euro' and '.' not in code: # Assume Latin-9 for @euro locales. This is bogus, # since some systems may use other encodings for these diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py --- a/Lib/test/test_locale.py +++ b/Lib/test/test_locale.py @@ -372,6 +372,64 @@ self.assertEqual('\xec\xa0\xbc'.split(), ['\xec\xa0\xbc']) +class NormalizeTest(unittest.TestCase): + def check(self, localename, expected): + self.assertEqual(locale.normalize(localename), expected, msg=localename) + + def test_locale_alias(self): + for localename, alias in locale.locale_alias.items(): + with self.subTest(locale=(localename, alias)): + self.check(localename, alias) + + def test_empty(self): + self.check('', '') + + def test_c(self): + self.check('c', 'C') + self.check('posix', 'C') + + def test_english(self): + self.check('en', 'en_US.ISO8859-1') + self.check('EN', 'en_US.ISO8859-1') + self.check('en_US', 'en_US.ISO8859-1') + self.check('en_us', 'en_US.ISO8859-1') + self.check('en_GB', 'en_GB.ISO8859-1') + self.check('en_US.UTF-8', 'en_US.UTF-8') + self.check('en_US.utf8', 'en_US.UTF-8') + self.check('en_US:UTF-8', 'en_US.UTF-8') + self.check('en_US.ISO8859-1', 'en_US.ISO8859-1') + self.check('en_US.US-ASCII', 'en_US.ISO8859-1') + self.check('english', 'en_EN.ISO8859-1') + + def test_hyphenated_encoding(self): + self.check('az_AZ.iso88599e', 'az_AZ.ISO8859-9E') + self.check('az_AZ.ISO8859-9E', 'az_AZ.ISO8859-9E') + self.check('tt_RU.koi8c', 'tt_RU.KOI8-C') + self.check('tt_RU.KOI8-C', 'tt_RU.KOI8-C') + self.check('lo_LA.cp1133', 'lo_LA.IBM-CP1133') + self.check('lo_LA.ibmcp1133', 'lo_LA.IBM-CP1133') + self.check('lo_LA.IBM-CP1133', 'lo_LA.IBM-CP1133') + self.check('uk_ua.microsoftcp1251', 'uk_UA.CP1251') + self.check('uk_ua.microsoft-cp1251', 'uk_UA.CP1251') + self.check('ka_ge.georgianacademy', 'ka_GE.GEORGIAN-ACADEMY') + self.check('ka_GE.GEORGIAN-ACADEMY', 'ka_GE.GEORGIAN-ACADEMY') + self.check('cs_CZ.iso88592', 'cs_CZ.ISO8859-2') + self.check('cs_CZ.ISO8859-2', 'cs_CZ.ISO8859-2') + + def test_euro_modifier(self): + self.check('de_DE at euro', 'de_DE.ISO8859-15') + self.check('en_US.ISO8859-15 at euro', 'en_US.ISO8859-15') + + def test_latin_modifier(self): + self.check('be_BY.UTF-8 at latin', 'be_BY.UTF-8 at latin') + self.check('sr_RS.UTF-8 at latin', 'sr_RS.UTF-8 at latin') + + def test_valencia_modifier(self): + self.check('ca_ES.UTF-8 at valencia', 'ca_ES.UTF-8 at valencia') + self.check('ca_ES at valencia', 'ca_ES.ISO8859-1 at valencia') + self.check('ca at valencia', 'ca_ES.ISO8859-1 at valencia') + + class TestMiscellaneous(unittest.TestCase): def test_getpreferredencoding(self): # Invoke getpreferredencoding to make sure it does not cause exceptions. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -27,6 +27,9 @@ Library ------- +- Issue #5815: Fixed support for locales with modifiers. Fixed support for + locale encodings with hyphens. + - Issue #20026: Fix the sqlite module to handle correctly invalid isolation level (wrong type). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 20:21:57 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 19 Dec 2013 20:21:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzU4MTU6?= =?utf-8?q?_Fixed_support_for_locales_with_modifiers=2E__Fixed_support_for?= Message-ID: <3dljdK1KPgz7Ljh@mail.python.org> http://hg.python.org/cpython/rev/28883e89f335 changeset: 88070:28883e89f335 branch: 3.3 parent: 88066:68ec8949dbf1 user: Serhiy Storchaka date: Thu Dec 19 21:21:25 2013 +0200 summary: Issue #5815: Fixed support for locales with modifiers. Fixed support for locale encodings with hyphens. files: Lib/locale.py | 114 +++++++++++++++++---------- Lib/test/test_locale.py | 58 ++++++++++++++ Misc/NEWS | 3 + 3 files changed, 134 insertions(+), 41 deletions(-) diff --git a/Lib/locale.py b/Lib/locale.py --- a/Lib/locale.py +++ b/Lib/locale.py @@ -336,6 +336,22 @@ # overridden below) _setlocale = setlocale +def _replace_encoding(code, encoding): + if '.' in code: + langname = code[:code.index('.')] + else: + langname = code + # Convert the encoding to a C lib compatible encoding string + norm_encoding = encodings.normalize_encoding(encoding) + #print('norm encoding: %r' % norm_encoding) + norm_encoding = encodings.aliases.aliases.get(norm_encoding, + norm_encoding) + #print('aliased encoding: %r' % norm_encoding) + encoding = locale_encoding_alias.get(norm_encoding, + norm_encoding) + #print('found encoding %r' % encoding) + return langname + '.' + encoding + def normalize(localename): """ Returns a normalized locale code for the given locale @@ -352,55 +368,71 @@ does. """ - # Normalize the locale name and extract the encoding - fullname = localename.lower() - if ':' in fullname: + # Normalize the locale name and extract the encoding and modifier + code = localename.lower() + if ':' in code: # ':' is sometimes used as encoding delimiter. - fullname = fullname.replace(':', '.') - if '.' in fullname: - langname, encoding = fullname.split('.')[:2] - fullname = langname + '.' + encoding + code = code.replace(':', '.') + if '@' in code: + code, modifier = code.split('@', 1) else: - langname = fullname + modifier = '' + if '.' in code: + langname, encoding = code.split('.')[:2] + else: + langname = code encoding = '' - # First lookup: fullname (possibly with encoding) - norm_encoding = encoding.replace('-', '') - norm_encoding = norm_encoding.replace('_', '') - lookup_name = langname + '.' + encoding + # First lookup: fullname (possibly with encoding and modifier) + lang_enc = langname + if encoding: + norm_encoding = encoding.replace('-', '') + norm_encoding = norm_encoding.replace('_', '') + lang_enc += '.' + norm_encoding + lookup_name = lang_enc + if modifier: + lookup_name += '@' + modifier code = locale_alias.get(lookup_name, None) if code is not None: return code - #print 'first lookup failed' + #print('first lookup failed') - # Second try: langname (without encoding) - code = locale_alias.get(langname, None) - if code is not None: - #print 'langname lookup succeeded' - if '.' in code: - langname, defenc = code.split('.') - else: - langname = code - defenc = '' - if encoding: - # Convert the encoding to a C lib compatible encoding string - norm_encoding = encodings.normalize_encoding(encoding) - #print 'norm encoding: %r' % norm_encoding - norm_encoding = encodings.aliases.aliases.get(norm_encoding, - norm_encoding) - #print 'aliased encoding: %r' % norm_encoding - encoding = locale_encoding_alias.get(norm_encoding, - norm_encoding) - else: - encoding = defenc - #print 'found encoding %r' % encoding - if encoding: - return langname + '.' + encoding - else: - return langname + if modifier: + # Second try: fullname without modifier (possibly with encoding) + code = locale_alias.get(lang_enc, None) + if code is not None: + #print('lookup without modifier succeeded') + if '@' not in code: + return code + '@' + modifier + if code.split('@', 1)[1].lower() == modifier: + return code + #print('second lookup failed') - else: - return localename + if encoding: + # Third try: langname (without encoding, possibly with modifier) + lookup_name = langname + if modifier: + lookup_name += '@' + modifier + code = locale_alias.get(lookup_name, None) + if code is not None: + #print('lookup without encoding succeeded') + if '@' not in code: + return _replace_encoding(code, encoding) + code, modifier = code.split('@', 1) + return _replace_encoding(code, encoding) + '@' + modifier + + if modifier: + # Fourth try: langname (without encoding and modifier) + code = locale_alias.get(langname, None) + if code is not None: + #print('lookup without modifier and encoding succeeded') + if '@' not in code: + return _replace_encoding(code, encoding) + '@' + modifier + code, defmod = code.split('@', 1) + if defmod.lower() == modifier: + return _replace_encoding(code, encoding) + '@' + defmod + + return localename def _parse_localename(localename): @@ -419,7 +451,7 @@ code = normalize(localename) if '@' in code: # Deal with locale modifiers - code, modifier = code.split('@') + code, modifier = code.split('@', 1) if modifier == 'euro' and '.' not in code: # Assume Latin-9 for @euro locales. This is bogus, # since some systems may use other encodings for these diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py --- a/Lib/test/test_locale.py +++ b/Lib/test/test_locale.py @@ -365,6 +365,64 @@ self.assertLess(locale.strxfrm('?'), locale.strxfrm('b')) +class NormalizeTest(unittest.TestCase): + def check(self, localename, expected): + self.assertEqual(locale.normalize(localename), expected, msg=localename) + + def test_locale_alias(self): + for localename, alias in locale.locale_alias.items(): + with self.subTest(locale=(localename, alias)): + self.check(localename, alias) + + def test_empty(self): + self.check('', '') + + def test_c(self): + self.check('c', 'C') + self.check('posix', 'C') + + def test_english(self): + self.check('en', 'en_US.ISO8859-1') + self.check('EN', 'en_US.ISO8859-1') + self.check('en_US', 'en_US.ISO8859-1') + self.check('en_us', 'en_US.ISO8859-1') + self.check('en_GB', 'en_GB.ISO8859-1') + self.check('en_US.UTF-8', 'en_US.UTF-8') + self.check('en_US.utf8', 'en_US.UTF-8') + self.check('en_US:UTF-8', 'en_US.UTF-8') + self.check('en_US.ISO8859-1', 'en_US.ISO8859-1') + self.check('en_US.US-ASCII', 'en_US.ISO8859-1') + self.check('english', 'en_EN.ISO8859-1') + + def test_hyphenated_encoding(self): + self.check('az_AZ.iso88599e', 'az_AZ.ISO8859-9E') + self.check('az_AZ.ISO8859-9E', 'az_AZ.ISO8859-9E') + self.check('tt_RU.koi8c', 'tt_RU.KOI8-C') + self.check('tt_RU.KOI8-C', 'tt_RU.KOI8-C') + self.check('lo_LA.cp1133', 'lo_LA.IBM-CP1133') + self.check('lo_LA.ibmcp1133', 'lo_LA.IBM-CP1133') + self.check('lo_LA.IBM-CP1133', 'lo_LA.IBM-CP1133') + self.check('uk_ua.microsoftcp1251', 'uk_UA.CP1251') + self.check('uk_ua.microsoft-cp1251', 'uk_UA.CP1251') + self.check('ka_ge.georgianacademy', 'ka_GE.GEORGIAN-ACADEMY') + self.check('ka_GE.GEORGIAN-ACADEMY', 'ka_GE.GEORGIAN-ACADEMY') + self.check('cs_CZ.iso88592', 'cs_CZ.ISO8859-2') + self.check('cs_CZ.ISO8859-2', 'cs_CZ.ISO8859-2') + + def test_euro_modifier(self): + self.check('de_DE at euro', 'de_DE.ISO8859-15') + self.check('en_US.ISO8859-15 at euro', 'en_US.ISO8859-15') + + def test_latin_modifier(self): + self.check('be_BY.UTF-8 at latin', 'be_BY.UTF-8 at latin') + self.check('sr_RS.UTF-8 at latin', 'sr_RS.UTF-8 at latin') + + def test_valencia_modifier(self): + self.check('ca_ES.UTF-8 at valencia', 'ca_ES.UTF-8 at valencia') + self.check('ca_ES at valencia', 'ca_ES.ISO8859-1 at valencia') + self.check('ca at valencia', 'ca_ES.ISO8859-1 at valencia') + + class TestMiscellaneous(unittest.TestCase): def test_getpreferredencoding(self): # Invoke getpreferredencoding to make sure it does not cause exceptions. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,9 @@ Library ------- +- Issue #5815: Fixed support for locales with modifiers. Fixed support for + locale encodings with hyphens. + - Issue #20026: Fix the sqlite module to handle correctly invalid isolation level (wrong type). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 20:21:58 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 19 Dec 2013 20:21:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=235815=3A_Fixed_sup?= =?utf-8?q?port_for_locales_with_modifiers=2E__Fixed_support_for?= Message-ID: <3dljdL55qRz7LmN@mail.python.org> http://hg.python.org/cpython/rev/b50971bccfc3 changeset: 88071:b50971bccfc3 parent: 88068:2b31b7eff99c user: Serhiy Storchaka date: Thu Dec 19 21:21:40 2013 +0200 summary: Issue #5815: Fixed support for locales with modifiers. Fixed support for locale encodings with hyphens. files: Lib/locale.py | 114 +++++++++++++++++---------- Lib/test/test_locale.py | 58 ++++++++++++++ 2 files changed, 131 insertions(+), 41 deletions(-) diff --git a/Lib/locale.py b/Lib/locale.py --- a/Lib/locale.py +++ b/Lib/locale.py @@ -336,6 +336,22 @@ # overridden below) _setlocale = setlocale +def _replace_encoding(code, encoding): + if '.' in code: + langname = code[:code.index('.')] + else: + langname = code + # Convert the encoding to a C lib compatible encoding string + norm_encoding = encodings.normalize_encoding(encoding) + #print('norm encoding: %r' % norm_encoding) + norm_encoding = encodings.aliases.aliases.get(norm_encoding, + norm_encoding) + #print('aliased encoding: %r' % norm_encoding) + encoding = locale_encoding_alias.get(norm_encoding, + norm_encoding) + #print('found encoding %r' % encoding) + return langname + '.' + encoding + def normalize(localename): """ Returns a normalized locale code for the given locale @@ -352,55 +368,71 @@ does. """ - # Normalize the locale name and extract the encoding - fullname = localename.lower() - if ':' in fullname: + # Normalize the locale name and extract the encoding and modifier + code = localename.lower() + if ':' in code: # ':' is sometimes used as encoding delimiter. - fullname = fullname.replace(':', '.') - if '.' in fullname: - langname, encoding = fullname.split('.')[:2] - fullname = langname + '.' + encoding + code = code.replace(':', '.') + if '@' in code: + code, modifier = code.split('@', 1) else: - langname = fullname + modifier = '' + if '.' in code: + langname, encoding = code.split('.')[:2] + else: + langname = code encoding = '' - # First lookup: fullname (possibly with encoding) - norm_encoding = encoding.replace('-', '') - norm_encoding = norm_encoding.replace('_', '') - lookup_name = langname + '.' + encoding + # First lookup: fullname (possibly with encoding and modifier) + lang_enc = langname + if encoding: + norm_encoding = encoding.replace('-', '') + norm_encoding = norm_encoding.replace('_', '') + lang_enc += '.' + norm_encoding + lookup_name = lang_enc + if modifier: + lookup_name += '@' + modifier code = locale_alias.get(lookup_name, None) if code is not None: return code - #print 'first lookup failed' + #print('first lookup failed') - # Second try: langname (without encoding) - code = locale_alias.get(langname, None) - if code is not None: - #print 'langname lookup succeeded' - if '.' in code: - langname, defenc = code.split('.') - else: - langname = code - defenc = '' - if encoding: - # Convert the encoding to a C lib compatible encoding string - norm_encoding = encodings.normalize_encoding(encoding) - #print 'norm encoding: %r' % norm_encoding - norm_encoding = encodings.aliases.aliases.get(norm_encoding, - norm_encoding) - #print 'aliased encoding: %r' % norm_encoding - encoding = locale_encoding_alias.get(norm_encoding, - norm_encoding) - else: - encoding = defenc - #print 'found encoding %r' % encoding - if encoding: - return langname + '.' + encoding - else: - return langname + if modifier: + # Second try: fullname without modifier (possibly with encoding) + code = locale_alias.get(lang_enc, None) + if code is not None: + #print('lookup without modifier succeeded') + if '@' not in code: + return code + '@' + modifier + if code.split('@', 1)[1].lower() == modifier: + return code + #print('second lookup failed') - else: - return localename + if encoding: + # Third try: langname (without encoding, possibly with modifier) + lookup_name = langname + if modifier: + lookup_name += '@' + modifier + code = locale_alias.get(lookup_name, None) + if code is not None: + #print('lookup without encoding succeeded') + if '@' not in code: + return _replace_encoding(code, encoding) + code, modifier = code.split('@', 1) + return _replace_encoding(code, encoding) + '@' + modifier + + if modifier: + # Fourth try: langname (without encoding and modifier) + code = locale_alias.get(langname, None) + if code is not None: + #print('lookup without modifier and encoding succeeded') + if '@' not in code: + return _replace_encoding(code, encoding) + '@' + modifier + code, defmod = code.split('@', 1) + if defmod.lower() == modifier: + return _replace_encoding(code, encoding) + '@' + defmod + + return localename def _parse_localename(localename): @@ -419,7 +451,7 @@ code = normalize(localename) if '@' in code: # Deal with locale modifiers - code, modifier = code.split('@') + code, modifier = code.split('@', 1) if modifier == 'euro' and '.' not in code: # Assume Latin-9 for @euro locales. This is bogus, # since some systems may use other encodings for these diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py --- a/Lib/test/test_locale.py +++ b/Lib/test/test_locale.py @@ -365,6 +365,64 @@ self.assertLess(locale.strxfrm('?'), locale.strxfrm('b')) +class NormalizeTest(unittest.TestCase): + def check(self, localename, expected): + self.assertEqual(locale.normalize(localename), expected, msg=localename) + + def test_locale_alias(self): + for localename, alias in locale.locale_alias.items(): + with self.subTest(locale=(localename, alias)): + self.check(localename, alias) + + def test_empty(self): + self.check('', '') + + def test_c(self): + self.check('c', 'C') + self.check('posix', 'C') + + def test_english(self): + self.check('en', 'en_US.ISO8859-1') + self.check('EN', 'en_US.ISO8859-1') + self.check('en_US', 'en_US.ISO8859-1') + self.check('en_us', 'en_US.ISO8859-1') + self.check('en_GB', 'en_GB.ISO8859-1') + self.check('en_US.UTF-8', 'en_US.UTF-8') + self.check('en_US.utf8', 'en_US.UTF-8') + self.check('en_US:UTF-8', 'en_US.UTF-8') + self.check('en_US.ISO8859-1', 'en_US.ISO8859-1') + self.check('en_US.US-ASCII', 'en_US.ISO8859-1') + self.check('english', 'en_EN.ISO8859-1') + + def test_hyphenated_encoding(self): + self.check('az_AZ.iso88599e', 'az_AZ.ISO8859-9E') + self.check('az_AZ.ISO8859-9E', 'az_AZ.ISO8859-9E') + self.check('tt_RU.koi8c', 'tt_RU.KOI8-C') + self.check('tt_RU.KOI8-C', 'tt_RU.KOI8-C') + self.check('lo_LA.cp1133', 'lo_LA.IBM-CP1133') + self.check('lo_LA.ibmcp1133', 'lo_LA.IBM-CP1133') + self.check('lo_LA.IBM-CP1133', 'lo_LA.IBM-CP1133') + self.check('uk_ua.microsoftcp1251', 'uk_UA.CP1251') + self.check('uk_ua.microsoft-cp1251', 'uk_UA.CP1251') + self.check('ka_ge.georgianacademy', 'ka_GE.GEORGIAN-ACADEMY') + self.check('ka_GE.GEORGIAN-ACADEMY', 'ka_GE.GEORGIAN-ACADEMY') + self.check('cs_CZ.iso88592', 'cs_CZ.ISO8859-2') + self.check('cs_CZ.ISO8859-2', 'cs_CZ.ISO8859-2') + + def test_euro_modifier(self): + self.check('de_DE at euro', 'de_DE.ISO8859-15') + self.check('en_US.ISO8859-15 at euro', 'en_US.ISO8859-15') + + def test_latin_modifier(self): + self.check('be_BY.UTF-8 at latin', 'be_BY.UTF-8 at latin') + self.check('sr_RS.UTF-8 at latin', 'sr_RS.UTF-8 at latin') + + def test_valencia_modifier(self): + self.check('ca_ES.UTF-8 at valencia', 'ca_ES.UTF-8 at valencia') + self.check('ca_ES at valencia', 'ca_ES.ISO8859-1 at valencia') + self.check('ca at valencia', 'ca_ES.ISO8859-1 at valencia') + + class TestMiscellaneous(unittest.TestCase): def test_getpreferredencoding(self): # Invoke getpreferredencoding to make sure it does not cause exceptions. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 20:24:23 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 19 Dec 2013 20:24:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge_with_3=2E3?= Message-ID: <3dljh76NhGz7LjY@mail.python.org> http://hg.python.org/cpython/rev/e0b99423107a changeset: 88072:e0b99423107a parent: 88071:b50971bccfc3 parent: 88070:28883e89f335 user: Serhiy Storchaka date: Thu Dec 19 21:24:06 2013 +0200 summary: Merge with 3.3 files: Misc/NEWS | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #5815: Fixed support for locales with modifiers. Fixed support for + locale encodings with hyphens. + - Issue #20026: Fix the sqlite module to handle correctly invalid isolation level (wrong type). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 20:47:43 2013 From: python-checkins at python.org (zach.ware) Date: Thu, 19 Dec 2013 20:47:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5Njgz?= =?utf-8?q?=3A_Removed_empty_tests_from_test=5Fminidom=2E?= Message-ID: <3dlkC361K6z7Ljh@mail.python.org> http://hg.python.org/cpython/rev/2737c0e7ba71 changeset: 88073:2737c0e7ba71 branch: 2.7 parent: 88069:3d805bee06e2 user: Zachary Ware date: Thu Dec 19 13:44:19 2013 -0600 summary: Issue #19683: Removed empty tests from test_minidom. Initial patch by Ajitesh Gupta. files: Lib/test/test_minidom.py | 69 ---------------------------- Misc/ACKS | 1 + Misc/NEWS | 3 + 3 files changed, 4 insertions(+), 69 deletions(-) diff --git a/Lib/test/test_minidom.py b/Lib/test/test_minidom.py --- a/Lib/test/test_minidom.py +++ b/Lib/test/test_minidom.py @@ -340,19 +340,6 @@ and el.getAttribute("spam2") == "bam2") dom.unlink() - def testGetAttrList(self): - pass - - def testGetAttrValues(self): pass - - def testGetAttrLength(self): pass - - def testGetAttribute(self): pass - - def testGetAttributeNS(self): pass - - def testGetAttributeNode(self): pass - def testGetElementsByTagNameNS(self): d=""" @@ -423,8 +410,6 @@ self.confirm(str(node) == repr(node)) dom.unlink() - def testTextNodeRepr(self): pass - def testWriteXML(self): str = '' dom = parseString(str) @@ -488,14 +473,6 @@ and pi.localName is None and pi.namespaceURI == xml.dom.EMPTY_NAMESPACE) - def testProcessingInstructionRepr(self): pass - - def testTextRepr(self): pass - - def testWriteText(self): pass - - def testDocumentElement(self): pass - def testTooManyDocumentElements(self): doc = parseString("") elem = doc.createElement("extra") @@ -504,26 +481,6 @@ elem.unlink() doc.unlink() - def testCreateElementNS(self): pass - - def testCreateAttributeNS(self): pass - - def testParse(self): pass - - def testParseString(self): pass - - def testComment(self): pass - - def testAttrListItem(self): pass - - def testAttrListItems(self): pass - - def testAttrListItemNS(self): pass - - def testAttrListKeys(self): pass - - def testAttrListKeysNS(self): pass - def testRemoveNamedItem(self): doc = parseString("") e = doc.documentElement @@ -543,32 +500,6 @@ self.assertRaises(xml.dom.NotFoundErr, attrs.removeNamedItemNS, "http://xml.python.org/", "b") - def testAttrListValues(self): pass - - def testAttrListLength(self): pass - - def testAttrList__getitem__(self): pass - - def testAttrList__setitem__(self): pass - - def testSetAttrValueandNodeValue(self): pass - - def testParseElement(self): pass - - def testParseAttributes(self): pass - - def testParseElementNamespaces(self): pass - - def testParseAttributeNamespaces(self): pass - - def testParseProcessingInstructions(self): pass - - def testChildNodes(self): pass - - def testFirstChild(self): pass - - def testHasChildNodes(self): pass - def _testCloneElementCopiesAttributes(self, e1, e2, test): attrs1 = e1.attributes attrs2 = e2.attributes diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -380,6 +380,7 @@ Dag Gruneau Filip Gruszczy?ski Thomas Guettler +Ajitesh Gupta Michael Guravage Lars Gust?bel Thomas G?ttler diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -119,6 +119,9 @@ Tests ----- +- Issue #19683: Removed empty tests from test_minidom. Initial patch by + Ajitesh Gupta. + - Issue #19928: Implemented a test for repr() of cell objects. - Issue #19595, #19987: Re-enabled a long-disabled test in test_winsound. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 20:47:45 2013 From: python-checkins at python.org (zach.ware) Date: Thu, 19 Dec 2013 20:47:45 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5Njgz?= =?utf-8?q?=3A_Removed_empty_tests_from_test=5Fminidom=2E__Patch_by_Ajites?= =?utf-8?q?h_Gupta=2E?= Message-ID: <3dlkC51s5Yz7LlB@mail.python.org> http://hg.python.org/cpython/rev/5e510117b71a changeset: 88074:5e510117b71a branch: 3.3 parent: 88070:28883e89f335 user: Zachary Ware date: Thu Dec 19 13:44:56 2013 -0600 summary: Issue #19683: Removed empty tests from test_minidom. Patch by Ajitesh Gupta. files: Lib/test/test_minidom.py | 65 ---------------------------- Misc/ACKS | 1 + Misc/NEWS | 2 + 3 files changed, 3 insertions(+), 65 deletions(-) diff --git a/Lib/test/test_minidom.py b/Lib/test/test_minidom.py --- a/Lib/test/test_minidom.py +++ b/Lib/test/test_minidom.py @@ -360,15 +360,6 @@ and el.getAttribute("spam2") == "bam2") dom.unlink() - def testGetAttrList(self): - pass - - def testGetAttrValues(self): - pass - - def testGetAttrLength(self): - pass - def testGetAttribute(self): dom = Document() child = dom.appendChild( @@ -389,8 +380,6 @@ self.assertEqual(child2.getAttributeNS("http://www.python.org", "missing"), '') - def testGetAttributeNode(self): pass - def testGetElementsByTagNameNS(self): d=""" @@ -461,8 +450,6 @@ self.confirm(str(node) == repr(node)) dom.unlink() - def testTextNodeRepr(self): pass - def testWriteXML(self): str = '' dom = parseString(str) @@ -526,14 +513,6 @@ and pi.localName is None and pi.namespaceURI == xml.dom.EMPTY_NAMESPACE) - def testProcessingInstructionRepr(self): pass - - def testTextRepr(self): pass - - def testWriteText(self): pass - - def testDocumentElement(self): pass - def testTooManyDocumentElements(self): doc = parseString("") elem = doc.createElement("extra") @@ -542,26 +521,6 @@ elem.unlink() doc.unlink() - def testCreateElementNS(self): pass - - def testCreateAttributeNS(self): pass - - def testParse(self): pass - - def testParseString(self): pass - - def testComment(self): pass - - def testAttrListItem(self): pass - - def testAttrListItems(self): pass - - def testAttrListItemNS(self): pass - - def testAttrListKeys(self): pass - - def testAttrListKeysNS(self): pass - def testRemoveNamedItem(self): doc = parseString("") e = doc.documentElement @@ -581,30 +540,6 @@ self.assertRaises(xml.dom.NotFoundErr, attrs.removeNamedItemNS, "http://xml.python.org/", "b") - def testAttrListValues(self): pass - - def testAttrListLength(self): pass - - def testAttrList__getitem__(self): pass - - def testAttrList__setitem__(self): pass - - def testSetAttrValueandNodeValue(self): pass - - def testParseElement(self): pass - - def testParseAttributes(self): pass - - def testParseElementNamespaces(self): pass - - def testParseAttributeNamespaces(self): pass - - def testParseProcessingInstructions(self): pass - - def testChildNodes(self): pass - - def testFirstChild(self): pass - def testHasChildNodes(self): dom = parseString("") doc = dom.documentElement diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -459,6 +459,7 @@ Dag Gruneau Filip Gruszczy?ski Thomas Guettler +Ajitesh Gupta Michael Guravage Lars Gust?bel Thomas G?ttler diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -165,6 +165,8 @@ Tests ----- +- Issue #19683: Removed empty tests from test_minidom. Patch by Ajitesh Gupta. + - Issue #19919: Fix flacky SSL test. connect_ex() sometimes returns EWOULDBLOCK on Windows or VMs hosted on Windows. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 20:47:46 2013 From: python-checkins at python.org (zach.ware) Date: Thu, 19 Dec 2013 20:47:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Null_merge_with_3=2E3=2E?= Message-ID: <3dlkC63mdjz7LlB@mail.python.org> http://hg.python.org/cpython/rev/1313dca0e2a5 changeset: 88075:1313dca0e2a5 parent: 88072:e0b99423107a parent: 88074:5e510117b71a user: Zachary Ware date: Thu Dec 19 13:47:40 2013 -0600 summary: Null merge with 3.3. The empty tests should be implemented on this branch. files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 21:32:50 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 19 Dec 2013 21:32:50 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogRG9uJ3QgdXNlIHNl?= =?utf-8?q?bTest=28=29_in_tests_for_issue_=235815=2E?= Message-ID: <3dllC61yjmz7LkL@mail.python.org> http://hg.python.org/cpython/rev/e0675408f4af changeset: 88076:e0675408f4af branch: 2.7 parent: 88069:3d805bee06e2 user: Serhiy Storchaka date: Thu Dec 19 22:28:25 2013 +0200 summary: Don't use sebTest() in tests for issue #5815. files: Lib/test/test_locale.py | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py --- a/Lib/test/test_locale.py +++ b/Lib/test/test_locale.py @@ -378,8 +378,7 @@ def test_locale_alias(self): for localename, alias in locale.locale_alias.items(): - with self.subTest(locale=(localename, alias)): - self.check(localename, alias) + self.check(localename, alias) def test_empty(self): self.check('', '') -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 21:32:51 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 19 Dec 2013 21:32:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf-8?q?_Merge_heads?= Message-ID: <3dllC73x8nz7LkZ@mail.python.org> http://hg.python.org/cpython/rev/3bec2d0fb668 changeset: 88077:3bec2d0fb668 branch: 2.7 parent: 88076:e0675408f4af parent: 88073:2737c0e7ba71 user: Serhiy Storchaka date: Thu Dec 19 22:31:23 2013 +0200 summary: Merge heads files: Lib/test/test_minidom.py | 69 ---------------------------- Misc/ACKS | 1 + Misc/NEWS | 3 + 3 files changed, 4 insertions(+), 69 deletions(-) diff --git a/Lib/test/test_minidom.py b/Lib/test/test_minidom.py --- a/Lib/test/test_minidom.py +++ b/Lib/test/test_minidom.py @@ -340,19 +340,6 @@ and el.getAttribute("spam2") == "bam2") dom.unlink() - def testGetAttrList(self): - pass - - def testGetAttrValues(self): pass - - def testGetAttrLength(self): pass - - def testGetAttribute(self): pass - - def testGetAttributeNS(self): pass - - def testGetAttributeNode(self): pass - def testGetElementsByTagNameNS(self): d=""" @@ -423,8 +410,6 @@ self.confirm(str(node) == repr(node)) dom.unlink() - def testTextNodeRepr(self): pass - def testWriteXML(self): str = '' dom = parseString(str) @@ -488,14 +473,6 @@ and pi.localName is None and pi.namespaceURI == xml.dom.EMPTY_NAMESPACE) - def testProcessingInstructionRepr(self): pass - - def testTextRepr(self): pass - - def testWriteText(self): pass - - def testDocumentElement(self): pass - def testTooManyDocumentElements(self): doc = parseString("") elem = doc.createElement("extra") @@ -504,26 +481,6 @@ elem.unlink() doc.unlink() - def testCreateElementNS(self): pass - - def testCreateAttributeNS(self): pass - - def testParse(self): pass - - def testParseString(self): pass - - def testComment(self): pass - - def testAttrListItem(self): pass - - def testAttrListItems(self): pass - - def testAttrListItemNS(self): pass - - def testAttrListKeys(self): pass - - def testAttrListKeysNS(self): pass - def testRemoveNamedItem(self): doc = parseString("") e = doc.documentElement @@ -543,32 +500,6 @@ self.assertRaises(xml.dom.NotFoundErr, attrs.removeNamedItemNS, "http://xml.python.org/", "b") - def testAttrListValues(self): pass - - def testAttrListLength(self): pass - - def testAttrList__getitem__(self): pass - - def testAttrList__setitem__(self): pass - - def testSetAttrValueandNodeValue(self): pass - - def testParseElement(self): pass - - def testParseAttributes(self): pass - - def testParseElementNamespaces(self): pass - - def testParseAttributeNamespaces(self): pass - - def testParseProcessingInstructions(self): pass - - def testChildNodes(self): pass - - def testFirstChild(self): pass - - def testHasChildNodes(self): pass - def _testCloneElementCopiesAttributes(self, e1, e2, test): attrs1 = e1.attributes attrs2 = e2.attributes diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -380,6 +380,7 @@ Dag Gruneau Filip Gruszczy?ski Thomas Guettler +Ajitesh Gupta Michael Guravage Lars Gust?bel Thomas G?ttler diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -119,6 +119,9 @@ Tests ----- +- Issue #19683: Removed empty tests from test_minidom. Initial patch by + Ajitesh Gupta. + - Issue #19928: Implemented a test for repr() of cell objects. - Issue #19595, #19987: Re-enabled a long-disabled test in test_winsound. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 21:32:52 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 19 Dec 2013 21:32:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogRG9uJ3QgdXNlIHNl?= =?utf-8?q?bTest=28=29_in_tests_for_issue_=235815=2E?= Message-ID: <3dllC85c5cz7Llm@mail.python.org> http://hg.python.org/cpython/rev/ed16f6695638 changeset: 88078:ed16f6695638 branch: 3.3 parent: 88074:5e510117b71a user: Serhiy Storchaka date: Thu Dec 19 22:31:46 2013 +0200 summary: Don't use sebTest() in tests for issue #5815. files: Lib/test/test_locale.py | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py --- a/Lib/test/test_locale.py +++ b/Lib/test/test_locale.py @@ -371,8 +371,7 @@ def test_locale_alias(self): for localename, alias in locale.locale_alias.items(): - with self.subTest(locale=(localename, alias)): - self.check(localename, alias) + self.check(localename, alias) def test_empty(self): self.check('', '') -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 21:32:54 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 19 Dec 2013 21:32:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Null_merge?= Message-ID: <3dllCB1VDnz7Llk@mail.python.org> http://hg.python.org/cpython/rev/88b6b37eb7cf changeset: 88079:88b6b37eb7cf parent: 88075:1313dca0e2a5 parent: 88078:ed16f6695638 user: Serhiy Storchaka date: Thu Dec 19 22:32:40 2013 +0200 summary: Null merge files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 21:47:47 2013 From: python-checkins at python.org (guido.van.rossum) Date: Thu, 19 Dec 2013 21:47:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Shorten_lines=2E?= Message-ID: <3dllXM4KjWz7Ljb@mail.python.org> http://hg.python.org/cpython/rev/5f06f6a623be changeset: 88080:5f06f6a623be user: Guido van Rossum date: Thu Dec 19 12:47:38 2013 -0800 summary: Shorten lines. files: Lib/asyncio/locks.py | 4 ++-- Lib/test/test_asyncio/test_events.py | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Lib/asyncio/locks.py b/Lib/asyncio/locks.py --- a/Lib/asyncio/locks.py +++ b/Lib/asyncio/locks.py @@ -138,7 +138,7 @@ class Event: - """An Event implementation, asynchronous equivalent to threading.Event. + """Asynchronous equivalent to threading.Event. Class implementing event objects. An event manages a flag that can be set to true with the set() method and reset to false with the clear() method. @@ -204,7 +204,7 @@ class Condition: - """A Condition implementation, asynchronous equivalent to threading.Condition. + """Asynchronous equivalent to threading.Condition. This class implements condition variable objects. A condition variable allows one or more coroutines to wait until they are notified by another diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -720,7 +720,8 @@ # incorrect server_hostname f_c = self.loop.create_connection(MyProto, host, port, ssl=sslcontext_client) - with self.assertRaisesRegex(ssl.CertificateError, + with self.assertRaisesRegex( + ssl.CertificateError, "hostname '127.0.0.1' doesn't match 'localhost'"): self.loop.run_until_complete(f_c) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 22:43:22 2013 From: python-checkins at python.org (victor.stinner) Date: Thu, 19 Dec 2013 22:43:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Close_=2319967=3A_Thanks_t?= =?utf-8?q?o_the_PEP_442=2C_asyncio=2EFuture_can_use_a_destructor_in?= Message-ID: <3dlmmV3qJwz7LjY@mail.python.org> http://hg.python.org/cpython/rev/5e9728ebb1d3 changeset: 88081:5e9728ebb1d3 user: Victor Stinner date: Thu Dec 19 22:42:40 2013 +0100 summary: Close #19967: Thanks to the PEP 442, asyncio.Future can use a destructor in Python 3.4 to log "uncatched" exceptions, instead of the dedicated _TracebackLogger class. files: Lib/asyncio/futures.py | 32 +++++++++++++--- Lib/test/test_asyncio/test_tasks.py | 2 + 2 files changed, 27 insertions(+), 7 deletions(-) diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -7,6 +7,7 @@ import concurrent.futures._base import logging +import sys import traceback from . import events @@ -17,6 +18,8 @@ _CANCELLED = 'CANCELLED' _FINISHED = 'FINISHED' +_PY34 = sys.version_info >= (3, 4) + # TODO: Do we really want to depend on concurrent.futures internals? Error = concurrent.futures._base.Error CancelledError = concurrent.futures.CancelledError @@ -128,7 +131,8 @@ _blocking = False # proper use of future (yield vs yield from) - _tb_logger = None + _traceback = None # Used for Python 3.4 and later + _tb_logger = None # Used for Python 3.3 only def __init__(self, *, loop=None): """Initialize the future. @@ -162,6 +166,12 @@ res += '<{}>'.format(self._state) return res + if _PY34: + def __del__(self): + if self._traceback is not None: + logger.error('Future/Task exception was never retrieved:\n%s', + ''.join(self._traceback)) + def cancel(self): """Cancel the future and schedule callbacks. @@ -214,9 +224,10 @@ raise CancelledError if self._state != _FINISHED: raise InvalidStateError('Result is not ready.') + self._traceback = None if self._tb_logger is not None: self._tb_logger.clear() - self._tb_logger = None + self._tb_logger = None if self._exception is not None: raise self._exception return self._result @@ -233,9 +244,10 @@ raise CancelledError if self._state != _FINISHED: raise InvalidStateError('Exception is not set.') + self._traceback = None if self._tb_logger is not None: self._tb_logger.clear() - self._tb_logger = None + self._tb_logger = None return self._exception def add_done_callback(self, fn): @@ -286,12 +298,18 @@ if self._state != _PENDING: raise InvalidStateError('{}: {!r}'.format(self._state, self)) self._exception = exception - self._tb_logger = _TracebackLogger(exception) self._state = _FINISHED self._schedule_callbacks() - # Arrange for the logger to be activated after all callbacks - # have had a chance to call result() or exception(). - self._loop.call_soon(self._tb_logger.activate) + if _PY34: + self._traceback = traceback.format_exception( + exception.__class__, + exception, + exception.__traceback__) + else: + self._tb_logger = _TracebackLogger(exception) + # Arrange for the logger to be activated after all callbacks + # have had a chance to call result() or exception(). + self._loop.call_soon(self._tb_logger.activate) # Truly internal methods. diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -1352,6 +1352,7 @@ c.set_result(3) d.cancel() e.set_exception(RuntimeError()) + e.exception() def test_return_exceptions(self): a, b, c, d = [futures.Future(loop=self.one_loop) for i in range(4)] @@ -1431,6 +1432,7 @@ c.set_result(3) d.cancel() e.set_exception(RuntimeError()) + e.exception() def test_result_exception_one_cancellation(self): a, b, c, d, e, f = [futures.Future(loop=self.one_loop) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 19 22:49:41 2013 From: python-checkins at python.org (guido.van.rossum) Date: Thu, 19 Dec 2013 22:49:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_Clean_up_format?= =?utf-8?q?ting=2E?= Message-ID: <3dlmvn1XBVz7LjY@mail.python.org> http://hg.python.org/cpython/rev/13a505260f17 changeset: 88082:13a505260f17 user: Guido van Rossum date: Thu Dec 19 13:49:32 2013 -0800 summary: asyncio: Clean up formatting. files: Lib/asyncio/futures.py | 4 +--- Lib/test/test_asyncio/test_tasks.py | 4 +++- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -302,9 +302,7 @@ self._schedule_callbacks() if _PY34: self._traceback = traceback.format_exception( - exception.__class__, - exception, - exception.__traceback__) + exception.__class__, exception, exception.__traceback__) else: self._tb_logger = _TracebackLogger(exception) # Arrange for the logger to be activated after all callbacks diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -1115,6 +1115,7 @@ def test_current_task(self): self.assertIsNone(tasks.Task.current_task(loop=self.loop)) + @tasks.coroutine def coro(loop): self.assertTrue(tasks.Task.current_task(loop=loop) is task) @@ -1146,7 +1147,8 @@ task1 = tasks.Task(coro1(self.loop), loop=self.loop) task2 = tasks.Task(coro2(self.loop), loop=self.loop) - self.loop.run_until_complete(tasks.wait((task1, task2), loop=self.loop)) + self.loop.run_until_complete(tasks.wait((task1, task2), + loop=self.loop)) self.assertIsNone(tasks.Task.current_task(loop=self.loop)) # Some thorough tests for cancellation propagation through -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Fri Dec 20 09:44:51 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 20 Dec 2013 09:44:51 +0100 Subject: [Python-checkins] Daily reference leaks (13a505260f17): sum=0 Message-ID: results for 13a505260f17 on branch "default" -------------------------------------------- test_site leaked [2, 0, -2] references, sum=0 test_site leaked [2, 0, -2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogQgVwnp', '-x'] From python-checkins at python.org Fri Dec 20 13:18:00 2013 From: python-checkins at python.org (nick.coghlan) Date: Fri, 20 Dec 2013 13:18:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319946=3A_use_publ?= =?utf-8?q?ic_API_for_multiprocessing_start_methods?= Message-ID: <3dm89h3sNyz7LjZ@mail.python.org> http://hg.python.org/cpython/rev/00d09afb57ca changeset: 88083:00d09afb57ca user: Nick Coghlan date: Fri Dec 20 22:14:03 2013 +1000 summary: Issue #19946: use public API for multiprocessing start methods This should appease the OpenIndiana buildbot. Also lengthened the worst case timeout to try to eliminate the inconsistent failure on one of the Windows 7 buildbots. files: Lib/test/test_multiprocessing_main_handling.py | 12 +++++----- 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_multiprocessing_main_handling.py b/Lib/test/test_multiprocessing_main_handling.py --- a/Lib/test/test_multiprocessing_main_handling.py +++ b/Lib/test/test_multiprocessing_main_handling.py @@ -15,9 +15,9 @@ assert_python_ok, assert_python_failure, temp_dir, spawn_python, kill_python) -# We look inside the context module to find out which -# start methods we can check -from multiprocessing.context import _concrete_contexts +# Look up which start methods are available to test +import multiprocessing +AVAILABLE_START_METHODS = set(multiprocessing.get_all_start_methods()) verbose = support.verbose @@ -51,7 +51,7 @@ p = Pool(5) results = [] p.map_async(f, [1, 2, 3], callback=results.extend) - deadline = time.time() + 2 # up to 2 s to report the results + deadline = time.time() + 5 # up to 5 s to report the results while not results: time.sleep(0.05) if time.time() > deadline: @@ -79,7 +79,7 @@ p = Pool(5) results = [] p.map_async(int, [1, 4, 9], callback=results.extend) -deadline = time.time() + 2 # up to 2 s to report the results +deadline = time.time() + 5 # up to 5 s to report the results while not results: time.sleep(0.05) if time.time() > deadline: @@ -131,7 +131,7 @@ maxDiff = None # Show full tracebacks on subprocess failure def setUp(self): - if self.start_method not in _concrete_contexts: + if self.start_method not in AVAILABLE_START_METHODS: self.skipTest("%r start method not available" % self.start_method) def _check_output(self, script_name, exit_code, out, err): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 17:24:29 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 20 Dec 2013 17:24:29 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIwMDM0?= =?utf-8?q?=3A_Updated_alias_mapping_to_most_recent_locale=2Ealias_file?= Message-ID: <3dmFf52dpMz7Ljn@mail.python.org> http://hg.python.org/cpython/rev/81f8375e60ce changeset: 88084:81f8375e60ce branch: 2.7 parent: 88077:3bec2d0fb668 user: Serhiy Storchaka date: Fri Dec 20 18:22:38 2013 +0200 summary: Issue #20034: Updated alias mapping to most recent locale.alias file from X.org distribution using makelocalealias.py. files: Lib/locale.py | 54 +++++++++++++++++++++++++++++++------- Misc/NEWS | 3 ++ 2 files changed, 46 insertions(+), 11 deletions(-) diff --git a/Lib/locale.py b/Lib/locale.py --- a/Lib/locale.py +++ b/Lib/locale.py @@ -766,11 +766,30 @@ # updated 'sr_yu.utf8 at cyrillic' -> 'sr_CS.UTF-8' to 'sr_RS.UTF-8' # updated 'sr_yu at cyrillic' -> 'sr_CS.ISO8859-5' to 'sr_RS.UTF-8' # +# SS 2013-12-20: +# Updated alias mapping to most recent locale.alias file +# from X.org distribution using makelocalealias.py. +# +# These are the differences compared to the old mapping (Python 2.7.6 +# and older): +# +# updated 'a3' -> 'a3_AZ.KOI8-C' to 'az_AZ.KOI8-C' +# updated 'a3_az' -> 'a3_AZ.KOI8-C' to 'az_AZ.KOI8-C' +# updated 'a3_az.koi8c' -> 'a3_AZ.KOI8-C' to 'az_AZ.KOI8-C' +# updated 'cs_cs.iso88592' -> 'cs_CS.ISO8859-2' to 'cs_CZ.ISO8859-2' +# updated 'hebrew' -> 'iw_IL.ISO8859-8' to 'he_IL.ISO8859-8' +# updated 'hebrew.iso88598' -> 'iw_IL.ISO8859-8' to 'he_IL.ISO8859-8' +# updated 'sd' -> 'sd_IN at devanagari.UTF-8' to 'sd_IN.UTF-8' +# updated 'sr at latn' -> 'sr_RS.UTF-8 at latin' to 'sr_CS.UTF-8 at latin' +# updated 'sr_cs' -> 'sr_RS.UTF-8' to 'sr_CS.UTF-8' +# updated 'sr_cs.utf8 at latn' -> 'sr_RS.UTF-8 at latin' to 'sr_CS.UTF-8 at latin' +# updated 'sr_cs at latn' -> 'sr_RS.UTF-8 at latin' to 'sr_CS.UTF-8 at latin' locale_alias = { - 'a3': 'a3_AZ.KOI8-C', - 'a3_az': 'a3_AZ.KOI8-C', - 'a3_az.koi8c': 'a3_AZ.KOI8-C', + 'a3': 'az_AZ.KOI8-C', + 'a3_az': 'az_AZ.KOI8-C', + 'a3_az.koi8c': 'az_AZ.KOI8-C', + 'a3_az.koic': 'az_AZ.KOI8-C', 'af': 'af_ZA.ISO8859-1', 'af_za': 'af_ZA.ISO8859-1', 'af_za.iso88591': 'af_ZA.ISO8859-1', @@ -789,6 +808,7 @@ 'ar_dz.iso88596': 'ar_DZ.ISO8859-6', 'ar_eg': 'ar_EG.ISO8859-6', 'ar_eg.iso88596': 'ar_EG.ISO8859-6', + 'ar_in': 'ar_IN.UTF-8', 'ar_iq': 'ar_IQ.ISO8859-6', 'ar_iq.iso88596': 'ar_IQ.ISO8859-6', 'ar_jo': 'ar_JO.ISO8859-6', @@ -818,6 +838,7 @@ 'arabic': 'ar_AA.ISO8859-6', 'arabic.iso88596': 'ar_AA.ISO8859-6', 'as': 'as_IN.UTF-8', + 'as_in': 'as_IN.UTF-8', 'az': 'az_AZ.ISO8859-9E', 'az_az': 'az_AZ.ISO8859-9E', 'az_az.iso88599e': 'az_AZ.ISO8859-9E', @@ -835,6 +856,7 @@ 'bg_bg.koi8r': 'bg_BG.KOI8-R', 'bg_bg.microsoftcp1251': 'bg_BG.CP1251', 'bn_in': 'bn_IN.UTF-8', + 'bo_in': 'bo_IN.UTF-8', 'bokmal': 'nb_NO.ISO8859-1', 'bokm\xe5l': 'nb_NO.ISO8859-1', 'br': 'br_FR.ISO8859-1', @@ -852,6 +874,7 @@ 'c': 'C', 'c-french': 'fr_CA.ISO8859-1', 'c-french.iso88591': 'fr_CA.ISO8859-1', + 'c.ascii': 'C', 'c.en': 'C', 'c.iso88591': 'en_US.ISO8859-1', 'c_c': 'C', @@ -889,7 +912,7 @@ 'croatian': 'hr_HR.ISO8859-2', 'cs': 'cs_CZ.ISO8859-2', 'cs_cs': 'cs_CZ.ISO8859-2', - 'cs_cs.iso88592': 'cs_CS.ISO8859-2', + 'cs_cs.iso88592': 'cs_CZ.ISO8859-2', 'cs_cz': 'cs_CZ.ISO8859-2', 'cs_cz.iso88592': 'cs_CZ.ISO8859-2', 'cy': 'cy_GB.ISO8859-1', @@ -1193,12 +1216,13 @@ 'he_il.cp1255': 'he_IL.CP1255', 'he_il.iso88598': 'he_IL.ISO8859-8', 'he_il.microsoftcp1255': 'he_IL.CP1255', - 'hebrew': 'iw_IL.ISO8859-8', - 'hebrew.iso88598': 'iw_IL.ISO8859-8', + 'hebrew': 'he_IL.ISO8859-8', + 'hebrew.iso88598': 'he_IL.ISO8859-8', 'hi': 'hi_IN.ISCII-DEV', 'hi_in': 'hi_IN.ISCII-DEV', 'hi_in.isciidev': 'hi_IN.ISCII-DEV', 'hne': 'hne_IN.UTF-8', + 'hne_in': 'hne_IN.UTF-8', 'hr': 'hr_HR.ISO8859-2', 'hr_hr': 'hr_HR.ISO8859-2', 'hr_hr.iso88592': 'hr_HR.ISO8859-2', @@ -1286,6 +1310,7 @@ 'korean': 'ko_KR.eucKR', 'korean.euc': 'ko_KR.eucKR', 'ks': 'ks_IN.UTF-8', + 'ks_in': 'ks_IN.UTF-8', 'ks_in at devanagari': 'ks_IN at devanagari.UTF-8', 'kw': 'kw_GB.ISO8859-1', 'kw_gb': 'kw_GB.ISO8859-1', @@ -1310,6 +1335,7 @@ 'lv_lv.iso885913': 'lv_LV.ISO8859-13', 'lv_lv.iso88594': 'lv_LV.ISO8859-4', 'mai': 'mai_IN.UTF-8', + 'mai_in': 'mai_IN.UTF-8', 'mi': 'mi_NZ.ISO8859-1', 'mi_nz': 'mi_NZ.ISO8859-1', 'mi_nz.iso88591': 'mi_NZ.ISO8859-1', @@ -1319,6 +1345,7 @@ 'mk_mk.iso88595': 'mk_MK.ISO8859-5', 'mk_mk.microsoftcp1251': 'mk_MK.CP1251', 'ml': 'ml_IN.UTF-8', + 'ml_in': 'ml_IN.UTF-8', 'mr': 'mr_IN.UTF-8', 'mr_in': 'mr_IN.UTF-8', 'ms': 'ms_MY.ISO8859-1', @@ -1333,6 +1360,7 @@ 'nb_no.iso88591': 'nb_NO.ISO8859-1', 'nb_no.iso885915': 'nb_NO.ISO8859-15', 'nb_no at euro': 'nb_NO.ISO8859-15', + 'ne_np': 'ne_NP.UTF-8', 'nl': 'nl_NL.ISO8859-1', 'nl.iso885915': 'nl_NL.ISO8859-15', 'nl_be': 'nl_BE.ISO8859-1', @@ -1385,6 +1413,7 @@ 'oc_fr.iso885915': 'oc_FR.ISO8859-15', 'oc_fr at euro': 'oc_FR.ISO8859-15', 'or': 'or_IN.UTF-8', + 'or_in': 'or_IN.UTF-8', 'pa': 'pa_IN.UTF-8', 'pa_in': 'pa_IN.UTF-8', 'pd': 'pd_US.ISO8859-1', @@ -1446,7 +1475,9 @@ 'rw': 'rw_RW.ISO8859-1', 'rw_rw': 'rw_RW.ISO8859-1', 'rw_rw.iso88591': 'rw_RW.ISO8859-1', - 'sd': 'sd_IN at devanagari.UTF-8', + 'sd': 'sd_IN.UTF-8', + 'sd at devanagari': 'sd_IN at devanagari.UTF-8', + 'sd_in at devanagari': 'sd_IN at devanagari.UTF-8', 'se_no': 'se_NO.UTF-8', 'serbocroatian': 'sr_RS.UTF-8 at latin', 'sh': 'sr_RS.UTF-8 at latin', @@ -1480,13 +1511,13 @@ 'sr': 'sr_RS.UTF-8', 'sr at cyrillic': 'sr_RS.UTF-8', 'sr at latin': 'sr_RS.UTF-8 at latin', - 'sr at latn': 'sr_RS.UTF-8 at latin', - 'sr_cs': 'sr_RS.UTF-8', + 'sr at latn': 'sr_CS.UTF-8 at latin', + 'sr_cs': 'sr_CS.UTF-8', 'sr_cs.iso88592': 'sr_CS.ISO8859-2', 'sr_cs.iso88592 at latn': 'sr_CS.ISO8859-2', 'sr_cs.iso88595': 'sr_CS.ISO8859-5', - 'sr_cs.utf8 at latn': 'sr_RS.UTF-8 at latin', - 'sr_cs at latn': 'sr_RS.UTF-8 at latin', + 'sr_cs.utf8 at latn': 'sr_CS.UTF-8 at latin', + 'sr_cs at latn': 'sr_CS.UTF-8 at latin', 'sr_me': 'sr_ME.UTF-8', 'sr_rs': 'sr_RS.UTF-8', 'sr_rs.utf8 at latn': 'sr_RS.UTF-8 at latin', @@ -1564,6 +1595,7 @@ 'universal': 'en_US.utf', 'universal.utf8 at ucs4': 'en_US.UTF-8', 'ur': 'ur_PK.CP1256', + 'ur_in': 'ur_IN.UTF-8', 'ur_pk': 'ur_PK.CP1256', 'ur_pk.cp1256': 'ur_PK.CP1256', 'ur_pk.microsoftcp1256': 'ur_PK.CP1256', diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -27,6 +27,9 @@ Library ------- +- Issue #20034: Updated alias mapping to most recent locale.alias file + from X.org distribution using makelocalealias.py. + - Issue #5815: Fixed support for locales with modifiers. Fixed support for locale encodings with hyphens. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 17:24:30 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 20 Dec 2013 17:24:30 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzIwMDM0?= =?utf-8?q?=3A_Updated_alias_mapping_to_most_recent_locale=2Ealias_file?= Message-ID: <3dmFf65jCGz7LkW@mail.python.org> http://hg.python.org/cpython/rev/ed62c4c70c4d changeset: 88085:ed62c4c70c4d branch: 3.3 parent: 88078:ed16f6695638 user: Serhiy Storchaka date: Fri Dec 20 18:23:26 2013 +0200 summary: Issue #20034: Updated alias mapping to most recent locale.alias file from X.org distribution using makelocalealias.py. files: Lib/locale.py | 54 +++++++++++++++++++++++++++++++------- Misc/NEWS | 3 ++ 2 files changed, 46 insertions(+), 11 deletions(-) diff --git a/Lib/locale.py b/Lib/locale.py --- a/Lib/locale.py +++ b/Lib/locale.py @@ -775,11 +775,30 @@ # updated 'sr_yu.utf8 at cyrillic' -> 'sr_CS.UTF-8' to 'sr_RS.UTF-8' # updated 'sr_yu at cyrillic' -> 'sr_CS.ISO8859-5' to 'sr_RS.UTF-8' # +# SS 2013-12-20: +# Updated alias mapping to most recent locale.alias file +# from X.org distribution using makelocalealias.py. +# +# These are the differences compared to the old mapping (Python 3.3.3 +# and older): +# +# updated 'a3' -> 'a3_AZ.KOI8-C' to 'az_AZ.KOI8-C' +# updated 'a3_az' -> 'a3_AZ.KOI8-C' to 'az_AZ.KOI8-C' +# updated 'a3_az.koi8c' -> 'a3_AZ.KOI8-C' to 'az_AZ.KOI8-C' +# updated 'cs_cs.iso88592' -> 'cs_CS.ISO8859-2' to 'cs_CZ.ISO8859-2' +# updated 'hebrew' -> 'iw_IL.ISO8859-8' to 'he_IL.ISO8859-8' +# updated 'hebrew.iso88598' -> 'iw_IL.ISO8859-8' to 'he_IL.ISO8859-8' +# updated 'sd' -> 'sd_IN at devanagari.UTF-8' to 'sd_IN.UTF-8' +# updated 'sr at latn' -> 'sr_RS.UTF-8 at latin' to 'sr_CS.UTF-8 at latin' +# updated 'sr_cs' -> 'sr_RS.UTF-8' to 'sr_CS.UTF-8' +# updated 'sr_cs.utf8 at latn' -> 'sr_RS.UTF-8 at latin' to 'sr_CS.UTF-8 at latin' +# updated 'sr_cs at latn' -> 'sr_RS.UTF-8 at latin' to 'sr_CS.UTF-8 at latin' locale_alias = { - 'a3': 'a3_AZ.KOI8-C', - 'a3_az': 'a3_AZ.KOI8-C', - 'a3_az.koi8c': 'a3_AZ.KOI8-C', + 'a3': 'az_AZ.KOI8-C', + 'a3_az': 'az_AZ.KOI8-C', + 'a3_az.koi8c': 'az_AZ.KOI8-C', + 'a3_az.koic': 'az_AZ.KOI8-C', 'af': 'af_ZA.ISO8859-1', 'af_za': 'af_ZA.ISO8859-1', 'af_za.iso88591': 'af_ZA.ISO8859-1', @@ -798,6 +817,7 @@ 'ar_dz.iso88596': 'ar_DZ.ISO8859-6', 'ar_eg': 'ar_EG.ISO8859-6', 'ar_eg.iso88596': 'ar_EG.ISO8859-6', + 'ar_in': 'ar_IN.UTF-8', 'ar_iq': 'ar_IQ.ISO8859-6', 'ar_iq.iso88596': 'ar_IQ.ISO8859-6', 'ar_jo': 'ar_JO.ISO8859-6', @@ -827,6 +847,7 @@ 'arabic': 'ar_AA.ISO8859-6', 'arabic.iso88596': 'ar_AA.ISO8859-6', 'as': 'as_IN.UTF-8', + 'as_in': 'as_IN.UTF-8', 'az': 'az_AZ.ISO8859-9E', 'az_az': 'az_AZ.ISO8859-9E', 'az_az.iso88599e': 'az_AZ.ISO8859-9E', @@ -844,6 +865,7 @@ 'bg_bg.koi8r': 'bg_BG.KOI8-R', 'bg_bg.microsoftcp1251': 'bg_BG.CP1251', 'bn_in': 'bn_IN.UTF-8', + 'bo_in': 'bo_IN.UTF-8', 'bokmal': 'nb_NO.ISO8859-1', 'bokm\xe5l': 'nb_NO.ISO8859-1', 'br': 'br_FR.ISO8859-1', @@ -861,6 +883,7 @@ 'c': 'C', 'c-french': 'fr_CA.ISO8859-1', 'c-french.iso88591': 'fr_CA.ISO8859-1', + 'c.ascii': 'C', 'c.en': 'C', 'c.iso88591': 'en_US.ISO8859-1', 'c_c': 'C', @@ -898,7 +921,7 @@ 'croatian': 'hr_HR.ISO8859-2', 'cs': 'cs_CZ.ISO8859-2', 'cs_cs': 'cs_CZ.ISO8859-2', - 'cs_cs.iso88592': 'cs_CS.ISO8859-2', + 'cs_cs.iso88592': 'cs_CZ.ISO8859-2', 'cs_cz': 'cs_CZ.ISO8859-2', 'cs_cz.iso88592': 'cs_CZ.ISO8859-2', 'cy': 'cy_GB.ISO8859-1', @@ -1202,12 +1225,13 @@ 'he_il.cp1255': 'he_IL.CP1255', 'he_il.iso88598': 'he_IL.ISO8859-8', 'he_il.microsoftcp1255': 'he_IL.CP1255', - 'hebrew': 'iw_IL.ISO8859-8', - 'hebrew.iso88598': 'iw_IL.ISO8859-8', + 'hebrew': 'he_IL.ISO8859-8', + 'hebrew.iso88598': 'he_IL.ISO8859-8', 'hi': 'hi_IN.ISCII-DEV', 'hi_in': 'hi_IN.ISCII-DEV', 'hi_in.isciidev': 'hi_IN.ISCII-DEV', 'hne': 'hne_IN.UTF-8', + 'hne_in': 'hne_IN.UTF-8', 'hr': 'hr_HR.ISO8859-2', 'hr_hr': 'hr_HR.ISO8859-2', 'hr_hr.iso88592': 'hr_HR.ISO8859-2', @@ -1295,6 +1319,7 @@ 'korean': 'ko_KR.eucKR', 'korean.euc': 'ko_KR.eucKR', 'ks': 'ks_IN.UTF-8', + 'ks_in': 'ks_IN.UTF-8', 'ks_in at devanagari': 'ks_IN at devanagari.UTF-8', 'kw': 'kw_GB.ISO8859-1', 'kw_gb': 'kw_GB.ISO8859-1', @@ -1319,6 +1344,7 @@ 'lv_lv.iso885913': 'lv_LV.ISO8859-13', 'lv_lv.iso88594': 'lv_LV.ISO8859-4', 'mai': 'mai_IN.UTF-8', + 'mai_in': 'mai_IN.UTF-8', 'mi': 'mi_NZ.ISO8859-1', 'mi_nz': 'mi_NZ.ISO8859-1', 'mi_nz.iso88591': 'mi_NZ.ISO8859-1', @@ -1328,6 +1354,7 @@ 'mk_mk.iso88595': 'mk_MK.ISO8859-5', 'mk_mk.microsoftcp1251': 'mk_MK.CP1251', 'ml': 'ml_IN.UTF-8', + 'ml_in': 'ml_IN.UTF-8', 'mr': 'mr_IN.UTF-8', 'mr_in': 'mr_IN.UTF-8', 'ms': 'ms_MY.ISO8859-1', @@ -1342,6 +1369,7 @@ 'nb_no.iso88591': 'nb_NO.ISO8859-1', 'nb_no.iso885915': 'nb_NO.ISO8859-15', 'nb_no at euro': 'nb_NO.ISO8859-15', + 'ne_np': 'ne_NP.UTF-8', 'nl': 'nl_NL.ISO8859-1', 'nl.iso885915': 'nl_NL.ISO8859-15', 'nl_be': 'nl_BE.ISO8859-1', @@ -1394,6 +1422,7 @@ 'oc_fr.iso885915': 'oc_FR.ISO8859-15', 'oc_fr at euro': 'oc_FR.ISO8859-15', 'or': 'or_IN.UTF-8', + 'or_in': 'or_IN.UTF-8', 'pa': 'pa_IN.UTF-8', 'pa_in': 'pa_IN.UTF-8', 'pd': 'pd_US.ISO8859-1', @@ -1455,7 +1484,9 @@ 'rw': 'rw_RW.ISO8859-1', 'rw_rw': 'rw_RW.ISO8859-1', 'rw_rw.iso88591': 'rw_RW.ISO8859-1', - 'sd': 'sd_IN at devanagari.UTF-8', + 'sd': 'sd_IN.UTF-8', + 'sd at devanagari': 'sd_IN at devanagari.UTF-8', + 'sd_in at devanagari': 'sd_IN at devanagari.UTF-8', 'se_no': 'se_NO.UTF-8', 'serbocroatian': 'sr_RS.UTF-8 at latin', 'sh': 'sr_RS.UTF-8 at latin', @@ -1489,13 +1520,13 @@ 'sr': 'sr_RS.UTF-8', 'sr at cyrillic': 'sr_RS.UTF-8', 'sr at latin': 'sr_RS.UTF-8 at latin', - 'sr at latn': 'sr_RS.UTF-8 at latin', - 'sr_cs': 'sr_RS.UTF-8', + 'sr at latn': 'sr_CS.UTF-8 at latin', + 'sr_cs': 'sr_CS.UTF-8', 'sr_cs.iso88592': 'sr_CS.ISO8859-2', 'sr_cs.iso88592 at latn': 'sr_CS.ISO8859-2', 'sr_cs.iso88595': 'sr_CS.ISO8859-5', - 'sr_cs.utf8 at latn': 'sr_RS.UTF-8 at latin', - 'sr_cs at latn': 'sr_RS.UTF-8 at latin', + 'sr_cs.utf8 at latn': 'sr_CS.UTF-8 at latin', + 'sr_cs at latn': 'sr_CS.UTF-8 at latin', 'sr_me': 'sr_ME.UTF-8', 'sr_rs': 'sr_RS.UTF-8', 'sr_rs.utf8 at latn': 'sr_RS.UTF-8 at latin', @@ -1573,6 +1604,7 @@ 'universal': 'en_US.utf', 'universal.utf8 at ucs4': 'en_US.UTF-8', 'ur': 'ur_PK.CP1256', + 'ur_in': 'ur_IN.UTF-8', 'ur_pk': 'ur_PK.CP1256', 'ur_pk.cp1256': 'ur_PK.CP1256', 'ur_pk.microsoftcp1256': 'ur_PK.CP1256', diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,9 @@ Library ------- +- Issue #20034: Updated alias mapping to most recent locale.alias file + from X.org distribution using makelocalealias.py. + - Issue #5815: Fixed support for locales with modifiers. Fixed support for locale encodings with hyphens. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 17:51:45 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Fri, 20 Dec 2013 17:51:45 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2320034=3A_Updated_alias_mapping_to_most_recent_l?= =?utf-8?q?ocale=2Ealias_file?= Message-ID: <3dmGFY0wrrz7LjT@mail.python.org> http://hg.python.org/cpython/rev/72e68af9e2fa changeset: 88086:72e68af9e2fa parent: 88083:00d09afb57ca parent: 88085:ed62c4c70c4d user: Serhiy Storchaka date: Fri Dec 20 18:50:32 2013 +0200 summary: Issue #20034: Updated alias mapping to most recent locale.alias file from X.org distribution using makelocalealias.py. files: Lib/locale.py | 54 +++++++++++++++++++++++++++++++------- Misc/NEWS | 3 ++ 2 files changed, 46 insertions(+), 11 deletions(-) diff --git a/Lib/locale.py b/Lib/locale.py --- a/Lib/locale.py +++ b/Lib/locale.py @@ -764,11 +764,30 @@ # updated 'sr_yu.utf8 at cyrillic' -> 'sr_CS.UTF-8' to 'sr_RS.UTF-8' # updated 'sr_yu at cyrillic' -> 'sr_CS.ISO8859-5' to 'sr_RS.UTF-8' # +# SS 2013-12-20: +# Updated alias mapping to most recent locale.alias file +# from X.org distribution using makelocalealias.py. +# +# These are the differences compared to the old mapping (Python 3.3.3 +# and older): +# +# updated 'a3' -> 'a3_AZ.KOI8-C' to 'az_AZ.KOI8-C' +# updated 'a3_az' -> 'a3_AZ.KOI8-C' to 'az_AZ.KOI8-C' +# updated 'a3_az.koi8c' -> 'a3_AZ.KOI8-C' to 'az_AZ.KOI8-C' +# updated 'cs_cs.iso88592' -> 'cs_CS.ISO8859-2' to 'cs_CZ.ISO8859-2' +# updated 'hebrew' -> 'iw_IL.ISO8859-8' to 'he_IL.ISO8859-8' +# updated 'hebrew.iso88598' -> 'iw_IL.ISO8859-8' to 'he_IL.ISO8859-8' +# updated 'sd' -> 'sd_IN at devanagari.UTF-8' to 'sd_IN.UTF-8' +# updated 'sr at latn' -> 'sr_RS.UTF-8 at latin' to 'sr_CS.UTF-8 at latin' +# updated 'sr_cs' -> 'sr_RS.UTF-8' to 'sr_CS.UTF-8' +# updated 'sr_cs.utf8 at latn' -> 'sr_RS.UTF-8 at latin' to 'sr_CS.UTF-8 at latin' +# updated 'sr_cs at latn' -> 'sr_RS.UTF-8 at latin' to 'sr_CS.UTF-8 at latin' locale_alias = { - 'a3': 'a3_AZ.KOI8-C', - 'a3_az': 'a3_AZ.KOI8-C', - 'a3_az.koi8c': 'a3_AZ.KOI8-C', + 'a3': 'az_AZ.KOI8-C', + 'a3_az': 'az_AZ.KOI8-C', + 'a3_az.koi8c': 'az_AZ.KOI8-C', + 'a3_az.koic': 'az_AZ.KOI8-C', 'af': 'af_ZA.ISO8859-1', 'af_za': 'af_ZA.ISO8859-1', 'af_za.iso88591': 'af_ZA.ISO8859-1', @@ -787,6 +806,7 @@ 'ar_dz.iso88596': 'ar_DZ.ISO8859-6', 'ar_eg': 'ar_EG.ISO8859-6', 'ar_eg.iso88596': 'ar_EG.ISO8859-6', + 'ar_in': 'ar_IN.UTF-8', 'ar_iq': 'ar_IQ.ISO8859-6', 'ar_iq.iso88596': 'ar_IQ.ISO8859-6', 'ar_jo': 'ar_JO.ISO8859-6', @@ -816,6 +836,7 @@ 'arabic': 'ar_AA.ISO8859-6', 'arabic.iso88596': 'ar_AA.ISO8859-6', 'as': 'as_IN.UTF-8', + 'as_in': 'as_IN.UTF-8', 'az': 'az_AZ.ISO8859-9E', 'az_az': 'az_AZ.ISO8859-9E', 'az_az.iso88599e': 'az_AZ.ISO8859-9E', @@ -833,6 +854,7 @@ 'bg_bg.koi8r': 'bg_BG.KOI8-R', 'bg_bg.microsoftcp1251': 'bg_BG.CP1251', 'bn_in': 'bn_IN.UTF-8', + 'bo_in': 'bo_IN.UTF-8', 'bokmal': 'nb_NO.ISO8859-1', 'bokm\xe5l': 'nb_NO.ISO8859-1', 'br': 'br_FR.ISO8859-1', @@ -850,6 +872,7 @@ 'c': 'C', 'c-french': 'fr_CA.ISO8859-1', 'c-french.iso88591': 'fr_CA.ISO8859-1', + 'c.ascii': 'C', 'c.en': 'C', 'c.iso88591': 'en_US.ISO8859-1', 'c_c': 'C', @@ -887,7 +910,7 @@ 'croatian': 'hr_HR.ISO8859-2', 'cs': 'cs_CZ.ISO8859-2', 'cs_cs': 'cs_CZ.ISO8859-2', - 'cs_cs.iso88592': 'cs_CS.ISO8859-2', + 'cs_cs.iso88592': 'cs_CZ.ISO8859-2', 'cs_cz': 'cs_CZ.ISO8859-2', 'cs_cz.iso88592': 'cs_CZ.ISO8859-2', 'cy': 'cy_GB.ISO8859-1', @@ -1191,12 +1214,13 @@ 'he_il.cp1255': 'he_IL.CP1255', 'he_il.iso88598': 'he_IL.ISO8859-8', 'he_il.microsoftcp1255': 'he_IL.CP1255', - 'hebrew': 'iw_IL.ISO8859-8', - 'hebrew.iso88598': 'iw_IL.ISO8859-8', + 'hebrew': 'he_IL.ISO8859-8', + 'hebrew.iso88598': 'he_IL.ISO8859-8', 'hi': 'hi_IN.ISCII-DEV', 'hi_in': 'hi_IN.ISCII-DEV', 'hi_in.isciidev': 'hi_IN.ISCII-DEV', 'hne': 'hne_IN.UTF-8', + 'hne_in': 'hne_IN.UTF-8', 'hr': 'hr_HR.ISO8859-2', 'hr_hr': 'hr_HR.ISO8859-2', 'hr_hr.iso88592': 'hr_HR.ISO8859-2', @@ -1284,6 +1308,7 @@ 'korean': 'ko_KR.eucKR', 'korean.euc': 'ko_KR.eucKR', 'ks': 'ks_IN.UTF-8', + 'ks_in': 'ks_IN.UTF-8', 'ks_in at devanagari': 'ks_IN at devanagari.UTF-8', 'kw': 'kw_GB.ISO8859-1', 'kw_gb': 'kw_GB.ISO8859-1', @@ -1308,6 +1333,7 @@ 'lv_lv.iso885913': 'lv_LV.ISO8859-13', 'lv_lv.iso88594': 'lv_LV.ISO8859-4', 'mai': 'mai_IN.UTF-8', + 'mai_in': 'mai_IN.UTF-8', 'mi': 'mi_NZ.ISO8859-1', 'mi_nz': 'mi_NZ.ISO8859-1', 'mi_nz.iso88591': 'mi_NZ.ISO8859-1', @@ -1317,6 +1343,7 @@ 'mk_mk.iso88595': 'mk_MK.ISO8859-5', 'mk_mk.microsoftcp1251': 'mk_MK.CP1251', 'ml': 'ml_IN.UTF-8', + 'ml_in': 'ml_IN.UTF-8', 'mr': 'mr_IN.UTF-8', 'mr_in': 'mr_IN.UTF-8', 'ms': 'ms_MY.ISO8859-1', @@ -1331,6 +1358,7 @@ 'nb_no.iso88591': 'nb_NO.ISO8859-1', 'nb_no.iso885915': 'nb_NO.ISO8859-15', 'nb_no at euro': 'nb_NO.ISO8859-15', + 'ne_np': 'ne_NP.UTF-8', 'nl': 'nl_NL.ISO8859-1', 'nl.iso885915': 'nl_NL.ISO8859-15', 'nl_be': 'nl_BE.ISO8859-1', @@ -1383,6 +1411,7 @@ 'oc_fr.iso885915': 'oc_FR.ISO8859-15', 'oc_fr at euro': 'oc_FR.ISO8859-15', 'or': 'or_IN.UTF-8', + 'or_in': 'or_IN.UTF-8', 'pa': 'pa_IN.UTF-8', 'pa_in': 'pa_IN.UTF-8', 'pd': 'pd_US.ISO8859-1', @@ -1444,7 +1473,9 @@ 'rw': 'rw_RW.ISO8859-1', 'rw_rw': 'rw_RW.ISO8859-1', 'rw_rw.iso88591': 'rw_RW.ISO8859-1', - 'sd': 'sd_IN at devanagari.UTF-8', + 'sd': 'sd_IN.UTF-8', + 'sd at devanagari': 'sd_IN at devanagari.UTF-8', + 'sd_in at devanagari': 'sd_IN at devanagari.UTF-8', 'se_no': 'se_NO.UTF-8', 'serbocroatian': 'sr_RS.UTF-8 at latin', 'sh': 'sr_RS.UTF-8 at latin', @@ -1478,13 +1509,13 @@ 'sr': 'sr_RS.UTF-8', 'sr at cyrillic': 'sr_RS.UTF-8', 'sr at latin': 'sr_RS.UTF-8 at latin', - 'sr at latn': 'sr_RS.UTF-8 at latin', - 'sr_cs': 'sr_RS.UTF-8', + 'sr at latn': 'sr_CS.UTF-8 at latin', + 'sr_cs': 'sr_CS.UTF-8', 'sr_cs.iso88592': 'sr_CS.ISO8859-2', 'sr_cs.iso88592 at latn': 'sr_CS.ISO8859-2', 'sr_cs.iso88595': 'sr_CS.ISO8859-5', - 'sr_cs.utf8 at latn': 'sr_RS.UTF-8 at latin', - 'sr_cs at latn': 'sr_RS.UTF-8 at latin', + 'sr_cs.utf8 at latn': 'sr_CS.UTF-8 at latin', + 'sr_cs at latn': 'sr_CS.UTF-8 at latin', 'sr_me': 'sr_ME.UTF-8', 'sr_rs': 'sr_RS.UTF-8', 'sr_rs.utf8 at latn': 'sr_RS.UTF-8 at latin', @@ -1562,6 +1593,7 @@ 'universal': 'en_US.utf', 'universal.utf8 at ucs4': 'en_US.UTF-8', 'ur': 'ur_PK.CP1256', + 'ur_in': 'ur_IN.UTF-8', 'ur_pk': 'ur_PK.CP1256', 'ur_pk.cp1256': 'ur_PK.CP1256', 'ur_pk.microsoftcp1256': 'ur_PK.CP1256', diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #20034: Updated alias mapping to most recent locale.alias file + from X.org distribution using makelocalealias.py. + - Issue #5815: Fixed support for locales with modifiers. Fixed support for locale encodings with hyphens. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 20:26:32 2013 From: python-checkins at python.org (zach.ware) Date: Fri, 20 Dec 2013 20:26:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogVXBkYXRlIHRlc3Qu?= =?utf-8?q?outstanding=5Fbugs=2Epy?= Message-ID: <3dmKh84wmxz7LjS@mail.python.org> http://hg.python.org/cpython/rev/49dd8e1ef4c3 changeset: 88087:49dd8e1ef4c3 branch: 3.3 parent: 88085:ed62c4c70c4d user: Zachary Ware date: Fri Dec 20 13:25:07 2013 -0600 summary: Update test.outstanding_bugs.py files: Lib/test/outstanding_bugs.py | 74 +----------------------- 1 files changed, 2 insertions(+), 72 deletions(-) diff --git a/Lib/test/outstanding_bugs.py b/Lib/test/outstanding_bugs.py --- a/Lib/test/outstanding_bugs.py +++ b/Lib/test/outstanding_bugs.py @@ -10,79 +10,9 @@ from test import support # -# One test case for outstanding bugs at the moment: +# No test cases for outstanding bugs at the moment. # -# test_io -import io -class TextIOWrapperTest(unittest.TestCase): - - def setUp(self): - self.testdata = b"AAA\r\nBBB\rCCC\r\nDDD\nEEE\r\n" - self.normalized = b"AAA\nBBB\nCCC\nDDD\nEEE\n".decode("ASCII") - - def tearDown(self): - support.unlink(support.TESTFN) - - - def test_issue1395_1(self): - txt = io.TextIOWrapper(io.BytesIO(self.testdata), encoding="ASCII") - - # read one char at a time - reads = "" - while True: - c = txt.read(1) - if not c: - break - reads += c - self.assertEqual(reads, self.normalized) - - def test_issue1395_2(self): - txt = io.TextIOWrapper(io.BytesIO(self.testdata), encoding="ASCII") - txt._CHUNK_SIZE = 4 - - reads = "" - while True: - c = txt.read(4) - if not c: - break - reads += c - self.assertEqual(reads, self.normalized) - - def test_issue1395_3(self): - txt = io.TextIOWrapper(io.BytesIO(self.testdata), encoding="ASCII") - txt._CHUNK_SIZE = 4 - - reads = txt.read(4) - reads += txt.read(4) - reads += txt.readline() - reads += txt.readline() - reads += txt.readline() - self.assertEqual(reads, self.normalized) - - def test_issue1395_4(self): - txt = io.TextIOWrapper(io.BytesIO(self.testdata), encoding="ASCII") - txt._CHUNK_SIZE = 4 - - reads = txt.read(4) - reads += txt.read() - self.assertEqual(reads, self.normalized) - - def test_issue1395_5(self): - txt = io.TextIOWrapper(io.BytesIO(self.testdata), encoding="ASCII") - txt._CHUNK_SIZE = 4 - - reads = txt.read(4) - pos = txt.tell() - txt.seek(0) - txt.seek(pos) - self.assertEqual(txt.read(4), "BBB\n") - - - -def test_main(): - support.run_unittest( - TextIOWrapperTest) if __name__ == "__main__": - test_main() + unittest.main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 20:26:34 2013 From: python-checkins at python.org (zach.ware) Date: Fri, 20 Dec 2013 20:26:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Update_test=2Eoutstanding=5Fbugs=2Epy?= Message-ID: <3dmKhB0hKmz7LjS@mail.python.org> http://hg.python.org/cpython/rev/be22ffb4fdf1 changeset: 88088:be22ffb4fdf1 parent: 88086:72e68af9e2fa parent: 88087:49dd8e1ef4c3 user: Zachary Ware date: Fri Dec 20 13:25:53 2013 -0600 summary: Update test.outstanding_bugs.py files: Lib/test/outstanding_bugs.py | 74 +----------------------- 1 files changed, 2 insertions(+), 72 deletions(-) diff --git a/Lib/test/outstanding_bugs.py b/Lib/test/outstanding_bugs.py --- a/Lib/test/outstanding_bugs.py +++ b/Lib/test/outstanding_bugs.py @@ -10,79 +10,9 @@ from test import support # -# One test case for outstanding bugs at the moment: +# No test cases for outstanding bugs at the moment. # -# test_io -import io -class TextIOWrapperTest(unittest.TestCase): - - def setUp(self): - self.testdata = b"AAA\r\nBBB\rCCC\r\nDDD\nEEE\r\n" - self.normalized = b"AAA\nBBB\nCCC\nDDD\nEEE\n".decode("ASCII") - - def tearDown(self): - support.unlink(support.TESTFN) - - - def test_issue1395_1(self): - txt = io.TextIOWrapper(io.BytesIO(self.testdata), encoding="ASCII") - - # read one char at a time - reads = "" - while True: - c = txt.read(1) - if not c: - break - reads += c - self.assertEqual(reads, self.normalized) - - def test_issue1395_2(self): - txt = io.TextIOWrapper(io.BytesIO(self.testdata), encoding="ASCII") - txt._CHUNK_SIZE = 4 - - reads = "" - while True: - c = txt.read(4) - if not c: - break - reads += c - self.assertEqual(reads, self.normalized) - - def test_issue1395_3(self): - txt = io.TextIOWrapper(io.BytesIO(self.testdata), encoding="ASCII") - txt._CHUNK_SIZE = 4 - - reads = txt.read(4) - reads += txt.read(4) - reads += txt.readline() - reads += txt.readline() - reads += txt.readline() - self.assertEqual(reads, self.normalized) - - def test_issue1395_4(self): - txt = io.TextIOWrapper(io.BytesIO(self.testdata), encoding="ASCII") - txt._CHUNK_SIZE = 4 - - reads = txt.read(4) - reads += txt.read() - self.assertEqual(reads, self.normalized) - - def test_issue1395_5(self): - txt = io.TextIOWrapper(io.BytesIO(self.testdata), encoding="ASCII") - txt._CHUNK_SIZE = 4 - - reads = txt.read(4) - pos = txt.tell() - txt.seek(0) - txt.seek(pos) - self.assertEqual(txt.read(4), "BBB\n") - - - -def test_main(): - support.run_unittest( - TextIOWrapperTest) if __name__ == "__main__": - test_main() + unittest.main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 21:11:56 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 20 Dec 2013 21:11:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_email_package_to_whats?= =?utf-8?q?new_list_of_significantly_improved_stdlib_modules=2E?= Message-ID: <3dmLhX4NX3z7LjS@mail.python.org> http://hg.python.org/cpython/rev/9ac04353aa98 changeset: 88089:9ac04353aa98 user: R David Murray date: Fri Dec 20 13:10:43 2013 -0500 summary: Add email package to whatsnew list of significantly improved stdlib modules. files: Doc/whatsnew/3.4.rst | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -115,6 +115,9 @@ * TLSv1.1 and TLSv1.2 support for :mod:`ssl`. * :mod:`multiprocessing` now has option to avoid using :func:`os.fork` on Unix (:issue:`8713`). +* :mod:`email` has a new submodule, :mod:`~email.contentmanager`, and + a new :mod:`~email.message.Message` subclass + (:class:`~email.contentmanager.EmailMessage`) that simplify MIME handling. CPython implementation improvements: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 21:11:57 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 20 Dec 2013 21:11:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_versionadded_to_enum_d?= =?utf-8?q?ocs=2E?= Message-ID: <3dmLhY675Dz7Ljq@mail.python.org> http://hg.python.org/cpython/rev/471ba7045ed9 changeset: 88090:471ba7045ed9 user: R David Murray date: Fri Dec 20 14:20:49 2013 -0500 summary: Add versionadded to enum docs. files: Doc/library/enum.rst | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -9,6 +9,8 @@ .. :sectionauthor:: Eli Bendersky , .. :sectionauthor:: Ethan Furman +.. versionadded:: 3.4 + **Source code:** :source:`Lib/enum.py` ---------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 21:11:59 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 20 Dec 2013 21:11:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_Sphinx_markup_for_asyn?= =?utf-8?q?cio_modules_so_=3Amod=3A=60asyncio=60_links_work=2E?= Message-ID: <3dmLhb0mzQz7Lk6@mail.python.org> http://hg.python.org/cpython/rev/42f52935ccee changeset: 88091:42f52935ccee user: R David Murray date: Fri Dec 20 14:37:39 2013 -0500 summary: Fix Sphinx markup for asyncio modules so :mod:`asyncio` links work. files: Doc/library/asyncio-eventloop.rst | 2 +- Doc/library/asyncio-protocol.rst | 2 +- Doc/library/asyncio-sync.rst | 1 + Doc/library/asyncio-task.rst | 2 +- 4 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -1,4 +1,4 @@ -.. module:: asyncio +.. currentmodule:: asyncio .. _event-loop: diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -1,4 +1,4 @@ -.. module:: asyncio +.. currentmodule:: asyncio ++++++++++++++++++++++++ Transports and protocols diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst --- a/Doc/library/asyncio-sync.rst +++ b/Doc/library/asyncio-sync.rst @@ -1,3 +1,4 @@ +.. currentmodule:: asyncio .. _sync: Synchronization primitives diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -1,4 +1,4 @@ -.. module:: asyncio +.. currentmodule:: asyncio Tasks and coroutines ==================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 21:12:00 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 20 Dec 2013 21:12:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Move_versionadded_for_ensu?= =?utf-8?q?repip_into_the_standard_location_=28right_after_title=29?= Message-ID: <3dmLhc2Z59z7LkZ@mail.python.org> http://hg.python.org/cpython/rev/75e4068bb796 changeset: 88092:75e4068bb796 user: R David Murray date: Fri Dec 20 14:40:11 2013 -0500 summary: Move versionadded for ensurepip into the standard location (right after title) files: Doc/library/ensurepip.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/ensurepip.rst b/Doc/library/ensurepip.rst --- a/Doc/library/ensurepip.rst +++ b/Doc/library/ensurepip.rst @@ -5,6 +5,8 @@ :synopsis: Bootstrapping the ``pip`` installer into an existing Python installation or virtual environment. +.. versionadded:: 3.4 + The :mod:`ensurepip` package provides support for bootstrapping the ``pip`` installer into an existing Python installation or virtual environment. This bootstrapping approach reflects the fact that ``pip`` is an independent @@ -18,8 +20,6 @@ when creating a virtual environment) or after explicitly uninstalling ``pip``. -.. versionadded:: 3.4 - .. note:: This module *does not* access the internet. All of the components -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 21:12:01 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 20 Dec 2013 21:12:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Move_versionadded_for_trac?= =?utf-8?q?emalloc_into_the_standard_location_=28right_after_title=29?= Message-ID: <3dmLhd4Yf9z7Ll6@mail.python.org> http://hg.python.org/cpython/rev/eba17a83bab1 changeset: 88093:eba17a83bab1 user: R David Murray date: Fri Dec 20 14:48:50 2013 -0500 summary: Move versionadded for tracemalloc into the standard location (right after title) files: Doc/library/tracemalloc.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/tracemalloc.rst b/Doc/library/tracemalloc.rst --- a/Doc/library/tracemalloc.rst +++ b/Doc/library/tracemalloc.rst @@ -4,6 +4,8 @@ .. module:: tracemalloc :synopsis: Trace memory allocations. +.. versionadded:: 3.4 + The tracemalloc module is a debug tool to trace memory blocks allocated by Python. It provides the following information: @@ -23,8 +25,6 @@ :envvar:`PYTHONTRACEMALLOC` environment variable to ``25``, or use the :option:`-X` ``tracemalloc=25`` command line option. -.. versionadded:: 3.4 - Examples ======== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 21:12:02 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 20 Dec 2013 21:12:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Make_the_=27PEP_446=27_str?= =?utf-8?q?ing_link_to_the_PEP_in_whatsnew_new_feature_description=2E?= Message-ID: <3dmLhf6Lv9z7LkW@mail.python.org> http://hg.python.org/cpython/rev/759e989a229e changeset: 88094:759e989a229e user: R David Murray date: Fri Dec 20 14:50:12 2013 -0500 summary: Make the 'PEP 446' string link to the PEP in whatsnew new feature description. I also further qualified the cross reference link, since those are global names. files: Doc/whatsnew/3.4.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -101,7 +101,7 @@ New expected features for Python implementations: -* :ref:`PEP 446: Make newly created file descriptors non-inheritable `. +* :pep:`446`: :ref:`Make newly created file descriptors non-inheritable `. * command line option for :ref:`isolated mode `, (:issue:`16499`). * :ref:`improvements ` in the handling of @@ -170,7 +170,7 @@ Donald Stufft, Nick Coghlan, Martin von L?wis and Ned Deily. -.. _pep-446: +.. _whatsnew-pep-446: PEP 446: Make newly created file descriptors non-inheritable ============================================================ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 21:12:04 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 20 Dec 2013 21:12:04 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_spelling_error_in_sing?= =?utf-8?q?le_dispatch_whatsnew_summary=2C_and_add_link=2E?= Message-ID: <3dmLhh0zJwz7Lk6@mail.python.org> http://hg.python.org/cpython/rev/9fcd376bc525 changeset: 88095:9fcd376bc525 user: R David Murray date: Fri Dec 20 15:00:54 2013 -0500 summary: Fix spelling error in single dispatch whatsnew summary, and add link. files: Doc/whatsnew/3.4.rst | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -109,7 +109,8 @@ Significantly Improved Library Modules: -* Single-dispatch generic functions in :mod:`functoools` (:pep:`443`) +* :ref:`Single-dispatch generic functions ` in + :mod:`functools` (:pep:`443`) * New :mod:`pickle` protocol 4 (:pep:`3154`) * SHA-3 (Keccak) support for :mod:`hashlib`. * TLSv1.1 and TLSv1.2 support for :mod:`ssl`. @@ -553,6 +554,8 @@ (Contributed by Alon Horev and Nick Coghlan in :issue:`4331`) +.. _whatsnew-singledispatch: + The new :func:`~functools.singledispatch` decorator brings support for single-dispatch generic functions to the Python standard library. Where object oriented programming focuses on grouping multiple operations on a -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 22:39:42 2013 From: python-checkins at python.org (antoine.pitrou) Date: Fri, 20 Dec 2013 22:39:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_Add_asyncio_=22experts=22?= Message-ID: <3dmNdp0tnVz7LjM@mail.python.org> http://hg.python.org/devguide/rev/d0d47f6852e6 changeset: 657:d0d47f6852e6 user: Antoine Pitrou date: Fri Dec 20 22:39:37 2013 +0100 summary: Add asyncio "experts" files: experts.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/experts.rst b/experts.rst --- a/experts.rst +++ b/experts.rst @@ -56,7 +56,7 @@ array ast benjamin.peterson asynchat josiahcarlson, giampaolo.rodola, stutzbach -asyncio +asyncio gvanrossum, haypo, pitrou asyncore josiahcarlson, giampaolo.rodola, stutzbach atexit audioop -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Fri Dec 20 23:16:58 2013 From: python-checkins at python.org (antoine.pitrou) Date: Fri, 20 Dec 2013 23:16:58 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Bump_Python_version?= Message-ID: <3dmPSp3mYHzRwm@mail.python.org> http://hg.python.org/peps/rev/5342f21e4321 changeset: 5327:5342f21e4321 user: Antoine Pitrou date: Fri Dec 20 23:16:54 2013 +0100 summary: Bump Python version files: pep-0455.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0455.txt b/pep-0455.txt --- a/pep-0455.txt +++ b/pep-0455.txt @@ -8,7 +8,7 @@ Type: Standards Track Content-Type: text/x-rst Created: 13-Sep-2013 -Python-Version: 3.4 +Python-Version: 3.5 Post-History: -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri Dec 20 23:17:35 2013 From: python-checkins at python.org (guido.van.rossum) Date: Fri, 20 Dec 2013 23:17:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_Export_all_abst?= =?utf-8?q?ract_protocol_and_transport_classes=2E_Fixes_issue_=2320029=2E?= Message-ID: <3dmPTW5xnpz7LjZ@mail.python.org> http://hg.python.org/cpython/rev/0a135e790ce5 changeset: 88096:0a135e790ce5 user: Guido van Rossum date: Fri Dec 20 14:16:21 2013 -0800 summary: asyncio: Export all abstract protocol and transport classes. Fixes issue #20029. files: Lib/asyncio/protocols.py | 3 ++- Lib/asyncio/transports.py | 4 +++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Lib/asyncio/protocols.py b/Lib/asyncio/protocols.py --- a/Lib/asyncio/protocols.py +++ b/Lib/asyncio/protocols.py @@ -1,6 +1,7 @@ """Abstract Protocol class.""" -__all__ = ['Protocol', 'DatagramProtocol'] +__all__ = ['BaseProtocol', 'Protocol', 'DatagramProtocol', + 'SubprocessProtocol'] class BaseProtocol: diff --git a/Lib/asyncio/transports.py b/Lib/asyncio/transports.py --- a/Lib/asyncio/transports.py +++ b/Lib/asyncio/transports.py @@ -4,7 +4,9 @@ PY34 = sys.version_info >= (3, 4) -__all__ = ['ReadTransport', 'WriteTransport', 'Transport'] +__all__ = ['BaseTransport', 'ReadTransport', 'WriteTransport', + 'Transport', 'DatagramTransport', 'SubprocessTransport', + ] class BaseTransport: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 23:28:15 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 20 Dec 2013 23:28:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_s/no_new_synta?= =?utf-8?q?x_features_planned/no_new_syntax_features_added/?= Message-ID: <3dmPjq6SYKz7LjM@mail.python.org> http://hg.python.org/cpython/rev/4bf26388e892 changeset: 88097:4bf26388e892 user: R David Murray date: Fri Dec 20 16:04:29 2013 -0500 summary: whatsnew: s/no new syntax features planned/no new syntax features added/ files: Doc/whatsnew/3.4.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -86,7 +86,7 @@ New syntax features: -* No new syntax features are planned for Python 3.4. +* No new syntax features were added in Python 3.4. New library modules: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 23:28:17 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 20 Dec 2013 23:28:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_No_need_for_single_para_de?= =?utf-8?q?scriptions_to_have_separate_contributed_by_para=2E?= Message-ID: <3dmPjs0zDwz7LjM@mail.python.org> http://hg.python.org/cpython/rev/ede4801020e8 changeset: 88098:ede4801020e8 user: R David Murray date: Fri Dec 20 16:12:28 2013 -0500 summary: No need for single para descriptions to have separate contributed by para. files: Doc/whatsnew/3.4.rst | 14 -------------- 1 files changed, 0 insertions(+), 14 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -573,7 +573,6 @@ ------- New :func:`hashlib.pbkdf2_hmac` function. - (Contributed by Christian Heimes in :issue:`18582`) @@ -582,7 +581,6 @@ Added a new :func:`html.unescape` function that converts HTML5 character references to the corresponding Unicode characters. - (Contributed by Ezio Melotti in :issue:`2927`) Added a new *convert_charrefs* keyword argument to @@ -590,11 +588,9 @@ all character references. For backward-compatibility, its value defaults to ``False``, but it will change to ``True`` in future versions, so you are invited to set it explicitly and update your code to use this new feature. - (Contributed by Ezio Melotti in :issue:`13633`) The *strict* argument of :class:`~html.parser.HTMLParser` is now deprecated. - (Contributed by Ezio Melotti in :issue:`15114`) @@ -623,7 +619,6 @@ ---- mmap objects can now be weakref'ed. - (Contributed by Valerie Lambert in :issue:`4885`.) @@ -711,7 +706,6 @@ The repr of :ref:`regex objects ` now includes the pattern and the flags; the repr of :ref:`match objects ` now includes the start, end, and the part of the string that matched. - (Contributed by Serhiy Storchaka in :issue:`13592` and :issue:`17087`.) @@ -746,7 +740,6 @@ --- TLSv1.1 and TLSv1.2 support. - (Contributed by Michele Orr? and Antoine Pitrou in :issue:`16692`) * New diagnostic functions :func:`~ssl.get_default_verify_paths`, @@ -761,7 +754,6 @@ Support for server-side SNI using the new :meth:`ssl.SSLContext.set_servername_callback` method. - (Contributed by Daniel Black in :issue:`8109`.) @@ -779,7 +771,6 @@ ------ Streaming struct unpacking using :func:`struct.iter_unpack`. - (Contributed by Antoine Pitrou in :issue:`17804`.) @@ -804,7 +795,6 @@ ------ Add support.for ``data:`` URLs in :mod:`urllib.request`. - (Contributed by Mathias Panzenb?ck in :issue:`16423`.) @@ -813,7 +803,6 @@ Support for easy dynamically-generated subtests using the :meth:`~unittest.TestCase.subTest` context manager. - (Contributed by Antoine Pitrou in :issue:`16997`.) @@ -844,7 +833,6 @@ Add an event-driven parser for non-blocking applications, :class:`~xml.etree.ElementTree.XMLPullParser`. - (Contributed by Antoine Pitrou in :issue:`17741`.) @@ -853,7 +841,6 @@ Add a filter function to ignore some packages (tests for instance), :meth:`~zipfile.PyZipFile.writepy`. - (Contributed by Christian Tismer in :issue:`19274`.) @@ -861,7 +848,6 @@ ================== Tab-completion is now enabled by default in the interactive interpreter. - (Contributed by Antoine Pitrou and ?ric Araujo in :issue:`5845`.) Python invocation changes -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 23:28:18 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 20 Dec 2013 23:28:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Include_sha3_addition_in_h?= =?utf-8?q?ashlib_section_of_whatsnew=2E?= Message-ID: <3dmPjt2gWFz7Ljp@mail.python.org> http://hg.python.org/cpython/rev/f08e41a99cb6 changeset: 88099:f08e41a99cb6 user: R David Murray date: Fri Dec 20 16:33:52 2013 -0500 summary: Include sha3 addition in hashlib section of whatsnew. And link to it from the summary, and link to the Hash Algorithms section of hashlib from the linked description. files: Doc/library/hashlib.rst | 4 +++- Doc/whatsnew/3.4.rst | 8 +++++++- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -33,6 +33,8 @@ also" section at the end. +.. _hash-algorithms: + Hash algorithms --------------- @@ -63,7 +65,7 @@ the OpenSSL library that Python uses on your platform. .. versionchanged:: 3.4 - Add sha3 family of hash algorithms. + Added sha3 family of hash algorithms. For example, to obtain the digest of the byte string ``b'Nobody inspects the spammish repetition'``:: diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -112,7 +112,7 @@ * :ref:`Single-dispatch generic functions ` in :mod:`functools` (:pep:`443`) * New :mod:`pickle` protocol 4 (:pep:`3154`) -* SHA-3 (Keccak) support for :mod:`hashlib`. +* :ref:`SHA-3 (Keccak) support ` for :mod:`hashlib`. * TLSv1.1 and TLSv1.2 support for :mod:`ssl`. * :mod:`multiprocessing` now has option to avoid using :func:`os.fork` on Unix (:issue:`8713`). @@ -575,6 +575,12 @@ New :func:`hashlib.pbkdf2_hmac` function. (Contributed by Christian Heimes in :issue:`18582`) +.. _whatsnew-sha3: + +New :ref:`hash algorithms ` ``sah3_224()``, ``sha3_256()``, +``sha3_384()``, and ``sha3_512()``. (Contributed by Christian Heimes in +:issue:`16113`.) + html ---- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 23:28:19 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 20 Dec 2013 23:28:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_link_from_pickle_proto?= =?utf-8?q?col_4_in_summary_to_the_section_about_it=2E?= Message-ID: <3dmPjv4MY1z7LkR@mail.python.org> http://hg.python.org/cpython/rev/c4238e91532e changeset: 88100:c4238e91532e user: R David Murray date: Fri Dec 20 16:38:09 2013 -0500 summary: Add link from pickle protocol 4 in summary to the section about it. files: Doc/whatsnew/3.4.rst | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -111,7 +111,7 @@ * :ref:`Single-dispatch generic functions ` in :mod:`functools` (:pep:`443`) -* New :mod:`pickle` protocol 4 (:pep:`3154`) +* New :mod:`pickle` :ref:`protocol 4 ` (:pep:`3154`) * :ref:`SHA-3 (Keccak) support ` for :mod:`hashlib`. * TLSv1.1 and TLSv1.2 support for :mod:`ssl`. * :mod:`multiprocessing` now has option to avoid using :func:`os.fork` @@ -291,6 +291,8 @@ the new methods. +.. _whatsnew-protocol-4: + Pickle protocol 4 ================= -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 23:28:20 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 20 Dec 2013 23:28:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Tidy_up_ssl_whatsnew_refer?= =?utf-8?q?ences=2C_make_ssl_section_formatting_consistent=2E?= Message-ID: <3dmPjw61c9z7Lk7@mail.python.org> http://hg.python.org/cpython/rev/c0a23ff41238 changeset: 88101:c0a23ff41238 user: R David Murray date: Fri Dec 20 17:08:39 2013 -0500 summary: Tidy up ssl whatsnew references, make ssl section formatting consistent. Also remove some extra blank lines in the ssl doc acctions for tls1.1/1.2, and reflow a paragraph. files: Doc/library/ssl.rst | 5 +---- Doc/whatsnew/3.4.rst | 22 ++++++++++++---------- 2 files changed, 13 insertions(+), 14 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -547,7 +547,6 @@ .. data:: PROTOCOL_TLSv1_1 - Selects TLS version 1.1 as the channel encryption protocol. Available only with openssl version 1.0.1+. @@ -555,11 +554,9 @@ .. data:: PROTOCOL_TLSv1_2 - Selects TLS version 1.2 as the channel encryption protocol. This is the most modern version, and probably the best choice for maximum protection, if both - sides can speak it. - Available only with openssl version 1.0.1+. + sides can speak it. Available only with openssl version 1.0.1+. .. versionadded:: 3.4 diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -113,7 +113,7 @@ :mod:`functools` (:pep:`443`) * New :mod:`pickle` :ref:`protocol 4 ` (:pep:`3154`) * :ref:`SHA-3 (Keccak) support ` for :mod:`hashlib`. -* TLSv1.1 and TLSv1.2 support for :mod:`ssl`. +* :ref:`TLSv1.1 and TLSv1.2 support ` for :mod:`ssl`. * :mod:`multiprocessing` now has option to avoid using :func:`os.fork` on Unix (:issue:`8713`). * :mod:`email` has a new submodule, :mod:`~email.contentmanager`, and @@ -747,18 +747,20 @@ ssl --- -TLSv1.1 and TLSv1.2 support. -(Contributed by Michele Orr? and Antoine Pitrou in :issue:`16692`) +.. _whatsnew-tls-11-12: -* New diagnostic functions :func:`~ssl.get_default_verify_paths`, - :meth:`~ssl.SSLContext.cert_store_stats` and - :meth:`~ssl.SSLContext.get_ca_certs` +:data:`~ssl.PROTOCOL_TLSv1_1` and :data:`~ssl.PROTOCOL_TLSv1_2` (TLSv1.1 and +TLSv1.2 support) have been added; support for these protocols is only available if +Python is linked with OpenSSL 1.0.1 or later. (Contributed by Michele Orr? and +Antoine Pitrou in :issue:`16692`) -* Add :func:`ssl.enum_cert_store` to retrieve certificates and CRL from Windows' - cert store. +New diagnostic functions :func:`~ssl.get_default_verify_paths`, +:meth:`~ssl.SSLContext.cert_store_stats` and +:meth:`~ssl.SSLContext.get_ca_certs` (Contributed by Christian Heimes +in :issue:`18143` and :issue:`18147`) -(Contributed by Christian Heimes in :issue:`18143`, :issue:`18147` and -:issue:`17134`.) +Add :func:`ssl.enum_cert_store` to retrieve certificates and CRL from Windows' +cert store. (Contributed by Christian Heimes in :issue:`17134`.) Support for server-side SNI using the new :meth:`ssl.SSLContext.set_servername_callback` method. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 23:28:22 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 20 Dec 2013 23:28:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_links_to_discussion_of?= =?utf-8?q?_multiprocessing_spawn/forkserver_feature=2E?= Message-ID: <3dmPjy0k78z7Lkg@mail.python.org> http://hg.python.org/cpython/rev/e36d03a1fadb changeset: 88102:e36d03a1fadb user: R David Murray date: Fri Dec 20 17:23:57 2013 -0500 summary: Add links to discussion of multiprocessing spawn/forkserver feature. files: Doc/library/multiprocessing.rst | 4 +++- Doc/whatsnew/3.4.rst | 19 +++++++++++-------- 2 files changed, 14 insertions(+), 9 deletions(-) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -101,6 +101,8 @@ Contexts and start methods ~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. _multiprocessing-start-methods: + Depending on the platform, :mod:`multiprocessing` supports three ways to start a process. These *start methods* are @@ -147,7 +149,7 @@ is a serious matter since the system allows only a limited number, and they will not be automatically unlinked until the next reboot.) -To select the a start method you use the :func:`set_start_method` in +To select a start method you use the :func:`set_start_method` in the ``if __name__ == '__main__'`` clause of the main module. For example:: diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -114,8 +114,8 @@ * New :mod:`pickle` :ref:`protocol 4 ` (:pep:`3154`) * :ref:`SHA-3 (Keccak) support ` for :mod:`hashlib`. * :ref:`TLSv1.1 and TLSv1.2 support ` for :mod:`ssl`. -* :mod:`multiprocessing` now has option to avoid using :func:`os.fork` - on Unix (:issue:`8713`). +* :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork + on Unix ` (:issue:`8713`). * :mod:`email` has a new submodule, :mod:`~email.contentmanager`, and a new :mod:`~email.message.Message` subclass (:class:`~email.contentmanager.EmailMessage`) that simplify MIME handling. @@ -633,14 +633,17 @@ multiprocessing --------------- -On Unix, two new *start methods* (``spawn`` and ``forkserver``) have been -added for starting processes using :mod:`multiprocessing`. These make -the mixing of processes with threads more robust, and the ``spawn`` -method matches the semantics that multiprocessing has always used on -Windows. (Contributed by Richard Oudkerk in :issue:`8713`). +.. _whatsnew-multiprocessing-no-fork: + +On Unix, two new :ref:`start methods ` +(``spawn`` and ``forkserver``) have been added for starting processes using +:mod:`multiprocessing`. These make the mixing of processes with threads more +robust, and the ``spawn`` method matches the semantics that multiprocessing has +always used on Windows. (Contributed by Richard Oudkerk in :issue:`8713`). Also, except when using the old *fork* start method, child processes -will no longer inherit unneeded handles/file descriptors from their parents. +will no longer inherit unneeded handles/file descriptors from their parents +(part of :issue:`8713`). :mod:`multiprocessing` now relies on :mod:`runpy` (which implements the ``-m`` switch) to initialise ``__main__`` appropriately in child processes -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 20 23:28:23 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 20 Dec 2013 23:28:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_link_from_email_summar?= =?utf-8?q?y_entry_to_entry_in_changed_module_section=2E?= Message-ID: <3dmPjz2L25z7LkM@mail.python.org> http://hg.python.org/cpython/rev/b9895b64d2d9 changeset: 88103:b9895b64d2d9 user: R David Murray date: Fri Dec 20 17:26:52 2013 -0500 summary: Add link from email summary entry to entry in changed module section. files: Doc/whatsnew/3.4.rst | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -118,7 +118,8 @@ on Unix ` (:issue:`8713`). * :mod:`email` has a new submodule, :mod:`~email.contentmanager`, and a new :mod:`~email.message.Message` subclass - (:class:`~email.contentmanager.EmailMessage`) that simplify MIME handling. + (:class:`~email.contentmanager.EmailMessage`) that :ref:`simplify MIME + handling `. CPython implementation improvements: @@ -529,6 +530,8 @@ (Contributed by R. David Murray in :issue:`18600`.) +.. _whatsnew_email_contentmanager: + A pair of new subclasses of :class:`~email.message.Message` have been added, along with a new sub-module, :mod:`~email.contentmanager`. All documentation is currently in the new module, which is being added as part of the new -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 21 00:20:41 2013 From: python-checkins at python.org (victor.stinner) Date: Sat, 21 Dec 2013 00:20:41 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319967=3A_Defer_th?= =?utf-8?q?e_formating_of_the_traceback_in_asyncio=2EFuture_destructor?= Message-ID: <3dmQtK5Ybkz7Ljp@mail.python.org> http://hg.python.org/cpython/rev/06ed1691efdc changeset: 88104:06ed1691efdc user: Victor Stinner date: Sat Dec 21 00:19:33 2013 +0100 summary: Issue #19967: Defer the formating of the traceback in asyncio.Future destructor files: Lib/asyncio/futures.py | 25 ++++++++++++++----------- 1 files changed, 14 insertions(+), 11 deletions(-) diff --git a/Lib/asyncio/futures.py b/Lib/asyncio/futures.py --- a/Lib/asyncio/futures.py +++ b/Lib/asyncio/futures.py @@ -131,8 +131,8 @@ _blocking = False # proper use of future (yield vs yield from) - _traceback = None # Used for Python 3.4 and later - _tb_logger = None # Used for Python 3.3 only + _log_traceback = False # Used for Python 3.4 and later + _tb_logger = None # Used for Python 3.3 only def __init__(self, *, loop=None): """Initialize the future. @@ -168,9 +168,13 @@ if _PY34: def __del__(self): - if self._traceback is not None: - logger.error('Future/Task exception was never retrieved:\n%s', - ''.join(self._traceback)) + if not self._log_traceback: + # set_exception() was not called, or result() or exception() + # has consumed the exception + return + exc = self._exception + logger.error('Future/Task exception was never retrieved:', + exc_info=(exc.__class__, exc, exc.__traceback__)) def cancel(self): """Cancel the future and schedule callbacks. @@ -224,10 +228,10 @@ raise CancelledError if self._state != _FINISHED: raise InvalidStateError('Result is not ready.') - self._traceback = None + self._log_traceback = False if self._tb_logger is not None: self._tb_logger.clear() - self._tb_logger = None + self._tb_logger = None if self._exception is not None: raise self._exception return self._result @@ -244,10 +248,10 @@ raise CancelledError if self._state != _FINISHED: raise InvalidStateError('Exception is not set.') - self._traceback = None + self._log_traceback = False if self._tb_logger is not None: self._tb_logger.clear() - self._tb_logger = None + self._tb_logger = None return self._exception def add_done_callback(self, fn): @@ -301,8 +305,7 @@ self._state = _FINISHED self._schedule_callbacks() if _PY34: - self._traceback = traceback.format_exception( - exception.__class__, exception, exception.__traceback__) + self._log_traceback = True else: self._tb_logger = _TracebackLogger(exception) # Arrange for the logger to be activated after all callbacks -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 21 01:16:17 2013 From: python-checkins at python.org (donald.stufft) Date: Sat, 21 Dec 2013 01:16:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Upgrade_pip_to_1=2E5rc2_an?= =?utf-8?q?d_setuptools_to_2=2E0=2E1?= Message-ID: <3dmS6T0Q5yz7LjM@mail.python.org> http://hg.python.org/cpython/rev/4336a0a5074d changeset: 88105:4336a0a5074d user: Donald Stufft date: Fri Dec 20 19:03:18 2013 -0500 summary: Upgrade pip to 1.5rc2 and setuptools to 2.0.1 files: Lib/ensurepip/__init__.py | 4 ++-- Lib/ensurepip/_bundled/pip-1.5rc1-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/pip-1.5rc2-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/setuptools-1.3.2-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/setuptools-2.0.1-py2.py3-none-any.whl | Bin 5 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -10,9 +10,9 @@ __all__ = ["version", "bootstrap"] -_SETUPTOOLS_VERSION = "1.3.2" +_SETUPTOOLS_VERSION = "2.0.1" -_PIP_VERSION = "1.5rc1" +_PIP_VERSION = "1.5rc2" _PROJECTS = [ ("setuptools", _SETUPTOOLS_VERSION), diff --git a/Lib/ensurepip/_bundled/pip-1.5rc1-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-1.5rc1-py2.py3-none-any.whl deleted file mode 100644 index 6418895a582a1a9243a73d79a8b73a5140d46e78..0000000000000000000000000000000000000000 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/pip-1.5rc2-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-1.5rc2-py2.py3-none-any.whl new file mode 100644 index 0000000000000000000000000000000000000000..719bd84505dbdcbf13516df0b9ab979afa178d78 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/setuptools-1.3.2-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-1.3.2-py2.py3-none-any.whl deleted file mode 100644 index 81962b10814b495bb8f7f0edc34e5274a0580865..0000000000000000000000000000000000000000 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/setuptools-2.0.1-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-2.0.1-py2.py3-none-any.whl new file mode 100644 index 0000000000000000000000000000000000000000..67ef8a1a9757835bb1de367c0f8c3b5b579ffa15 GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 21 05:47:18 2013 From: python-checkins at python.org (guido.van.rossum) Date: Sat, 21 Dec 2013 05:47:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_asyncio=3A_Fix_space_in_lo?= =?utf-8?q?g_message_about_poll_time=2E?= Message-ID: <3dmZ7B5CpfzScj@mail.python.org> http://hg.python.org/cpython/rev/46de42d47cb6 changeset: 88106:46de42d47cb6 user: Guido van Rossum date: Fri Dec 20 20:37:41 2013 -0800 summary: asyncio: Fix space in log message about poll time. files: Lib/asyncio/base_events.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -613,7 +613,7 @@ t0 = self.time() event_list = self._selector.select(timeout) t1 = self.time() - argstr = '' if timeout is None else '{:.3f}'.format(timeout) + argstr = '' if timeout is None else ' {:.3f}'.format(timeout) if t1-t0 >= 1: level = logging.INFO else: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 21 05:47:20 2013 From: python-checkins at python.org (guido.van.rossum) Date: Sat, 21 Dec 2013 05:47:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_some_asyncio_news_for_?= =?utf-8?q?beta_2=2E?= Message-ID: <3dmZ7D02FzzScj@mail.python.org> http://hg.python.org/cpython/rev/320fbfd40691 changeset: 88107:320fbfd40691 user: Guido van Rossum date: Fri Dec 20 20:47:06 2013 -0800 summary: Add some asyncio news for beta 2. files: Misc/NEWS | 14 ++++++++++++++ 1 files changed, 14 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,20 @@ Library ------- +- Issue #19967: Thanks to the PEP 442, asyncio.Future now uses a + destructor to log uncaught exceptions, instead of the dedicated + _TracebackLogger class. + +- Added a Task.current_task() class method to asyncio. + +- Issue #19850: Set SA_RESTART in asyncio when registering a signal + handler to limit EINTR occurrences. + +- Implemented write flow control in asyncio for proactor event loop (Windows). + +- Change write buffer in asyncio use to avoid O(N**2) behavior. Make + write()/sendto() accept bytearray/memoryview. + - Issue #20034: Updated alias mapping to most recent locale.alias file from X.org distribution using makelocalealias.py. -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sat Dec 21 09:49:04 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 21 Dec 2013 09:49:04 +0100 Subject: [Python-checkins] Daily reference leaks (4336a0a5074d): sum=8 Message-ID: results for 4336a0a5074d on branch "default" -------------------------------------------- test_asyncio leaked [0, 0, 4] memory blocks, sum=4 test_site leaked [2, -2, 2] references, sum=2 test_site leaked [2, -2, 2] memory blocks, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog6HH6f_', '-x'] From python-checkins at python.org Sat Dec 21 13:36:40 2013 From: python-checkins at python.org (nick.coghlan) Date: Sat, 21 Dec 2013 13:36:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Withdraw_the_obsolete_PEP_395?= Message-ID: <3dmmXm3Q2qzQph@mail.python.org> http://hg.python.org/peps/rev/16b61233cdca changeset: 5328:16b61233cdca user: Nick Coghlan date: Sat Dec 21 22:36:31 2013 +1000 summary: Withdraw the obsolete PEP 395 files: pep-0395.txt | 18 +++++++++++++++++- 1 files changed, 17 insertions(+), 1 deletions(-) diff --git a/pep-0395.txt b/pep-0395.txt --- a/pep-0395.txt +++ b/pep-0395.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan -Status: Deferred +Status: Withdrawn Type: Standards Track Content-Type: text/x-rst Created: 4-Mar-2011 @@ -11,6 +11,22 @@ Post-History: 5-Mar-2011, 19-Nov-2011 +PEP Withdrawal +============== + +This PEP was withdrawn by the author in December 2013, as other significant +changes in the time since it was written have rendered several aspects +obsolete. Most notably :pep:`420` namespace packages rendered some of the +proposals related to package detection unworkable and :pep:`451` module +specifications resolved the multiprocessing issues and provide a possible +means to tackle the pickle compatibility issues. + +A future PEP to resolve the remaining issues would still be appropriate, +but it's worth starting any such effort as a fresh PEP restating the +remaining problems in an updated context rather than trying to build on +this one directly. + + Abstract ======== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sat Dec 21 13:43:18 2013 From: python-checkins at python.org (nick.coghlan) Date: Sat, 21 Dec 2013 13:43:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Withdraw_the_now_obsolete_PEP?= =?utf-8?q?_406?= Message-ID: <3dmmhQ1VgnzRNv@mail.python.org> http://hg.python.org/peps/rev/1ed784aa9f3c changeset: 5329:1ed784aa9f3c user: Nick Coghlan date: Sat Dec 21 22:43:08 2013 +1000 summary: Withdraw the now obsolete PEP 406 files: pep-0406.txt | 19 ++++++++++--------- 1 files changed, 10 insertions(+), 9 deletions(-) diff --git a/pep-0406.txt b/pep-0406.txt --- a/pep-0406.txt +++ b/pep-0406.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan , Greg Slodkowicz -Status: Deferred +Status: Withdrawn Type: Standards Track Content-Type: text/x-rst Created: 4-Jul-2011 @@ -30,16 +30,17 @@ invalidating path cache entries when ``sys.path`` is modified). -PEP Deferral -============ +PEP Withdrawal +============== -The import system is already seeing substantial changes in Python 3.3, to -natively handle packages split across multiple directories (PEP 382) and -(potentially) to make the import semantics in the main module better match -those in other modules (PEP 395). +The import system has seen substantial changes since this PEP was originally +written, as part of :pep:`420` in Python 3.3 and :pep:`451` in Python 3.4. -Accordingly, the proposal in this PEP will not be seriously considered until -Python 3.4 at the earliest. +While providing an encapsulation of the import state is still highly +desirable, it is better tackled in a new PEP using PEP 451 as a foundation, +and permitting only the use of PEP 451 compatible finders and loaders (as +those avoid many of the issues of direct manipulation of global state +associated with the previous loader API). Rationale -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sat Dec 21 13:44:27 2013 From: python-checkins at python.org (nick.coghlan) Date: Sat, 21 Dec 2013 13:44:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Fix_PEP_type_for_PEP_453?= Message-ID: <3dmmjl2dMDzRNv@mail.python.org> http://hg.python.org/peps/rev/3fd3daed7670 changeset: 5330:3fd3daed7670 user: Nick Coghlan date: Sat Dec 21 22:44:18 2013 +1000 summary: Fix PEP type for PEP 453 files: pep-0453.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0453.txt b/pep-0453.txt --- a/pep-0453.txt +++ b/pep-0453.txt @@ -6,7 +6,7 @@ Nick Coghlan BDFL-Delegate: Martin von L?wis Status: Final -Type: Process +Type: Standards Track Content-Type: text/x-rst Created: 10-Aug-2013 Post-History: 30-Aug-2013, 15-Sep-2013, 18-Sep-2013, 19-Sep-2013, -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sat Dec 21 13:58:11 2013 From: python-checkins at python.org (nick.coghlan) Date: Sat, 21 Dec 2013 13:58:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Withdraw_my_faster_stdlib_upd?= =?utf-8?q?ates_PEP?= Message-ID: <3dmn1b1l51zRSx@mail.python.org> http://hg.python.org/peps/rev/1ae534736fd8 changeset: 5331:1ae534736fd8 user: Nick Coghlan date: Sat Dec 21 22:57:57 2013 +1000 summary: Withdraw my faster stdlib updates PEP files: pep-0413.txt | 19 ++++++++++++++++++- 1 files changed, 18 insertions(+), 1 deletions(-) diff --git a/pep-0413.txt b/pep-0413.txt --- a/pep-0413.txt +++ b/pep-0413.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan -Status: Deferred +Status: Withdrawn Type: Process Content-Type: text/x-rst Created: 2012-02-24 @@ -11,6 +11,23 @@ Resolution: TBD +PEP Withdrawal +============== + +With the acceptance of :pep:`453` meaning that ``pip`` will be available to +most new Python users by default, this will hopefully reduce the pressure +to add new modules to the standard library before they are sufficiently +mature. + +The last couple of years have also seen increased usage of the model where +a standard library package also has an equivalent available from the Python +Package Index that also supports older versions of Python. + +Given these two developments and the level of engagement throughout the +Python 3.4 release cycle, the PEP author no longer feels it would be +appropriate to make such a fundamental change to the standard library +development process. + Abstract ======== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sat Dec 21 14:03:28 2013 From: python-checkins at python.org (nick.coghlan) Date: Sat, 21 Dec 2013 14:03:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_First_draft_of_standard_metad?= =?utf-8?q?ata_extensions_PEP?= Message-ID: <3dmn7h4nRCzRFl@mail.python.org> http://hg.python.org/peps/rev/4fe2d2212465 changeset: 5332:4fe2d2212465 user: Nick Coghlan date: Sat Dec 21 23:02:32 2013 +1000 summary: First draft of standard metadata extensions PEP This is part of a larger refactoring of PEP 426, but I needed to commit this early in order to claim the PEP number and cross reference it from PEP 426 and 440. files: pep-0459.txt | 697 +++++++++++++++++++++++++++++++++++++++ 1 files changed, 697 insertions(+), 0 deletions(-) diff --git a/pep-0459.txt b/pep-0459.txt new file mode 100644 --- /dev/null +++ b/pep-0459.txt @@ -0,0 +1,697 @@ +PEP: 459 +Title: Standard Metadata Extensions for Python Software Packages +Version: $Revision$ +Last-Modified: $Date$ +Author: Nick Coghlan +BDFL-Delegate: Nick Coghlan +Discussions-To: Distutils SIG +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Requires: 426 +Created: 11 Nov 2013 +Post-History: + + +Abstract +======== + +This PEP describes several standard extensions to the Python metadata. + +Like all metadata extensions, each standard extension format is +independently versioned. Changing any of the formats requires an update +to this PEP, but does not require an update to the core packaging metadata. + +.. note:: + + These extensions may eventually be separated out into their own PEPs, + but we're already suffering from PEP overload in the packaging + metadata space. + + This PEP was initially created by slicing out large sections of earlier + drafts of PEP 426 and making them extensions, so some of the specifics + may still be rough in the new context. + + +Standard Extension Namespace +============================ + +The ``python`` project on the Python Package Index refers to the CPython +reference interpreter. This namespace is used as the namespace for the +standard metadata extensions. + +The currently defined standard extensions are: + +* ``python.details`` +* ``python.project`` +* ``python.integrator`` +* ``python.commands`` +* ``python.exports`` +* ``python.metadata_hooks`` + + +The ``details`` extension +========================= + +The ``details`` extension allows for more information to be provided +regarding the software distribution. + +The ``details`` extension contains three subfields: + +* ``license``: the copyright license for the distribution +* ``keywords``: package index keywords for the distribution +* ``classifiers``: package index Trove classifiers for the distribution +* ``document_names``: the names of additional metadata files + +All of these fields are optional. Automated tools MUST operate correctly if +a distribution does not provide them, including failing cleanly when an +operation depending on one of these fields is requested. + + +License +------- + +A short string summarising the license used for this distribution. + +Note that distributions that provide this field should still specify any +applicable license Trove classifiers in the `Classifiers`_ field. Even +when an appropriate Trove classifier is available, the license summary can +be a good way to specify a particular version of that license, or to +indicate any variations or exception to the license. + +This field SHOULD contain fewer than 512 characters and MUST contain fewer +than 2048. + +This field SHOULD NOT contain any line breaks. + +The full license text SHOULD be included as a separate file in the source +archive for the distribution. See `Document names`_ for details. + +Example:: + + "license": "GPL version 3, excluding DRM provisions" + + +Keywords +-------- + +A list of additional keywords to be used to assist searching for the +distribution in a larger catalog. + +Example:: + + "keywords": ["comfy", "chair", "cushions", "too silly", "monty python"] + + +Classifiers +----------- + +A list of strings, with each giving a single classification value +for the distribution. Classifiers are described in PEP 301 [2]. + +Example:: + + "classifiers": [ + "Development Status :: 4 - Beta", + "Environment :: Console (Text Based)", + "License :: OSI Approved :: GNU General Public License v3 (GPLv3)" + ] + + +Document names +-------------- + +Filenames for supporting documents included in the distribution's +``dist-info`` metadata directory. + +The following supporting documents can be named: + +* ``description``: a file containing a long description of the distribution +* ``license``: a file with the full text of the distribution's license +* ``changelog``: a file describing changes made to the distribution + +Supporting documents MUST be included directly in the ``dist-info`` +directory. Directory separators are NOT permitted in document names. + +The markup format (if any) for the file is indicated by the file extension. +This allows index servers and other automated tools to render included +text documents correctly and provide feedback on rendering errors, rather +than having to guess the intended format. + +If the filename has no extension, or the extension is not recognised, the +default rendering format MUST be plain text. + +The following markup renderers SHOULD be used for the specified file +extensions: + +* Plain text: ``.txt``, no extension, unknown extension +* reStructured Text: ``.rst`` +* Markdown: ``.md`` +* AsciiDoc: ``.adoc``, ``.asc``, ``.asciidoc`` +* HTML: ``.html``, ``.htm`` + +Automated tools MAY render one or more of the specified formats as plain +text and MAY render other markup formats beyond those listed. + +Automated tools SHOULD NOT make any assumptions regarding the maximum length +of supporting document content, except as necessary to protect the +integrity of a service. + +Example:: + + "document_names": { + "description": "README.rst", + "license": "LICENSE.rst", + "changelog": "NEWS" + } + + +The ``project`` extension +========================= + +The ``project`` extension allows for more information to be provided +regarding the creation and maintenance of the distribution. + +The ``project`` extension contains three subfields: + +* ``contacts``: key contact points for the distribution +* ``contributors``: other contributors to the distribution +* ``project_urls``: relevant URLs for the distribution + + +Contact information +------------------- + +Details on individuals and organisations are recorded as mappings with +the following subfields: + +* ``name``: the name of an individual or group +* ``email``: an email address (this may be a mailing list) +* ``url``: a URL (such as a profile page on a source code hosting service) +* ``role``: one of ``"author"``, ``"maintainer"`` or ``"contributor"`` + +The ``name`` subfield is required, the other subfields are optional. + +If no specific role is stated, the default is ``contributor``. + +Email addresses must be in the form ``local-part at domain`` where the +local-part may be up to 64 characters long and the entire email address +contains no more than 254 characters. The formal specification of the +format is in RFC 5322 (sections 3.2.3 and 3.4.1) and RFC 5321, with a more +readable form given in the informational RFC 3696 and the associated errata. + +The defined contributor roles are as follows: + +* ``author``: the original creator of a distribution +* ``maintainer``: the current lead contributor for a distribution, when + they are not the original creator +* ``contributor``: any other individuals or organizations involved in the + creation of the distribution + +Contact and contributor metadata is optional. Automated tools MUST operate +correctly if a distribution does not provide it, including failing cleanly +when an operation depending on one of these fields is requested. + + +Contacts +-------- + +A list of contributor entries giving the recommended contact points for +getting more information about the project. + +The example below would be suitable for a project that was in the process +of handing over from the original author to a new lead maintainer, while +operating as part of a larger development group. + +Example:: + + "contacts": [ + { + "name": "Python Packaging Authority/Distutils-SIG", + "email": "distutils-sig at python.org", + "url": "https://bitbucket.org/pypa/" + }, + { + "name": "Samantha C.", + "role": "maintainer", + "email": "dontblameme at example.org" + }, + { + "name": "Charlotte C.", + "role": "author", + "email": "iambecomingasketchcomedian at example.com" + } + ] + + +Contributors +------------ + +A list of contributor entries for other contributors not already listed as +current project points of contact. The subfields within the list elements +are the same as those for the main contact field. + +Example:: + + "contributors": [ + {"name": "John C."}, + {"name": "Erik I."}, + {"name": "Terry G."}, + {"name": "Mike P."}, + {"name": "Graeme C."}, + {"name": "Terry J."} + ] + + +Project URLs +------------ + +A mapping of arbitrary text labels to additional URLs relevant to the +project. + +While projects are free to choose their own labels and specific URLs, +it is RECOMMENDED that home page, source control, issue tracker and +documentation links be provided using the labels in the example below. + +URL labels MUST be treated as case insensitive by automated tools, but they +are not required to be valid Python identifiers. Any legal JSON string is +permitted as a URL label. + +Example:: + + "project_urls": { + "Documentation": "https://distlib.readthedocs.org", + "Home": "https://bitbucket.org/pypa/distlib", + "Repository": "https://bitbucket.org/pypa/distlib/src", + "Tracker": "https://bitbucket.org/pypa/distlib/issues" + } + + +The ``integrator`` extension +============================ + +Structurally, this extension is identical to the ``project`` extension. + +However, where the ``project`` metadata refers to the upstream creators +of the software, the ``integrator`` metadata refers to the downstream +redistributor of a modified version. + +If the software is being redistributed unmodified, then typically this +extension will not be used. However, if the software has been patched (for +example, backporting compatible fixes from a later version, or addressing +a platform compatibility issue), then this extension SHOULD be used, and +an integrator suffix added to the package version number. + +If there are multiple redistributors in the chain, each one just overwrites +this extension with their particular metadata. + + +The ``exports`` extension +========================= + +Most Python distributions expose packages and modules for import through +the Python module namespace. Distributions may also expose other +interfaces when installed. + +The ``exports`` extension contains three subfields: + +* ``modules``: modules exported by the distribution +* ``namespaces``: namespace packages that the distribution contributes to +* ``exports``: other Python interfaces exported by the distribution + + +Export specifiers +----------------- + +An export specifier is a string consisting of a fully qualified name, as +well as an optional extra name enclosed in square brackets. This gives the +following four possible forms for an export specifier:: + + module + module:name + module[requires_extra] + module:name[requires_extra] + +.. note:: + + The jsonschema file currently restricts qualified names using the + Python 2 ASCII identifier rules. This may need to be reconsidered + given the more relaxed identifier rules in Python 3. + +The meaning of the subfields is as follows: + +* ``module``: the module providing the export +* ``name``: if applicable, the qualified name of the export within the module +* ``requires_extra``: indicates the export will only work correctly if the + additional dependencies named in the given extra are available in the + installed environment + +.. note:: + + I tried this as a mapping with subfields, and it made the examples below + unreadable. While this PEP is mostly for tool use, readability still + matters to some degree for debugging purposes, and because I expect + snippets of the format to be reused elsewhere. + + +Modules +------- + +A list of qualified names of modules and packages that the distribution +provides for import. + +.. note:: + + The jsonschema file currently restricts qualified names using the + Python 2 ASCII identifier rules. This may need to be reconsidered + given the more relaxed identifier rules in Python 3. + +For names that contain dots, the portion of the name before the final dot +MUST appear either in the installed module list or in the namespace package +list. + +To help avoid name conflicts, it is RECOMMENDED that distributions provide +a single top level module or package that matches the distribution name +(or a lower case equivalent). This requires that the distribution name also +meet the requirements of a Python identifier, which are stricter than +those for distribution names). This practice will also make it easier to +find authoritative sources for modules. + +Index servers SHOULD allow multiple distributions to publish the same +modules, but MAY notify distribution authors of potential conflicts. + +Installation tools SHOULD report an error when asked to install a +distribution that provides a module that is also provided by a different, +previously installed, distribution. + +Note that attempting to import some declared modules may result in an +exception if the appropriate extras are not installed. + +Example:: + + "modules": ["chair", "chair.cushions", "python_sketches.nobody_expects"] + +.. note:: + + Making this a list of export specifiers instead would allow a distribution + to declare when a particular module requires a particular extra in order + to run correctly. On the other hand, there's an argument to be made that + that is the point where it starts to become worthwhile to split out a + separate distribution rather than using extras. + + +Namespaces +---------- + +A list of qualified names of namespace packages that the distribution +contributes modules to. + +.. note:: + + The jsonschema file currently restricts qualified names using the + Python 2 ASCII identifier rules. This may need to be reconsidered + given the more relaxed identifier rules in Python 3. + +On versions of Python prior to Python 3.3 (which provides native namespace +package support), installation tools SHOULD emit a suitable ``__init__.py`` +file to properly initialise the namespace rather than using a distribution +provided file. + +Installation tools SHOULD emit a warning and MAY emit an error if a +distribution declares a namespace package that conflicts with the name of +an already installed module or vice-versa. + +Example:: + + "namespaces": ["python_sketches"] + + +Exports +------- + +The ``exports`` field is a mapping containing prefixed names as keys. Each +key identifies an export group containing one or more exports published by +the distribution. + +Export group names are defined by distributions that will then make use of +the published export information in some way. The primary use case is for +distributions that support a plugin model: defining an export group allows +other distributions to indicate which plugins they provide, how they +can be imported and accessed, and which additional dependencies (if any) +are needed for the plugin to work correctly. + +To reduce the chance of name conflicts, export group names SHOULD use a +prefix that corresponds to a module name in the distribution that defines +the meaning of the export group. This practice will also make it easier to +find authoritative documentation for export groups. + +Each individual export group is then a mapping of arbitrary non-empty string +keys to export specifiers. The meaning of export names within an export +group is up to the distribution that defines the export group. Creating an +appropriate definition for the export name format can allow the importing +distribution to determine whether or not an export is relevant without +needing to import every exporting module. + +Example:: + + "exports": { + "nose.plugins.0.10": { + "chairtest": "chair:NosePlugin" + } + } + + +The ``commands`` extension +========================== + +The ``commands`` extension contains three subfields: + +* ``wrap_console``: console wrapper scripts to be generated by the installer +* ``wrap_gui``: GUI wrapper scripts to be generated by the installer +* ``prebuilt``: scripts created by the distribution's build process and + installed directly to the configured scripts directory + +``wrap_console`` and ``wrap_gui`` are both mappings of script names to +export specifiers. The script names must follow the same naming rules as +distribution names. + +The export specifiers for wrapper scripts must refer to either a package +with a __main__ submodule (if no ``name`` subfield is given in the export +specifier) or else to a callable inside the named module. + +Installation tools should generate appropriate wrappers as part of the +installation process. + +.. note:: + + Still needs more detail on what "appropriate wrappers" means. For now, + refer to what setuptools and zc.buildout generate as wrapper scripts. + +``prebuilt`` is a list of script paths, relative to the scripts directory in +a wheel file or following installation. They are provided for informational +purpose only - installing them is handled through the normal processes for +files created when building a distribution. + +Index servers SHOULD allow multiple distributions to publish the same +commands, but MAY notify distribution authors of potential conflicts. + +Installation tools SHOULD report an error when asked to install a +distribution that provides a command that is also provided by a different, +previously installed, distribution. + +Example:: + + "commands": { + "wrap_console": [{"wrapwithpython": "chair:run_cli"}], + "wrap_gui": [{"wrapwithpythonw": "chair:run_gui"}], + "prebuilt": ["notawrapper"] + } + + + +The ``metadata_hooks`` extension +================================ + +The ``metadata_hooks`` extension is used to define operations to be +invoked on the distribution in the following situations: + +* a relevant distribution has been installed or changed on the system +* a relevant distribution has been completely uninstalled from the system + +The following subfields determine when hooks are triggered: + +* ``export_groups``: trigger based on named export groups +* ``extensions``: trigger based on named extensions + +Note that distributions *do* trigger their own install hooks, but do +*not* trigger their own uninstall hooks. + + +Hook signatures +--------------- + +The currently defined metadata hooks are: + +* ``install``: run after a relevant distribution has been installed, + upgraded or downgraded on the current system. May also be run as part + of a system state resync operation. If the hook is not defined, it + indicates no distribution specific actions are needed following + installation. +* ``uninstall``: run after a relevant distribution has been completely + removed from the current system. If the hook is not defined, it indicates + no distribution specific actions are needed following uninstallation. + +The required signatures of these hooks are as follows:: + + def install(current_meta, previous_meta=None): + """Run following installation or upgrade of a relevant distribution + + *current_meta* is the distribution metadata for the version now + installed on the current system + *previous_meta* is either omitted or ``None`` (indicating a fresh + install) or else the distribution metadata for the version that + was previously installed (indicating an upgrade, downgrade or + resynchronisation of the system state). + """ + + def uninstall(previous_meta): + """Run after complete uninstallation of a relevant distribution + + *previous_meta* is the distribution metadata for the version that + was previously installed on the current system + """ + +Export group hooks +------------------ + +Export group hooks are named after the export group of interest:: + + "export_groups": { + "ComfyChair.plugins": { + "install": "ComfyChair.plugins:install_hook", + "uininstall": "ComfyChair.plugins:uninstall_hook" + } + } + +The nominated hooks will then be invoked appropriately for any distribution +that publishes that export group as part of their ``python.exports`` +extension metadata. + +A trailing ".*" may be used to request prefix matching rather than +requiring an exact match on the export group name. + + +Extension hooks +--------------- + +Extension hooks are named after the metadata extension of interest:: + + "extensions": { + "python.exports": { + "install": "pip.export_group_hooks:run_install_hooks", + "uininstall": "pip.export_group_hooks:run_uninstall_hooks" + } + "python.commands": { + "install": "pip.command_hook:install_wrapper_scripts", + } + } + +(Note: this is just an example, but the intent is that pip *could* implement +that functionality that way if it wanted to. + +A trailing ".*" may be used to request prefix matching rather than +requiring an exact match on the extension name. + + +Guidelines for metadata hook invocation +--------------------------------------- + +.. note:: + + Metadata hooks are likely to run with elevated privileges, this needs + to be considered carefully (e.g. by *requiring* that metadata hook + installation be opt in). + +The given parameter names are considered part of the hook signature. +Installation tools MUST call metadata hooks solely with keyword arguments. + +When metadata hooks are defined, it is assumed that they MUST be executed +to obtain a properly working installation of the distribution, and to +properly remove the distribution from a system. + +Installation tools MUST ensure the new or updated distribution is fully +installed, and available through the import system and installation database +when invoking install hooks. + +Installation tools MUST ensure the removed distribution is fully uninstalled, +and no longer available through the import system and installation database +when invoking uninstall hooks. + +Installation tools MUST call metadata hooks with full metadata (including +all extensions), rather than only the core metadata. + +Installation tools SHOULD invoke metadata hooks automatically after +installing a distribution from a binary archive. + +When installing from an sdist, source archive or VCS checkout, installation +tools SHOULD create a binary archive using ``setup.py bdist_wheel`` and +then install the binary archive normally (including invocation of any +metadata hooks). Installation tools SHOULD NOT invoke ``setup.py install`` +directly. + +Installation tools SHOULD treat an exception thrown by a metadata install +hook as a failure of the installation and revert any other changes made +to the system. The installed distribution responsible for the hook that +failed should be clearly indicated to the user. + +Installation tools SHOULD provide a warning to the user for any exception +thrown by a metadata uninstall hook, again clearly indicating to the user +the installed distribution that triggered the warning. + +Installation tools MUST NOT silently ignore metadata hooks, as failing +to call these hooks may result in a misconfigured installation that fails +unexpectedly at runtime. Installation tools MAY refuse to install +distributions that define metadata hooks, or require that users +explicitly opt in to permitting the installation of packages that +define such hooks. + + +Guidelines for metadata hook implementations +-------------------------------------------- + +The given parameter names are considered part of the hook signature. +Metadata hook implementations MUST use the given parameter names. + +Metadata hooks SHOULD NOT be used to provide functionality that is +expected to be provided by installation tools (such as rewriting of +shebang lines and generation of executable wrappers for Windows). + +Metadata hook implementations MUST NOT make any assumptions regarding the +current working directory when they are invoked, and MUST NOT make +persistent alterations to the working directory or any other process global +state (other than potentially importing additional modules, or other +expected side effects of running the distribution). + +Metadata install hooks have access to the full metadata for the release being +installed, that of the previous/next release (as appropriate), as well as +to all the normal runtime information (such as available imports). Hook +implementations can use this information to perform additional platform +specific installation steps. To check for the presence or absence of +"extras", hook implementations should use the same runtime checks that +would be used during normal operation (such as checking for the availability +of the relevant dependencies). + + +Copyright +========= + +This document has been placed in the public domain. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + End: -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sat Dec 21 14:52:15 2013 From: python-checkins at python.org (nick.coghlan) Date: Sat, 21 Dec 2013 14:52:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Major_update_to_metadata_2=2E?= =?utf-8?q?0_draft_specs?= Message-ID: <3dmpCz4qklz7Ljq@mail.python.org> http://hg.python.org/peps/rev/d18629859154 changeset: 5333:d18629859154 user: Nick Coghlan date: Sat Dec 21 23:52:04 2013 +1000 summary: Major update to metadata 2.0 draft specs - split many fields out to standard extensions - support for local versions in PEP 440 - addressed several other issues noted in https://bitbucket.org/pypa/pypi-metadata-formats/issues files: pep-0426.txt | 953 ++++++++------------------------------ pep-0440.txt | 248 ++++++++- pep-0459.txt | 35 +- 3 files changed, 434 insertions(+), 802 deletions(-) diff --git a/pep-0426.txt b/pep-0426.txt --- a/pep-0426.txt +++ b/pep-0426.txt @@ -13,7 +13,8 @@ Requires: 440 Created: 30 Aug 2012 Post-History: 14 Nov 2012, 5 Feb 2013, 7 Feb 2013, 9 Feb 2013, - 27 May 2013, 20 Jun 2013, 23 Jun 2013, 14 Jul 2013 + 27 May 2013, 20 Jun 2013, 23 Jun 2013, 14 Jul 2013, + 21 Dec 2013 Replaces: 345 @@ -62,16 +63,14 @@ * this PEP, covering the core metadata format * PEP 440, covering the versioning identification and selection scheme - * a new PEP to define v2.0 of the sdist format + * PEP 459, covering several standard extensions + * a yet-to-be-written PEP to define v2.0 of the sdist format * an updated wheel PEP (v1.1) to add pydist.json (and possibly convert the wheel metadata file from Key:Value to JSON) - * an updated installation database PEP both for pydist.json (and possibly convert - the wheel metadata file from Key:Value to JSON) - * an alternative to \*.pth files that avoids system global side effects - and better supports runtime selection of dependencies - * a new static config PEP to standardise metadata generation and - creation of sdists - * a PEP to cover bundling ``pip`` with the CPython installers + * an updated installation database PEP to add pydist.json + * a PEP to standardise the expected command line interface for setup.py + as an interface to an application's build system (rather than requiring + that the build system support the distutils command system) It's going to take a while to work through all of these and make them a reality. The main change from our last attempt at this is that we're @@ -87,23 +86,6 @@ an irrelevant distraction for future readers. -A Note on Time Frames -===================== - -There's a lot of work going on in the Python packaging space at the moment. -In the near term (up until the release of Python 3.4), those efforts will be -focused on the existing metadata standards, both those defined in Python -Enhancement Proposals, and the de facto standards defined by the setuptools -project. - -This PEP is about setting out a longer term goal for the ecosystem that -captures those existing capabilities in a format that is easier to work -with. There are still a number of key open questions (mostly related to -source based distribution), and those won't be able to receive proper -attention from the development community until the other near term -concerns have been resolved. - - Purpose ======= @@ -114,7 +96,9 @@ Python code by those doing the analysis. Another aim is to encourage good software distribution practices by default, while continuing to support the current practices of almost all existing users of the Python Package Index -(both publishers and integrators). +(both publishers and integrators). Finally, the aim is to support an upgrade +path from the existing setuptools defined dependency and entry point +metadata formats that is transparent to end users. The design draws on the Python community's 15 years of experience with distutils based software distribution, and incorporates ideas and concepts @@ -122,6 +106,36 @@ other projects, Ruby's gems, Perl's CPAN, Node.js's npm, PHP's composer and Linux packaging systems such as RPM and APT. +While the specifics of this format are aimed at the Python ecosystem, some +of the ideas may also be useful in the future evolution of other dependency +management ecosystems. + + +A Note on Time Frames +===================== + +There's a lot of work going on in the Python packaging space at the moment. +In the near term (up until the release of Python 3.4), those efforts are +focused on the existing metadata standards, both those defined in Python +Enhancement Proposals, and the de facto standards defined by the setuptools +project. + +This PEP is about setting out a longer term goal for the ecosystem that +captures those existing capabilities in a format that is easier to work +with. There are still a number of key open questions (mostly related to +source based distribution), and those won't be able to receive proper +attention from the development community until the other near term +concerns have been resolved. + +At this point in time, the PEP is quite possibly still overengineered, as +we're still trying to make sure we have all the use cases covered. The +"transparent upgrade path from setuptools" goal brings in a lot of required +functionality though, and then the aim of supporting automated creation of +policy compliant downstream packages for Linux distributions adds more. +However, we've at least reached the point where we're taking a critical +look at the core metadata, and are pushing as much functionality out to +standard metadata extensions as we can. + Development, Distribution and Deployment of Python Software =========================================================== @@ -138,8 +152,8 @@ * Software publication: this phase involves taking the developed software and making it available for use by software integrators. This includes - creating the descriptive metadata defined in this PEP, as well making the - software available (typically by uploading it to an index server). + creating the descriptive metadata defined in this PEP, as well as making + the software available (typically by uploading it to an index server). * Software integration: this phase involves taking published software components and combining them into a coherent, integrated system. This @@ -153,7 +167,9 @@ The publication and integration phases are collectively referred to as the distribution phase, and the individual software components distributed -in that phase are referred to as "distributions". +in that phase are formally referred to as "distributions", but are more +colloquially known as "packages" (relying on context to disambiguate them +from the "module with submodules" kind of Python package). The exact details of these phases will vary greatly for particular use cases. Deploying a web application to a public Platform-as-a-Service provider, @@ -185,6 +201,10 @@ "Distributions" are the packaged files which are used to publish and distribute a release. +Depending on context, "package" may refer to either a distribution, or +to an importable Python module that has a ``__path__`` attribute and hence +may also have importable submodules. + "Source archive" and "VCS checkout" both refer to the raw source code for a release, prior to creation of an sdist or binary archive. @@ -419,7 +439,9 @@ metadata. Each metadata file consists of a single serialised mapping, with fields as -described in this PEP. +described in this PEP. When serialising metadata, automated tools SHOULD +lexically sort any keys and list elements in order to simplify reviews +of any changes. There are three standard locations for these metadata files: @@ -457,91 +479,6 @@ in Appendix A. -Essential dependency resolution metadata ----------------------------------------- - -For dependency resolution purposes, it is useful to have a minimal subset -of the metadata that contains only those fields needed to identify -distributions and resolve dependencies between them. - -The essential dependency resolution metadata consists of the following -fields: - -* ``metadata_version`` -* ``generator`` -* ``name`` -* ``version`` -* ``source_label`` -* ``source_url`` -* ``extras`` -* ``meta_requires`` -* ``run_requires`` -* ``test_requires`` -* ``build_requires`` -* ``dev_requires`` -* ``provides`` -* ``obsoleted_by`` -* ``supports_environments`` - -When serialised to a file, the name used for this metadata set SHOULD -be ``pydist-dependencies.json``. - - -Export metadata ---------------- - -Distributions may define components that are intended for use by other -distributions (such as plugins). As it can be beneficial to know whether or -not a distribution defines any such exports without needing to parse any -metadata, a suitable subset is defined for serialisation to a separate file -in the ``dist-info`` metadata directory. - -The external command metadata consists of the following fields: - -* ``metadata_version`` -* ``generator`` -* ``name`` -* ``version`` -* ``exports`` - -When serialised to a file, the name used for this metadata set SHOULD -be ``pydist-exports.json``. - - -Command metadata ----------------- - -Distributions may define commands that will be available from the command -line following installation. As it can be beneficial to know whether or not -a distribution has such commands without needing to parse any metadata, -a suitable subset is defined for serialisation to a separate file in the -``dist-info`` metadata directory. - -The external command metadata consists of the following fields: - -* ``metadata_version`` -* ``generator`` -* ``name`` -* ``version`` -* ``commands`` - -When serialised to a file, the name used for this metadata set SHOULD -be ``pydist-commands.json``. - - -Included documents ------------------- - -Rather than being incorporated directly into the structured metadata, some -supporting documents are included alongside the metadata file in the -``dist-info`` metadata directory. - -To accommodate the variety of existing naming conventions for these files, -they are explicitly identified in the ``document_names`` field, rather -than expecting index servers and other automated tools to identify them -automatically. - - Metadata validation ------------------- @@ -555,6 +492,12 @@ Except where otherwise noted, all URL fields in the metadata MUST comply with RFC 3986. +.. note:: + + The current version of the schema file covers the previous draft of the + PEP, and has not yet been updated for the split into the essential + dependency resolution metadata and multiple standard extensions. + Core metadata ============= @@ -678,7 +621,8 @@ This field SHOULD NOT contain any line breaks. A more complete description SHOULD be included as a separate file in the -sdist for the distribution. See `Document names`_ for details. +sdist for the distribution. Refer to the ``python-details`` extension in +:pep:`459` for more information. Example:: @@ -734,13 +678,14 @@ suitable VCS checkout. It is intended primarily for integrators that wish to recreate the distribution from the original source form. -All source URL references SHOULD specify a secure transport -mechanism (such as ``https``), include an expected hash value in the -URL for verification purposes, or both. If an insecure transport is specified -without any hash information, with hash information that the tool doesn't -understand, or with a selected hash algorithm that the tool considers too -weak to trust, automated tools SHOULD at least emit a warning and MAY -refuse to rely on the URL. +All source URL references SHOULD specify a secure transport mechanism +(such as ``https``) AND include an expected hash value in the URL for +verification purposes. If a source URL is specified without any hash +information, with hash information that the tool doesn't understand, or +with a selected hash algorithm that the tool considers too weak to trust, +automated tools SHOULD at least emit a warning and MAY refuse to rely on +the URL. If such a source URL also uses an insecure transport, automated +tools SHOULD NOT rely on the URL. It is RECOMMENDED that only hashes which are unconditionally provided by the latest version of the standard library's ``hashlib`` module be used @@ -753,236 +698,31 @@ fragment. For version control references, the ``VCS+protocol`` scheme SHOULD be -used to identify both the version control system and the secure transport. - -To support version control systems that do not support including commit or +used to identify both the version control system and the secure transport, +and a version control system with hash based commit identifiers SHOULD be +used. Automated tools MAY omit warnings about missing hashes for version +control systems that do not provide hash based commit identifiers. + +To handle version control systems that do not support including commit or tag references directly in the URL, that information may be appended to the -end of the URL using the ``@`` notation. +end of the URL using the ``@`` or the ``@#`` +notation. + +.. note:: + + This isn't *quite* the same as the existing VCS reference notation + supported by pip. Firstly, the distribution name is moved in front rather + than embedded as part of the URL. Secondly, the commit hash is included + even when retrieving based on a tag, in order to meet the requirement + above that *every* link should include a hash to make things harder to + forge (creating a malicious repo with a particular tag is easy, creating + one with a specific *hash*, less so). Example:: - "source_url": "https://github.com/pypa/pip/archive/1.3.1.zip" - "source_url": "http://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686" - "source_url": "git+https://github.com/pypa/pip.git at 1.3.1" - - -Additional descriptive metadata -=============================== - -This section specifies fields that provide additional information regarding -the distribution and its contents. - -All of these fields are optional. Automated tools MUST operate correctly if -a distribution does not provide them, including failing cleanly when an -operation depending on one of these fields is requested. - - -License -------- - -A short string summarising the license used for this distribution. - -Note that distributions that provide this field should still specify any -applicable license Trove classifiers in the `Classifiers`_ field. Even -when an appropriate Trove classifier is available, the license summary can -be a good way to specify a particular version of that license, or to -indicate any variations or exception to the license. - -This field SHOULD contain fewer than 512 characters and MUST contain fewer -than 2048. - -This field SHOULD NOT contain any line breaks. - -The full license text SHOULD be included as a separate file in the source -archive for the distribution. See `Document names`_ for details. - -Example:: - - "license": "GPL version 3, excluding DRM provisions" - - -Keywords --------- - -A list of additional keywords to be used to assist searching for the -distribution in a larger catalog. - -Example:: - - "keywords": ["comfy", "chair", "cushions", "too silly", "monty python"] - - -Classifiers ------------ - -A list of strings, with each giving a single classification value -for the distribution. Classifiers are described in PEP 301 [2]. - -Example:: - - "classifiers": [ - "Development Status :: 4 - Beta", - "Environment :: Console (Text Based)", - "License :: OSI Approved :: GNU General Public License v3 (GPLv3)" - ] - - -Document names --------------- - -Filenames for supporting documents included in the distribution's -``dist-info`` metadata directory. - -The following supporting documents can be named: - -* ``description``: a file containing a long description of the distribution -* ``license``: a file with the full text of the distribution's license -* ``changelog``: a file describing changes made to the distribution - -Supporting documents MUST be included directly in the ``dist-info`` -directory. Directory separators are NOT permitted in document names. - -The markup format (if any) for the file is indicated by the file extension. -This allows index servers and other automated tools to render included -text documents correctly and provide feedback on rendering errors, rather -than having to guess the intended format. - -If the filename has no extension, or the extension is not recognised, the -default rendering format MUST be plain text. - -The following markup renderers SHOULD be used for the specified file -extensions: - -* Plain text: ``.txt``, no extension, unknown extension -* reStructured Text: ``.rst`` -* Markdown: ``.md`` -* AsciiDoc: ``.adoc``, ``.asc``, ``.asciidoc`` -* HTML: ``.html``, ``.htm`` - -Automated tools MAY render one or more of the specified formats as plain -text and MAY render other markup formats beyond those listed. - -Automated tools SHOULD NOT make any assumptions regarding the maximum length -of supporting document content, except as necessary to protect the -integrity of a service. - -Example:: - - "document_names": { - "description": "README.rst", - "license": "LICENSE.rst", - "changelog": "NEWS" - } - - -Contributor metadata -==================== - -Contributor metadata for a distribution is provided to allow users to get -access to more information about the distribution and its maintainers. - -These details are recorded as mappings with the following subfields: - -* ``name``: the name of an individual or group -* ``email``: an email address (this may be a mailing list) -* ``url``: a URL (such as a profile page on a source code hosting service) -* ``role``: one of ``"author"``, ``"maintainer"`` or ``"contributor"`` - -The ``name`` subfield is required, the other subfields are optional. - -If no specific role is stated, the default is ``contributor``. - -Email addresses must be in the form ``local-part at domain`` where the -local-part may be up to 64 characters long and the entire email address -contains no more than 254 characters. The formal specification of the -format is in RFC 5322 (sections 3.2.3 and 3.4.1) and RFC 5321, with a more -readable form given in the informational RFC 3696 and the associated errata. - -The defined contributor roles are as follows: - -* ``author``: the original creator of a distribution -* ``maintainer``: the current lead contributor for a distribution, when - they are not the original creator -* ``contributor``: any other individuals or organizations involved in the - creation of the distribution - -Contact and contributor metadata is optional. Automated tools MUST operate -correctly if a distribution does not provide it, including failing cleanly -when an operation depending on one of these fields is requested. - - -Contacts --------- - -A list of contributor entries giving the recommended contact points for -getting more information about the project. - -The example below would be suitable for a project that was in the process -of handing over from the original author to a new lead maintainer, while -operating as part of a larger development group. - -Example:: - - "contacts": [ - { - "name": "Python Packaging Authority/Distutils-SIG", - "email": "distutils-sig at python.org", - "url": "https://bitbucket.org/pypa/" - }, - { - "name": "Samantha C.", - "role": "maintainer", - "email": "dontblameme at example.org" - }, - { - "name": "Charlotte C.", - "role": "author", - "email": "iambecomingasketchcomedian at example.com" - } - ] - - -Contributors ------------- - -A list of contributor entries for other contributors not already listed as -current project points of contact. The subfields within the list elements -are the same as those for the main contact field. - -Example:: - - "contributors": [ - {"name": "John C."}, - {"name": "Erik I."}, - {"name": "Terry G."}, - {"name": "Mike P."}, - {"name": "Graeme C."}, - {"name": "Terry J."} - ] - - -Project URLs ------------- - -A mapping of arbitrary text labels to additional URLs relevant to the -project. - -While projects are free to choose their own labels and specific URLs, -it is RECOMMENDED that home page, source control, issue tracker and -documentation links be provided using the labels in the example below. - -URL labels MUST be treated as case insensitive by automated tools, but they -are not required to be valid Python identifiers. Any legal JSON string is -permitted as a URL label. - -Example:: - - "project_urls": { - "Documentation": "https://distlib.readthedocs.org" - "Home": "https://bitbucket.org/pypa/distlib" - "Repository": "https://bitbucket.org/pypa/distlib/src" - "Tracker": "https://bitbucket.org/pypa/distlib/issues" - } + "source_url": "https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686" + "source_url": "git+https://github.com/pypa/pip.git at 1.3.1#7921be1537eac1e97bc40179a57f0349c2aee67d" + "source_url": "git+https://github.com/pypa/pip.git at 7921be1537eac1e97bc40179a57f0349c2aee67d" Semantic dependencies @@ -1001,10 +741,10 @@ Distributions may declare five differents kinds of dependency: +* Runtime dependencies: other distributions that are needed to actually use + this distribution (but are not considered subdistributions). * "Meta" dependencies: subdistributions that are grouped together into a single larger metadistribution for ease of reference and installation. -* Runtime dependencies: other distributions that are needed to actually use - this distribution (but are not considered subdistributions). * Test dependencies: other distributions that are needed to run the automated test suite for this distribution (but are not needed just to use it). @@ -1105,24 +845,24 @@ * Implied runtime dependencies: + * ``run_requires`` * ``meta_requires`` - * ``run_requires`` * Implied build dependencies: * ``build_requires`` * If running the distribution's test suite as part of the build process, - request the ``:meta:``, ``:run:`` and ``:test:`` extras to also + request the ``:run:``, ``:meta:``, and ``:test:`` extras to also install: + * ``run_requires`` * ``meta_requires`` - * ``run_requires`` * ``test_requires`` * Implied development and publication dependencies: + * ``run_requires`` * ``meta_requires`` - * ``run_requires`` * ``build_requires`` * ``test_requires`` * ``dev_requires`` @@ -1153,6 +893,32 @@ "extras": ["warmup"] +Run requires +------------ + +A list of other distributions needed to actually run this distribution. + +Automated tools MUST NOT allow strict version matching clauses or direct +references in this field - if permitted at all, such clauses should appear +in ``meta_requires`` instead. + +Example:: + + "run_requires": + { + "requires": ["SciPy", "PasteDeploy", "zope.interface (>3.5.0)"] + }, + { + "requires": ["pywin32 (>1.0)"], + "environment": "sys_platform == 'win32'" + }, + { + "requires": ["SoftCushions"], + "extra": "warmup" + } + ] + + Meta requires ------------- @@ -1183,33 +949,7 @@ }, { "requires": ["CupOfTeaAtEleven (== 1.0a2)"], - "environment": "'linux' in sys.platform" - } - ] - - -Run requires ------------- - -A list of other distributions needed to actually run this distribution. - -Automated tools MUST NOT allow strict version matching clauses or direct -references in this field - if permitted at all, such clauses should appear -in ``meta_requires`` instead. - -Example:: - - "run_requires": - { - "requires": ["SciPy", "PasteDeploy", "zope.interface (>3.5.0)"] - }, - { - "requires": ["pywin32 (>1.0)"], - "environment": "sys.platform == 'win32'" - }, - { - "requires": ["SoftCushions"], - "extra": "warmup" + "environment": "'linux' in sys_platform" } ] @@ -1234,7 +974,7 @@ }, { "requires": ["pywin32 (>1.0)"], - "environment": "sys.platform == 'win32'" + "environment": "sys_platform == 'win32'" }, { "requires": ["CompressPadding"], @@ -1266,7 +1006,7 @@ }, { "requires": ["pywin32 (>1.0)"], - "environment": "sys.platform == 'win32'" + "environment": "sys_platform == 'win32'" }, { "requires": ["cython"], @@ -1302,7 +1042,7 @@ }, { "requires": ["pywin32 (>1.0)"], - "environment": "sys.platform == 'win32'" + "environment": "sys_platform == 'win32'" } ] @@ -1369,6 +1109,11 @@ A version declaration may be supplied and must follow the rules described in PEP 440. +An inactive project may be explicitly indicated by setting this field to +``None`` (which is serialised as ``null`` in JSON as usual). + +Automated tools SHOULD report a warning when installing an obsolete project. + Possible uses for this field include handling project name changes and project mergers. @@ -1441,321 +1186,39 @@ "supports_environments": ["python_version >= '2.6' and sys_platform != 'win32'", "python_version >= '3.3' and sys_platform == 'win32'"] -Installed interfaces -==================== - -Most Python distributions expose packages and modules for import through -the Python module namespace. Distributions may also expose other -interfaces when installed. - -Export specifiers ------------------ - -An export specifier is a string consisting of a fully qualified name, as -well as an optional extra name enclosed in square brackets. This gives the -following four possible forms for an export specifier:: - - module - module:name - module[requires_extra] - module:name[requires_extra] - -.. note:: - - The jsonschema file currently restricts qualified names using the - Python 2 ASCII identifier rules. This may need to be reconsidered - given the more relaxed identifier rules in Python 3. - -The meaning of the subfields is as follows: - -* ``module``: the module providing the export -* ``name``: if applicable, the qualified name of the export within the module -* ``requires_extra``: indicates the export will only work correctly if the - additional dependencies named in the given extra are available in the - installed environment - -.. note:: - - I tried this as a mapping with subfields, and it made the examples below - unreadable. While this PEP is mostly for tool use, readability still - matters to some degree for debugging purposes, and because I expect - snippets of the format to be reused elsewhere. - - -Modules -------- - -A list of qualified names of modules and packages that the distribution -provides for import. - -.. note:: - - The jsonschema file currently restricts qualified names using the - Python 2 ASCII identifier rules. This may need to be reconsidered - given the more relaxed identifier rules in Python 3. - -For names that contain dots, the portion of the name before the final dot -MUST appear either in the installed module list or in the namespace package -list. - -To help avoid name conflicts, it is RECOMMENDED that distributions provide -a single top level module or package that matches the distribution name -(or a lower case equivalent). This requires that the distribution name also -meet the requirements of a Python identifier, which are stricter than -those for distribution names). This practice will also make it easier to -find authoritative sources for modules. - -Index servers SHOULD allow multiple distributions to publish the same -modules, but MAY notify distribution authors of potential conflicts. - -Installation tools SHOULD report an error when asked to install a -distribution that provides a module that is also provided by a different, -previously installed, distribution. - -Note that attempting to import some declared modules may result in an -exception if the appropriate extras are not installed. - -Example:: - - "modules": ["chair", "chair.cushions", "python_sketches.nobody_expects"] - -.. note:: - - Making this a list of export specifiers instead would allow a distribution - to declare when a particular module requires a particular extra in order - to run correctly. On the other hand, there's an argument to be made that - that is the point where it starts to become worthwhile to split out a - separate distribution rather than using extras. - - -Namespaces ----------- - -A list of qualified names of namespace packages that the distribution -contributes modules to. - -.. note:: - - The jsonschema file currently restricts qualified names using the - Python 2 ASCII identifier rules. This may need to be reconsidered - given the more relaxed identifier rules in Python 3. - -On versions of Python prior to Python 3.3 (which provides native namespace -package support), installation tools SHOULD emit a suitable ``__init__.py`` -file to properly initialise the namespace rather than using a distribution -provided file. - -Installation tools SHOULD emit a warning and MAY emit an error if a -distribution declares a namespace package that conflicts with the name of -an already installed module or vice-versa. - -Example:: - - "namespaces": ["python_sketches"] - - -Commands --------- - -The ``commands`` mapping contains three subfields: - -* ``wrap_console``: console wrapper scripts to be generated by the installer -* ``wrap_gui``: GUI wrapper scripts to be generated by the installer -* ``prebuilt``: scripts created by the distribution's build process and - installed directly to the configured scripts directory - -``wrap_console`` and ``wrap_gui`` are both mappings of script names to -export specifiers. The script names must follow the same naming rules as -distribution names. - -The export specifiers for wrapper scripts must refer to either a package -with a __main__ submodule (if no ``name`` subfield is given in the export -specifier) or else to a callable inside the named module. - -Installation tools should generate appropriate wrappers as part of the -installation process. - -.. note:: - - Still needs more detail on what "appropriate wrappers" means. For now, - refer to what setuptools and zc.buildout generate as wrapper scripts. - -``prebuilt`` is a list of script paths, relative to the scripts directory in -a wheel file or following installation. They are provided for informational -purpose only - installing them is handled through the normal processes for -files created when building a distribution. - -Index servers SHOULD allow multiple distributions to publish the same -commands, but MAY notify distribution authors of potential conflicts. - -Installation tools SHOULD report an error when asked to install a -distribution that provides a command that is also provided by a different, -previously installed, distribution. - -Example:: - - "commands": { - "wrap_console": [{"wrapwithpython": "chair:run_cli"}], - "wrap_gui": [{"wrapwithpythonw": "chair:run_gui"}], - "prebuilt": ["notawrapper"] - } - - - -Exports -------- - -The ``exports`` field is a mapping containing prefixed names as keys. Each -key identifies an export group containing one or more exports published by -the distribution. - -Export group names are defined by distributions that will then make use of -the published export information in some way. The primary use case is for -distributions that support a plugin model: defining an export group allows -other distributions to indicate which plugins they provide, how they -can be imported and accessed, and which additional dependencies (if any) -are needed for the plugin to work correctly. - -To reduce the chance of name conflicts, export group names SHOULD use a -prefix that corresponds to a module name in the distribution that defines -the meaning of the export group. This practice will also make it easier to -find authoritative documentation for export groups. - -Each individual export group is then a mapping of arbitrary non-empty string -keys to export specifiers. The meaning of export names within an export -group is up to the distribution that defines the export group. Creating an -appropriate definition for the export name format can allow the importing -distribution to determine whether or not an export is relevant without -needing to import every exporting module. - -Example:: - - "exports": { - "nose.plugins.0.10": { - "chairtest": "chair:NosePlugin" - } - } - - -Install hooks -============= - -The ``install_hooks`` field is used to define operations to be -invoked on the distribution in the following situations: - -* Installing to a deployment system -* Uninstalling from a deployment system - -Distributions may define handlers for each of these operations as an -"entry point", which is a reference to a Python callable, with the module -name separated from the reference within the module by a colon (``:``). - -Example install hooks:: - - "install_hooks": { - "postinstall": "ComfyChair.install_hooks:postinstall", - "preuininstall": "ComfyChair.install_hooks:preuninstall" - } - -The currently defined install hooks are: - -* ``postinstall``: run after the distribution has been installed to a - target deployment system (or after it has been upgraded). If the hook is - not defined, it indicates no distribution specific actions are needed - following installation. -* ``preuninstall``: run before the distribution has been uninstalled from a - deployment system (or before it is upgraded). If the hook is not defined, - it indicates no distribution specific actions are needed prior to - uninstallation. - -The required signatures of these hooks are as follows:: - - def postinstall(current_meta, previous_meta=None): - """Run following installation or upgrade of the distribution - - *current_meta* is the distribution metadata for the version now - installed on the current system - *previous_meta* is either omitted or ``None`` (indicating a fresh - install) or else the distribution metadata for the version that - was previously installed (indicating an upgrade or downgrade). - """ - - def preuninstall(current_meta, next_meta=None): - """Run prior to uninstallation or upgrade of the distribution - - *current_meta* is the distribution metadata for the version now - installed on the current system - *next_meta* is either omitted or ``None`` (indicating complete - uninstallation) or else the distribution metadata for the version - that is about to be installed (indicating an upgrade or downgrade). - """ - -When install hooks are defined, it is assumed that they MUST be executed -to obtain a properly working installation of the distribution, and to -properly remove the distribution from a system. - -Install hooks SHOULD NOT be used to provide functionality that is -expected to be provided by installation tools (such as rewriting of -shebang lines and generation of executable wrappers for Windows). - -Installation tools MUST ensure the distribution is fully installed, and -available through the import system and installation database when invoking -install hooks. - -Installation tools MUST call install hooks with full metadata, rather than -only the essential dependency resolution metadata. - -The given parameter names are considered part of the hook signature. -Installation tools MUST call install hooks solely with keyword arguments. -Install hook implementations MUST use the given parameter names. - -Installation tools SHOULD invoke install hooks automatically after -installing a distribution from a binary archive. - -When installing from an sdist, source archive or VCS checkout, installation -tools SHOULD create a binary archive using ``setup.py bdist_wheel`` and -then install binary archive normally (including invocation of any install -hooks). Installation tools SHOULD NOT invoke ``setup.py install`` directly. - -Installation tools SHOULD treat an exception thrown by a postinstall hook -as a failure of the installation and revert any other changes made to the -system. - -Installation tools SHOULD treat an exception thrown by a preuninstall hook -as an indication the removal of the distribution should be aborted. - -Installation tools MUST NOT silently ignore install hooks, as failing -to call these hooks may result in a misconfigured installation that fails -unexpectedly at runtime. Installation tools MAY refuse to install -distributions that define install hooks, or require that users -explicitly opt in to permitting the execution of such hooks. - -Install hook implementations MUST NOT make any assumptions regarding the -current working directory when they are invoked, and MUST NOT make -persistent alterations to the working directory or any other process global -state (other than potentially importing additional modules, or other -expected side effects of running the distribution). - -Install hooks have access to the full metadata for the release being -installed, that of the previous/next release (as appropriate), as well as -to all the normal runtime information (such as available imports). Hook -implementations can use this information to perform additional platform -specific installation steps. To check for the presence or absence of -"extras", hook implementations should use the same runtime checks that -would be used during normal operation (such as checking for the availability -of the relevant dependencies). - Metadata Extensions =================== -Extensions to the metadata may be present in a mapping under the -'extensions' key. The keys must be valid prefixed names, while -the values may be any type natively supported in JSON:: +Extensions to the metadata MAY be present in a mapping under the +``extensions`` key. The keys MUST be valid prefixed names, while +the values MUST themselves be nested mappings. + +The following example shows the ``python.details`` and ``python.commands`` +standard extensions from :pep:`459`:: "extensions" : { - "chili" : { "type" : "Poblano", "heat" : "Mild" }, - "languages" : [ "French", "Italian", "Hebrew" ] + "python.details": { + "license": "GPL version 3, excluding DRM provisions", + "keywords": [ + "comfy", "chair", "cushions", "too silly", "monty python" + ], + "classifiers": [ + "Development Status :: 4 - Beta", + "Environment :: Console (Text Based)", + "License :: OSI Approved :: GNU General Public License v3 (GPLv3)" + ], + "document_names": { + "description": "README.rst", + "license": "LICENSE.rst", + "changelog": "NEWS" + } + }, + "python.commands": { + "wrap_console": [{"chair": "chair:run_cli"}], + "wrap_gui": [{"chair-gui": "chair:run_gui"}], + "prebuilt": ["reduniforms"] + }, } Extension names are defined by distributions that will then make use of @@ -1766,11 +1229,22 @@ the meaning of the extension. This practice will also make it easier to find authoritative documentation for metadata extensions. +Extensions MUST be versioned, using the ``extension_version`` key. +However, if this key is omitted, then the implied version is ``1.0``. + +Automated tools consuming extension metadata SHOULD warn if +``extension_version`` is greater than the highest version they support, +and MUST fail if ``extension_version`` has a greater major version than +the highest version they support (as described in PEP 440, the major +version is the value before the first dot). + +For broader compatibility, build tools MAY choose to produce +extension metadata using the lowest metadata version that includes +all of the needed fields. + Metadata extensions allow development tools to record information in the -metadata that may be useful during later phases of distribution. For -example, a build tool could include default build options in a metadata -extension when creating an sdist, and use those when creating the wheel -files later. +metadata that may be useful during later phases of distribution, but is +not essential for dependency resolution or building the software. Extras (optional dependencies) @@ -1989,7 +1463,12 @@ generation Python Package Index implementation * the `distlib project `__ which is derived from the core packaging infrastructure created for the - ``distutils2`` project and + ``distutils2`` project + +.. note:: + + These tools have yet to be updated for the switch to standard extensions + for several fields. While it is expected that there may be some edge cases where manual intervention is needed for clean conversion, the specification has been @@ -2071,9 +1550,8 @@ * The "Extras" optional dependency mechanism -* A well-defined metadata extension mechanism - -* Install hook system +* A well-defined metadata extension mechanism, and migration of any fields + not needed for dependency resolution to standard extensions. * Clarify and simplify various aspects of environment markers: @@ -2081,14 +1559,16 @@ * consistently use underscores instead of periods in the variable names * allow ordered string comparisons and chained comparisons +* New system for defining supported environments + +* Updated obsolescence mechanism + +* Metadata hook system + * More flexible system for defining contact points and contributors * Defined a recommended set of project URLs -* New system for defining supported environments - -* Updated obsolescence mechanism - * Identification of supporting documents in the ``dist-info`` directory: * Allows markup formats to be indicated through file extensions @@ -2160,8 +1640,8 @@ versioning scheme. -Build labels ------------- +Source labels +------------- See PEP 440 for the rationale behind the addition of this field. @@ -2189,7 +1669,7 @@ conflicting with the existing terminology in setuptools and previous versions of the metadata standard. Specifically, the names ``requires``, ``install_requires`` and ``setup_requires`` are not used, which will -hopefully reduce confustion when converting legacy metadata to the new +hopefully reduce confusion when converting legacy metadata to the new standard. @@ -2227,10 +1707,13 @@ packages, and inclusion of CVE references to mark security releases. -Support for install hooks +Support for metadata hooks --------------------------- -The new install hook system is designed to allow the wheel format to fully +This feature is provided by the ``python.metadata_hooks`` extension in +:pep:`459`. + +The new metadata hook system is designed to allow the wheel format to fully replace direct installation on deployment targets, by allowing projects to explicitly define code that should be executed following installation from a wheel file. @@ -2252,7 +1735,7 @@ meaning they cannot be deferred to implicit execution on first use of the distribution. -The install hook and metadata extension systems allow support for such +The metadata hook and metadata extension systems allow support for such activities to be pursued independently by the individual platform communities, while still interoperating with the cross-platform Python tools. @@ -2291,6 +1774,9 @@ Updated contact information --------------------------- +This feature is provided by the ``python.project`` and +``python.integrator`` extensions in :pep:`459`. + The switch to JSON made it possible to provide a more flexible system for defining multiple contact points for a project, as well as listing other contributors. @@ -2303,6 +1789,9 @@ Changes to project URLs ----------------------- +This feature is provided by the ``python.project`` and +``python.integrator`` extensions in :pep:`459`. + In addition to allow arbitrary strings as project URL labels, the new metadata standard also defines a recommend set of four URL labels for a distribution's home page, documentation, source control and issue tracker. @@ -2340,6 +1829,8 @@ Included text documents ----------------------- +This feature is provided by the ``python.details`` extension in :pep:`459`. + Currently, PyPI attempts to determine the description's markup format by rendering it as reStructuredText, and if that fails, treating it as plain text. @@ -2425,40 +1916,14 @@ than a little strange. -Module and file listings ------------------------- - -Derived metadata giving the modules and files included in built -distributions may be useful at some point in the future. (At least RPM -provides this, and I believe the APT equivalent does as well) - -Explicitly providing a list of public module names will likely help -with enabling features in RPM like "Requires: python(requests)", as well -as providing richer static metadata for analysis from PyPI. - -However, this is just extra info that doesn't impact reliably installing -from wheels, so it is a good candidate for postponing to metadata 2.1 -(at the earliest). - - Additional install hooks ------------------------ -In addition to the postinstall and preuninstall hooks described in the PEP, -other distribution systems (like RPM) include the notion of preinstall -and postuninstall hooks. These hooks would run with the runtime dependencies -installed, but without the distribution itself. These have been deliberately -omitted, as they're well suited to being explored further as metadata -extensions. - -Similarly, the idea of "optional" postinstall and preuninstall hooks can -be pursued as a metadata extension. - -By contrast, the mandatory postinstall and preuninstall hooks have been -included directly in the PEP, specifically to ensure installation tools -don't silently ignore them. This ensures users will either be able to -install such distributions, or else receive an explicit error at installation -time. +In addition to the postinstall and postuninstall hooks currently described +in :pep:`459`, other distribution systems (like RPM) include the notion of +preinstall and postuninstall hooks. These hooks would run with the runtime +dependencies installed, but without the distribution itself. These have +been deliberately omitted for the time being. Metabuild system diff --git a/pep-0440.txt b/pep-0440.txt --- a/pep-0440.txt +++ b/pep-0440.txt @@ -9,7 +9,7 @@ Type: Standards Track Content-Type: text/x-rst Created: 18 Mar 2013 -Post-History: 30 Mar 2013, 27 May 2013, 20 Jun 2013 +Post-History: 30 Mar 2013, 27 May 2013, 20 Jun 2013, 21 Dec 2013 Replaces: 386 @@ -91,6 +91,15 @@ Any given release will be a "final release", "pre-release", "post-release" or "developmental release" as defined in the following sections. +All numeric components MUST be non-negative integers. + +All numeric components MUST be interpreted and ordered according to their +numeric value, not as text strings. + +All numeric components MAY be zero. Except as described below for the +release segment, a numeric component of zero has no special significance +aside from always being the lowest possible value in the version ordering. + .. note:: Some hard to read version identifiers are permitted by this scheme in @@ -103,6 +112,41 @@ sections. +Local version identifiers +------------------------- + +Local version identifiers MUST comply with the following scheme:: + + [-N[.N]+] + +Local version identifiers are used to denote fully API compatible patched +versions of upstream projects. These are created by application developers +and system integrators when upgrading to a new upstream release would be too +disruptive to the application or other integrated system (such as a Linux +distribution). + +Local version identifiers may be used anywhere a public version identifier +is expected. + +Local version identifiers MUST NOT include leading or trailing whitespace. + +Numeric components in the integrator suffix are interpreted in the same way +as the numeric components of the release segment. + +The additional segment after the hyphen is referred to as the "integrator +suffix", and makes it possible to differentiate upstream releases from +potentially altered rebuilds by downstream integrators. The inclusion of an +integrator suffix does not affect the kind of a release, but indicates that +it may not contain the exact same code as the corresponding upstream release. + +Public index servers SHOULD NOT allow the use of local version identifiers +in uploaded distributions. Local version identifiers are intended as a tool +for software integrators rather than publishers. + +Distributions using a local version identifier SHOULD provide the +``python.integrator`` extension metadata (as defined in :pep:`459`). + + Source labels ------------- @@ -146,12 +190,6 @@ segments with different numbers of components, the shorter segment is padded out with additional zeroes as necessary. -Date based release numbers are declared to be incompatible with this scheme, -as their use is not consistent with the expected API versioning semantics -described below. Accordingly, automated tools SHOULD report an error -when encountering a leading release component greater than or equal -to ``1980``. - While any number of additional components after the first are permitted under this scheme, the most common variants are to use two components ("major.minor") or three components ("major.minor.micro"). @@ -169,6 +207,7 @@ 1.1 2.0 2.0.1 + ... A release series is any set of final release numbers that start with a common prefix. For example, ``3.3.1``, ``3.3.5`` and ``3.3.9.45`` are all @@ -181,6 +220,21 @@ form to ``X.Y.0`` when comparing it to any release segment that includes three components. +Date based release segments are also permitted, and are treated differently +in some cases when used in version specifiers. Any version identifier where +the leading component in the release segment is greater than or equal to +``1980`` is considered to be a date based release. + +An example of a date based release scheme using the year and month of the +release:: + + 2012.04 + 2012.07 + 2012.10 + 2013.01 + 2013.06 + ... + Pre-releases ------------ @@ -359,6 +413,18 @@ 1.1.dev1 ... +Date based releases, using an incrementing serial within each year, skipping +zero:: + + 2012.1 + 2012.2 + 2012.3 + ... + 2012.15 + 2013.1 + 2013.2 + ... + Summary of permitted suffixes and relative ordering --------------------------------------------------- @@ -423,6 +489,19 @@ 1.0.post456 1.1.dev1 +The integrator suffix of local version identifiers that share a common +public version identifier prefix MUST be sorted in the same order as +Python's tuple sorting when the integrator suffix is parsed as follows +(this is the same definition as is used for the release segment):: + + tuple(map(int, integrator_suffix.split("."))) + +All integrator suffixes involved in the comparison MUST be converted to a +consistent length by padding shorter segments with zeroes as needed. + +All local version identifiers (even the ``-0`` suffix) are sorted *after* +the corresponding unqualified public version identifier. + Version ordering across different metadata versions --------------------------------------------------- @@ -485,7 +564,7 @@ plus sign (builds - clause 11) are *not* compatible with this PEP and are not permitted in the public version field. -One possible mechanism to translate such semantic versioning based build +One possible mechanism to translate such semantic versioning based source labels to compatible public versions is to use the ``.devN`` suffix to specify the appropriate version order. @@ -505,20 +584,6 @@ used to record the original DVCS based version label. -Date based versions -~~~~~~~~~~~~~~~~~~~ - -As with other incompatible version schemes, date based versions can be -stored in the source label field. Translating them to a compliant -public version is straightforward: use a leading ``"0."`` prefix in the -public version label, with the date based version number as the remaining -components in the release segment. - -This has the dual benefit of allowing subsequent migration to version -numbering based on API compatibility, as well as triggering more appropriate -version comparison semantics. - - Olson database versioning ~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -526,13 +591,13 @@ Olson timezone database versioning scheme: the year followed by a lowercase character indicating the version of the database within that year. -This can be translated to a compliant 3-part version identifier as -``0..``, where the serial starts at zero (for the 'a' +This can be translated to a compliant public version identifier as +``.``, where the serial starts at zero (for the 'a' release) and is incremented with each subsequent database update within the year. As with other translated version identifiers, the corresponding Olson -database version would be recorded in the source label field. +database version could be recorded in the source label field. Version specifiers @@ -546,7 +611,8 @@ The comparison operator (or lack thereof) determines the kind of version clause: -* No operator: equivalent to ``~=`` +* No operator: equivalent to ``>=`` for date based releases, and to ``~=`` + otherwise * ``~=``: `Compatible release`_ clause * ``==``: `Version matching`_ clause * ``!=``: `Version exclusion`_ clause @@ -578,6 +644,10 @@ The specified version identifier must be in the standard format described in `Version scheme`_. +Automated tools SHOULD report an error when this operator is used in +conjunction with a date based version identifier, as it assumes the use +of semantic API versioning. + For a given release identifier ``V.N``, the compatible release clause is approximately equivalent to the pair of comparison clauses:: @@ -627,11 +697,16 @@ The specified version identifier must be in the standard format described in `Version scheme`_, but a trailing ``.*`` is permitted as described below. +If the specified version identifier is a public version identifier (no +integrator suffix), then the integrator suffix of any candidate versions +MUST be ignored when matching versions. + By default, the version matching operator is based on a strict equality comparison: the specified version must be exactly the same as the requested version. The *only* substitution performed is the zero padding of the release segment to ensure the release segments are compared with the same -length. +length (and similarly for the integrator suffix, if matching against a +specified local version identifier). Whether or not strict version matching is appropriate depends on the specific use case for the version specifier. Automated tools SHOULD at least issue @@ -670,6 +745,10 @@ those of the `Version matching`_ operator, except that the sense of any match is inverted. +If the specified version identifier is a public version identifier (no +integrator suffix), then the integrator suffix of any candidate versions +MUST be ignored when excluding versions. + For example, given the version ``1.1.post1``, the following clauses would match or not as shown:: @@ -692,6 +771,10 @@ As with version matching, the release segment is zero padded as necessary to ensure the release segments are compared with the same length. +Local version identifiers are handled according to the combination of their +handling by the version matching operator and the consistent ordering +defined by the standard version scheme. + Exclusive ordered comparison ---------------------------- @@ -711,6 +794,10 @@ the given version, even if acceptance of pre-releases is enabled as described in the section below. +Local version identifiers are handled according to the combination of their +handling by the version exclusion operator and the consistent ordering +defined by the standard version scheme. + Handling of pre-releases ------------------------ @@ -788,13 +875,14 @@ pip (from file:///localbuilds/pip-1.3.1-py33-none-any.whl) -All direct references that do not refer to a local file URL SHOULD -specify a secure transport mechanism (such as ``https``), include an -expected hash value in the URL for verification purposes, or both. If an -insecure transport is specified without any hash information, with hash -information that the tool doesn't understand, or with a selected hash -algorithm that the tool considers too weak to trust, automated tools -SHOULD at least emit a warning and MAY refuse to rely on the URL. +All direct references that do not refer to a local file URL SHOULD specify +a secure transport mechanism (such as ``https``) AND include an expected +hash value in the URL for verification purposes. If a direct reference is +specified without any hash information, with hash information that the +tool doesn't understand, or with a selected hash algorithm that the tool +considers too weak to trust, automated tools SHOULD at least emit a warning +and MAY refuse to rely on the URL. If such a direct reference also uses an +insecure transport, automated tools SHOULD NOT rely on the URL. It is RECOMMENDED that only hashes which are unconditionally provided by the latest version of the standard library's ``hashlib`` module be used @@ -806,18 +894,33 @@ specified by including a ``=`` entry as part of the URL fragment. -Version control references, the ``VCS+protocol`` scheme SHOULD be -used to identify both the version control system and the secure transport. +For version control references, the ``VCS+protocol`` scheme SHOULD be +used to identify both the version control system and the secure transport, +and a version control system with hash based commit identifiers SHOULD be +used. Automated tools MAY omit warnings about missing hashes for version +control systems that do not provide hash based commit identifiers. -To support version control systems that do not support including commit or +To handle version control systems that do not support including commit or tag references directly in the URL, that information may be appended to the -end of the URL using the ``@`` notation. +end of the URL using the ``@`` or the ``@#`` +notation. + +.. note:: + + This isn't *quite* the same as the existing VCS reference notation + supported by pip. Firstly, the distribution name is moved in front rather + than embedded as part of the URL. Secondly, the commit hash is included + even when retrieving based on a tag, in order to meet the requirement + above that *every* link should include a hash to make things harder to + forge (creating a malicious repo with a particular tag is easy, creating + one with a specific *hash*, less so). Remote URL examples:: - pip (from https://github.com/pypa/pip/archive/1.3.1.zip) - pip (from http://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686) - pip (from git+https://github.com/pypa/pip.git at 1.3.1) + pip (from https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686) + pip (from git+https://github.com/pypa/pip.git at 7921be1537eac1e97bc40179a57f0349c2aee67d) + pip (from git+https://github.com/pypa/pip.git at 1.3.1#7921be1537eac1e97bc40179a57f0349c2aee67d) + Updating the versioning specification @@ -840,7 +943,11 @@ on DVCS hashes * Added the "direct reference" concept as a standard notation for direct - references to resources (rather than each tool needing to invents its own) + references to resources (rather than each tool needing to invent its own) + +* Added the "local version identifier" and "integrator suffix" concepts to + allow system integrators to indicate patched builds in a way that is + supported by the upstream tools * Added the "compatible release" clause @@ -853,7 +960,7 @@ * Explicit exclusion of leading or trailing whitespace -* Explicit criterion for the exclusion of date based versions +* Explicit support for date based versions * Implicitly exclude pre-releases unless they're already present or needed to satisfy a dependency @@ -1039,6 +1146,29 @@ ordered comparison clauses. +Support for date based version identifiers +------------------------------------------ + +Excluding date based versions caused significant problems in migrating +``pytz`` to the new metadata standards. It also caused concerns for the +OpenStack developers, as they use a date based versioning scheme and would +like to be able to migrate to the new metadata standards without changing +it. + +The approach now adopted in the PEP is to: + +* consider a leading release segment component greater than or equal to + ``1980`` to denote a "date based release" +* using ``>=`` rather than ``~=`` as the default comparison operator for + version specifier clauses based on a date based release +* recommend reporting an error if ``~=`` is used with a date based release + +This approach means that date based version identifiers should "just work" +for ``pytz`` and any other projects with stable APIs, and at least be usable +(through the use of appropriate version specifiers on the consumer side) for +projects with less stable APIs. + + Adding direct references ------------------------ @@ -1057,6 +1187,38 @@ as well as reducing PyPI's own apparent reliability. +Adding local version identifiers +-------------------------------- + +It's a fact of life that downstream integrators often need to backport +upstream bug fixes to older versions. It's one of the services that gets +Linux distro vendors paid, and application developers may also apply patches +they need to bundled dependencies. + +Historically, this practice has been invisible to cross-platform language +specific distribution tools - the reported "version" in the upstream +metadata is the same as for the unmodified code. This inaccuracy then +can then cause problems when attempting to work with a mixture of integrator +provided code and unmodified upstream code, or even just attempting to +identify exactly which version of the software is installed. + +The introduction of local version identifiers and the "integrator suffix" +into the versioning scheme, with the corresponding ``python.integrator`` +metadata extension allows this kind of activity to be represented +accurately, which should improve interoperability between the upstream +tools and various integrated platforms. + +The exact scheme chosen is largely modelled on the existing behaviour of +``pkg_resources.parse_version`` and ``pkg_resources.parse_requirements``, +with the main distinction being that where ``pkg_resources`` currently always +takes the suffix into account when comparing versions for exact matches, +the PEP requires that the integrator suffix of the candidate version be +ignored when no integrator suffix is present in the version specifier clause. + +This change is designed to ensure that an integrator provided version like +``pip 1.5-1`` will still satisfy a version specifier like ``pip (== 1.1)``. + + References ========== diff --git a/pep-0459.txt b/pep-0459.txt --- a/pep-0459.txt +++ b/pep-0459.txt @@ -10,7 +10,7 @@ Content-Type: text/x-rst Requires: 426 Created: 11 Nov 2013 -Post-History: +Post-History: 21 Dec 2013 Abstract @@ -49,6 +49,10 @@ * ``python.exports`` * ``python.metadata_hooks`` +All standard extensions are currently at version ``1.0``, and thus the +``extension_metadata`` field may be omitted without losing access to any +functionality. + The ``details`` extension ========================= @@ -502,9 +506,9 @@ Example:: "commands": { - "wrap_console": [{"wrapwithpython": "chair:run_cli"}], - "wrap_gui": [{"wrapwithpythonw": "chair:run_gui"}], - "prebuilt": ["notawrapper"] + "wrap_console": [{"chair": "chair:run_cli"}], + "wrap_gui": [{"chair-gui": "chair:run_gui"}], + "prebuilt": ["reduniforms"] } @@ -532,18 +536,18 @@ The currently defined metadata hooks are: -* ``install``: run after a relevant distribution has been installed, +* ``postinstall``: run after a relevant distribution has been installed, upgraded or downgraded on the current system. May also be run as part of a system state resync operation. If the hook is not defined, it indicates no distribution specific actions are needed following installation. -* ``uninstall``: run after a relevant distribution has been completely +* ``postuninstall``: run after a relevant distribution has been completely removed from the current system. If the hook is not defined, it indicates no distribution specific actions are needed following uninstallation. The required signatures of these hooks are as follows:: - def install(current_meta, previous_meta=None): + def postinstall(current_meta, previous_meta=None): """Run following installation or upgrade of a relevant distribution *current_meta* is the distribution metadata for the version now @@ -554,7 +558,7 @@ resynchronisation of the system state). """ - def uninstall(previous_meta): + def postuninstall(previous_meta): """Run after complete uninstallation of a relevant distribution *previous_meta* is the distribution metadata for the version that @@ -568,8 +572,8 @@ "export_groups": { "ComfyChair.plugins": { - "install": "ComfyChair.plugins:install_hook", - "uininstall": "ComfyChair.plugins:uninstall_hook" + "postinstall": "ComfyChair.plugins:install_hook", + "postuininstall": "ComfyChair.plugins:uninstall_hook" } } @@ -588,16 +592,16 @@ "extensions": { "python.exports": { - "install": "pip.export_group_hooks:run_install_hooks", - "uininstall": "pip.export_group_hooks:run_uninstall_hooks" + "postinstall": "pip.export_group_hooks:run_install_hooks", + "postuininstall": "pip.export_group_hooks:run_uninstall_hooks" } "python.commands": { - "install": "pip.command_hook:install_wrapper_scripts", + "postinstall": "pip.command_hook:install_wrapper_scripts", } } (Note: this is just an example, but the intent is that pip *could* implement -that functionality that way if it wanted to. +that functionality that way if it wanted to). A trailing ".*" may be used to request prefix matching rather than requiring an exact match on the extension name. @@ -610,7 +614,8 @@ Metadata hooks are likely to run with elevated privileges, this needs to be considered carefully (e.g. by *requiring* that metadata hook - installation be opt in). + installation be opt in when using the standard tools and running with + elevated privileges). The given parameter names are considered part of the hook signature. Installation tools MUST call metadata hooks solely with keyword arguments. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sat Dec 21 15:47:38 2013 From: python-checkins at python.org (christian.heimes) Date: Sat, 21 Dec 2013 15:47:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_skip_multiprocessing_main_?= =?utf-8?q?handling_test_when_Python_is_built_without_threads?= Message-ID: <3dmqRt2gmmz7LjY@mail.python.org> http://hg.python.org/cpython/rev/239faf6b6e8d changeset: 88108:239faf6b6e8d user: Christian Heimes date: Sat Dec 21 15:47:24 2013 +0100 summary: skip multiprocessing main handling test when Python is built without threads files: Lib/test/test_multiprocessing_main_handling.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_multiprocessing_main_handling.py b/Lib/test/test_multiprocessing_main_handling.py --- a/Lib/test/test_multiprocessing_main_handling.py +++ b/Lib/test/test_multiprocessing_main_handling.py @@ -15,6 +15,8 @@ assert_python_ok, assert_python_failure, temp_dir, spawn_python, kill_python) +# Skip tests if _multiprocessing wasn't built. +_multiprocessing = support.import_module('_multiprocessing') # Look up which start methods are available to test import multiprocessing AVAILABLE_START_METHODS = set(multiprocessing.get_all_start_methods()) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 21 15:52:36 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 21 Dec 2013 15:52:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320037=3A_Avoid_cr?= =?utf-8?q?ashes_when_doing_text_I/O_late_at_interpreter_shutdown=2E?= Message-ID: <3dmqYc2GYGz7LjY@mail.python.org> http://hg.python.org/cpython/rev/9158f201f6d0 changeset: 88109:9158f201f6d0 user: Antoine Pitrou date: Sat Dec 21 15:51:54 2013 +0100 summary: Issue #20037: Avoid crashes when doing text I/O late at interpreter shutdown. files: Lib/test/test_io.py | 41 +++++++++++++++++++++++++++- Lib/test/test_logging.py | 20 +++++++++++++ Misc/NEWS | 3 ++ Modules/_io/_iomodule.c | 14 +++++++++ Modules/_io/_iomodule.h | 3 +- Modules/_io/bufferedio.c | 4 ++- Modules/_io/fileio.c | 6 ++- Modules/_io/iobase.c | 4 ++- Modules/_io/textio.c | 9 ++++- 9 files changed, 96 insertions(+), 8 deletions(-) diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -36,6 +36,7 @@ from collections import deque, UserList from itertools import cycle, count from test import support +from test.script_helper import assert_python_ok import codecs import io # C implementation of io @@ -2589,8 +2590,46 @@ encoding='quopri_codec') self.assertRaises(TypeError, t.read) + def _check_create_at_shutdown(self, **kwargs): + # Issue #20037: creating a TextIOWrapper at shutdown + # shouldn't crash the interpreter. + iomod = self.io.__name__ + code = """if 1: + import codecs + import {iomod} as io + + # Avoid looking up codecs at shutdown + codecs.lookup('utf-8') + + class C: + def __init__(self): + self.buf = io.BytesIO() + def __del__(self): + io.TextIOWrapper(self.buf, **{kwargs}) + print("ok") + c = C() + """.format(iomod=iomod, kwargs=kwargs) + return assert_python_ok("-c", code) + + def test_create_at_shutdown_without_encoding(self): + rc, out, err = self._check_create_at_shutdown() + if err: + # Can error out with a RuntimeError if the module state + # isn't found. + self.assertIn("RuntimeError: could not find io module state", + err.decode()) + else: + self.assertEqual("ok", out.decode().strip()) + + def test_create_at_shutdown_with_encoding(self): + rc, out, err = self._check_create_at_shutdown(encoding='utf-8', + errors='strict') + self.assertFalse(err) + self.assertEqual("ok", out.decode().strip()) + class CTextIOWrapperTest(TextIOWrapperTest): + io = io def test_initialization(self): r = self.BytesIO(b"\xc3\xa9\n\n") @@ -2634,7 +2673,7 @@ class PyTextIOWrapperTest(TextIOWrapperTest): - pass + io = pyio class IncrementalNewlineDecoderTest(unittest.TestCase): diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -41,6 +41,7 @@ import struct import sys import tempfile +from test.script_helper import assert_python_ok from test.support import (captured_stdout, run_with_locale, run_unittest, patch, requires_zlib, TestHandler, Matcher) import textwrap @@ -3397,6 +3398,25 @@ logging.setLoggerClass(logging.Logger) self.assertEqual(logging.getLoggerClass(), logging.Logger) + def test_logging_at_shutdown(self): + # Issue #20037 + code = """if 1: + import logging + + class A: + def __del__(self): + try: + raise ValueError("some error") + except Exception: + logging.exception("exception in __del__") + + a = A()""" + rc, out, err = assert_python_ok("-c", code) + err = err.decode() + self.assertIn("exception in __del__", err) + self.assertIn("ValueError: some error", err) + + class LogRecordTest(BaseTest): def test_str_rep(self): r = logging.makeLogRecord({}) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #20037: Avoid crashes when opening a text file late at interpreter + shutdown. + - Issue #19967: Thanks to the PEP 442, asyncio.Future now uses a destructor to log uncaught exceptions, instead of the dedicated _TracebackLogger class. diff --git a/Modules/_io/_iomodule.c b/Modules/_io/_iomodule.c --- a/Modules/_io/_iomodule.c +++ b/Modules/_io/_iomodule.c @@ -539,6 +539,20 @@ } +_PyIO_State * +_PyIO_get_module_state(void) +{ + PyObject *mod = PyState_FindModule(&_PyIO_Module); + _PyIO_State *state; + if (mod == NULL || (state = IO_MOD_STATE(mod)) == NULL) { + PyErr_SetString(PyExc_RuntimeError, + "could not find io module state " + "(interpreter shutdown?)"); + return NULL; + } + return state; +} + PyObject * _PyIO_get_locale_module(_PyIO_State *state) { diff --git a/Modules/_io/_iomodule.h b/Modules/_io/_iomodule.h --- a/Modules/_io/_iomodule.h +++ b/Modules/_io/_iomodule.h @@ -135,8 +135,9 @@ } _PyIO_State; #define IO_MOD_STATE(mod) ((_PyIO_State *)PyModule_GetState(mod)) -#define IO_STATE IO_MOD_STATE(PyState_FindModule(&_PyIO_Module)) +#define IO_STATE() _PyIO_get_module_state() +extern _PyIO_State *_PyIO_get_module_state(void); extern PyObject *_PyIO_get_locale_module(_PyIO_State *); extern PyObject *_PyIO_str_close; diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c --- a/Modules/_io/bufferedio.c +++ b/Modules/_io/bufferedio.c @@ -91,7 +91,9 @@ static PyObject * bufferediobase_unsupported(const char *message) { - PyErr_SetString(IO_STATE->unsupported_operation, message); + _PyIO_State *state = IO_STATE(); + if (state != NULL) + PyErr_SetString(state->unsupported_operation, message); return NULL; } diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c --- a/Modules/_io/fileio.c +++ b/Modules/_io/fileio.c @@ -493,8 +493,10 @@ static PyObject * err_mode(char *action) { - PyErr_Format(IO_STATE->unsupported_operation, - "File not open for %s", action); + _PyIO_State *state = IO_STATE(); + if (state != NULL) + PyErr_Format(state->unsupported_operation, + "File not open for %s", action); return NULL; } diff --git a/Modules/_io/iobase.c b/Modules/_io/iobase.c --- a/Modules/_io/iobase.c +++ b/Modules/_io/iobase.c @@ -69,7 +69,9 @@ static PyObject * iobase_unsupported(const char *message) { - PyErr_SetString(IO_STATE->unsupported_operation, message); + _PyIO_State *state = IO_STATE(); + if (state != NULL) + PyErr_SetString(state->unsupported_operation, message); return NULL; } diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c --- a/Modules/_io/textio.c +++ b/Modules/_io/textio.c @@ -45,7 +45,9 @@ static PyObject * _unsupported(const char *message) { - PyErr_SetString(IO_STATE->unsupported_operation, message); + _PyIO_State *state = IO_STATE(); + if (state != NULL) + PyErr_SetString(state->unsupported_operation, message); return NULL; } @@ -852,7 +854,7 @@ char *errors = NULL; char *newline = NULL; int line_buffering = 0, write_through = 0; - _PyIO_State *state = IO_STATE; + _PyIO_State *state = NULL; PyObject *res; int r; @@ -891,6 +893,9 @@ if (encoding == NULL) { /* Try os.device_encoding(fileno) */ PyObject *fileno; + state = IO_STATE(); + if (state == NULL) + goto error; fileno = _PyObject_CallMethodId(buffer, &PyId_fileno, NULL); /* Ignore only AttributeError and UnsupportedOperation */ if (fileno == NULL) { -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 21 16:20:17 2013 From: python-checkins at python.org (christian.heimes) Date: Sat, 21 Dec 2013 16:20:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2316136=3A_Remove_V?= =?utf-8?q?MS_support_and_VMS-related_code?= Message-ID: <3dmr9Y1V6Pz7LjY@mail.python.org> http://hg.python.org/cpython/rev/568391b3eda9 changeset: 88110:568391b3eda9 parent: 88108:239faf6b6e8d user: Christian Heimes date: Sat Dec 21 16:19:10 2013 +0100 summary: Issue #16136: Remove VMS support and VMS-related code files: Doc/whatsnew/3.4.rst | 1 + Include/pyerrors.h | 3 - Lib/test/test_pep3151.py | 2 - Misc/NEWS | 2 + Modules/_cryptmodule.c | 7 - Modules/fcntlmodule.c | 4 - Modules/fpectlmodule.c | 29 ------- Modules/getpath.c | 6 +- Modules/grpmodule.c | 5 - Modules/main.c | 29 ------- Modules/mmapmodule.c | 6 - Modules/posixmodule.c | 23 ----- Modules/pwdmodule.c | 8 -- Modules/selectmodule.c | 25 ++--- Modules/socketmodule.c | 106 ++------------------------ Objects/exceptions.c | 6 - Parser/myreadline.c | 8 -- Python/dynload_shlib.c | 26 ------ Python/random.c | 37 +-------- Python/sysmodule.c | 19 ---- 20 files changed, 27 insertions(+), 325 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -977,6 +977,7 @@ * OS/2 * Windows 2000 +* VMS Deprecated Python modules, functions and methods diff --git a/Include/pyerrors.h b/Include/pyerrors.h --- a/Include/pyerrors.h +++ b/Include/pyerrors.h @@ -199,9 +199,6 @@ #ifdef MS_WINDOWS PyAPI_DATA(PyObject *) PyExc_WindowsError; #endif -#ifdef __VMS -PyAPI_DATA(PyObject *) PyExc_VMSError; -#endif PyAPI_DATA(PyObject *) PyExc_RecursionErrorInst; diff --git a/Lib/test/test_pep3151.py b/Lib/test/test_pep3151.py --- a/Lib/test/test_pep3151.py +++ b/Lib/test/test_pep3151.py @@ -157,8 +157,6 @@ e.characters_written = 5 self.assertEqual(e.characters_written, 5) - # XXX VMSError not tested - class ExplicitSubclassingTest(unittest.TestCase): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -256,6 +256,8 @@ Build ----- +- Issue #16136: Remove VMS support + - Issue #18215: Add script Tools/ssl/test_multiple_versions.py to compile and run Python's unit tests with multiple versions of OpenSSL. diff --git a/Modules/_cryptmodule.c b/Modules/_cryptmodule.c --- a/Modules/_cryptmodule.c +++ b/Modules/_cryptmodule.c @@ -5,19 +5,12 @@ #include -#ifdef __VMS -#include -#endif - /* Module crypt */ static PyObject *crypt_crypt(PyObject *self, PyObject *args) { char *word, *salt; -#ifndef __VMS - extern char * crypt(const char *, const char *); -#endif if (!PyArg_ParseTuple(args, "ss:crypt", &word, &salt)) { return NULL; diff --git a/Modules/fcntlmodule.c b/Modules/fcntlmodule.c --- a/Modules/fcntlmodule.c +++ b/Modules/fcntlmodule.c @@ -208,11 +208,7 @@ return NULL; } Py_BEGIN_ALLOW_THREADS -#ifdef __VMS - ret = ioctl(fd, code, (void *)arg); -#else ret = ioctl(fd, code, arg); -#endif Py_END_ALLOW_THREADS if (ret < 0) { PyErr_SetFromErrno(PyExc_IOError); diff --git a/Modules/fpectlmodule.c b/Modules/fpectlmodule.c --- a/Modules/fpectlmodule.c +++ b/Modules/fpectlmodule.c @@ -70,10 +70,6 @@ #if defined(__FreeBSD__) # include -#elif defined(__VMS) -#define __NEW_STARLET -#include -#include #endif #ifndef WANT_SIGFPE_HANDLER @@ -182,23 +178,6 @@ ieee_set_fp_control(fp_control); PyOS_setsig(SIGFPE, handler); -/*-- DEC ALPHA VMS --------------------------------------------------------*/ -#elif defined(__ALPHA) && defined(__VMS) - IEEE clrmsk; - IEEE setmsk; - clrmsk.ieee$q_flags = - IEEE$M_TRAP_ENABLE_UNF | IEEE$M_TRAP_ENABLE_INE | - IEEE$M_MAP_UMZ; - setmsk.ieee$q_flags = - IEEE$M_TRAP_ENABLE_INV | IEEE$M_TRAP_ENABLE_DZE | - IEEE$M_TRAP_ENABLE_OVF; - sys$ieee_set_fp_control(&clrmsk, &setmsk, 0); - PyOS_setsig(SIGFPE, handler); - -/*-- HP IA64 VMS --------------------------------------------------------*/ -#elif defined(__ia64) && defined(__VMS) - PyOS_setsig(SIGFPE, handler); - /*-- Cray Unicos ----------------------------------------------------------*/ #elif defined(cray) /* UNICOS delivers SIGFPE by default, but no matherr */ @@ -251,14 +230,6 @@ #ifdef __FreeBSD__ fpresetsticky(fpgetsticky()); fpsetmask(0); -#elif defined(__VMS) - IEEE clrmsk; - clrmsk.ieee$q_flags = - IEEE$M_TRAP_ENABLE_UNF | IEEE$M_TRAP_ENABLE_INE | - IEEE$M_MAP_UMZ | IEEE$M_TRAP_ENABLE_INV | - IEEE$M_TRAP_ENABLE_DZE | IEEE$M_TRAP_ENABLE_OVF | - IEEE$M_INHERIT; - sys$ieee_set_fp_control(&clrmsk, 0, 0); #else fputs("Operation not implemented\n", stderr); #endif diff --git a/Modules/getpath.c b/Modules/getpath.c --- a/Modules/getpath.c +++ b/Modules/getpath.c @@ -110,11 +110,7 @@ #endif #ifndef PREFIX -# ifdef __VMS -# define PREFIX "" -# else -# define PREFIX "/usr/local" -# endif +# define PREFIX "/usr/local" #endif #ifndef EXEC_PREFIX diff --git a/Modules/grpmodule.c b/Modules/grpmodule.c --- a/Modules/grpmodule.c +++ b/Modules/grpmodule.c @@ -58,17 +58,12 @@ #define SET(i,val) PyStructSequence_SET_ITEM(v, i, val) SET(setIndex++, PyUnicode_DecodeFSDefault(p->gr_name)); -#ifdef __VMS - SET(setIndex++, Py_None); - Py_INCREF(Py_None); -#else if (p->gr_passwd) SET(setIndex++, PyUnicode_DecodeFSDefault(p->gr_passwd)); else { SET(setIndex++, Py_None); Py_INCREF(Py_None); } -#endif SET(setIndex++, _PyLong_FromGid(p->gr_gid)); SET(setIndex++, w); #undef SET diff --git a/Modules/main.c b/Modules/main.c --- a/Modules/main.c +++ b/Modules/main.c @@ -5,11 +5,6 @@ #include -#ifdef __VMS -#error "PEP 11: VMS is now unsupported, code will be removed in Python 3.4" -#include -#endif - #if defined(MS_WINDOWS) || defined(__CYGWIN__) #include #ifdef HAVE_FCNTL_H @@ -124,19 +119,7 @@ fprintf(f, usage_5, DELIM, PYTHONHOMEHELP); fputs(usage_6, f); } -#if defined(__VMS) - if (exitcode == 0) { - /* suppress 'error' message */ - return 1; - } - else { - /* STS$M_INHIB_MSG + SS$_ABORT */ - return 0x1000002c; - } -#else return exitcode; -#endif - /*NOTREACHED*/ } static void RunStartupFile(PyCompilerFlags *cf) @@ -580,14 +563,7 @@ if (command == NULL && module == NULL && _PyOS_optind < argc && wcscmp(argv[_PyOS_optind], L"-") != 0) { -#ifdef __VMS - filename = decc$translate_vms(argv[_PyOS_optind]); - if (filename == (char *)0 || filename == (char *)-1) - filename = argv[_PyOS_optind]; - -#else filename = argv[_PyOS_optind]; -#endif } stdin_is_interactive = Py_FdIsInteractive(stdin, (char *)0); @@ -623,11 +599,6 @@ #endif /* !MS_WINDOWS */ /* Leave stderr alone - it should be unbuffered anyway. */ } -#ifdef __VMS - else { - setvbuf (stdout, (char *)NULL, _IOLBF, BUFSIZ); - } -#endif /* __VMS */ #ifdef __APPLE__ /* On MacOS X, when the Python interpreter is embedded in an diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c --- a/Modules/mmapmodule.c +++ b/Modules/mmapmodule.c @@ -1158,12 +1158,6 @@ (void)fcntl(fd, F_FULLFSYNC); #endif #ifdef HAVE_FSTAT -# ifdef __VMS - /* on OpenVMS we must ensure that all bytes are written to the file */ - if (fd != -1) { - fsync(fd); - } -# endif if (fd != -1 && fstat(fd, &st) == 0 && S_ISREG(st.st_mode)) { if (map_size == 0) { if (st.st_size == 0) { diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -29,11 +29,6 @@ #include "posixmodule.h" #endif -#if defined(__VMS) -# error "PEP 11: VMS is now unsupported, code will be removed in Python 3.4" -# include -#endif /* defined(__VMS) */ - #ifdef __cplusplus extern "C" { #endif @@ -164,9 +159,6 @@ #define HAVE_FSYNC 1 #define fsync _commit #else -#if defined(__VMS) -/* Everything needed is defined in vms/pyconfig.h */ -#else /* all other compilers */ /* Unix functions that the configure script doesn't check for */ #define HAVE_EXECV 1 #define HAVE_FORK 1 @@ -184,7 +176,6 @@ #define HAVE_SYSTEM 1 #define HAVE_WAIT 1 #define HAVE_TTYNAME 1 -#endif /* __VMS */ #endif /* _MSC_VER */ #endif /* __BORLANDC__ */ #endif /* ! __WATCOMC__ || __QNX__ */ @@ -2765,17 +2756,7 @@ { char *ret; -#if defined(__VMS) - /* file descriptor 0 only, the default input device (stdin) */ - if (fd == 0) { - ret = ttyname(); - } - else { - ret = NULL; - } -#else ret = ttyname(fd); -#endif if (ret == NULL) posix_error(); return ret; @@ -8340,10 +8321,6 @@ int res; if (!PyArg_ParseTuple(args, "i:fstat", &fd)) return NULL; -#ifdef __VMS - /* on OpenVMS we must ensure that all bytes are written to the file */ - fsync(fd); -#endif Py_BEGIN_ALLOW_THREADS res = FSTAT(fd, &st); Py_END_ALLOW_THREADS diff --git a/Modules/pwdmodule.c b/Modules/pwdmodule.c --- a/Modules/pwdmodule.c +++ b/Modules/pwdmodule.c @@ -69,18 +69,10 @@ #define SETS(i,val) sets(v, i, val) SETS(setIndex++, p->pw_name); -#ifdef __VMS - SETS(setIndex++, ""); -#else SETS(setIndex++, p->pw_passwd); -#endif PyStructSequence_SET_ITEM(v, setIndex++, _PyLong_FromUid(p->pw_uid)); PyStructSequence_SET_ITEM(v, setIndex++, _PyLong_FromGid(p->pw_gid)); -#ifdef __VMS - SETS(setIndex++, ""); -#else SETS(setIndex++, p->pw_gecos); -#endif SETS(setIndex++, p->pw_dir); SETS(setIndex++, p->pw_shell); diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -52,9 +52,6 @@ # include #else # define SOCKET int -# if defined(__VMS) -# include -# endif #endif /* list of Python objects and their file descriptor */ @@ -1708,17 +1705,17 @@ * kevent is not standard and its members vary across BSDs. */ #if !defined(__OpenBSD__) -# define IDENT_TYPE T_UINTPTRT -# define IDENT_CAST Py_intptr_t -# define DATA_TYPE T_INTPTRT +# define IDENT_TYPE T_UINTPTRT +# define IDENT_CAST Py_intptr_t +# define DATA_TYPE T_INTPTRT # define DATA_FMT_UNIT INTPTRT_FMT_UNIT -# define IDENT_AsType PyLong_AsUintptr_t +# define IDENT_AsType PyLong_AsUintptr_t #else -# define IDENT_TYPE T_UINT -# define IDENT_CAST int -# define DATA_TYPE T_INT +# define IDENT_TYPE T_UINT +# define IDENT_CAST int +# define DATA_TYPE T_INT # define DATA_FMT_UNIT "i" -# define IDENT_AsType PyLong_AsUnsignedLong +# define IDENT_AsType PyLong_AsUnsignedLong #endif /* Unfortunately, we can't store python objects in udata, because @@ -1770,7 +1767,7 @@ if (PyLong_Check(pfd) #if IDENT_TYPE == T_UINT - && PyLong_AsUnsignedLong(pfd) <= UINT_MAX + && PyLong_AsUnsignedLong(pfd) <= UINT_MAX #endif ) { self->e.ident = IDENT_AsType(pfd); @@ -2253,7 +2250,7 @@ that are ready.\n\ \n\ *** IMPORTANT NOTICE ***\n\ -On Windows and OpenVMS, only sockets are supported; on Unix, all file\n\ +On Windows only sockets are supported; on Unix, all file\n\ descriptors can be used."); static PyMethodDef select_methods[] = { @@ -2271,7 +2268,7 @@ "This module supports asynchronous I/O on multiple file descriptors.\n\ \n\ *** IMPORTANT NOTICE ***\n\ -On Windows and OpenVMS, only sockets are supported; on Unix, all file descriptors."); +On Windows only sockets are supported; on Unix, all file descriptors."); static struct PyModuleDef selectmodule = { diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -188,7 +188,7 @@ #if defined(WITH_THREAD) && (defined(__APPLE__) || \ (defined(__FreeBSD__) && __FreeBSD_version+0 < 503000) || \ defined(__OpenBSD__) || defined(__NetBSD__) || \ - defined(__VMS) || !defined(HAVE_GETADDRINFO)) + !defined(HAVE_GETADDRINFO)) #define USE_GETADDRINFO_LOCK #endif @@ -212,10 +212,6 @@ # include #endif -#if defined(__VMS) -# include -#endif - #ifdef __APPLE__ # include #endif @@ -403,11 +399,6 @@ #endif #endif -#ifdef __VMS -/* TCP/IP Services for VMS uses a maximum send/recv buffer length */ -#define SEGMENT_SIZE (32 * 1024 -1) -#endif - /* Convert "sock_addr_t *" to "struct sockaddr *". */ #define SAS2SA(x) (&((x)->sa)) @@ -556,37 +547,13 @@ return NULL; } -#ifdef __VMS -/* Function to send in segments */ -static int -sendsegmented(int sock_fd, char *buf, int len, int flags) -{ - int n = 0; - int remaining = len; - - while (remaining > 0) { - unsigned int segment; - - segment = (remaining >= SEGMENT_SIZE ? SEGMENT_SIZE : remaining); - n = send(sock_fd, buf, segment, flags); - if (n < 0) { - return n; - } - remaining -= segment; - buf += segment; - } /* end while */ - - return len; -} -#endif - /* Function to perform the setting of socket blocking mode internally. block = (1 | 0). */ static int internal_setblocking(PySocketSockObject *s, int block) { #if !defined(MS_WINDOWS) \ - && !((defined(HAVE_SYS_IOCTL_H) && defined(FIONBIO)) || defined(__VMS)) + && !((defined(HAVE_SYS_IOCTL_H) && defined(FIONBIO))) int delay_flag, new_delay_flag; #endif #ifdef SOCK_NONBLOCK @@ -598,7 +565,7 @@ Py_BEGIN_ALLOW_THREADS #ifndef MS_WINDOWS -#if (defined(HAVE_SYS_IOCTL_H) && defined(FIONBIO)) || defined(__VMS) +#if (defined(HAVE_SYS_IOCTL_H) && defined(FIONBIO)) block = !block; ioctl(s->sock_fd, FIONBIO, (unsigned int *)&block); #else @@ -2227,13 +2194,7 @@ return s->errorhandler(); return PyLong_FromLong(flag); } -#ifdef __VMS - /* socklen_t is unsigned so no negative test is needed, - test buflen == 0 is previously done */ - if (buflen > 1024) { -#else if (buflen <= 0 || buflen > 1024) { -#endif PyErr_SetString(PyExc_OSError, "getsockopt buflen out of range"); return NULL; @@ -2603,10 +2564,6 @@ { Py_ssize_t outlen = -1; int timeout; -#ifdef __VMS - int remaining; - char *read_buf; -#endif if (!IS_SELECTABLE(s)) { select_error(); @@ -2617,7 +2574,6 @@ return 0; } -#ifndef __VMS BEGIN_SELECT_LOOP(s) Py_BEGIN_ALLOW_THREADS timeout = internal_select_ex(s, 0, interval); @@ -2643,48 +2599,6 @@ s->errorhandler(); return -1; } -#else - read_buf = cbuf; - remaining = len; - while (remaining != 0) { - unsigned int segment; - int nread = -1; - - segment = remaining /SEGMENT_SIZE; - if (segment != 0) { - segment = SEGMENT_SIZE; - } - else { - segment = remaining; - } - - BEGIN_SELECT_LOOP(s) - Py_BEGIN_ALLOW_THREADS - timeout = internal_select_ex(s, 0, interval); - if (!timeout) - nread = recv(s->sock_fd, read_buf, segment, flags); - Py_END_ALLOW_THREADS - if (timeout == 1) { - PyErr_SetString(socket_timeout, "timed out"); - return -1; - } - END_SELECT_LOOP(s) - - if (nread < 0) { - s->errorhandler(); - return -1; - } - if (nread != remaining) { - read_buf += nread; - break; - } - - remaining -= segment; - read_buf += segment; - } - outlen = read_buf - cbuf; -#endif /* !__VMS */ - return outlen; } @@ -3318,9 +3232,7 @@ Py_BEGIN_ALLOW_THREADS timeout = internal_select_ex(s, 1, interval); if (!timeout) { -#ifdef __VMS - n = sendsegmented(s->sock_fd, buf, len, flags); -#elif defined(MS_WINDOWS) +#ifdef MS_WINDOWS if (len > INT_MAX) len = INT_MAX; n = send(s->sock_fd, buf, (int)len, flags); @@ -3375,9 +3287,7 @@ timeout = internal_select(s, 1); n = -1; if (!timeout) { -#ifdef __VMS - n = sendsegmented(s->sock_fd, buf, len, flags); -#elif defined(MS_WINDOWS) +#ifdef MS_WINDOWS if (len > INT_MAX) len = INT_MAX; n = send(s->sock_fd, buf, (int)len, flags); @@ -5310,9 +5220,9 @@ #if defined(__APPLE__) && defined(AI_NUMERICSERV) if ((flags & AI_NUMERICSERV) && (pptr == NULL || (pptr[0] == '0' && pptr[1] == 0))) { /* On OSX upto at least OSX 10.8 getaddrinfo crashes - * if AI_NUMERICSERV is set and the servname is NULL or "0". - * This workaround avoids a segfault in libsystem. - */ + * if AI_NUMERICSERV is set and the servname is NULL or "0". + * This workaround avoids a segfault in libsystem. + */ pptr = "00"; } #endif diff --git a/Objects/exceptions.c b/Objects/exceptions.c --- a/Objects/exceptions.c +++ b/Objects/exceptions.c @@ -16,9 +16,6 @@ #ifdef MS_WINDOWS PyObject *PyExc_WindowsError = NULL; #endif -#ifdef __VMS -PyObject *PyExc_VMSError = NULL; -#endif /* The dict map from errno codes to OSError subclasses */ static PyObject *errnomap = NULL; @@ -2473,9 +2470,6 @@ #ifdef MS_WINDOWS INIT_ALIAS(WindowsError, OSError) #endif -#ifdef __VMS - INIT_ALIAS(VMSError, OSError) -#endif POST_INIT(EOFError) POST_INIT(RuntimeError) POST_INIT(NotImplementedError) diff --git a/Parser/myreadline.c b/Parser/myreadline.c --- a/Parser/myreadline.c +++ b/Parser/myreadline.c @@ -15,10 +15,6 @@ #include "windows.h" #endif /* MS_WINDOWS */ -#ifdef __VMS -extern char* vms__StdioReadline(FILE *sys_stdin, FILE *sys_stdout, char *prompt); -#endif - PyThreadState* _PyOS_ReadlineTState; @@ -189,11 +185,7 @@ if (PyOS_ReadlineFunctionPointer == NULL) { -#ifdef __VMS - PyOS_ReadlineFunctionPointer = vms__StdioReadline; -#else PyOS_ReadlineFunctionPointer = PyOS_StdioReadline; -#endif } #ifdef WITH_THREAD diff --git a/Python/dynload_shlib.c b/Python/dynload_shlib.c --- a/Python/dynload_shlib.c +++ b/Python/dynload_shlib.c @@ -36,25 +36,16 @@ #ifdef __CYGWIN__ ".dll", #else /* !__CYGWIN__ */ -#ifdef __VMS - ".exe", - ".EXE", -#else /* !__VMS */ "." SOABI ".so", ".abi" PYTHON_ABI_STRING ".so", ".so", -#endif /* __VMS */ #endif /* __CYGWIN__ */ NULL, }; static struct { dev_t dev; -#ifdef __VMS - ino_t ino[3]; -#else ino_t ino; -#endif void *handle; } handles[128]; static int nhandles = 0; @@ -95,29 +86,12 @@ } if (nhandles < 128) { handles[nhandles].dev = statb.st_dev; -#ifdef __VMS - handles[nhandles].ino[0] = statb.st_ino[0]; - handles[nhandles].ino[1] = statb.st_ino[1]; - handles[nhandles].ino[2] = statb.st_ino[2]; -#else handles[nhandles].ino = statb.st_ino; -#endif } } dlopenflags = PyThreadState_GET()->interp->dlopenflags; -#ifdef __VMS - /* VMS currently don't allow a pathname, use a logical name instead */ - /* Concatenate 'python_module_' and shortname */ - /* so "import vms.bar" will use the logical python_module_bar */ - /* As C module use only one name space this is probably not a */ - /* important limitation */ - PyOS_snprintf(pathbuf, sizeof(pathbuf), "python_module_%-.200s", - shortname); - pathname = pathbuf; -#endif - handle = dlopen(pathname, dlopenflags); if (handle == NULL) { diff --git a/Python/random.c b/Python/random.c --- a/Python/random.c +++ b/Python/random.c @@ -68,28 +68,7 @@ #endif /* MS_WINDOWS */ -#ifdef __VMS -/* Use openssl random routine */ -#include -static int -vms_urandom(unsigned char *buffer, Py_ssize_t size, int raise) -{ - if (RAND_pseudo_bytes(buffer, size) < 0) { - if (raise) { - PyErr_Format(PyExc_ValueError, - "RAND_pseudo_bytes"); - } else { - Py_FatalError("Failed to initialize the randomized hash " - "secret using RAND_pseudo_bytes"); - } - return -1; - } - return 0; -} -#endif /* __VMS */ - - -#if !defined(MS_WINDOWS) && !defined(__VMS) +#ifndef MS_WINDOWS static int urandom_fd = -1; /* Read size bytes from /dev/urandom into buffer. @@ -195,7 +174,7 @@ } } -#endif /* !defined(MS_WINDOWS) && !defined(__VMS) */ +#endif /* MS_WINDOWS */ /* Fill buffer with pseudo-random bytes generated by a linear congruent generator (LCG): @@ -237,11 +216,7 @@ #ifdef MS_WINDOWS return win32_urandom((unsigned char *)buffer, size, 1); #else -# ifdef __VMS - return vms_urandom((unsigned char *)buffer, size, 1); -# else return dev_urandom_python((char*)buffer, size); -# endif #endif } @@ -285,12 +260,8 @@ else { #ifdef MS_WINDOWS (void)win32_urandom(secret, secret_size, 0); -#else /* #ifdef MS_WINDOWS */ -# ifdef __VMS - vms_urandom(secret, secret_size, 0); -# else +#else dev_urandom_noraise(secret, secret_size); -# endif #endif } } @@ -298,7 +269,7 @@ void _PyRandom_Fini(void) { -#if !defined(MS_WINDOWS) && !defined(__VMS) +#ifndef MS_WINDOWS dev_urandom_close(); #endif } diff --git a/Python/sysmodule.c b/Python/sysmodule.c --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -32,10 +32,6 @@ extern const char *PyWin_DLLVersionString; #endif -#ifdef __VMS -#include -#endif - #ifdef HAVE_LANGINFO_H #include #include @@ -1867,22 +1863,7 @@ if (av != NULL) { int i; for (i = 0; i < argc; i++) { -#ifdef __VMS - PyObject *v; - - /* argv[0] is the script pathname if known */ - if (i == 0) { - char* fn = decc$translate_vms(argv[0]); - if ((fn == (char *)0) || fn == (char *)-1) - v = PyUnicode_FromString(argv[0]); - else - v = PyUnicode_FromString( - decc$translate_vms(argv[0])); - } else - v = PyUnicode_FromString(argv[i]); -#else PyObject *v = PyUnicode_FromWideChar(argv[i], -1); -#endif if (v == NULL) { Py_DECREF(av); av = NULL; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 21 16:20:18 2013 From: python-checkins at python.org (christian.heimes) Date: Sat, 21 Dec 2013 16:20:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?q?=29=3A_merge?= Message-ID: <3dmr9Z4p3Mz7Ljq@mail.python.org> http://hg.python.org/cpython/rev/006c3e3c76b4 changeset: 88111:006c3e3c76b4 parent: 88110:568391b3eda9 parent: 88109:9158f201f6d0 user: Christian Heimes date: Sat Dec 21 16:19:57 2013 +0100 summary: merge files: Lib/test/test_io.py | 41 +++++++++++++++++++++++++++- Lib/test/test_logging.py | 20 +++++++++++++ Misc/NEWS | 3 ++ Modules/_io/_iomodule.c | 14 +++++++++ Modules/_io/_iomodule.h | 3 +- Modules/_io/bufferedio.c | 4 ++- Modules/_io/fileio.c | 6 ++- Modules/_io/iobase.c | 4 ++- Modules/_io/textio.c | 9 ++++- 9 files changed, 96 insertions(+), 8 deletions(-) diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -36,6 +36,7 @@ from collections import deque, UserList from itertools import cycle, count from test import support +from test.script_helper import assert_python_ok import codecs import io # C implementation of io @@ -2589,8 +2590,46 @@ encoding='quopri_codec') self.assertRaises(TypeError, t.read) + def _check_create_at_shutdown(self, **kwargs): + # Issue #20037: creating a TextIOWrapper at shutdown + # shouldn't crash the interpreter. + iomod = self.io.__name__ + code = """if 1: + import codecs + import {iomod} as io + + # Avoid looking up codecs at shutdown + codecs.lookup('utf-8') + + class C: + def __init__(self): + self.buf = io.BytesIO() + def __del__(self): + io.TextIOWrapper(self.buf, **{kwargs}) + print("ok") + c = C() + """.format(iomod=iomod, kwargs=kwargs) + return assert_python_ok("-c", code) + + def test_create_at_shutdown_without_encoding(self): + rc, out, err = self._check_create_at_shutdown() + if err: + # Can error out with a RuntimeError if the module state + # isn't found. + self.assertIn("RuntimeError: could not find io module state", + err.decode()) + else: + self.assertEqual("ok", out.decode().strip()) + + def test_create_at_shutdown_with_encoding(self): + rc, out, err = self._check_create_at_shutdown(encoding='utf-8', + errors='strict') + self.assertFalse(err) + self.assertEqual("ok", out.decode().strip()) + class CTextIOWrapperTest(TextIOWrapperTest): + io = io def test_initialization(self): r = self.BytesIO(b"\xc3\xa9\n\n") @@ -2634,7 +2673,7 @@ class PyTextIOWrapperTest(TextIOWrapperTest): - pass + io = pyio class IncrementalNewlineDecoderTest(unittest.TestCase): diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -41,6 +41,7 @@ import struct import sys import tempfile +from test.script_helper import assert_python_ok from test.support import (captured_stdout, run_with_locale, run_unittest, patch, requires_zlib, TestHandler, Matcher) import textwrap @@ -3397,6 +3398,25 @@ logging.setLoggerClass(logging.Logger) self.assertEqual(logging.getLoggerClass(), logging.Logger) + def test_logging_at_shutdown(self): + # Issue #20037 + code = """if 1: + import logging + + class A: + def __del__(self): + try: + raise ValueError("some error") + except Exception: + logging.exception("exception in __del__") + + a = A()""" + rc, out, err = assert_python_ok("-c", code) + err = err.decode() + self.assertIn("exception in __del__", err) + self.assertIn("ValueError: some error", err) + + class LogRecordTest(BaseTest): def test_str_rep(self): r = logging.makeLogRecord({}) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #20037: Avoid crashes when opening a text file late at interpreter + shutdown. + - Issue #19967: Thanks to the PEP 442, asyncio.Future now uses a destructor to log uncaught exceptions, instead of the dedicated _TracebackLogger class. diff --git a/Modules/_io/_iomodule.c b/Modules/_io/_iomodule.c --- a/Modules/_io/_iomodule.c +++ b/Modules/_io/_iomodule.c @@ -539,6 +539,20 @@ } +_PyIO_State * +_PyIO_get_module_state(void) +{ + PyObject *mod = PyState_FindModule(&_PyIO_Module); + _PyIO_State *state; + if (mod == NULL || (state = IO_MOD_STATE(mod)) == NULL) { + PyErr_SetString(PyExc_RuntimeError, + "could not find io module state " + "(interpreter shutdown?)"); + return NULL; + } + return state; +} + PyObject * _PyIO_get_locale_module(_PyIO_State *state) { diff --git a/Modules/_io/_iomodule.h b/Modules/_io/_iomodule.h --- a/Modules/_io/_iomodule.h +++ b/Modules/_io/_iomodule.h @@ -135,8 +135,9 @@ } _PyIO_State; #define IO_MOD_STATE(mod) ((_PyIO_State *)PyModule_GetState(mod)) -#define IO_STATE IO_MOD_STATE(PyState_FindModule(&_PyIO_Module)) +#define IO_STATE() _PyIO_get_module_state() +extern _PyIO_State *_PyIO_get_module_state(void); extern PyObject *_PyIO_get_locale_module(_PyIO_State *); extern PyObject *_PyIO_str_close; diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c --- a/Modules/_io/bufferedio.c +++ b/Modules/_io/bufferedio.c @@ -91,7 +91,9 @@ static PyObject * bufferediobase_unsupported(const char *message) { - PyErr_SetString(IO_STATE->unsupported_operation, message); + _PyIO_State *state = IO_STATE(); + if (state != NULL) + PyErr_SetString(state->unsupported_operation, message); return NULL; } diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c --- a/Modules/_io/fileio.c +++ b/Modules/_io/fileio.c @@ -493,8 +493,10 @@ static PyObject * err_mode(char *action) { - PyErr_Format(IO_STATE->unsupported_operation, - "File not open for %s", action); + _PyIO_State *state = IO_STATE(); + if (state != NULL) + PyErr_Format(state->unsupported_operation, + "File not open for %s", action); return NULL; } diff --git a/Modules/_io/iobase.c b/Modules/_io/iobase.c --- a/Modules/_io/iobase.c +++ b/Modules/_io/iobase.c @@ -69,7 +69,9 @@ static PyObject * iobase_unsupported(const char *message) { - PyErr_SetString(IO_STATE->unsupported_operation, message); + _PyIO_State *state = IO_STATE(); + if (state != NULL) + PyErr_SetString(state->unsupported_operation, message); return NULL; } diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c --- a/Modules/_io/textio.c +++ b/Modules/_io/textio.c @@ -45,7 +45,9 @@ static PyObject * _unsupported(const char *message) { - PyErr_SetString(IO_STATE->unsupported_operation, message); + _PyIO_State *state = IO_STATE(); + if (state != NULL) + PyErr_SetString(state->unsupported_operation, message); return NULL; } @@ -852,7 +854,7 @@ char *errors = NULL; char *newline = NULL; int line_buffering = 0, write_through = 0; - _PyIO_State *state = IO_STATE; + _PyIO_State *state = NULL; PyObject *res; int r; @@ -891,6 +893,9 @@ if (encoding == NULL) { /* Try os.device_encoding(fileno) */ PyObject *fileno; + state = IO_STATE(); + if (state == NULL) + goto error; fileno = _PyObject_CallMethodId(buffer, &PyId_fileno, NULL); /* Ignore only AttributeError and UnsupportedOperation */ if (fileno == NULL) { -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 21 18:32:37 2013 From: python-checkins at python.org (r.david.murray) Date: Sat, 21 Dec 2013 18:32:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Make_CPython_PEP_summary_e?= =?utf-8?q?ntry_links_consistent=2C_add_pep_456_discussion=2E?= Message-ID: <3dmv6F68TTz7LjP@mail.python.org> http://hg.python.org/cpython/rev/ae9b74c2b357 changeset: 88112:ae9b74c2b357 user: R David Murray date: Sat Dec 21 12:32:10 2013 -0500 summary: Make CPython PEP summary entry links consistent, add pep 456 discussion. files: Doc/whatsnew/3.4.rst | 29 ++++++++++++++++++++++++----- 1 files changed, 24 insertions(+), 5 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -123,9 +123,9 @@ CPython implementation improvements: -* :ref:`PEP 442: Safe object finalization ` -* :ref:`PEP 445: Configurable memory allocators ` -* :pep:`456` Secure and interchangeable hash algorithm +* :pep:`442`: :ref:`Safe object finalization ` +* :pep:`445`: :ref:`Configurable memory allocators ` +* :pep:`456`: :ref:`Secure and interchangeable hash algorithm ` * Improve finalization of Python modules to avoid setting their globals to None, in most cases (:issue:`18214`). * A more efficient :mod:`marshal` format (:issue:`16475`). @@ -909,7 +909,7 @@ ============================== -.. _pep-445: +.. _whatsnew-pep-445: PEP 445: Customization of CPython memory allocators --------------------------------------------------- @@ -923,7 +923,7 @@ PEP written and implemented by Victor Stinner. -.. _pep-442: +.. _whatsnew-pep-442: PEP 442: Safe object finalization --------------------------------- @@ -943,6 +943,25 @@ PEP written and implemented by Antoine Pitrou. +.. _whatsnew-pep-456: + +PEP 456: Secure and Interchangeable Hash Algorithm +-------------------------------------------------- + +:pep:`456` follows up on earlier security fix work done on Python's hash +algorithm to address certain DOS attacks to which public facing APIs backed by +dictionary lookups may be subject. (See :issue:`14621` for the start of the +current round of improvements.) The PEP unifies CPython's hash code to make it +easier for a packager to substitute a different hash algorithm, and switches +Python's default implementation to a SipHash implementation on platforms that +have a 64 bit data type. Any performance differences in comparison with the +older FNV algorithm are trivial. + +The PEP adds additional fields to the :func:`sys.hash_info` struct sequence to +describe the hash algorithm in use by the currently executing binary. Otherwise, +the PEP does not alter any existing CPython APIs. + + Other build and C API changes ----------------------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 21 22:19:21 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 21 Dec 2013 22:19:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE4ODc5?= =?utf-8?q?=3A_When_a_method_is_looked_up_on_a_temporary_file=2C_avoid_clo?= =?utf-8?q?sing_the?= Message-ID: <3dn07s1hVTz7LjT@mail.python.org> http://hg.python.org/cpython/rev/f3b7a76fb778 changeset: 88113:f3b7a76fb778 branch: 3.3 parent: 88087:49dd8e1ef4c3 user: Antoine Pitrou date: Sat Dec 21 22:14:56 2013 +0100 summary: Issue #18879: When a method is looked up on a temporary file, avoid closing the file before the method is possibly called. files: Lib/tempfile.py | 102 +++++++++++++++++-------- Lib/test/test_tempfile.py | 17 ++++ Misc/NEWS | 3 + 3 files changed, 87 insertions(+), 35 deletions(-) diff --git a/Lib/tempfile.py b/Lib/tempfile.py --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -27,6 +27,7 @@ # Imports. +import functools as _functools import warnings as _warnings import sys as _sys import io as _io @@ -349,13 +350,10 @@ "No usable temporary filename found") -class _TemporaryFileWrapper: - """Temporary file wrapper - - This class provides a wrapper around files opened for - temporary use. In particular, it seeks to automatically - remove the file when it is no longer needed. - """ +class _TemporaryFileCloser: + """A separate object allowing proper closing of a temporary file's + underlying file object, without adding a __del__ method to the + temporary file.""" def __init__(self, file, name, delete=True): self.file = file @@ -363,26 +361,6 @@ self.close_called = False self.delete = delete - def __getattr__(self, name): - # Attribute lookups are delegated to the underlying file - # and cached for non-numeric results - # (i.e. methods are cached, closed and friends are not) - file = self.__dict__['file'] - a = getattr(file, name) - if not isinstance(a, int): - setattr(self, name, a) - return a - - # The underlying __enter__ method returns the wrong object - # (self.file) so override it to return the wrapper - def __enter__(self): - self.file.__enter__() - return self - - # iter() doesn't use __getattr__ to find the __iter__ method - def __iter__(self): - return iter(self.file) - # NT provides delete-on-close as a primitive, so we don't need # the wrapper to do anything special. We still use it so that # file.name is useful (i.e. not "(fdopen)") with NamedTemporaryFile. @@ -401,18 +379,72 @@ if self.delete: self.unlink(self.name) + # Need to ensure the file is deleted on __del__ def __del__(self): self.close() - # Need to trap __exit__ as well to ensure the file gets - # deleted when used in a with statement - def __exit__(self, exc, value, tb): - result = self.file.__exit__(exc, value, tb) - self.close() - return result else: - def __exit__(self, exc, value, tb): - self.file.__exit__(exc, value, tb) + def close(self): + if not self.close_called: + self.close_called = True + self.file.close() + + +class _TemporaryFileWrapper: + """Temporary file wrapper + + This class provides a wrapper around files opened for + temporary use. In particular, it seeks to automatically + remove the file when it is no longer needed. + """ + + def __init__(self, file, name, delete=True): + self.file = file + self.name = name + self.delete = delete + self._closer = _TemporaryFileCloser(file, name, delete) + + def __getattr__(self, name): + # Attribute lookups are delegated to the underlying file + # and cached for non-numeric results + # (i.e. methods are cached, closed and friends are not) + file = self.__dict__['file'] + a = getattr(file, name) + if hasattr(a, '__call__'): + func = a + @_functools.wraps(func) + def func_wrapper(*args, **kwargs): + return func(*args, **kwargs) + # Avoid closing the file as long as the wrapper is alive, + # see issue #18879. + func_wrapper._closer = self._closer + a = func_wrapper + if not isinstance(a, int): + setattr(self, name, a) + return a + + # The underlying __enter__ method returns the wrong object + # (self.file) so override it to return the wrapper + def __enter__(self): + self.file.__enter__() + return self + + # Need to trap __exit__ as well to ensure the file gets + # deleted when used in a with statement + def __exit__(self, exc, value, tb): + result = self.file.__exit__(exc, value, tb) + self.close() + return result + + def close(self): + """ + Close the temporary file, possibly deleting it. + """ + self._closer.close() + + # iter() doesn't use __getattr__ to find the __iter__ method + def __iter__(self): + return iter(self.file) def NamedTemporaryFile(mode='w+b', buffering=-1, encoding=None, diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -8,6 +8,7 @@ import re import warnings import contextlib +import weakref import unittest from test import support @@ -674,6 +675,22 @@ self.do_create(pre="a", suf="b") self.do_create(pre="aa", suf=".txt") + def test_method_lookup(self): + # Issue #18879: Looking up a temporary file method should keep it + # alive long enough. + f = self.do_create() + wr = weakref.ref(f) + write = f.write + write2 = f.write + del f + write(b'foo') + del write + write2(b'bar') + del write2 + if support.check_impl_detail(cpython=True): + # No reference cycle was created. + self.assertIsNone(wr()) + def test_creates_named(self): # NamedTemporaryFile creates files with names f = tempfile.NamedTemporaryFile() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,9 @@ Library ------- +- Issue #18879: When a method is looked up on a temporary file, avoid closing + the file before the method is possibly called. + - Issue #20034: Updated alias mapping to most recent locale.alias file from X.org distribution using makelocalealias.py. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 21 22:19:22 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 21 Dec 2013 22:19:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2318879=3A_When_a_method_is_looked_up_on_a_tempor?= =?utf-8?q?ary_file=2C_avoid_closing_the?= Message-ID: <3dn07t4nh0z7Ljn@mail.python.org> http://hg.python.org/cpython/rev/d68ab2eb7a77 changeset: 88114:d68ab2eb7a77 parent: 88112:ae9b74c2b357 parent: 88113:f3b7a76fb778 user: Antoine Pitrou date: Sat Dec 21 22:16:19 2013 +0100 summary: Issue #18879: When a method is looked up on a temporary file, avoid closing the file before the method is possibly called. files: Lib/tempfile.py | 102 +++++++++++++++++-------- Lib/test/test_tempfile.py | 17 ++++ Misc/NEWS | 3 + 3 files changed, 87 insertions(+), 35 deletions(-) diff --git a/Lib/tempfile.py b/Lib/tempfile.py --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -27,6 +27,7 @@ # Imports. +import functools as _functools import warnings as _warnings import sys as _sys import io as _io @@ -329,13 +330,10 @@ "No usable temporary filename found") -class _TemporaryFileWrapper: - """Temporary file wrapper - - This class provides a wrapper around files opened for - temporary use. In particular, it seeks to automatically - remove the file when it is no longer needed. - """ +class _TemporaryFileCloser: + """A separate object allowing proper closing of a temporary file's + underlying file object, without adding a __del__ method to the + temporary file.""" def __init__(self, file, name, delete=True): self.file = file @@ -343,26 +341,6 @@ self.close_called = False self.delete = delete - def __getattr__(self, name): - # Attribute lookups are delegated to the underlying file - # and cached for non-numeric results - # (i.e. methods are cached, closed and friends are not) - file = self.__dict__['file'] - a = getattr(file, name) - if not isinstance(a, int): - setattr(self, name, a) - return a - - # The underlying __enter__ method returns the wrong object - # (self.file) so override it to return the wrapper - def __enter__(self): - self.file.__enter__() - return self - - # iter() doesn't use __getattr__ to find the __iter__ method - def __iter__(self): - return iter(self.file) - # NT provides delete-on-close as a primitive, so we don't need # the wrapper to do anything special. We still use it so that # file.name is useful (i.e. not "(fdopen)") with NamedTemporaryFile. @@ -381,18 +359,72 @@ if self.delete: self.unlink(self.name) + # Need to ensure the file is deleted on __del__ def __del__(self): self.close() - # Need to trap __exit__ as well to ensure the file gets - # deleted when used in a with statement - def __exit__(self, exc, value, tb): - result = self.file.__exit__(exc, value, tb) - self.close() - return result else: - def __exit__(self, exc, value, tb): - self.file.__exit__(exc, value, tb) + def close(self): + if not self.close_called: + self.close_called = True + self.file.close() + + +class _TemporaryFileWrapper: + """Temporary file wrapper + + This class provides a wrapper around files opened for + temporary use. In particular, it seeks to automatically + remove the file when it is no longer needed. + """ + + def __init__(self, file, name, delete=True): + self.file = file + self.name = name + self.delete = delete + self._closer = _TemporaryFileCloser(file, name, delete) + + def __getattr__(self, name): + # Attribute lookups are delegated to the underlying file + # and cached for non-numeric results + # (i.e. methods are cached, closed and friends are not) + file = self.__dict__['file'] + a = getattr(file, name) + if hasattr(a, '__call__'): + func = a + @_functools.wraps(func) + def func_wrapper(*args, **kwargs): + return func(*args, **kwargs) + # Avoid closing the file as long as the wrapper is alive, + # see issue #18879. + func_wrapper._closer = self._closer + a = func_wrapper + if not isinstance(a, int): + setattr(self, name, a) + return a + + # The underlying __enter__ method returns the wrong object + # (self.file) so override it to return the wrapper + def __enter__(self): + self.file.__enter__() + return self + + # Need to trap __exit__ as well to ensure the file gets + # deleted when used in a with statement + def __exit__(self, exc, value, tb): + result = self.file.__exit__(exc, value, tb) + self.close() + return result + + def close(self): + """ + Close the temporary file, possibly deleting it. + """ + self._closer.close() + + # iter() doesn't use __getattr__ to find the __iter__ method + def __iter__(self): + return iter(self.file) def NamedTemporaryFile(mode='w+b', buffering=-1, encoding=None, diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -8,6 +8,7 @@ import re import warnings import contextlib +import weakref import unittest from test import support @@ -689,6 +690,22 @@ self.do_create(pre="a", suf="b") self.do_create(pre="aa", suf=".txt") + def test_method_lookup(self): + # Issue #18879: Looking up a temporary file method should keep it + # alive long enough. + f = self.do_create() + wr = weakref.ref(f) + write = f.write + write2 = f.write + del f + write(b'foo') + del write + write2(b'bar') + del write2 + if support.check_impl_detail(cpython=True): + # No reference cycle was created. + self.assertIsNone(wr()) + def test_creates_named(self): # NamedTemporaryFile creates files with names f = tempfile.NamedTemporaryFile() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #18879: When a method is looked up on a temporary file, avoid closing + the file before the method is possibly called. + - Issue #20037: Avoid crashes when opening a text file late at interpreter shutdown. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 21 22:20:36 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 21 Dec 2013 22:20:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Fix_Deprecatio?= =?utf-8?q?nWarnings_in_test_suite?= Message-ID: <3dn09J2GLJz7LkZ@mail.python.org> http://hg.python.org/cpython/rev/282043a0ab22 changeset: 88115:282043a0ab22 branch: 3.3 parent: 88113:f3b7a76fb778 user: Antoine Pitrou date: Sat Dec 21 22:19:46 2013 +0100 summary: Fix DeprecationWarnings in test suite files: Lib/distutils/tests/test_dist.py | 16 ++++++++-------- Lib/distutils/tests/test_upload.py | 10 +++++----- Lib/test/test_shutil.py | 14 +++++++------- Lib/test/test_strftime.py | 6 +++--- 4 files changed, 23 insertions(+), 23 deletions(-) diff --git a/Lib/distutils/tests/test_dist.py b/Lib/distutils/tests/test_dist.py --- a/Lib/distutils/tests/test_dist.py +++ b/Lib/distutils/tests/test_dist.py @@ -406,14 +406,14 @@ PKG_INFO.seek(0) metadata.read_pkg_file(PKG_INFO) - self.assertEquals(metadata.name, "package") - self.assertEquals(metadata.version, "1.0") - self.assertEquals(metadata.description, "xxx") - self.assertEquals(metadata.download_url, 'http://example.com') - self.assertEquals(metadata.keywords, ['one', 'two']) - self.assertEquals(metadata.platforms, ['UNKNOWN']) - self.assertEquals(metadata.obsoletes, None) - self.assertEquals(metadata.requires, ['foo']) + self.assertEqual(metadata.name, "package") + self.assertEqual(metadata.version, "1.0") + self.assertEqual(metadata.description, "xxx") + self.assertEqual(metadata.download_url, 'http://example.com') + self.assertEqual(metadata.keywords, ['one', 'two']) + self.assertEqual(metadata.platforms, ['UNKNOWN']) + self.assertEqual(metadata.obsoletes, None) + self.assertEqual(metadata.requires, ['foo']) def test_suite(): suite = unittest.TestSuite() diff --git a/Lib/distutils/tests/test_upload.py b/Lib/distutils/tests/test_upload.py --- a/Lib/distutils/tests/test_upload.py +++ b/Lib/distutils/tests/test_upload.py @@ -114,11 +114,11 @@ # what did we send ? headers = dict(self.last_open.req.headers) self.assertEqual(headers['Content-length'], '2087') - self.assert_(headers['Content-type'].startswith('multipart/form-data')) - self.assertEquals(self.last_open.req.get_method(), 'POST') - self.assertEquals(self.last_open.req.get_full_url(), - 'http://pypi.python.org/pypi') - self.assert_(b'xxx' in self.last_open.req.data) + self.assertTrue(headers['Content-type'].startswith('multipart/form-data')) + self.assertEqual(self.last_open.req.get_method(), 'POST') + self.assertEqual(self.last_open.req.get_full_url(), + 'http://pypi.python.org/pypi') + self.assertIn(b'xxx', self.last_open.req.data) def test_suite(): return unittest.makeSuite(uploadTestCase) diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -742,15 +742,15 @@ os.chmod(restrictive_subdir, 0o600) shutil.copytree(src_dir, dst_dir) - self.assertEquals(os.stat(src_dir).st_mode, os.stat(dst_dir).st_mode) - self.assertEquals(os.stat(os.path.join(src_dir, 'permissive.txt')).st_mode, - os.stat(os.path.join(dst_dir, 'permissive.txt')).st_mode) - self.assertEquals(os.stat(os.path.join(src_dir, 'restrictive.txt')).st_mode, - os.stat(os.path.join(dst_dir, 'restrictive.txt')).st_mode) + self.assertEqual(os.stat(src_dir).st_mode, os.stat(dst_dir).st_mode) + self.assertEqual(os.stat(os.path.join(src_dir, 'permissive.txt')).st_mode, + os.stat(os.path.join(dst_dir, 'permissive.txt')).st_mode) + self.assertEqual(os.stat(os.path.join(src_dir, 'restrictive.txt')).st_mode, + os.stat(os.path.join(dst_dir, 'restrictive.txt')).st_mode) restrictive_subdir_dst = os.path.join(dst_dir, os.path.split(restrictive_subdir)[1]) - self.assertEquals(os.stat(restrictive_subdir).st_mode, - os.stat(restrictive_subdir_dst).st_mode) + self.assertEqual(os.stat(restrictive_subdir).st_mode, + os.stat(restrictive_subdir_dst).st_mode) @unittest.skipIf(os.name == 'nt', 'temporarily disabled on Windows') @unittest.skipUnless(hasattr(os, 'link'), 'requires os.link') diff --git a/Lib/test/test_strftime.py b/Lib/test/test_strftime.py --- a/Lib/test/test_strftime.py +++ b/Lib/test/test_strftime.py @@ -189,15 +189,15 @@ @unittest.skipIf(sys.platform == "win32", "Doesn't apply on Windows") def test_y_before_1900_nonwin(self): - self.assertEquals( + self.assertEqual( time.strftime("%y", (1899, 1, 1, 0, 0, 0, 0, 0, 0)), "99") def test_y_1900(self): - self.assertEquals( + self.assertEqual( time.strftime("%y", (1900, 1, 1, 0, 0, 0, 0, 0, 0)), "00") def test_y_after_1900(self): - self.assertEquals( + self.assertEqual( time.strftime("%y", (2013, 1, 1, 0, 0, 0, 0, 0, 0)), "13") def test_main(): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 21 22:20:37 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 21 Dec 2013 22:20:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Null_merge?= Message-ID: <3dn09K3wqJz7LkH@mail.python.org> http://hg.python.org/cpython/rev/eae6966d9734 changeset: 88116:eae6966d9734 parent: 88114:d68ab2eb7a77 parent: 88115:282043a0ab22 user: Antoine Pitrou date: Sat Dec 21 22:20:29 2013 +0100 summary: Null merge files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 21 22:52:32 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 21 Dec 2013 22:52:32 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIwMDQ4?= =?utf-8?q?=3A_Fixed_ZipExtFile=2Epeek=28=29_when_it_is_called_on_the_boun?= =?utf-8?q?dary_of?= Message-ID: <3dn0t80LYwz7Ljb@mail.python.org> http://hg.python.org/cpython/rev/8b097d07488d changeset: 88117:8b097d07488d branch: 2.7 parent: 88084:81f8375e60ce user: Serhiy Storchaka date: Sat Dec 21 23:51:15 2013 +0200 summary: Issue #20048: Fixed ZipExtFile.peek() when it is called on the boundary of the uncompress buffer and read() goes through more than one readbuffer. This is partial backport of changeset 028e8e0b03e8. files: Lib/zipfile.py | 6 +++++- Misc/NEWS | 3 +++ 2 files changed, 8 insertions(+), 1 deletions(-) diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -606,7 +606,11 @@ """Returns buffered bytes without advancing the position.""" if n > len(self._readbuffer) - self._offset: chunk = self.read(n) - self._offset -= len(chunk) + if len(chunk) > self._offset: + self._readbuffer = chunk + self._readbuffer[self._offset:] + self._offset = 0 + else: + self._offset -= len(chunk) # Return up to 512 bytes to reduce allocation overhead for tight loops. return self._readbuffer[self._offset: self._offset + 512] diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -27,6 +27,9 @@ Library ------- +- Issue #20048: Fixed ZipExtFile.peek() when it is called on the boundary of + the uncompress buffer and read() goes through more than one readbuffer. + - Issue #20034: Updated alias mapping to most recent locale.alias file from X.org distribution using makelocalealias.py. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 21 22:59:14 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 21 Dec 2013 22:59:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzIwMDQ1?= =?utf-8?q?=3A_Fix_=22setup=2Epy_register_--list-classifiers=22=2E?= Message-ID: <3dn11t70p6z7Ljb@mail.python.org> http://hg.python.org/cpython/rev/cffed58b1bbd changeset: 88118:cffed58b1bbd branch: 3.3 parent: 88115:282043a0ab22 user: Antoine Pitrou date: Sat Dec 21 22:57:56 2013 +0100 summary: Issue #20045: Fix "setup.py register --list-classifiers". files: Lib/distutils/command/register.py | 5 +++- Lib/distutils/tests/support.py | 7 +++-- Lib/distutils/tests/test_register.py | 19 ++++++++++++++- Misc/NEWS | 2 + 4 files changed, 27 insertions(+), 6 deletions(-) diff --git a/Lib/distutils/command/register.py b/Lib/distutils/command/register.py --- a/Lib/distutils/command/register.py +++ b/Lib/distutils/command/register.py @@ -5,6 +5,7 @@ # created 2002/10/21, Richard Jones +import cgi import os, string, getpass import io import urllib.parse, urllib.request @@ -87,7 +88,9 @@ ''' url = self.repository+'?:action=list_classifiers' response = urllib.request.urlopen(url) - log.info(response.read()) + content_type = response.getheader('content-type', 'text/plain') + encoding = cgi.parse_header(content_type)[1].get('charset', 'ascii') + log.info(response.read().decode(encoding)) def verify_metadata(self): ''' Send the metadata to the package index server to be checked. diff --git a/Lib/distutils/tests/support.py b/Lib/distutils/tests/support.py --- a/Lib/distutils/tests/support.py +++ b/Lib/distutils/tests/support.py @@ -32,14 +32,15 @@ def _log(self, level, msg, args): if level not in (DEBUG, INFO, WARN, ERROR, FATAL): raise ValueError('%s wrong log level' % str(level)) + if not isinstance(msg, str): + raise TypeError("msg should be str, not '%.200s'" + % (type(msg).__name__)) self.logs.append((level, msg, args)) def get_logs(self, *levels): def _format(msg, args): - if len(args) == 0: - return msg return msg % args - return [_format(msg, args) for level, msg, args + return [msg % args for level, msg, args in self.logs if level in levels] def clear_logs(self): diff --git a/Lib/distutils/tests/test_register.py b/Lib/distutils/tests/test_register.py --- a/Lib/distutils/tests/test_register.py +++ b/Lib/distutils/tests/test_register.py @@ -10,6 +10,7 @@ from distutils.command import register as register_module from distutils.command.register import register from distutils.errors import DistutilsSetupError +from distutils.log import INFO from distutils.tests.test_config import PyPIRCCommandTestCase @@ -58,12 +59,18 @@ def __call__(self, *args): return self - def open(self, req): + def open(self, req, data=None, timeout=None): self.reqs.append(req) return self def read(self): - return 'xxx' + return b'xxx' + + def getheader(self, name, default=None): + return { + 'content-type': 'text/plain; charset=utf-8', + }.get(name.lower(), default) + class RegisterTestCase(PyPIRCCommandTestCase): @@ -285,6 +292,14 @@ cmd.check_metadata() self.assertEqual(len(w.warnings), 1) + def test_list_classifiers(self): + cmd = self._get_cmd() + cmd.list_classifiers = 1 + cmd.run() + results = self.get_logs(INFO) + self.assertEqual(results, ['running check', 'xxx']) + + def test_suite(): return unittest.makeSuite(RegisterTestCase) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,8 @@ Library ------- +- Issue #20045: Fix "setup.py register --list-classifiers". + - Issue #18879: When a method is looked up on a temporary file, avoid closing the file before the method is possibly called. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 21 22:59:16 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 21 Dec 2013 22:59:16 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2320045=3A_Fix_=22setup=2Epy_register_--list-clas?= =?utf-8?b?c2lmaWVycyIu?= Message-ID: <3dn11w2t6vz7Ljw@mail.python.org> http://hg.python.org/cpython/rev/597b69d3a74f changeset: 88119:597b69d3a74f parent: 88116:eae6966d9734 parent: 88118:cffed58b1bbd user: Antoine Pitrou date: Sat Dec 21 22:59:06 2013 +0100 summary: Issue #20045: Fix "setup.py register --list-classifiers". files: Lib/distutils/command/register.py | 5 +++- Lib/distutils/tests/support.py | 7 +++-- Lib/distutils/tests/test_register.py | 19 ++++++++++++++- Misc/NEWS | 2 + 4 files changed, 27 insertions(+), 6 deletions(-) diff --git a/Lib/distutils/command/register.py b/Lib/distutils/command/register.py --- a/Lib/distutils/command/register.py +++ b/Lib/distutils/command/register.py @@ -5,6 +5,7 @@ # created 2002/10/21, Richard Jones +import cgi import os, string, getpass import io import urllib.parse, urllib.request @@ -87,7 +88,9 @@ ''' url = self.repository+'?:action=list_classifiers' response = urllib.request.urlopen(url) - log.info(response.read()) + content_type = response.getheader('content-type', 'text/plain') + encoding = cgi.parse_header(content_type)[1].get('charset', 'ascii') + log.info(response.read().decode(encoding)) def verify_metadata(self): ''' Send the metadata to the package index server to be checked. diff --git a/Lib/distutils/tests/support.py b/Lib/distutils/tests/support.py --- a/Lib/distutils/tests/support.py +++ b/Lib/distutils/tests/support.py @@ -32,14 +32,15 @@ def _log(self, level, msg, args): if level not in (DEBUG, INFO, WARN, ERROR, FATAL): raise ValueError('%s wrong log level' % str(level)) + if not isinstance(msg, str): + raise TypeError("msg should be str, not '%.200s'" + % (type(msg).__name__)) self.logs.append((level, msg, args)) def get_logs(self, *levels): def _format(msg, args): - if len(args) == 0: - return msg return msg % args - return [_format(msg, args) for level, msg, args + return [msg % args for level, msg, args in self.logs if level in levels] def clear_logs(self): diff --git a/Lib/distutils/tests/test_register.py b/Lib/distutils/tests/test_register.py --- a/Lib/distutils/tests/test_register.py +++ b/Lib/distutils/tests/test_register.py @@ -10,6 +10,7 @@ from distutils.command import register as register_module from distutils.command.register import register from distutils.errors import DistutilsSetupError +from distutils.log import INFO from distutils.tests.test_config import PyPIRCCommandTestCase @@ -58,12 +59,18 @@ def __call__(self, *args): return self - def open(self, req): + def open(self, req, data=None, timeout=None): self.reqs.append(req) return self def read(self): - return 'xxx' + return b'xxx' + + def getheader(self, name, default=None): + return { + 'content-type': 'text/plain; charset=utf-8', + }.get(name.lower(), default) + class RegisterTestCase(PyPIRCCommandTestCase): @@ -285,6 +292,14 @@ cmd.check_metadata() self.assertEqual(len(w.warnings), 1) + def test_list_classifiers(self): + cmd = self._get_cmd() + cmd.list_classifiers = 1 + cmd.run() + results = self.get_logs(INFO) + self.assertEqual(results, ['running check', 'xxx']) + + def test_suite(): return unittest.makeSuite(RegisterTestCase) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,8 @@ Library ------- +- Issue #20045: Fix "setup.py register --list-classifiers". + - Issue #18879: When a method is looked up on a temporary file, avoid closing the file before the method is possibly called. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 22 00:44:42 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 22 Dec 2013 00:44:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogRml4IHVybGxpYi5y?= =?utf-8?q?equest=2Ebuild=5Fopener_mocking_in_test=5Fdistutils_=28should_f?= =?utf-8?q?ix_some?= Message-ID: <3dn3MZ0fHKz7Ljm@mail.python.org> http://hg.python.org/cpython/rev/927946b62d6b changeset: 88120:927946b62d6b branch: 3.3 parent: 88118:cffed58b1bbd user: Antoine Pitrou date: Sun Dec 22 00:44:01 2013 +0100 summary: Fix urllib.request.build_opener mocking in test_distutils (should fix some random buildbot failures) files: Lib/distutils/tests/test_register.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/distutils/tests/test_register.py b/Lib/distutils/tests/test_register.py --- a/Lib/distutils/tests/test_register.py +++ b/Lib/distutils/tests/test_register.py @@ -81,11 +81,13 @@ def _getpass(prompt): return 'password' getpass.getpass = _getpass + urllib.request._opener = None self.old_opener = urllib.request.build_opener self.conn = urllib.request.build_opener = FakeOpener() def tearDown(self): getpass.getpass = self._old_getpass + urllib.request._opener = None urllib.request.build_opener = self.old_opener super(RegisterTestCase, self).tearDown() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 22 00:44:43 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 22 Dec 2013 00:44:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Fix_urllib=2Erequest=2Ebuild=5Fopener_mocking_in_test=5F?= =?utf-8?q?distutils_=28should_fix_some?= Message-ID: <3dn3Mb2njDz7Ljw@mail.python.org> http://hg.python.org/cpython/rev/c3bbb5597f11 changeset: 88121:c3bbb5597f11 parent: 88119:597b69d3a74f parent: 88120:927946b62d6b user: Antoine Pitrou date: Sun Dec 22 00:44:28 2013 +0100 summary: Fix urllib.request.build_opener mocking in test_distutils (should fix some random buildbot failures) files: Lib/distutils/tests/test_register.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/distutils/tests/test_register.py b/Lib/distutils/tests/test_register.py --- a/Lib/distutils/tests/test_register.py +++ b/Lib/distutils/tests/test_register.py @@ -81,11 +81,13 @@ def _getpass(prompt): return 'password' getpass.getpass = _getpass + urllib.request._opener = None self.old_opener = urllib.request.build_opener self.conn = urllib.request.build_opener = FakeOpener() def tearDown(self): getpass.getpass = self._old_getpass + urllib.request._opener = None urllib.request.build_opener = self.old_opener super(RegisterTestCase, self).tearDown() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 22 01:36:00 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 22 Dec 2013 01:36:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEyMjI2?= =?utf-8?q?=3A_HTTPS_is_now_used_by_default_when_connecting_to_PyPI=2E?= Message-ID: <3dn4Vm6gm7z7Lkr@mail.python.org> http://hg.python.org/cpython/rev/32a39ec6bd75 changeset: 88122:32a39ec6bd75 branch: 2.7 parent: 88117:8b097d07488d user: Antoine Pitrou date: Sun Dec 22 01:35:53 2013 +0100 summary: Issue #12226: HTTPS is now used by default when connecting to PyPI. files: Lib/distutils/config.py | 2 +- Lib/distutils/tests/test_config.py | 4 ++-- Lib/distutils/tests/test_upload.py | 4 ++-- Misc/NEWS | 2 ++ 4 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Lib/distutils/config.py b/Lib/distutils/config.py --- a/Lib/distutils/config.py +++ b/Lib/distutils/config.py @@ -21,7 +21,7 @@ class PyPIRCCommand(Command): """Base command that knows how to handle the .pypirc file """ - DEFAULT_REPOSITORY = 'http://pypi.python.org/pypi' + DEFAULT_REPOSITORY = 'https://pypi.python.org/pypi' DEFAULT_REALM = 'pypi' repository = None realm = None diff --git a/Lib/distutils/tests/test_config.py b/Lib/distutils/tests/test_config.py --- a/Lib/distutils/tests/test_config.py +++ b/Lib/distutils/tests/test_config.py @@ -89,7 +89,7 @@ config = config.items() config.sort() waited = [('password', 'secret'), ('realm', 'pypi'), - ('repository', 'http://pypi.python.org/pypi'), + ('repository', 'https://pypi.python.org/pypi'), ('server', 'server1'), ('username', 'me')] self.assertEqual(config, waited) @@ -99,7 +99,7 @@ config = config.items() config.sort() waited = [('password', 'secret'), ('realm', 'pypi'), - ('repository', 'http://pypi.python.org/pypi'), + ('repository', 'https://pypi.python.org/pypi'), ('server', 'server-login'), ('username', 'tarek')] self.assertEqual(config, waited) diff --git a/Lib/distutils/tests/test_upload.py b/Lib/distutils/tests/test_upload.py --- a/Lib/distutils/tests/test_upload.py +++ b/Lib/distutils/tests/test_upload.py @@ -78,7 +78,7 @@ cmd.finalize_options() for attr, waited in (('username', 'me'), ('password', 'secret'), ('realm', 'pypi'), - ('repository', 'http://pypi.python.org/pypi')): + ('repository', 'https://pypi.python.org/pypi')): self.assertEqual(getattr(cmd, attr), waited) def test_saved_password(self): @@ -119,7 +119,7 @@ self.assertTrue(headers['Content-type'].startswith('multipart/form-data')) self.assertEqual(self.last_open.req.get_method(), 'POST') self.assertEqual(self.last_open.req.get_full_url(), - 'http://pypi.python.org/pypi') + 'https://pypi.python.org/pypi') self.assertIn('xxx', self.last_open.req.data) auth = self.last_open.req.headers['Authorization'] self.assertNotIn('\n', auth) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -27,6 +27,8 @@ Library ------- +- Issue #12226: HTTPS is now used by default when connecting to PyPI. + - Issue #20048: Fixed ZipExtFile.peek() when it is called on the boundary of the uncompress buffer and read() goes through more than one readbuffer. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 22 01:48:18 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 22 Dec 2013 01:48:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyMjI2?= =?utf-8?q?=3A_HTTPS_is_now_used_by_default_when_connecting_to_PyPI=2E?= Message-ID: <3dn4my0rT9z7Lkd@mail.python.org> http://hg.python.org/cpython/rev/2b5cd6d4d149 changeset: 88123:2b5cd6d4d149 branch: 3.2 parent: 86913:8c9769b17171 user: Antoine Pitrou date: Sun Dec 22 01:35:53 2013 +0100 summary: Issue #12226: HTTPS is now used by default when connecting to PyPI. files: Lib/distutils/config.py | 2 +- Lib/distutils/tests/test_config.py | 4 ++-- Lib/distutils/tests/test_upload.py | 8 ++++---- Misc/NEWS | 2 ++ 4 files changed, 9 insertions(+), 7 deletions(-) diff --git a/Lib/distutils/config.py b/Lib/distutils/config.py --- a/Lib/distutils/config.py +++ b/Lib/distutils/config.py @@ -21,7 +21,7 @@ class PyPIRCCommand(Command): """Base command that knows how to handle the .pypirc file """ - DEFAULT_REPOSITORY = 'http://pypi.python.org/pypi' + DEFAULT_REPOSITORY = 'https://pypi.python.org/pypi' DEFAULT_REALM = 'pypi' repository = None realm = None diff --git a/Lib/distutils/tests/test_config.py b/Lib/distutils/tests/test_config.py --- a/Lib/distutils/tests/test_config.py +++ b/Lib/distutils/tests/test_config.py @@ -87,7 +87,7 @@ config = list(sorted(config.items())) waited = [('password', 'secret'), ('realm', 'pypi'), - ('repository', 'http://pypi.python.org/pypi'), + ('repository', 'https://pypi.python.org/pypi'), ('server', 'server1'), ('username', 'me')] self.assertEqual(config, waited) @@ -96,7 +96,7 @@ config = cmd._read_pypirc() config = list(sorted(config.items())) waited = [('password', 'secret'), ('realm', 'pypi'), - ('repository', 'http://pypi.python.org/pypi'), + ('repository', 'https://pypi.python.org/pypi'), ('server', 'server-login'), ('username', 'tarek')] self.assertEqual(config, waited) diff --git a/Lib/distutils/tests/test_upload.py b/Lib/distutils/tests/test_upload.py --- a/Lib/distutils/tests/test_upload.py +++ b/Lib/distutils/tests/test_upload.py @@ -72,11 +72,11 @@ def setUp(self): super(uploadTestCase, self).setUp() - self.old_class = httpclient.HTTPConnection - self.conn = httpclient.HTTPConnection = FakeConnection() + self.old_class = httpclient.HTTPSConnection + self.conn = httpclient.HTTPSConnection = FakeConnection() def tearDown(self): - httpclient.HTTPConnection = self.old_class + httpclient.HTTPSConnection = self.old_class super(uploadTestCase, self).tearDown() def test_finalize_options(self): @@ -88,7 +88,7 @@ cmd.finalize_options() for attr, waited in (('username', 'me'), ('password', 'secret'), ('realm', 'pypi'), - ('repository', 'http://pypi.python.org/pypi')): + ('repository', 'https://pypi.python.org/pypi')): self.assertEqual(getattr(cmd, attr), waited) def test_saved_password(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,8 @@ Library ------- +- Issue #12226: HTTPS is now used by default when connecting to PyPI. + - Issue #19435: Fix directory traversal attack on CGIHttpRequestHandler. - Issue #14984: On POSIX systems, when netrc is called without a filename -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 22 01:48:19 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 22 Dec 2013 01:48:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMyk6?= =?utf-8?q?_Issue_=2312226=3A_HTTPS_is_now_used_by_default_when_connecting?= =?utf-8?q?_to_PyPI=2E?= Message-ID: <3dn4mz33nZz7Lkd@mail.python.org> http://hg.python.org/cpython/rev/e5a9755c967c changeset: 88124:e5a9755c967c branch: 3.3 parent: 88120:927946b62d6b parent: 88123:2b5cd6d4d149 user: Antoine Pitrou date: Sun Dec 22 01:45:42 2013 +0100 summary: Issue #12226: HTTPS is now used by default when connecting to PyPI. files: Lib/distutils/config.py | 2 +- Lib/distutils/tests/test_config.py | 4 ++-- Lib/distutils/tests/test_upload.py | 4 ++-- Misc/NEWS | 2 ++ 4 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Lib/distutils/config.py b/Lib/distutils/config.py --- a/Lib/distutils/config.py +++ b/Lib/distutils/config.py @@ -21,7 +21,7 @@ class PyPIRCCommand(Command): """Base command that knows how to handle the .pypirc file """ - DEFAULT_REPOSITORY = 'http://pypi.python.org/pypi' + DEFAULT_REPOSITORY = 'https://pypi.python.org/pypi' DEFAULT_REALM = 'pypi' repository = None realm = None diff --git a/Lib/distutils/tests/test_config.py b/Lib/distutils/tests/test_config.py --- a/Lib/distutils/tests/test_config.py +++ b/Lib/distutils/tests/test_config.py @@ -87,7 +87,7 @@ config = list(sorted(config.items())) waited = [('password', 'secret'), ('realm', 'pypi'), - ('repository', 'http://pypi.python.org/pypi'), + ('repository', 'https://pypi.python.org/pypi'), ('server', 'server1'), ('username', 'me')] self.assertEqual(config, waited) @@ -96,7 +96,7 @@ config = cmd._read_pypirc() config = list(sorted(config.items())) waited = [('password', 'secret'), ('realm', 'pypi'), - ('repository', 'http://pypi.python.org/pypi'), + ('repository', 'https://pypi.python.org/pypi'), ('server', 'server-login'), ('username', 'tarek')] self.assertEqual(config, waited) diff --git a/Lib/distutils/tests/test_upload.py b/Lib/distutils/tests/test_upload.py --- a/Lib/distutils/tests/test_upload.py +++ b/Lib/distutils/tests/test_upload.py @@ -77,7 +77,7 @@ cmd.finalize_options() for attr, waited in (('username', 'me'), ('password', 'secret'), ('realm', 'pypi'), - ('repository', 'http://pypi.python.org/pypi')): + ('repository', 'https://pypi.python.org/pypi')): self.assertEqual(getattr(cmd, attr), waited) def test_saved_password(self): @@ -117,7 +117,7 @@ self.assertTrue(headers['Content-type'].startswith('multipart/form-data')) self.assertEqual(self.last_open.req.get_method(), 'POST') self.assertEqual(self.last_open.req.get_full_url(), - 'http://pypi.python.org/pypi') + 'https://pypi.python.org/pypi') self.assertIn(b'xxx', self.last_open.req.data) def test_suite(): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,8 @@ Library ------- +- Issue #12226: HTTPS is now used by default when connecting to PyPI. + - Issue #20045: Fix "setup.py register --list-classifiers". - Issue #18879: When a method is looked up on a temporary file, avoid closing -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 22 01:48:20 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 22 Dec 2013 01:48:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Null_merge_=28=2312226_already_fixed_on_default=29?= Message-ID: <3dn4n05DxZz7LkQ@mail.python.org> http://hg.python.org/cpython/rev/9839aa0e5b28 changeset: 88125:9839aa0e5b28 parent: 88121:c3bbb5597f11 parent: 88124:e5a9755c967c user: Antoine Pitrou date: Sun Dec 22 01:48:06 2013 +0100 summary: Null merge (#12226 already fixed on default) files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 22 01:57:59 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 22 Dec 2013 01:57:59 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_s/lightweight/?= =?utf-8?q?minimal/=2C_as_per_issue_=2311379=2E?= Message-ID: <3dn5070YVLz7Ljv@mail.python.org> http://hg.python.org/cpython/rev/39ea24aaf0e7 changeset: 88126:39ea24aaf0e7 branch: 2.7 parent: 88122:32a39ec6bd75 user: Antoine Pitrou date: Sun Dec 22 01:57:01 2013 +0100 summary: s/lightweight/minimal/, as per issue #11379. files: Doc/library/xml.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/xml.rst b/Doc/library/xml.rst --- a/Doc/library/xml.rst +++ b/Doc/library/xml.rst @@ -32,7 +32,7 @@ .. * :mod:`xml.dom`: the DOM API definition -* :mod:`xml.dom.minidom`: a lightweight DOM implementation +* :mod:`xml.dom.minidom`: a minimal DOM implementation * :mod:`xml.dom.pulldom`: support for building partial DOM trees .. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 22 01:58:00 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 22 Dec 2013 01:58:00 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_s/lightweight/?= =?utf-8?q?minimal/=2C_as_per_issue_=2311379=2E?= Message-ID: <3dn5082gjkz7LkD@mail.python.org> http://hg.python.org/cpython/rev/b63258b6eb4d changeset: 88127:b63258b6eb4d branch: 3.3 parent: 88124:e5a9755c967c user: Antoine Pitrou date: Sun Dec 22 01:57:01 2013 +0100 summary: s/lightweight/minimal/, as per issue #11379. files: Doc/library/xml.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/xml.rst b/Doc/library/xml.rst --- a/Doc/library/xml.rst +++ b/Doc/library/xml.rst @@ -33,7 +33,7 @@ .. * :mod:`xml.dom`: the DOM API definition -* :mod:`xml.dom.minidom`: a lightweight DOM implementation +* :mod:`xml.dom.minidom`: a minimal DOM implementation * :mod:`xml.dom.pulldom`: support for building partial DOM trees .. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 22 01:58:01 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 22 Dec 2013 01:58:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_s/lightweight/minimal/=2C_as_per_issue_=2311379=2E?= Message-ID: <3dn5094mxjz7Ll6@mail.python.org> http://hg.python.org/cpython/rev/d659e7761d59 changeset: 88128:d659e7761d59 parent: 88125:9839aa0e5b28 parent: 88127:b63258b6eb4d user: Antoine Pitrou date: Sun Dec 22 01:57:45 2013 +0100 summary: s/lightweight/minimal/, as per issue #11379. files: Doc/library/xml.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/xml.rst b/Doc/library/xml.rst --- a/Doc/library/xml.rst +++ b/Doc/library/xml.rst @@ -33,7 +33,7 @@ .. * :mod:`xml.dom`: the DOM API definition -* :mod:`xml.dom.minidom`: a lightweight DOM implementation +* :mod:`xml.dom.minidom`: a minimal DOM implementation * :mod:`xml.dom.pulldom`: support for building partial DOM trees .. -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sun Dec 22 09:48:11 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 22 Dec 2013 09:48:11 +0100 Subject: [Python-checkins] Daily reference leaks (d659e7761d59): sum=2989 Message-ID: results for d659e7761d59 on branch "default" -------------------------------------------- test_site leaked [-2, 0, 0] references, sum=-2 test_site leaked [-2, 0, 0] memory blocks, sum=-2 test_urllib2net leaked [0, 0, 1587] references, sum=1587 test_urllib2net leaked [0, 0, 1406] memory blocks, sum=1406 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogVWQy57', '-x'] From python-checkins at python.org Sun Dec 22 18:15:03 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 22 Dec 2013 18:15:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Fix_TypeError_?= =?utf-8?q?on_=22setup=2Epy_upload_--show-response=22=2E?= Message-ID: <3dnVgW3ZdLz7LjQ@mail.python.org> http://hg.python.org/cpython/rev/bb42e10a5620 changeset: 88129:bb42e10a5620 branch: 3.3 parent: 88127:b63258b6eb4d user: Antoine Pitrou date: Sun Dec 22 18:13:51 2013 +0100 summary: Fix TypeError on "setup.py upload --show-response". files: Lib/distutils/command/register.py | 5 +---- Lib/distutils/command/upload.py | 3 ++- Lib/distutils/config.py | 7 +++++++ Lib/distutils/tests/test_upload.py | 17 ++++++++++++++++- Misc/NEWS | 2 ++ 5 files changed, 28 insertions(+), 6 deletions(-) diff --git a/Lib/distutils/command/register.py b/Lib/distutils/command/register.py --- a/Lib/distutils/command/register.py +++ b/Lib/distutils/command/register.py @@ -5,7 +5,6 @@ # created 2002/10/21, Richard Jones -import cgi import os, string, getpass import io import urllib.parse, urllib.request @@ -88,9 +87,7 @@ ''' url = self.repository+'?:action=list_classifiers' response = urllib.request.urlopen(url) - content_type = response.getheader('content-type', 'text/plain') - encoding = cgi.parse_header(content_type)[1].get('charset', 'ascii') - log.info(response.read().decode(encoding)) + log.info(self._read_pypi_response(response)) def verify_metadata(self): ''' Send the metadata to the package index server to be checked. diff --git a/Lib/distutils/command/upload.py b/Lib/distutils/command/upload.py --- a/Lib/distutils/command/upload.py +++ b/Lib/distutils/command/upload.py @@ -196,5 +196,6 @@ self.announce('Upload failed (%s): %s' % (status, reason), log.ERROR) if self.show_response: - msg = '\n'.join(('-' * 75, result.read(), '-' * 75)) + text = self._read_pypi_response(result) + msg = '\n'.join(('-' * 75, text, '-' * 75)) self.announce(msg, log.INFO) diff --git a/Lib/distutils/config.py b/Lib/distutils/config.py --- a/Lib/distutils/config.py +++ b/Lib/distutils/config.py @@ -3,6 +3,7 @@ Provides the PyPIRCCommand class, the base class for the command classes that uses .pypirc in the distutils.command package. """ +import cgi import os from configparser import ConfigParser @@ -101,6 +102,12 @@ return {} + def _read_pypi_response(self, response): + """Read and decode a PyPI HTTP response.""" + content_type = response.getheader('content-type', 'text/plain') + encoding = cgi.parse_header(content_type)[1].get('charset', 'ascii') + return response.read().decode(encoding) + def initialize_options(self): """Initialize options.""" self.repository = None diff --git a/Lib/distutils/tests/test_upload.py b/Lib/distutils/tests/test_upload.py --- a/Lib/distutils/tests/test_upload.py +++ b/Lib/distutils/tests/test_upload.py @@ -6,6 +6,7 @@ from distutils.command import upload as upload_mod from distutils.command.upload import upload from distutils.core import Distribution +from distutils.log import INFO from distutils.tests.test_config import PYPIRC, PyPIRCCommandTestCase @@ -48,6 +49,14 @@ self.req = None self.msg = 'OK' + def getheader(self, name, default=None): + return { + 'content-type': 'text/plain; charset=utf-8', + }.get(name.lower(), default) + + def read(self): + return b'xyzzy' + def getcode(self): return 200 @@ -108,10 +117,11 @@ # lets run it pkg_dir, dist = self.create_dist(dist_files=dist_files) cmd = upload(dist) + cmd.show_response = 1 cmd.ensure_finalized() cmd.run() - # what did we send ? + # what did we send ? headers = dict(self.last_open.req.headers) self.assertEqual(headers['Content-length'], '2087') self.assertTrue(headers['Content-type'].startswith('multipart/form-data')) @@ -120,6 +130,11 @@ 'https://pypi.python.org/pypi') self.assertIn(b'xxx', self.last_open.req.data) + # The PyPI response body was echoed + results = self.get_logs(INFO) + self.assertIn('xyzzy\n', results[-1]) + + def test_suite(): return unittest.makeSuite(uploadTestCase) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,8 @@ Library ------- +- Fix TypeError on "setup.py upload --show-response". + - Issue #12226: HTTPS is now used by default when connecting to PyPI. - Issue #20045: Fix "setup.py register --list-classifiers". -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 22 18:15:06 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 22 Dec 2013 18:15:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Fix_TypeError_on_=22setup=2Epy_upload_--show-response=22?= =?utf-8?q?=2E?= Message-ID: <3dnVgZ6MCMz7LkR@mail.python.org> http://hg.python.org/cpython/rev/d69ba246a3e4 changeset: 88130:d69ba246a3e4 parent: 88128:d659e7761d59 parent: 88129:bb42e10a5620 user: Antoine Pitrou date: Sun Dec 22 18:14:56 2013 +0100 summary: Fix TypeError on "setup.py upload --show-response". files: Lib/distutils/command/register.py | 5 +---- Lib/distutils/command/upload.py | 3 ++- Lib/distutils/config.py | 7 +++++++ Lib/distutils/tests/test_upload.py | 17 ++++++++++++++++- Misc/NEWS | 2 ++ 5 files changed, 28 insertions(+), 6 deletions(-) diff --git a/Lib/distutils/command/register.py b/Lib/distutils/command/register.py --- a/Lib/distutils/command/register.py +++ b/Lib/distutils/command/register.py @@ -5,7 +5,6 @@ # created 2002/10/21, Richard Jones -import cgi import os, string, getpass import io import urllib.parse, urllib.request @@ -88,9 +87,7 @@ ''' url = self.repository+'?:action=list_classifiers' response = urllib.request.urlopen(url) - content_type = response.getheader('content-type', 'text/plain') - encoding = cgi.parse_header(content_type)[1].get('charset', 'ascii') - log.info(response.read().decode(encoding)) + log.info(self._read_pypi_response(response)) def verify_metadata(self): ''' Send the metadata to the package index server to be checked. diff --git a/Lib/distutils/command/upload.py b/Lib/distutils/command/upload.py --- a/Lib/distutils/command/upload.py +++ b/Lib/distutils/command/upload.py @@ -196,5 +196,6 @@ self.announce('Upload failed (%s): %s' % (status, reason), log.ERROR) if self.show_response: - msg = '\n'.join(('-' * 75, result.read(), '-' * 75)) + text = self._read_pypi_response(result) + msg = '\n'.join(('-' * 75, text, '-' * 75)) self.announce(msg, log.INFO) diff --git a/Lib/distutils/config.py b/Lib/distutils/config.py --- a/Lib/distutils/config.py +++ b/Lib/distutils/config.py @@ -3,6 +3,7 @@ Provides the PyPIRCCommand class, the base class for the command classes that uses .pypirc in the distutils.command package. """ +import cgi import os from configparser import ConfigParser @@ -110,6 +111,12 @@ return {} + def _read_pypi_response(self, response): + """Read and decode a PyPI HTTP response.""" + content_type = response.getheader('content-type', 'text/plain') + encoding = cgi.parse_header(content_type)[1].get('charset', 'ascii') + return response.read().decode(encoding) + def initialize_options(self): """Initialize options.""" self.repository = None diff --git a/Lib/distutils/tests/test_upload.py b/Lib/distutils/tests/test_upload.py --- a/Lib/distutils/tests/test_upload.py +++ b/Lib/distutils/tests/test_upload.py @@ -6,6 +6,7 @@ from distutils.command import upload as upload_mod from distutils.command.upload import upload from distutils.core import Distribution +from distutils.log import INFO from distutils.tests.test_config import PYPIRC, PyPIRCCommandTestCase @@ -48,6 +49,14 @@ self.req = None self.msg = 'OK' + def getheader(self, name, default=None): + return { + 'content-type': 'text/plain; charset=utf-8', + }.get(name.lower(), default) + + def read(self): + return b'xyzzy' + def getcode(self): return 200 @@ -108,10 +117,11 @@ # lets run it pkg_dir, dist = self.create_dist(dist_files=dist_files) cmd = upload(dist) + cmd.show_response = 1 cmd.ensure_finalized() cmd.run() - # what did we send ? + # what did we send ? headers = dict(self.last_open.req.headers) self.assertEqual(headers['Content-length'], '2087') content_type = headers['Content-type'] @@ -121,6 +131,11 @@ self.assertEqual(self.last_open.req.get_full_url(), expected_url) self.assertTrue(b'xxx' in self.last_open.req.data) + # The PyPI response body was echoed + results = self.get_logs(INFO) + self.assertIn('xyzzy\n', results[-1]) + + def test_suite(): return unittest.makeSuite(uploadTestCase) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,8 @@ Library ------- +- Fix TypeError on "setup.py upload --show-response". + - Issue #20045: Fix "setup.py register --list-classifiers". - Issue #18879: When a method is looked up on a temporary file, avoid closing -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 22 18:24:56 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 22 Dec 2013 18:24:56 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?devguide=3A_Remove_assignment_to_inac?= =?utf-8?q?tive_maintainers?= Message-ID: <3dnVtw0SY2z7LjQ@mail.python.org> http://hg.python.org/devguide/rev/831cab51864a changeset: 658:831cab51864a user: Antoine Pitrou date: Sun Dec 22 18:24:51 2013 +0100 summary: Remove assignment to inactive maintainers files: experts.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/experts.rst b/experts.rst --- a/experts.rst +++ b/experts.rst @@ -96,7 +96,7 @@ decimal facundobatista, rhettinger, mark.dickinson difflib tim.peters (inactive) dis ncoghlan* -distutils tarek*, eric.araujo* +distutils tarek, eric.araujo doctest tim.peters (inactive) dummy_threading brett.cannon email barry, r.david.murray* -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Sun Dec 22 19:41:38 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 22 Dec 2013 19:41:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Fix_bootstrap_?= =?utf-8?q?issue_by_importing_the_cgi_module_lazily?= Message-ID: <3dnXbQ1xKWzPCC@mail.python.org> http://hg.python.org/cpython/rev/3c449a0cd4eb changeset: 88131:3c449a0cd4eb branch: 3.3 parent: 88129:bb42e10a5620 user: Antoine Pitrou date: Sun Dec 22 19:37:17 2013 +0100 summary: Fix bootstrap issue by importing the cgi module lazily files: Lib/distutils/config.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/distutils/config.py b/Lib/distutils/config.py --- a/Lib/distutils/config.py +++ b/Lib/distutils/config.py @@ -3,7 +3,6 @@ Provides the PyPIRCCommand class, the base class for the command classes that uses .pypirc in the distutils.command package. """ -import cgi import os from configparser import ConfigParser @@ -104,6 +103,7 @@ def _read_pypi_response(self, response): """Read and decode a PyPI HTTP response.""" + import cgi content_type = response.getheader('content-type', 'text/plain') encoding = cgi.parse_header(content_type)[1].get('charset', 'ascii') return response.read().decode(encoding) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 22 19:41:39 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 22 Dec 2013 19:41:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Fix_bootstrap_issue_by_importing_the_cgi_module_lazily?= Message-ID: <3dnXbR4b8qzPCC@mail.python.org> http://hg.python.org/cpython/rev/12197b52cb65 changeset: 88132:12197b52cb65 parent: 88130:d69ba246a3e4 parent: 88131:3c449a0cd4eb user: Antoine Pitrou date: Sun Dec 22 19:41:31 2013 +0100 summary: Fix bootstrap issue by importing the cgi module lazily files: Lib/distutils/config.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/distutils/config.py b/Lib/distutils/config.py --- a/Lib/distutils/config.py +++ b/Lib/distutils/config.py @@ -3,7 +3,6 @@ Provides the PyPIRCCommand class, the base class for the command classes that uses .pypirc in the distutils.command package. """ -import cgi import os from configparser import ConfigParser @@ -113,6 +112,7 @@ def _read_pypi_response(self, response): """Read and decode a PyPI HTTP response.""" + import cgi content_type = response.getheader('content-type', 'text/plain') encoding = cgi.parse_header(content_type)[1].get('charset', 'ascii') return response.read().decode(encoding) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 22 20:11:20 2013 From: python-checkins at python.org (r.david.murray) Date: Sun, 22 Dec 2013 20:11:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Make_argument_clinic_summa?= =?utf-8?q?ry_entry_parallel_to_other_PEP_entries=2E?= Message-ID: <3dnYFh52gwz7LjQ@mail.python.org> http://hg.python.org/cpython/rev/d9a8c3e50360 changeset: 88133:d9a8c3e50360 user: R David Murray date: Sun Dec 22 13:57:45 2013 -0500 summary: Make argument clinic summary entry parallel to other PEP entries. The existing entry repeated the same info that was in the body text, which means it didn't satisfy the sections "brevity" requirement :) files: Doc/whatsnew/3.4.rst | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -126,12 +126,10 @@ * :pep:`442`: :ref:`Safe object finalization ` * :pep:`445`: :ref:`Configurable memory allocators ` * :pep:`456`: :ref:`Secure and interchangeable hash algorithm ` +* :pep:`436`: :ref:`Argument Clinic `. * Improve finalization of Python modules to avoid setting their globals to None, in most cases (:issue:`18214`). * A more efficient :mod:`marshal` format (:issue:`16475`). -* "Argument Clinic", an initial step towards providing improved introspection - support for builtin and standard library extension types implemented in C - (:pep:`436`) Please read on for a comprehensive list of user-facing changes. @@ -976,6 +974,8 @@ marked as accepting ``const char *`` rather than ``char *`` (Contributed by Serhiy Storchaka in :issue:`1772673`). +.. _whatsnew-pep-436: + * "Argument Clinic" (:pep:`436`) is now part of the CPython build process and can be used to simplify the process of defining and maintaining accurate signatures for builtins and standard library extension modules -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 22 20:11:22 2013 From: python-checkins at python.org (r.david.murray) Date: Sun, 22 Dec 2013 20:11:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Rewrite_module-globals_sum?= =?utf-8?q?mary_entry=2C_and_expand_the_full_description_a_bit=2E?= Message-ID: <3dnYFk0y5nz7LjQ@mail.python.org> http://hg.python.org/cpython/rev/daa85dbabffb changeset: 88134:daa85dbabffb user: R David Murray date: Sun Dec 22 14:05:11 2013 -0500 summary: Rewrite module-globals summary entry, and expand the full description a bit. files: Doc/whatsnew/3.4.rst | 9 ++++++--- 1 files changed, 6 insertions(+), 3 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -124,11 +124,11 @@ CPython implementation improvements: * :pep:`442`: :ref:`Safe object finalization ` +* Leveraging :pep:`442`, :ref:`module globals are no longer set to None + during finalization `, in most cases (:issue:`18214`). * :pep:`445`: :ref:`Configurable memory allocators ` * :pep:`456`: :ref:`Secure and interchangeable hash algorithm ` * :pep:`436`: :ref:`Argument Clinic `. -* Improve finalization of Python modules to avoid setting their globals - to None, in most cases (:issue:`18214`). * A more efficient :mod:`marshal` format (:issue:`16475`). Please read on for a comprehensive list of user-facing changes. @@ -933,7 +933,10 @@ As part of this change, module globals are no longer forcibly set to :const:`None` during interpreter shutdown in most cases, instead relying -on the normal operation of the cyclic garbage collector. +on the normal operation of the cyclic garbage collector. This avoids a +whole class of interpreter-shutdown-time errors, usually involving +``__del__`` methods, that have plagued Python since the cyclic GC +was first introduced. .. seealso:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 22 20:11:23 2013 From: python-checkins at python.org (r.david.murray) Date: Sun, 22 Dec 2013 20:11:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Be_consistent_in_how_PEPs_?= =?utf-8?q?are_referenced_throughout_the_summary_section=2E?= Message-ID: <3dnYFl3557z7Ljj@mail.python.org> http://hg.python.org/cpython/rev/d431af9a20e3 changeset: 88135:d431af9a20e3 user: R David Murray date: Sun Dec 22 14:10:21 2013 -0500 summary: Be consistent in how PEPs are referenced throughout the summary section. files: Doc/whatsnew/3.4.rst | 12 +++++++----- 1 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -101,7 +101,8 @@ New expected features for Python implementations: -* :pep:`446`: :ref:`Make newly created file descriptors non-inheritable `. +* :ref:`Make newly created file descriptors non-inheritable ` + (:pep:`446`). * command line option for :ref:`isolated mode `, (:issue:`16499`). * :ref:`improvements ` in the handling of @@ -123,12 +124,13 @@ CPython implementation improvements: -* :pep:`442`: :ref:`Safe object finalization ` +* :ref:`Safe object finalization ` (:pep:`442`). * Leveraging :pep:`442`, :ref:`module globals are no longer set to None during finalization `, in most cases (:issue:`18214`). -* :pep:`445`: :ref:`Configurable memory allocators ` -* :pep:`456`: :ref:`Secure and interchangeable hash algorithm ` -* :pep:`436`: :ref:`Argument Clinic `. +* :ref:`Configurable memory allocators ` (:pep:`445`). +* :ref:`Secure and interchangeable hash algorithm ` + (:pep:`456`). +* :ref:`Argument Clinic ` (:pep:`436`). * A more efficient :mod:`marshal` format (:issue:`16475`). Please read on for a comprehensive list of user-facing changes. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 02:46:08 2013 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 23 Dec 2013 02:46:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_update_Barry?= =?utf-8?q?=27s_email_=28closes_=2319563=29?= Message-ID: <3dnk1D3dGVzQvN@mail.python.org> http://hg.python.org/cpython/rev/f07689845f55 changeset: 88136:f07689845f55 branch: 2.7 parent: 88126:39ea24aaf0e7 user: Benjamin Peterson date: Sun Dec 22 19:45:12 2013 -0600 summary: update Barry's email (closes #19563) files: Tools/i18n/pygettext.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Tools/i18n/pygettext.py b/Tools/i18n/pygettext.py --- a/Tools/i18n/pygettext.py +++ b/Tools/i18n/pygettext.py @@ -1,6 +1,6 @@ #! /usr/bin/env python # -*- coding: iso-8859-1 -*- -# Originally written by Barry Warsaw +# Originally written by Barry Warsaw # # Minimally patched to make it even more xgettext compatible # by Peter Funk -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 02:46:09 2013 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 23 Dec 2013 02:46:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_update_Barry?= =?utf-8?b?J3MgZW1haWwgKCMxOTU2Myk=?= Message-ID: <3dnk1F5dprzQvN@mail.python.org> http://hg.python.org/cpython/rev/9529c6c993fe changeset: 88137:9529c6c993fe branch: 3.3 parent: 88131:3c449a0cd4eb user: Benjamin Peterson date: Sun Dec 22 19:45:38 2013 -0600 summary: update Barry's email (#19563) files: Lib/test/test_email/data/msg_06.txt | 4 ++-- Lib/test/test_email/data/msg_08.txt | 2 +- Lib/test/test_email/data/msg_09.txt | 2 +- Lib/test/test_email/data/msg_10.txt | 2 +- Lib/test/test_email/data/msg_12.txt | 2 +- Lib/test/test_email/data/msg_12a.txt | 2 +- Tools/i18n/pygettext.py | 2 +- 7 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Lib/test/test_email/data/msg_06.txt b/Lib/test/test_email/data/msg_06.txt --- a/Lib/test/test_email/data/msg_06.txt +++ b/Lib/test/test_email/data/msg_06.txt @@ -5,7 +5,7 @@ Content-Description: forwarded message Content-Transfer-Encoding: 7bit Message-ID: <15265.9482.641338.555352 at python.org> -From: barry at zope.com (Barry A. Warsaw) +From: barry at python.org (Barry A. Warsaw) Sender: barry at python.org To: barry at python.org Subject: forwarded message from Barry A. Warsaw @@ -20,7 +20,7 @@ Return-Path: Delivered-To: barry at python.org Message-ID: <15265.9468.713530.98441 at python.org> -From: barry at zope.com (Barry A. Warsaw) +From: barry at python.org (Barry A. Warsaw) Sender: barry at python.org To: barry at python.org Subject: testing diff --git a/Lib/test/test_email/data/msg_08.txt b/Lib/test/test_email/data/msg_08.txt --- a/Lib/test/test_email/data/msg_08.txt +++ b/Lib/test/test_email/data/msg_08.txt @@ -1,5 +1,5 @@ MIME-Version: 1.0 -From: Barry Warsaw +From: Barry Warsaw To: Dingus Lovers Subject: Lyrics Date: Fri, 20 Apr 2001 19:35:02 -0400 diff --git a/Lib/test/test_email/data/msg_09.txt b/Lib/test/test_email/data/msg_09.txt --- a/Lib/test/test_email/data/msg_09.txt +++ b/Lib/test/test_email/data/msg_09.txt @@ -1,5 +1,5 @@ MIME-Version: 1.0 -From: Barry Warsaw +From: Barry Warsaw To: Dingus Lovers Subject: Lyrics Date: Fri, 20 Apr 2001 19:35:02 -0400 diff --git a/Lib/test/test_email/data/msg_10.txt b/Lib/test/test_email/data/msg_10.txt --- a/Lib/test/test_email/data/msg_10.txt +++ b/Lib/test/test_email/data/msg_10.txt @@ -1,5 +1,5 @@ MIME-Version: 1.0 -From: Barry Warsaw +From: Barry Warsaw To: Dingus Lovers Subject: Lyrics Date: Fri, 20 Apr 2001 19:35:02 -0400 diff --git a/Lib/test/test_email/data/msg_12.txt b/Lib/test/test_email/data/msg_12.txt --- a/Lib/test/test_email/data/msg_12.txt +++ b/Lib/test/test_email/data/msg_12.txt @@ -1,5 +1,5 @@ MIME-Version: 1.0 -From: Barry Warsaw +From: Barry Warsaw To: Dingus Lovers Subject: Lyrics Date: Fri, 20 Apr 2001 19:35:02 -0400 diff --git a/Lib/test/test_email/data/msg_12a.txt b/Lib/test/test_email/data/msg_12a.txt --- a/Lib/test/test_email/data/msg_12a.txt +++ b/Lib/test/test_email/data/msg_12a.txt @@ -1,5 +1,5 @@ MIME-Version: 1.0 -From: Barry Warsaw +From: Barry Warsaw To: Dingus Lovers Subject: Lyrics Date: Fri, 20 Apr 2001 19:35:02 -0400 diff --git a/Tools/i18n/pygettext.py b/Tools/i18n/pygettext.py --- a/Tools/i18n/pygettext.py +++ b/Tools/i18n/pygettext.py @@ -1,6 +1,6 @@ #! /usr/bin/env python3 # -*- coding: iso-8859-1 -*- -# Originally written by Barry Warsaw +# Originally written by Barry Warsaw # # Minimally patched to make it even more xgettext compatible # by Peter Funk -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 02:46:11 2013 From: python-checkins at python.org (benjamin.peterson) Date: Mon, 23 Dec 2013 02:46:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?b?KTogbWVyZ2UgMy4zICgjMTk1NjMp?= Message-ID: <3dnk1H0Sf6z7Ljh@mail.python.org> http://hg.python.org/cpython/rev/a3b61c770b40 changeset: 88138:a3b61c770b40 parent: 88135:d431af9a20e3 parent: 88137:9529c6c993fe user: Benjamin Peterson date: Sun Dec 22 19:45:53 2013 -0600 summary: merge 3.3 (#19563) files: Lib/test/test_email/data/msg_06.txt | 4 ++-- Lib/test/test_email/data/msg_08.txt | 2 +- Lib/test/test_email/data/msg_09.txt | 2 +- Lib/test/test_email/data/msg_10.txt | 2 +- Lib/test/test_email/data/msg_12.txt | 2 +- Lib/test/test_email/data/msg_12a.txt | 2 +- Tools/i18n/pygettext.py | 2 +- 7 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Lib/test/test_email/data/msg_06.txt b/Lib/test/test_email/data/msg_06.txt --- a/Lib/test/test_email/data/msg_06.txt +++ b/Lib/test/test_email/data/msg_06.txt @@ -5,7 +5,7 @@ Content-Description: forwarded message Content-Transfer-Encoding: 7bit Message-ID: <15265.9482.641338.555352 at python.org> -From: barry at zope.com (Barry A. Warsaw) +From: barry at python.org (Barry A. Warsaw) Sender: barry at python.org To: barry at python.org Subject: forwarded message from Barry A. Warsaw @@ -20,7 +20,7 @@ Return-Path: Delivered-To: barry at python.org Message-ID: <15265.9468.713530.98441 at python.org> -From: barry at zope.com (Barry A. Warsaw) +From: barry at python.org (Barry A. Warsaw) Sender: barry at python.org To: barry at python.org Subject: testing diff --git a/Lib/test/test_email/data/msg_08.txt b/Lib/test/test_email/data/msg_08.txt --- a/Lib/test/test_email/data/msg_08.txt +++ b/Lib/test/test_email/data/msg_08.txt @@ -1,5 +1,5 @@ MIME-Version: 1.0 -From: Barry Warsaw +From: Barry Warsaw To: Dingus Lovers Subject: Lyrics Date: Fri, 20 Apr 2001 19:35:02 -0400 diff --git a/Lib/test/test_email/data/msg_09.txt b/Lib/test/test_email/data/msg_09.txt --- a/Lib/test/test_email/data/msg_09.txt +++ b/Lib/test/test_email/data/msg_09.txt @@ -1,5 +1,5 @@ MIME-Version: 1.0 -From: Barry Warsaw +From: Barry Warsaw To: Dingus Lovers Subject: Lyrics Date: Fri, 20 Apr 2001 19:35:02 -0400 diff --git a/Lib/test/test_email/data/msg_10.txt b/Lib/test/test_email/data/msg_10.txt --- a/Lib/test/test_email/data/msg_10.txt +++ b/Lib/test/test_email/data/msg_10.txt @@ -1,5 +1,5 @@ MIME-Version: 1.0 -From: Barry Warsaw +From: Barry Warsaw To: Dingus Lovers Subject: Lyrics Date: Fri, 20 Apr 2001 19:35:02 -0400 diff --git a/Lib/test/test_email/data/msg_12.txt b/Lib/test/test_email/data/msg_12.txt --- a/Lib/test/test_email/data/msg_12.txt +++ b/Lib/test/test_email/data/msg_12.txt @@ -1,5 +1,5 @@ MIME-Version: 1.0 -From: Barry Warsaw +From: Barry Warsaw To: Dingus Lovers Subject: Lyrics Date: Fri, 20 Apr 2001 19:35:02 -0400 diff --git a/Lib/test/test_email/data/msg_12a.txt b/Lib/test/test_email/data/msg_12a.txt --- a/Lib/test/test_email/data/msg_12a.txt +++ b/Lib/test/test_email/data/msg_12a.txt @@ -1,5 +1,5 @@ MIME-Version: 1.0 -From: Barry Warsaw +From: Barry Warsaw To: Dingus Lovers Subject: Lyrics Date: Fri, 20 Apr 2001 19:35:02 -0400 diff --git a/Tools/i18n/pygettext.py b/Tools/i18n/pygettext.py --- a/Tools/i18n/pygettext.py +++ b/Tools/i18n/pygettext.py @@ -1,6 +1,6 @@ #! /usr/bin/env python3 # -*- coding: iso-8859-1 -*- -# Originally written by Barry Warsaw +# Originally written by Barry Warsaw # # Minimally patched to make it even more xgettext compatible # by Peter Funk -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 03:15:39 2013 From: python-checkins at python.org (r.david.murray) Date: Mon, 23 Dec 2013 03:15:39 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_description_of_marshal?= =?utf-8?q?_changes=2E?= Message-ID: <3dnkgH32Byz7LjP@mail.python.org> http://hg.python.org/cpython/rev/338f75e9ec69 changeset: 88139:338f75e9ec69 user: R David Murray date: Sun Dec 22 20:40:11 2013 -0500 summary: Add description of marshal changes. files: Doc/whatsnew/3.4.rst | 17 ++++++++++++++++- 1 files changed, 16 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -131,7 +131,8 @@ * :ref:`Secure and interchangeable hash algorithm ` (:pep:`456`). * :ref:`Argument Clinic ` (:pep:`436`). -* A more efficient :mod:`marshal` format (:issue:`16475`). +* The :mod:`marshal` format has been made :ref:`more compact and efficient + ` (:issue:`16475`). Please read on for a comprehensive list of user-facing changes. @@ -626,6 +627,20 @@ :issue:`19030`) +.. _whatsnew-marshal-3: + +marshal +------- + +The default :mod:`marshal` version has been bumped to 3. The code implementing +the new version restores the Python2 behavior of recording only one copy of +interned strings and preserving the interning on deserialization, and extends +this "one copy" ability to any object type (including handling recursive +references). This reduces both the size of ``.pyc`` files and the amount of +memory a module occupies in memory when it is loaded from a ``.pyc`` (or +``.pyo``) file. (Contributed by Kristj?n Valur J?nsson in :issue:`16475`.) + + mmap ---- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 03:15:40 2013 From: python-checkins at python.org (r.david.murray) Date: Mon, 23 Dec 2013 03:15:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Make_the_organization_of_t?= =?utf-8?q?he_body_and_the_summary_have_parallel_structure=2E?= Message-ID: <3dnkgJ61Bhz7Ljl@mail.python.org> http://hg.python.org/cpython/rev/5108330cb65c changeset: 88140:5108330cb65c user: R David Murray date: Sun Dec 22 20:49:40 2013 -0500 summary: Make the organization of the body and the summary have parallel structure. This means I moved the 'new expected features' section to the top of the summary, and made a new section with the same name at the start of the body, turning the previous top level sections there into subsections. I also added a line to the new first summary section for modulespec (pep 451). files: Doc/whatsnew/3.4.rst | 70 +++++++++++++++++-------------- 1 files changed, 39 insertions(+), 31 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -88,6 +88,17 @@ * No new syntax features were added in Python 3.4. +New expected features for Python implementations: + +* :ref:`Make newly created file descriptors non-inheritable ` + (:pep:`446`). +* command line option for :ref:`isolated mode `, + (:issue:`16499`). +* :ref:`improvements ` in the handling of + codecs that are not text encodings +* :ref:`A ModuleSpec Type ` for the Import System + (:pep:`451`). + New library modules: * :mod:`asyncio`: New provisonal API for asynchronous IO (:pep:`3156`). @@ -99,15 +110,6 @@ * :mod:`statistics`: A basic numerically stable statistics library (:pep:`450`). * :mod:`tracemalloc`: Trace Python memory allocations (:pep:`454`). -New expected features for Python implementations: - -* :ref:`Make newly created file descriptors non-inheritable ` - (:pep:`446`). -* command line option for :ref:`isolated mode `, - (:issue:`16499`). -* :ref:`improvements ` in the handling of - codecs that are not text encodings - Significantly Improved Library Modules: * :ref:`Single-dispatch generic functions ` in @@ -137,8 +139,11 @@ Please read on for a comprehensive list of user-facing changes. +New expected features for Python implementations +================================================ + PEP 453: Explicit bootstrapping of pip in Python installations -============================================================== +-------------------------------------------------------------- The new :mod:`ensurepip` module (defined in :pep:`453`) provides a standard cross-platform mechanism to boostrap the pip installer into Python @@ -176,7 +181,7 @@ .. _whatsnew-pep-446: PEP 446: Make newly created file descriptors non-inheritable -============================================================ +------------------------------------------------------------ :pep:`446` makes newly created file descriptors :ref:`non-inheritable `. New functions and methods: @@ -194,7 +199,7 @@ .. _codec-handling-improvements: Improvements to codec handling -============================== +------------------------------ Since it was first introduced, the :mod:`codecs` module has always been intended to operate as a type-neutral dynamic encoding and decoding @@ -272,10 +277,10 @@ (Contributed by Nick Coghlan in :issue:`7475`, , :issue:`17827`, :issue:`17828` and :issue:`19619`) -.. _pep-451: +.. _whatsnew-pep-451: PEP 451: A ModuleSpec Type for the Import System -================================================ +------------------------------------------------ :pep:`451` provides an encapsulation of the information about a module that the import machinery will use to load it, (i.e. a module spec). @@ -293,24 +298,8 @@ the new methods. -.. _whatsnew-protocol-4: - -Pickle protocol 4 -================= - -The new :mod:`pickle` protocol addresses a number of issues that were present -in previous protocols, such as the serialization of nested classes, very -large strings and containers, or classes whose :meth:`__new__` method takes -keyword-only arguments. It also brings a couple efficiency improvements. - -.. seealso:: - - :pep:`3154` - Pickle protocol 4 - PEP written by Antoine Pitrou and implemented by Alexandre Vassalotti. - - Other Language Changes -====================== +---------------------- Some smaller changes made to the core Python language are: @@ -697,6 +686,25 @@ (Contributed by Connor Osborn in :issue:`18764`.) +.. _whatsnew-protocol-4: + +Pickle +------ + +protocol 4 + +:mod:`pickle` now supports (but does not use by default) a new pickle protocol, +protocol 4. This new protocol addresses a number of issues that were present +in previous protocols, such as the serialization of nested classes, very large +strings and containers, or classes whose :meth:`__new__` method takes +keyword-only arguments. It also provides some efficiency improvements. + +.. seealso:: + + :pep:`3154` - Pickle protocol 4 + PEP written by Antoine Pitrou and implemented by Alexandre Vassalotti. + + poplib ------ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 03:15:42 2013 From: python-checkins at python.org (r.david.murray) Date: Mon, 23 Dec 2013 03:15:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Make_mailing_list_thread_r?= =?utf-8?q?eference_an_inline_link=2C_reword_i=2Ee=2E_abbreviation=2E?= Message-ID: <3dnkgL0n00z7LkQ@mail.python.org> http://hg.python.org/cpython/rev/82d20770d7dd changeset: 88141:82d20770d7dd user: R David Murray date: Sun Dec 22 21:05:04 2013 -0500 summary: Make mailing list thread reference an inline link, reword i.e. abbreviation. files: Doc/whatsnew/3.4.rst | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -97,7 +97,7 @@ * :ref:`improvements ` in the handling of codecs that are not text encodings * :ref:`A ModuleSpec Type ` for the Import System - (:pep:`451`). + (:pep:`451`). (Affects importer authors.) New library modules: @@ -283,12 +283,12 @@ ------------------------------------------------ :pep:`451` provides an encapsulation of the information about a module -that the import machinery will use to load it, (i.e. a module spec). +that the import machinery will use to load it (that is, a module specification). This helps simplify both the import implementation and several -import-related APIs. The change is also a stepping stone for several -future import-related improvements. +import-related APIs. The change is also a stepping stone for `several +future import-related improvements`__. -https://mail.python.org/pipermail/python-dev/2013-November/130111.html +__ https://mail.python.org/pipermail/python-dev/2013-November/130111.html The public-facing changes from the PEP are entirely backward-compatible. Furthermore, they should be transparent to everyone but importer -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 03:15:43 2013 From: python-checkins at python.org (r.david.murray) Date: Mon, 23 Dec 2013 03:15:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Reflow_=27modulespec=27_pa?= =?utf-8?q?ragraphs=2E?= Message-ID: <3dnkgM2n1fz7LjQ@mail.python.org> http://hg.python.org/cpython/rev/f65ac78d8e25 changeset: 88142:f65ac78d8e25 user: R David Murray date: Sun Dec 22 21:06:13 2013 -0500 summary: Reflow 'modulespec' paragraphs. files: Doc/whatsnew/3.4.rst | 19 +++++++++---------- 1 files changed, 9 insertions(+), 10 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -282,20 +282,19 @@ PEP 451: A ModuleSpec Type for the Import System ------------------------------------------------ -:pep:`451` provides an encapsulation of the information about a module -that the import machinery will use to load it (that is, a module specification). -This helps simplify both the import implementation and several -import-related APIs. The change is also a stepping stone for `several -future import-related improvements`__. +:pep:`451` provides an encapsulation of the information about a module that the +import machinery will use to load it (that is, a module specification). This +helps simplify both the import implementation and several import-related APIs. +The change is also a stepping stone for `several future import-related +improvements`__. __ https://mail.python.org/pipermail/python-dev/2013-November/130111.html The public-facing changes from the PEP are entirely backward-compatible. -Furthermore, they should be transparent to everyone but importer -authors. Key finder and loader methods have been deprecated, but they -will continue working. New importers should use the new methods -described in the PEP. Existing importers should be updated to implement -the new methods. +Furthermore, they should be transparent to everyone but importer authors. Key +finder and loader methods have been deprecated, but they will continue working. +New importers should use the new methods described in the PEP. Existing +importers should be updated to implement the new methods. Other Language Changes -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 03:15:44 2013 From: python-checkins at python.org (r.david.murray) Date: Mon, 23 Dec 2013 03:15:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Mention_deprecation_and_po?= =?utf-8?q?rting_sections_in_intro=2E?= Message-ID: <3dnkgN4fBDz7Ljy@mail.python.org> http://hg.python.org/cpython/rev/6836d6200674 changeset: 88143:6836d6200674 user: R David Murray date: Sun Dec 22 21:09:16 2013 -0500 summary: Mention deprecation and porting sections in intro. files: Doc/whatsnew/3.4.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -136,7 +136,8 @@ * The :mod:`marshal` format has been made :ref:`more compact and efficient ` (:issue:`16475`). -Please read on for a comprehensive list of user-facing changes. +Please read on for a comprehensive list of user-facing changes, including +sections on deprecations and porting issues. New expected features for Python implementations -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 07:16:27 2013 From: python-checkins at python.org (nick.coghlan) Date: Mon, 23 Dec 2013 07:16:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Close_=2319734=3A_ignore_p?= =?utf-8?q?ip_env_vars_in_ensurepip?= Message-ID: <3dnr174fBJz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/7a5a4d7c564d changeset: 88144:7a5a4d7c564d user: Nick Coghlan date: Mon Dec 23 16:16:07 2013 +1000 summary: Close #19734: ignore pip env vars in ensurepip files: Doc/library/ensurepip.rst | 6 ++++++ Lib/ensurepip/__init__.py | 8 ++++++++ Lib/test/test_ensurepip.py | 8 ++++++++ Lib/test/test_venv.py | 14 +++++--------- 4 files changed, 27 insertions(+), 9 deletions(-) diff --git a/Doc/library/ensurepip.rst b/Doc/library/ensurepip.rst --- a/Doc/library/ensurepip.rst +++ b/Doc/library/ensurepip.rst @@ -119,6 +119,12 @@ .. note:: + The bootstrapping process has side effects on both ``sys.path`` and + ``os.environ``. Invoking the command line interface in a subprocess + instead allows these side effects to be avoided. + + .. note:: + The bootstrapping process may install additional modules required by ``pip``, but other software should not assume those dependencies will always be present by default (as the dependencies may be removed in a diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -43,10 +43,18 @@ """ Bootstrap pip into the current Python installation (or the given root directory). + + Note that calling this function will alter both sys.path and os.environ. """ if altinstall and default_pip: raise ValueError("Cannot use altinstall and default_pip together") + # We deliberately ignore all pip environment variables + # See http://bugs.python.org/issue19734 for details + keys_to_remove = [k for k in os.environ if k.startswith("PIP_")] + for k in keys_to_remove: + del os.environ[k] + # By default, installing pip and setuptools installs all of the # following scripts (X.Y == running Python version): # diff --git a/Lib/test/test_ensurepip.py b/Lib/test/test_ensurepip.py --- a/Lib/test/test_ensurepip.py +++ b/Lib/test/test_ensurepip.py @@ -126,6 +126,14 @@ ensurepip.bootstrap(altinstall=True, default_pip=True) self.run_pip.assert_not_called() + def test_pip_environment_variables_removed(self): + # ensurepip deliberately ignores all pip environment variables + # See http://bugs.python.org/issue19734 for details + self.os_environ["PIP_THIS_SHOULD_GO_AWAY"] = "test fodder" + ensurepip.bootstrap() + self.assertNotIn("PIP_THIS_SHOULD_GO_AWAY", self.os_environ) + + @contextlib.contextmanager def fake_pip(version=ensurepip._PIP_VERSION): if version is None: diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py --- a/Lib/test/test_venv.py +++ b/Lib/test/test_venv.py @@ -294,11 +294,12 @@ # warnings in current versions of Python. Ensure related # environment settings don't cause venv to fail. envvars["PYTHONWARNINGS"] = "e" - # pip doesn't ignore environment variables when running in - # isolated mode, and we don't have an active virtualenv here, - # we're relying on the native venv support in 3.3+ + # ensurepip is different enough from a normal pip invocation + # that we want to ensure it ignores the normal pip environment + # variable settings. We set PIP_NO_INSTALL here specifically + # to check that ensurepip (and hence venv) ignores it. # See http://bugs.python.org/issue19734 for details - del envvars["PIP_REQUIRE_VIRTUALENV"] + envvars["PIP_NO_INSTALL"] = "1" try: self.run_with_capture(venv.create, self.env_dir, with_pip=True) except subprocess.CalledProcessError as exc: @@ -328,11 +329,6 @@ # installers works (at least in a virtual environment) cmd = [envpy, '-Im', 'ensurepip._uninstall'] with EnvironmentVarGuard() as envvars: - # pip doesn't ignore environment variables when running in - # isolated mode, and we don't have an active virtualenv here, - # we're relying on the native venv support in 3.3+ - # See http://bugs.python.org/issue19734 for details - del envvars["PIP_REQUIRE_VIRTUALENV"] p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) out, err = p.communicate() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 08:40:17 2013 From: python-checkins at python.org (nick.coghlan) Date: Mon, 23 Dec 2013 08:40:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319734=3A_ignore_p?= =?utf-8?q?ip_env_vars_in_ensurepip=2E=5Funinstall?= Message-ID: <3dnsss1LbTz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/98d8bf13a32c changeset: 88145:98d8bf13a32c user: Nick Coghlan date: Mon Dec 23 17:39:12 2013 +1000 summary: Issue #19734: ignore pip env vars in ensurepip._uninstall files: Lib/ensurepip/__init__.py | 21 +++++++++++++++------ Lib/test/test_ensurepip.py | 14 ++++++++++++++ 2 files changed, 29 insertions(+), 6 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -36,6 +36,14 @@ """ return _PIP_VERSION +def _clear_pip_environment_variables(): + # We deliberately ignore all pip environment variables + # when invoking pip + # See http://bugs.python.org/issue19734 for details + keys_to_remove = [k for k in os.environ if k.startswith("PIP_")] + for k in keys_to_remove: + del os.environ[k] + def bootstrap(*, root=None, upgrade=False, user=False, altinstall=False, default_pip=False, @@ -49,11 +57,7 @@ if altinstall and default_pip: raise ValueError("Cannot use altinstall and default_pip together") - # We deliberately ignore all pip environment variables - # See http://bugs.python.org/issue19734 for details - keys_to_remove = [k for k in os.environ if k.startswith("PIP_")] - for k in keys_to_remove: - del os.environ[k] + _clear_pip_environment_variables() # By default, installing pip and setuptools installs all of the # following scripts (X.Y == running Python version): @@ -101,7 +105,10 @@ _run_pip(args + [p[0] for p in _PROJECTS], additional_paths) def _uninstall(*, verbosity=0): - """Helper to support a clean default uninstall process on Windows""" + """Helper to support a clean default uninstall process on Windows + + Note that calling this function may alter os.environ. + """ # Nothing to do if pip was never installed, or has been removed try: import pip @@ -114,6 +121,8 @@ "({!r} installed, {!r} bundled)") raise RuntimeError(msg.format(pip.__version__, _PIP_VERSION)) + _clear_pip_environment_variables() + # Construct the arguments to be passed to the pip command args = ["uninstall", "-y"] if verbosity: diff --git a/Lib/test/test_ensurepip.py b/Lib/test/test_ensurepip.py --- a/Lib/test/test_ensurepip.py +++ b/Lib/test/test_ensurepip.py @@ -160,6 +160,13 @@ self.run_pip = run_pip_patch.start() self.addCleanup(run_pip_patch.stop) + # Avoid side effects on the actual os module + os_patch = unittest.mock.patch("ensurepip.os") + patched_os = os_patch.start() + self.addCleanup(os_patch.stop) + patched_os.path = os.path + self.os_environ = patched_os.environ = os.environ.copy() + def test_uninstall_skipped_when_not_installed(self): with fake_pip(None): ensurepip._uninstall() @@ -204,6 +211,13 @@ ["uninstall", "-y", "-vvv", "pip", "setuptools"] ) + def test_pip_environment_variables_removed(self): + # ensurepip deliberately ignores all pip environment variables + # See http://bugs.python.org/issue19734 for details + self.os_environ["PIP_THIS_SHOULD_GO_AWAY"] = "test fodder" + with fake_pip(): + ensurepip._uninstall() + self.assertNotIn("PIP_THIS_SHOULD_GO_AWAY", self.os_environ) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 08:42:14 2013 From: python-checkins at python.org (nick.coghlan) Date: Mon, 23 Dec 2013 08:42:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319734=3A_add_miss?= =?utf-8?q?ing_NEWS_entry?= Message-ID: <3dnsw63j7Fz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/3b3d1c312042 changeset: 88146:3b3d1c312042 user: Nick Coghlan date: Mon Dec 23 17:42:02 2013 +1000 summary: Issue #19734: add missing NEWS entry files: Misc/NEWS | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #19734: ensurepip now ignores all pip environment variables to avoid + odd behaviour based on user configuration settings + - Fix TypeError on "setup.py upload --show-response". - Issue #20045: Fix "setup.py register --list-classifiers". -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 09:17:33 2013 From: python-checkins at python.org (nick.coghlan) Date: Mon, 23 Dec 2013 09:17:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Further_increase_tolerance?= =?utf-8?q?_for_slow_buildbots?= Message-ID: <3dnths6tnCz7LjW@mail.python.org> http://hg.python.org/cpython/rev/64786a88c641 changeset: 88147:64786a88c641 user: Nick Coghlan date: Mon Dec 23 18:17:20 2013 +1000 summary: Further increase tolerance for slow buildbots files: Lib/test/test_multiprocessing_main_handling.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_multiprocessing_main_handling.py b/Lib/test/test_multiprocessing_main_handling.py --- a/Lib/test/test_multiprocessing_main_handling.py +++ b/Lib/test/test_multiprocessing_main_handling.py @@ -53,7 +53,7 @@ p = Pool(5) results = [] p.map_async(f, [1, 2, 3], callback=results.extend) - deadline = time.time() + 5 # up to 5 s to report the results + deadline = time.time() + 10 # up to 10 s to report the results while not results: time.sleep(0.05) if time.time() > deadline: @@ -81,7 +81,7 @@ p = Pool(5) results = [] p.map_async(int, [1, 4, 9], callback=results.extend) -deadline = time.time() + 5 # up to 5 s to report the results +deadline = time.time() + 10 # up to 10 s to report the results while not results: time.sleep(0.05) if time.time() > deadline: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 09:22:14 2013 From: python-checkins at python.org (nick.coghlan) Date: Mon, 23 Dec 2013 09:22:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319728=3A_fix_ensu?= =?utf-8?q?repip_name_clash_with_submodule?= Message-ID: <3dntpG0Tfvz7Llc@mail.python.org> http://hg.python.org/cpython/rev/cd62fc2488cf changeset: 88148:cd62fc2488cf user: Nick Coghlan date: Mon Dec 23 18:20:34 2013 +1000 summary: Issue #19728: fix ensurepip name clash with submodule Also added refactoring and added basic tests for the argument parsing in both ensurepip._main and ensurepip._uninstall._main. files: Lib/ensurepip/__init__.py | 63 ++++++++++++++++++- Lib/ensurepip/__main__.py | 64 +------------------ Lib/ensurepip/_uninstall.py | 8 +- Lib/test/test_ensurepip.py | 83 ++++++++++++++++++------ 4 files changed, 127 insertions(+), 91 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -104,7 +104,7 @@ _run_pip(args + [p[0] for p in _PROJECTS], additional_paths) -def _uninstall(*, verbosity=0): +def _uninstall_helper(*, verbosity=0): """Helper to support a clean default uninstall process on Windows Note that calling this function may alter os.environ. @@ -129,3 +129,64 @@ args += ["-" + "v" * verbosity] _run_pip(args + [p[0] for p in reversed(_PROJECTS)]) + + +def _main(argv=None): + import argparse + parser = argparse.ArgumentParser(prog="python -m ensurepip") + parser.add_argument( + "--version", + action="version", + version="pip {}".format(version()), + help="Show the version of pip that is bundled with this Python.", + ) + parser.add_argument( + "-v", "--verbose", + action="count", + default=0, + dest="verbosity", + help=("Give more output. Option is additive, and can be used up to 3 " + "times."), + ) + parser.add_argument( + "-U", "--upgrade", + action="store_true", + default=False, + help="Upgrade pip and dependencies, even if already installed.", + ) + parser.add_argument( + "--user", + action="store_true", + default=False, + help="Install using the user scheme.", + ) + parser.add_argument( + "--root", + default=None, + help="Install everything relative to this alternate root directory.", + ) + parser.add_argument( + "--altinstall", + action="store_true", + default=False, + help=("Make an alternate install, installing only the X.Y versioned" + "scripts (Default: pipX, pipX.Y, easy_install-X.Y)"), + ) + parser.add_argument( + "--default-pip", + action="store_true", + default=False, + help=("Make a default pip install, installing the unqualified pip " + "and easy_install in addition to the versioned scripts"), + ) + + args = parser.parse_args(argv) + + bootstrap( + root=args.root, + upgrade=args.upgrade, + user=args.user, + verbosity=args.verbosity, + altinstall=args.altinstall, + default_pip=args.default_pip, + ) diff --git a/Lib/ensurepip/__main__.py b/Lib/ensurepip/__main__.py --- a/Lib/ensurepip/__main__.py +++ b/Lib/ensurepip/__main__.py @@ -1,66 +1,4 @@ -import argparse import ensurepip - -def main(): - parser = argparse.ArgumentParser(prog="python -m ensurepip") - parser.add_argument( - "--version", - action="version", - version="pip {}".format(ensurepip.version()), - help="Show the version of pip that is bundled with this Python.", - ) - parser.add_argument( - "-v", "--verbose", - action="count", - default=0, - dest="verbosity", - help=("Give more output. Option is additive, and can be used up to 3 " - "times."), - ) - parser.add_argument( - "-U", "--upgrade", - action="store_true", - default=False, - help="Upgrade pip and dependencies, even if already installed.", - ) - parser.add_argument( - "--user", - action="store_true", - default=False, - help="Install using the user scheme.", - ) - parser.add_argument( - "--root", - default=None, - help="Install everything relative to this alternate root directory.", - ) - parser.add_argument( - "--altinstall", - action="store_true", - default=False, - help=("Make an alternate install, installing only the X.Y versioned" - "scripts (Default: pipX, pipX.Y, easy_install-X.Y)"), - ) - parser.add_argument( - "--default-pip", - action="store_true", - default=False, - help=("Make a default pip install, installing the unqualified pip " - "and easy_install in addition to the versioned scripts"), - ) - - args = parser.parse_args() - - ensurepip.bootstrap( - root=args.root, - upgrade=args.upgrade, - user=args.user, - verbosity=args.verbosity, - altinstall=args.altinstall, - default_pip=args.default_pip, - ) - - if __name__ == "__main__": - main() + ensurepip._main() diff --git a/Lib/ensurepip/_uninstall.py b/Lib/ensurepip/_uninstall.py --- a/Lib/ensurepip/_uninstall.py +++ b/Lib/ensurepip/_uninstall.py @@ -4,7 +4,7 @@ import ensurepip -def main(): +def _main(argv=None): parser = argparse.ArgumentParser(prog="python -m ensurepip._uninstall") parser.add_argument( "--version", @@ -21,10 +21,10 @@ "times."), ) - args = parser.parse_args() + args = parser.parse_args(argv) - ensurepip._uninstall(verbosity=args.verbosity) + ensurepip._uninstall_helper(verbosity=args.verbosity) if __name__ == "__main__": - main() + _main() diff --git a/Lib/test/test_ensurepip.py b/Lib/test/test_ensurepip.py --- a/Lib/test/test_ensurepip.py +++ b/Lib/test/test_ensurepip.py @@ -1,20 +1,20 @@ import unittest import unittest.mock -import ensurepip import test.support import os import os.path import contextlib import sys +import ensurepip +import ensurepip._uninstall class TestEnsurePipVersion(unittest.TestCase): def test_returns_version(self): self.assertEqual(ensurepip._PIP_VERSION, ensurepip.version()) - -class TestBootstrap(unittest.TestCase): +class EnsurepipMixin: def setUp(self): run_pip_patch = unittest.mock.patch("ensurepip._run_pip") @@ -28,6 +28,8 @@ patched_os.path = os.path self.os_environ = patched_os.environ = os.environ.copy() +class TestBootstrap(EnsurepipMixin, unittest.TestCase): + def test_basic_bootstrapping(self): ensurepip.bootstrap() @@ -153,35 +155,23 @@ else: sys.modules["pip"] = orig_pip -class TestUninstall(unittest.TestCase): - - def setUp(self): - run_pip_patch = unittest.mock.patch("ensurepip._run_pip") - self.run_pip = run_pip_patch.start() - self.addCleanup(run_pip_patch.stop) - - # Avoid side effects on the actual os module - os_patch = unittest.mock.patch("ensurepip.os") - patched_os = os_patch.start() - self.addCleanup(os_patch.stop) - patched_os.path = os.path - self.os_environ = patched_os.environ = os.environ.copy() +class TestUninstall(EnsurepipMixin, unittest.TestCase): def test_uninstall_skipped_when_not_installed(self): with fake_pip(None): - ensurepip._uninstall() + ensurepip._uninstall_helper() self.run_pip.assert_not_called() def test_uninstall_fails_with_wrong_version(self): with fake_pip("not a valid version"): with self.assertRaises(RuntimeError): - ensurepip._uninstall() + ensurepip._uninstall_helper() self.run_pip.assert_not_called() def test_uninstall(self): with fake_pip(): - ensurepip._uninstall() + ensurepip._uninstall_helper() self.run_pip.assert_called_once_with( ["uninstall", "-y", "pip", "setuptools"] @@ -189,7 +179,7 @@ def test_uninstall_with_verbosity_1(self): with fake_pip(): - ensurepip._uninstall(verbosity=1) + ensurepip._uninstall_helper(verbosity=1) self.run_pip.assert_called_once_with( ["uninstall", "-y", "-v", "pip", "setuptools"] @@ -197,7 +187,7 @@ def test_uninstall_with_verbosity_2(self): with fake_pip(): - ensurepip._uninstall(verbosity=2) + ensurepip._uninstall_helper(verbosity=2) self.run_pip.assert_called_once_with( ["uninstall", "-y", "-vv", "pip", "setuptools"] @@ -205,7 +195,7 @@ def test_uninstall_with_verbosity_3(self): with fake_pip(): - ensurepip._uninstall(verbosity=3) + ensurepip._uninstall_helper(verbosity=3) self.run_pip.assert_called_once_with( ["uninstall", "-y", "-vvv", "pip", "setuptools"] @@ -216,10 +206,57 @@ # See http://bugs.python.org/issue19734 for details self.os_environ["PIP_THIS_SHOULD_GO_AWAY"] = "test fodder" with fake_pip(): - ensurepip._uninstall() + ensurepip._uninstall_helper() self.assertNotIn("PIP_THIS_SHOULD_GO_AWAY", self.os_environ) +# Basic testing of the main functions and their argument parsing + +EXPECTED_VERSION_OUTPUT = "pip " + ensurepip._PIP_VERSION + +class TestBootstrappingMainFunction(EnsurepipMixin, unittest.TestCase): + + def test_bootstrap_version(self): + with test.support.captured_stdout() as stdout: + with self.assertRaises(SystemExit): + ensurepip._main(["--version"]) + result = stdout.getvalue().strip() + self.assertEqual(result, EXPECTED_VERSION_OUTPUT) + self.run_pip.assert_not_called() + + def test_basic_bootstrapping(self): + ensurepip._main([]) + + self.run_pip.assert_called_once_with( + [ + "install", "--no-index", "--find-links", + unittest.mock.ANY, "--pre", "setuptools", "pip", + ], + unittest.mock.ANY, + ) + + additional_paths = self.run_pip.call_args[0][1] + self.assertEqual(len(additional_paths), 2) + +class TestUninstallationMainFunction(EnsurepipMixin, unittest.TestCase): + + def test_uninstall_version(self): + with test.support.captured_stdout() as stdout: + with self.assertRaises(SystemExit): + ensurepip._uninstall._main(["--version"]) + result = stdout.getvalue().strip() + self.assertEqual(result, EXPECTED_VERSION_OUTPUT) + self.run_pip.assert_not_called() + + def test_basic_uninstall(self): + with fake_pip(): + ensurepip._uninstall._main([]) + + self.run_pip.assert_called_once_with( + ["uninstall", "-y", "pip", "setuptools"] + ) + + if __name__ == "__main__": test.support.run_unittest(__name__) -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Mon Dec 23 09:45:04 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 23 Dec 2013 09:45:04 +0100 Subject: [Python-checkins] Daily reference leaks (6836d6200674): sum=0 Message-ID: results for 6836d6200674 on branch "default" -------------------------------------------- test_asyncio leaked [4, 0, 0] memory blocks, sum=4 test_site leaked [0, 0, -2] references, sum=-2 test_site leaked [0, 0, -2] memory blocks, sum=-2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogGEWCAw', '-x'] From python-checkins at python.org Mon Dec 23 14:07:21 2013 From: python-checkins at python.org (nick.coghlan) Date: Mon, 23 Dec 2013 14:07:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319744=3A_improve_?= =?utf-8?q?ensurepip_error_when_ssl_is_missing?= Message-ID: <3dp17F0RYHz7Ll4@mail.python.org> http://hg.python.org/cpython/rev/f670d8db8ef3 changeset: 88149:f670d8db8ef3 user: Nick Coghlan date: Mon Dec 23 23:07:07 2013 +1000 summary: Issue #19744: improve ensurepip error when ssl is missing files: Lib/ensurepip/__init__.py | 15 ++++++ Lib/test/test_ensurepip.py | 57 ++++++++++++++++++++++++++ Lib/test/test_venv.py | 7 ++- Misc/NEWS | 4 + 4 files changed, 81 insertions(+), 2 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -14,6 +14,19 @@ _PIP_VERSION = "1.5rc2" +# pip currently requires ssl support, so we try to provide a nicer +# error message when that is missing (http://bugs.python.org/issue19744) +_MISSING_SSL_MESSAGE = ("pip {} requires SSL/TLS".format(_PIP_VERSION)) +try: + import ssl +except ImportError: + ssl = None + def _require_ssl_for_pip(): + raise RuntimeError(_MISSING_SSL_MESSAGE) +else: + def _require_ssl_for_pip(): + pass + _PROJECTS = [ ("setuptools", _SETUPTOOLS_VERSION), ("pip", _PIP_VERSION), @@ -57,6 +70,7 @@ if altinstall and default_pip: raise ValueError("Cannot use altinstall and default_pip together") + _require_ssl_for_pip() _clear_pip_environment_variables() # By default, installing pip and setuptools installs all of the @@ -121,6 +135,7 @@ "({!r} installed, {!r} bundled)") raise RuntimeError(msg.format(pip.__version__, _PIP_VERSION)) + _require_ssl_for_pip() _clear_pip_environment_variables() # Construct the arguments to be passed to the pip command diff --git a/Lib/test/test_ensurepip.py b/Lib/test/test_ensurepip.py --- a/Lib/test/test_ensurepip.py +++ b/Lib/test/test_ensurepip.py @@ -9,6 +9,20 @@ import ensurepip import ensurepip._uninstall +# pip currently requires ssl support, so we ensure we handle +# it being missing (http://bugs.python.org/issue19744) +ensurepip_no_ssl = test.support.import_fresh_module("ensurepip", + blocked=["ssl"]) +try: + import ssl +except ImportError: + def requires_usable_pip(f): + deco = unittest.skip(ensurepip._MISSING_SSL_MESSAGE) + return deco(f) +else: + def requires_usable_pip(f): + return f + class TestEnsurePipVersion(unittest.TestCase): def test_returns_version(self): @@ -28,8 +42,10 @@ patched_os.path = os.path self.os_environ = patched_os.environ = os.environ.copy() + class TestBootstrap(EnsurepipMixin, unittest.TestCase): + @requires_usable_pip def test_basic_bootstrapping(self): ensurepip.bootstrap() @@ -44,6 +60,7 @@ additional_paths = self.run_pip.call_args[0][1] self.assertEqual(len(additional_paths), 2) + @requires_usable_pip def test_bootstrapping_with_root(self): ensurepip.bootstrap(root="/foo/bar/") @@ -56,6 +73,7 @@ unittest.mock.ANY, ) + @requires_usable_pip def test_bootstrapping_with_user(self): ensurepip.bootstrap(user=True) @@ -67,6 +85,7 @@ unittest.mock.ANY, ) + @requires_usable_pip def test_bootstrapping_with_upgrade(self): ensurepip.bootstrap(upgrade=True) @@ -78,6 +97,7 @@ unittest.mock.ANY, ) + @requires_usable_pip def test_bootstrapping_with_verbosity_1(self): ensurepip.bootstrap(verbosity=1) @@ -89,6 +109,7 @@ unittest.mock.ANY, ) + @requires_usable_pip def test_bootstrapping_with_verbosity_2(self): ensurepip.bootstrap(verbosity=2) @@ -100,6 +121,7 @@ unittest.mock.ANY, ) + @requires_usable_pip def test_bootstrapping_with_verbosity_3(self): ensurepip.bootstrap(verbosity=3) @@ -111,14 +133,17 @@ unittest.mock.ANY, ) + @requires_usable_pip def test_bootstrapping_with_regular_install(self): ensurepip.bootstrap() self.assertEqual(self.os_environ["ENSUREPIP_OPTIONS"], "install") + @requires_usable_pip def test_bootstrapping_with_alt_install(self): ensurepip.bootstrap(altinstall=True) self.assertEqual(self.os_environ["ENSUREPIP_OPTIONS"], "altinstall") + @requires_usable_pip def test_bootstrapping_with_default_pip(self): ensurepip.bootstrap(default_pip=True) self.assertNotIn("ENSUREPIP_OPTIONS", self.os_environ) @@ -128,6 +153,7 @@ ensurepip.bootstrap(altinstall=True, default_pip=True) self.run_pip.assert_not_called() + @requires_usable_pip def test_pip_environment_variables_removed(self): # ensurepip deliberately ignores all pip environment variables # See http://bugs.python.org/issue19734 for details @@ -169,6 +195,7 @@ self.run_pip.assert_not_called() + @requires_usable_pip def test_uninstall(self): with fake_pip(): ensurepip._uninstall_helper() @@ -177,6 +204,7 @@ ["uninstall", "-y", "pip", "setuptools"] ) + @requires_usable_pip def test_uninstall_with_verbosity_1(self): with fake_pip(): ensurepip._uninstall_helper(verbosity=1) @@ -185,6 +213,7 @@ ["uninstall", "-y", "-v", "pip", "setuptools"] ) + @requires_usable_pip def test_uninstall_with_verbosity_2(self): with fake_pip(): ensurepip._uninstall_helper(verbosity=2) @@ -193,6 +222,7 @@ ["uninstall", "-y", "-vv", "pip", "setuptools"] ) + @requires_usable_pip def test_uninstall_with_verbosity_3(self): with fake_pip(): ensurepip._uninstall_helper(verbosity=3) @@ -201,6 +231,7 @@ ["uninstall", "-y", "-vvv", "pip", "setuptools"] ) + @requires_usable_pip def test_pip_environment_variables_removed(self): # ensurepip deliberately ignores all pip environment variables # See http://bugs.python.org/issue19734 for details @@ -210,6 +241,30 @@ self.assertNotIn("PIP_THIS_SHOULD_GO_AWAY", self.os_environ) +class TestMissingSSL(EnsurepipMixin, unittest.TestCase): + + def setUp(self): + sys.modules["ensurepip"] = ensurepip_no_ssl + @self.addCleanup + def restore_module(): + sys.modules["ensurepip"] = ensurepip + super().setUp() + + def test_bootstrap_requires_ssl(self): + self.os_environ["PIP_THIS_SHOULD_STAY"] = "test fodder" + with self.assertRaisesRegex(RuntimeError, "requires SSL/TLS"): + ensurepip_no_ssl.bootstrap() + self.run_pip.assert_not_called() + self.assertIn("PIP_THIS_SHOULD_STAY", self.os_environ) + + def test_uninstall_requires_ssl(self): + self.os_environ["PIP_THIS_SHOULD_STAY"] = "test fodder" + with self.assertRaisesRegex(RuntimeError, "requires SSL/TLS"): + with fake_pip(): + ensurepip_no_ssl._uninstall_helper() + self.run_pip.assert_not_called() + self.assertIn("PIP_THIS_SHOULD_STAY", self.os_environ) + # Basic testing of the main functions and their argument parsing EXPECTED_VERSION_OUTPUT = "pip " + ensurepip._PIP_VERSION @@ -224,6 +279,7 @@ self.assertEqual(result, EXPECTED_VERSION_OUTPUT) self.run_pip.assert_not_called() + @requires_usable_pip def test_basic_bootstrapping(self): ensurepip._main([]) @@ -248,6 +304,7 @@ self.assertEqual(result, EXPECTED_VERSION_OUTPUT) self.run_pip.assert_not_called() + @requires_usable_pip def test_basic_uninstall(self): with fake_pip(): ensurepip._uninstall._main([]) diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py --- a/Lib/test/test_venv.py +++ b/Lib/test/test_venv.py @@ -17,6 +17,9 @@ import textwrap import unittest import venv + +# pip currently requires ssl support, so we ensure we handle +# it being missing (http://bugs.python.org/issue19744) try: import ssl except ImportError: @@ -285,8 +288,8 @@ self.run_with_capture(venv.create, self.env_dir, with_pip=False) self.assert_pip_not_installed() - # Temporary skip for http://bugs.python.org/issue19744 - @unittest.skipIf(ssl is None, 'pip needs SSL support') + # Requesting pip fails without SSL (http://bugs.python.org/issue19744) + @unittest.skipIf(ssl is None, ensurepip._MISSING_SSL_MESSAGE) def test_with_pip(self): shutil.rmtree(self.env_dir) with EnvironmentVarGuard() as envvars: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,10 @@ Library ------- +- Issue #19744: ensurepip now provides a better error message when Python is + built without SSL/TLS support (pip currently requires that support to run, + even if only operating with local wheel files) + - Issue #19734: ensurepip now ignores all pip environment variables to avoid odd behaviour based on user configuration settings -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 17:22:22 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 23 Dec 2013 17:22:22 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Removed_spaces?= =?utf-8?q?_before_commas_and_periods=2E?= Message-ID: <3dp5SG1qstz7Ljg@mail.python.org> http://hg.python.org/cpython/rev/6aeef7be5752 changeset: 88150:6aeef7be5752 branch: 2.7 parent: 88136:f07689845f55 user: Serhiy Storchaka date: Mon Dec 23 18:19:34 2013 +0200 summary: Removed spaces before commas and periods. files: Doc/faq/library.rst | 2 +- Doc/howto/descriptor.rst | 2 +- Doc/howto/pyporting.rst | 2 +- Doc/howto/urllib2.rst | 2 +- Doc/library/bdb.rst | 2 +- Doc/library/bsddb.rst | 2 +- Doc/library/cookielib.rst | 2 +- Doc/library/ctypes.rst | 2 +- Doc/library/itertools.rst | 2 +- Doc/library/logging.rst | 2 +- Doc/library/ossaudiodev.rst | 2 +- Doc/library/pyexpat.rst | 2 +- Doc/library/sys.rst | 2 +- Doc/library/tkinter.rst | 4 ++-- Doc/library/trace.rst | 2 +- Doc/library/xml.dom.minidom.rst | 2 +- 16 files changed, 17 insertions(+), 17 deletions(-) diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -640,7 +640,7 @@ .. XXX check if wiki page is still up to date A summary of available frameworks is maintained by Paul Boddie at -http://wiki.python.org/moin/WebProgramming . +http://wiki.python.org/moin/WebProgramming\ . Cameron Laird maintains a useful set of pages about Python web technologies at http://phaseit.net/claird/comp.lang.python/web_python. diff --git a/Doc/howto/descriptor.rst b/Doc/howto/descriptor.rst --- a/Doc/howto/descriptor.rst +++ b/Doc/howto/descriptor.rst @@ -167,7 +167,7 @@ return self.val def __set__(self, obj, val): - print 'Updating' , self.name + print 'Updating', self.name self.val = val >>> class MyClass(object): diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -598,7 +598,7 @@ To get a complete idea of what issues you will need to deal with, see the `What's New in Python 3.0`_. Others have reorganized the data in other formats -such as http://docs.pythonsprints.com/python3_porting/py-porting.html . +such as http://docs.pythonsprints.com/python3_porting/py-porting.html\ . The following are some steps to take to try to support both Python 2 & 3 from the same source code. diff --git a/Doc/howto/urllib2.rst b/Doc/howto/urllib2.rst --- a/Doc/howto/urllib2.rst +++ b/Doc/howto/urllib2.rst @@ -150,7 +150,7 @@ to your HTTP request. Some websites [#]_ dislike being browsed by programs, or send different versions -to different browsers [#]_ . By default urllib2 identifies itself as +to different browsers [#]_. By default urllib2 identifies itself as ``Python-urllib/x.y`` (where ``x`` and ``y`` are the major and minor version numbers of the Python release, e.g. ``Python-urllib/2.5``), which may confuse the site, or just plain diff --git a/Doc/library/bdb.rst b/Doc/library/bdb.rst --- a/Doc/library/bdb.rst +++ b/Doc/library/bdb.rst @@ -20,7 +20,7 @@ The :mod:`bdb` module also defines two classes: -.. class:: Breakpoint(self, file, line, temporary=0, cond=None , funcname=None) +.. class:: Breakpoint(self, file, line, temporary=0, cond=None, funcname=None) This class implements temporary breakpoints, ignore counts, disabling and (re-)enabling, and conditionals. diff --git a/Doc/library/bsddb.rst b/Doc/library/bsddb.rst --- a/Doc/library/bsddb.rst +++ b/Doc/library/bsddb.rst @@ -52,7 +52,7 @@ Open the hash format file named *filename*. Files never intended to be preserved on disk may be created by passing ``None`` as the *filename*. The optional *flag* identifies the mode used to open the file. It may be ``'r'`` - (read only), ``'w'`` (read-write) , ``'c'`` (read-write - create if necessary; + (read only), ``'w'`` (read-write), ``'c'`` (read-write - create if necessary; the default) or ``'n'`` (read-write - truncate to zero length). The other arguments are rarely used and are just passed to the low-level :c:func:`dbopen` function. Consult the Berkeley DB documentation for their use and diff --git a/Doc/library/cookielib.rst b/Doc/library/cookielib.rst --- a/Doc/library/cookielib.rst +++ b/Doc/library/cookielib.rst @@ -308,7 +308,7 @@ ----------------------------------------------------------- The following :class:`CookieJar` subclasses are provided for reading and -writing . +writing. .. class:: MozillaCookieJar(filename, delayload=None, policy=None) diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -1714,7 +1714,7 @@ WINUSERAPI int WINAPI MessageBoxA( - HWND hWnd , + HWND hWnd, LPCSTR lpText, LPCSTR lpCaption, UINT uType); diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -57,7 +57,7 @@ :func:`islice` seq, [start,] stop [, step] elements from seq[start:stop:step] ``islice('ABCDEFG', 2, None) --> C D E F G`` :func:`imap` func, p, q, ... func(p0, q0), func(p1, q1), ... ``imap(pow, (2,3,10), (5,2,3)) --> 32 9 1000`` :func:`starmap` func, seq func(\*seq[0]), func(\*seq[1]), ... ``starmap(pow, [(2,5), (3,2), (10,3)]) --> 32 9 1000`` -:func:`tee` it, n it1, it2 , ... itn splits one iterator into n +:func:`tee` it, n it1, it2, ... itn splits one iterator into n :func:`takewhile` pred, seq seq[0], seq[1], until pred fails ``takewhile(lambda x: x<5, [1,4,6,4,1]) --> 1 4`` :func:`izip` p, q, ... (p[0], q[0]), (p[1], q[1]), ... ``izip('ABCD', 'xy') --> Ax By`` :func:`izip_longest` p, q, ... (p[0], q[0]), (p[1], q[1]), ... ``izip_longest('ABCD', 'xy', fillvalue='-') --> Ax By C- D-`` diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -691,7 +691,7 @@ --------------------- :class:`LoggerAdapter` instances are used to conveniently pass contextual -information into logging calls. For a usage example , see the section on +information into logging calls. For a usage example, see the section on :ref:`adding contextual information to your logging output `. .. versionadded:: 2.6 diff --git a/Doc/library/ossaudiodev.rst b/Doc/library/ossaudiodev.rst --- a/Doc/library/ossaudiodev.rst +++ b/Doc/library/ossaudiodev.rst @@ -48,7 +48,7 @@ the official documentation for the OSS C API The module defines a large number of constants supplied by the OSS device - driver; see ```` on either Linux or FreeBSD for a listing . + driver; see ```` on either Linux or FreeBSD for a listing. :mod:`ossaudiodev` defines the following variables and functions: diff --git a/Doc/library/pyexpat.rst b/Doc/library/pyexpat.rst --- a/Doc/library/pyexpat.rst +++ b/Doc/library/pyexpat.rst @@ -906,5 +906,5 @@ .. [#] The encoding string included in XML output should conform to the appropriate standards. For example, "UTF-8" is valid, but "UTF8" is not. See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl - and http://www.iana.org/assignments/character-sets . + and http://www.iana.org/assignments/character-sets\ . diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -1077,5 +1077,5 @@ .. rubric:: Citations -.. [C99] ISO/IEC 9899:1999. "Programming languages -- C." A public draft of this standard is available at http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf . +.. [C99] ISO/IEC 9899:1999. "Programming languages -- C." A public draft of this standard is available at http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf\ . diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -188,7 +188,7 @@ The Tk/Tcl development is largely taking place at ActiveState. `Tcl and the Tk Toolkit `_ - The book by John Ousterhout, the inventor of Tcl . + The book by John Ousterhout, the inventor of Tcl. `Practical Programming in Tcl and Tk `_ Brent Welch's encyclopedic book. @@ -625,7 +625,7 @@ preceded with an ``@``, as in ``"@/usr/contrib/bitmap/gumby.bit"``. boolean - You can pass integers 0 or 1 or the strings ``"yes"`` or ``"no"`` . + You can pass integers 0 or 1 or the strings ``"yes"`` or ``"no"``. callback This is any Python function that takes no arguments. For example:: diff --git a/Doc/library/trace.rst b/Doc/library/trace.rst --- a/Doc/library/trace.rst +++ b/Doc/library/trace.rst @@ -41,7 +41,7 @@ At least one of the following options must be specified when invoking :mod:`trace`. The :option:`--listfuncs <-l>` option is mutually exclusive with -the :option:`--trace <-t>` and :option:`--counts <-c>` options . When +the :option:`--trace <-t>` and :option:`--counts <-c>` options. When :option:`--listfuncs <-l>` is provided, neither :option:`--counts <-c>` nor :option:`--trace <-t>` are accepted, and vice versa. diff --git a/Doc/library/xml.dom.minidom.rst b/Doc/library/xml.dom.minidom.rst --- a/Doc/library/xml.dom.minidom.rst +++ b/Doc/library/xml.dom.minidom.rst @@ -276,4 +276,4 @@ .. [#] The encoding string included in XML output should conform to the appropriate standards. For example, "UTF-8" is valid, but "UTF8" is not. See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl - and http://www.iana.org/assignments/character-sets . + and http://www.iana.org/assignments/character-sets\ . -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 17:22:23 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 23 Dec 2013 17:22:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Removed_spaces?= =?utf-8?q?_before_commas_and_periods=2E?= Message-ID: <3dp5SH5d2Gz7Ljg@mail.python.org> http://hg.python.org/cpython/rev/86e54df4312a changeset: 88151:86e54df4312a branch: 3.3 parent: 88137:9529c6c993fe user: Serhiy Storchaka date: Mon Dec 23 18:20:51 2013 +0200 summary: Removed spaces before commas and periods. files: Doc/faq/library.rst | 2 +- Doc/howto/pyporting.rst | 2 +- Doc/howto/urllib2.rst | 2 +- Doc/library/ctypes.rst | 2 +- Doc/library/http.cookiejar.rst | 2 +- Doc/library/itertools.rst | 4 ++-- Doc/library/logging.rst | 2 +- Doc/library/ossaudiodev.rst | 2 +- Doc/library/pyexpat.rst | 2 +- Doc/library/re.rst | 2 +- Doc/library/stdtypes.rst | 2 +- Doc/library/sys.rst | 2 +- Doc/library/tkinter.rst | 4 ++-- Doc/library/trace.rst | 2 +- Doc/library/urllib.request.rst | 2 +- Doc/library/xml.dom.minidom.rst | 2 +- Doc/whatsnew/3.2.rst | 2 +- Doc/whatsnew/3.3.rst | 2 +- 18 files changed, 20 insertions(+), 20 deletions(-) diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -658,7 +658,7 @@ .. XXX check if wiki page is still up to date A summary of available frameworks is maintained by Paul Boddie at -http://wiki.python.org/moin/WebProgramming . +http://wiki.python.org/moin/WebProgramming\ . Cameron Laird maintains a useful set of pages about Python web technologies at http://phaseit.net/claird/comp.lang.python/web_python. diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -640,7 +640,7 @@ To get a complete idea of what issues you will need to deal with, see the `What's New in Python 3.0`_. Others have reorganized the data in other formats -such as http://docs.pythonsprints.com/python3_porting/py-porting.html . +such as http://docs.pythonsprints.com/python3_porting/py-porting.html\ . The following are some steps to take to try to support both Python 2 & 3 from the same source code. diff --git a/Doc/howto/urllib2.rst b/Doc/howto/urllib2.rst --- a/Doc/howto/urllib2.rst +++ b/Doc/howto/urllib2.rst @@ -160,7 +160,7 @@ to your HTTP request. Some websites [#]_ dislike being browsed by programs, or send different versions -to different browsers [#]_ . By default urllib identifies itself as +to different browsers [#]_. By default urllib identifies itself as ``Python-urllib/x.y`` (where ``x`` and ``y`` are the major and minor version numbers of the Python release, e.g. ``Python-urllib/2.5``), which may confuse the site, or just plain diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -1651,7 +1651,7 @@ WINUSERAPI int WINAPI MessageBoxA( - HWND hWnd , + HWND hWnd, LPCSTR lpText, LPCSTR lpCaption, UINT uType); diff --git a/Doc/library/http.cookiejar.rst b/Doc/library/http.cookiejar.rst --- a/Doc/library/http.cookiejar.rst +++ b/Doc/library/http.cookiejar.rst @@ -309,7 +309,7 @@ ----------------------------------------------------------- The following :class:`CookieJar` subclasses are provided for reading and -writing . +writing. .. class:: MozillaCookieJar(filename, delayload=None, policy=None) diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -56,7 +56,7 @@ :func:`islice` seq, [start,] stop [, step] elements from seq[start:stop:step] ``islice('ABCDEFG', 2, None) --> C D E F G`` :func:`starmap` func, seq func(\*seq[0]), func(\*seq[1]), ... ``starmap(pow, [(2,5), (3,2), (10,3)]) --> 32 9 1000`` :func:`takewhile` pred, seq seq[0], seq[1], until pred fails ``takewhile(lambda x: x<5, [1,4,6,4,1]) --> 1 4`` -:func:`tee` it, n it1, it2 , ... itn splits one iterator into n +:func:`tee` it, n it1, it2, ... itn splits one iterator into n :func:`zip_longest` p, q, ... (p[0], q[0]), (p[1], q[1]), ... ``zip_longest('ABCD', 'xy', fillvalue='-') --> Ax By C- D-`` ==================== ============================ ================================================= ============================================================= @@ -131,7 +131,7 @@ >>> inputs = repeat(x0, 36) # only the initial value is used >>> [format(x, '.2f') for x in accumulate(inputs, logistic_map)] ['0.40', '0.91', '0.30', '0.81', '0.60', '0.92', '0.29', '0.79', '0.63', - '0.88' ,'0.39', '0.90', '0.33', '0.84', '0.52', '0.95', '0.18', '0.57', + '0.88', '0.39', '0.90', '0.33', '0.84', '0.52', '0.95', '0.18', '0.57', '0.93', '0.25', '0.71', '0.79', '0.63', '0.88', '0.39', '0.91', '0.32', '0.83', '0.54', '0.95', '0.20', '0.60', '0.91', '0.30', '0.80', '0.60'] diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -809,7 +809,7 @@ --------------------- :class:`LoggerAdapter` instances are used to conveniently pass contextual -information into logging calls. For a usage example , see the section on +information into logging calls. For a usage example, see the section on :ref:`adding contextual information to your logging output `. .. class:: LoggerAdapter(logger, extra) diff --git a/Doc/library/ossaudiodev.rst b/Doc/library/ossaudiodev.rst --- a/Doc/library/ossaudiodev.rst +++ b/Doc/library/ossaudiodev.rst @@ -49,7 +49,7 @@ the official documentation for the OSS C API The module defines a large number of constants supplied by the OSS device - driver; see ```` on either Linux or FreeBSD for a listing . + driver; see ```` on either Linux or FreeBSD for a listing. :mod:`ossaudiodev` defines the following variables and functions: diff --git a/Doc/library/pyexpat.rst b/Doc/library/pyexpat.rst --- a/Doc/library/pyexpat.rst +++ b/Doc/library/pyexpat.rst @@ -861,5 +861,5 @@ .. [#] The encoding string included in XML output should conform to the appropriate standards. For example, "UTF-8" is valid, but "UTF8" is not. See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl - and http://www.iana.org/assignments/character-sets . + and http://www.iana.org/assignments/character-sets\ . diff --git a/Doc/library/re.rst b/Doc/library/re.rst --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -317,7 +317,7 @@ optional and can be omitted. For example, ``(<)?(\w+@\w+(?:\.\w+)+)(?(1)>|$)`` is a poor email matching pattern, which will match with ``''`` as well as ``'user at host.com'``, but - not with ``''`` . + not with ``''``. The special sequences consist of ``'\'`` and a character from the list below. diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1583,7 +1583,7 @@ .. method:: str.format_map(mapping) Similar to ``str.format(**mapping)``, except that ``mapping`` is - used directly and not copied to a :class:`dict` . This is useful + used directly and not copied to a :class:`dict`. This is useful if for example ``mapping`` is a dict subclass: >>> class Default(dict): diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -1187,5 +1187,5 @@ .. rubric:: Citations -.. [C99] ISO/IEC 9899:1999. "Programming languages -- C." A public draft of this standard is available at http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf . +.. [C99] ISO/IEC 9899:1999. "Programming languages -- C." A public draft of this standard is available at http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf\ . diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -180,7 +180,7 @@ The Tk/Tcl development is largely taking place at ActiveState. `Tcl and the Tk Toolkit `_ - The book by John Ousterhout, the inventor of Tcl . + The book by John Ousterhout, the inventor of Tcl. `Practical Programming in Tcl and Tk `_ Brent Welch's encyclopedic book. @@ -613,7 +613,7 @@ preceded with an ``@``, as in ``"@/usr/contrib/bitmap/gumby.bit"``. boolean - You can pass integers 0 or 1 or the strings ``"yes"`` or ``"no"`` . + You can pass integers 0 or 1 or the strings ``"yes"`` or ``"no"``. callback This is any Python function that takes no arguments. For example:: diff --git a/Doc/library/trace.rst b/Doc/library/trace.rst --- a/Doc/library/trace.rst +++ b/Doc/library/trace.rst @@ -41,7 +41,7 @@ At least one of the following options must be specified when invoking :mod:`trace`. The :option:`--listfuncs <-l>` option is mutually exclusive with -the :option:`--trace <-t>` and :option:`--counts <-c>` options . When +the :option:`--trace <-t>` and :option:`--counts <-c>` options. When :option:`--listfuncs <-l>` is provided, neither :option:`--counts <-c>` nor :option:`--trace <-t>` are accepted, and vice versa. diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -1072,7 +1072,7 @@ the returned bytes object to string once it determines or guesses the appropriate encoding. -The following W3C document, http://www.w3.org/International/O-charset , lists +The following W3C document, http://www.w3.org/International/O-charset\ , lists the various ways in which a (X)HTML or a XML document could have specified its encoding information. diff --git a/Doc/library/xml.dom.minidom.rst b/Doc/library/xml.dom.minidom.rst --- a/Doc/library/xml.dom.minidom.rst +++ b/Doc/library/xml.dom.minidom.rst @@ -252,4 +252,4 @@ "UTF8" is not valid in an XML document's declaration, even though Python accepts it as an encoding name. See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl - and http://www.iana.org/assignments/character-sets . + and http://www.iana.org/assignments/character-sets\ . diff --git a/Doc/whatsnew/3.2.rst b/Doc/whatsnew/3.2.rst --- a/Doc/whatsnew/3.2.rst +++ b/Doc/whatsnew/3.2.rst @@ -2469,7 +2469,7 @@ In addition to the existing Subversion code repository at http://svn.python.org there is now a `Mercurial `_ repository at -http://hg.python.org/ . +http://hg.python.org/\ . After the 3.2 release, there are plans to switch to Mercurial as the primary repository. This distributed version control system should make it easier for diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -194,7 +194,7 @@ are still permitted, but will always compare as unequal, regardless of view contents. -* For further changes see `Build and C API Changes`_ and `Porting C code`_ . +* For further changes see `Build and C API Changes`_ and `Porting C code`_. (Contributed by Stefan Krah in :issue:`10181`) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 17:22:25 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Mon, 23 Dec 2013 17:22:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Removed_spaces_before_commas_and_periods=2E?= Message-ID: <3dp5SK2Lwmz7Lm4@mail.python.org> http://hg.python.org/cpython/rev/26c938374e2e changeset: 88152:26c938374e2e parent: 88149:f670d8db8ef3 parent: 88151:86e54df4312a user: Serhiy Storchaka date: Mon Dec 23 18:21:57 2013 +0200 summary: Removed spaces before commas and periods. files: Doc/faq/library.rst | 2 +- Doc/howto/pyporting.rst | 2 +- Doc/howto/urllib2.rst | 2 +- Doc/library/ctypes.rst | 2 +- Doc/library/http.cookiejar.rst | 2 +- Doc/library/itertools.rst | 4 ++-- Doc/library/logging.rst | 2 +- Doc/library/ossaudiodev.rst | 2 +- Doc/library/pyexpat.rst | 2 +- Doc/library/re.rst | 2 +- Doc/library/stdtypes.rst | 2 +- Doc/library/sys.rst | 2 +- Doc/library/tkinter.rst | 4 ++-- Doc/library/trace.rst | 2 +- Doc/library/urllib.request.rst | 2 +- Doc/library/xml.dom.minidom.rst | 2 +- Doc/reference/import.rst | 2 +- Doc/whatsnew/3.2.rst | 2 +- Doc/whatsnew/3.3.rst | 2 +- Doc/whatsnew/3.4.rst | 2 +- 20 files changed, 22 insertions(+), 22 deletions(-) diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -662,7 +662,7 @@ .. XXX check if wiki page is still up to date A summary of available frameworks is maintained by Paul Boddie at -http://wiki.python.org/moin/WebProgramming . +http://wiki.python.org/moin/WebProgramming\ . Cameron Laird maintains a useful set of pages about Python web technologies at http://phaseit.net/claird/comp.lang.python/web_python. diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -640,7 +640,7 @@ To get a complete idea of what issues you will need to deal with, see the `What's New in Python 3.0`_. Others have reorganized the data in other formats -such as http://docs.pythonsprints.com/python3_porting/py-porting.html . +such as http://docs.pythonsprints.com/python3_porting/py-porting.html\ . The following are some steps to take to try to support both Python 2 & 3 from the same source code. diff --git a/Doc/howto/urllib2.rst b/Doc/howto/urllib2.rst --- a/Doc/howto/urllib2.rst +++ b/Doc/howto/urllib2.rst @@ -160,7 +160,7 @@ to your HTTP request. Some websites [#]_ dislike being browsed by programs, or send different versions -to different browsers [#]_ . By default urllib identifies itself as +to different browsers [#]_. By default urllib identifies itself as ``Python-urllib/x.y`` (where ``x`` and ``y`` are the major and minor version numbers of the Python release, e.g. ``Python-urllib/2.5``), which may confuse the site, or just plain diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -1651,7 +1651,7 @@ WINUSERAPI int WINAPI MessageBoxA( - HWND hWnd , + HWND hWnd, LPCSTR lpText, LPCSTR lpCaption, UINT uType); diff --git a/Doc/library/http.cookiejar.rst b/Doc/library/http.cookiejar.rst --- a/Doc/library/http.cookiejar.rst +++ b/Doc/library/http.cookiejar.rst @@ -309,7 +309,7 @@ ----------------------------------------------------------- The following :class:`CookieJar` subclasses are provided for reading and -writing . +writing. .. class:: MozillaCookieJar(filename, delayload=None, policy=None) diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -56,7 +56,7 @@ :func:`islice` seq, [start,] stop [, step] elements from seq[start:stop:step] ``islice('ABCDEFG', 2, None) --> C D E F G`` :func:`starmap` func, seq func(\*seq[0]), func(\*seq[1]), ... ``starmap(pow, [(2,5), (3,2), (10,3)]) --> 32 9 1000`` :func:`takewhile` pred, seq seq[0], seq[1], until pred fails ``takewhile(lambda x: x<5, [1,4,6,4,1]) --> 1 4`` -:func:`tee` it, n it1, it2 , ... itn splits one iterator into n +:func:`tee` it, n it1, it2, ... itn splits one iterator into n :func:`zip_longest` p, q, ... (p[0], q[0]), (p[1], q[1]), ... ``zip_longest('ABCD', 'xy', fillvalue='-') --> Ax By C- D-`` ==================== ============================ ================================================= ============================================================= @@ -131,7 +131,7 @@ >>> inputs = repeat(x0, 36) # only the initial value is used >>> [format(x, '.2f') for x in accumulate(inputs, logistic_map)] ['0.40', '0.91', '0.30', '0.81', '0.60', '0.92', '0.29', '0.79', '0.63', - '0.88' ,'0.39', '0.90', '0.33', '0.84', '0.52', '0.95', '0.18', '0.57', + '0.88', '0.39', '0.90', '0.33', '0.84', '0.52', '0.95', '0.18', '0.57', '0.93', '0.25', '0.71', '0.79', '0.63', '0.88', '0.39', '0.91', '0.32', '0.83', '0.54', '0.95', '0.20', '0.60', '0.91', '0.30', '0.80', '0.60'] diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -809,7 +809,7 @@ --------------------- :class:`LoggerAdapter` instances are used to conveniently pass contextual -information into logging calls. For a usage example , see the section on +information into logging calls. For a usage example, see the section on :ref:`adding contextual information to your logging output `. .. class:: LoggerAdapter(logger, extra) diff --git a/Doc/library/ossaudiodev.rst b/Doc/library/ossaudiodev.rst --- a/Doc/library/ossaudiodev.rst +++ b/Doc/library/ossaudiodev.rst @@ -49,7 +49,7 @@ the official documentation for the OSS C API The module defines a large number of constants supplied by the OSS device - driver; see ```` on either Linux or FreeBSD for a listing . + driver; see ```` on either Linux or FreeBSD for a listing. :mod:`ossaudiodev` defines the following variables and functions: diff --git a/Doc/library/pyexpat.rst b/Doc/library/pyexpat.rst --- a/Doc/library/pyexpat.rst +++ b/Doc/library/pyexpat.rst @@ -861,5 +861,5 @@ .. [#] The encoding string included in XML output should conform to the appropriate standards. For example, "UTF-8" is valid, but "UTF8" is not. See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl - and http://www.iana.org/assignments/character-sets . + and http://www.iana.org/assignments/character-sets\ . diff --git a/Doc/library/re.rst b/Doc/library/re.rst --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -317,7 +317,7 @@ optional and can be omitted. For example, ``(<)?(\w+@\w+(?:\.\w+)+)(?(1)>|$)`` is a poor email matching pattern, which will match with ``''`` as well as ``'user at host.com'``, but - not with ``''`` . + not with ``''``. The special sequences consist of ``'\'`` and a character from the list below. diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1583,7 +1583,7 @@ .. method:: str.format_map(mapping) Similar to ``str.format(**mapping)``, except that ``mapping`` is - used directly and not copied to a :class:`dict` . This is useful + used directly and not copied to a :class:`dict`. This is useful if for example ``mapping`` is a dict subclass: >>> class Default(dict): diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -1221,5 +1221,5 @@ .. rubric:: Citations -.. [C99] ISO/IEC 9899:1999. "Programming languages -- C." A public draft of this standard is available at http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf . +.. [C99] ISO/IEC 9899:1999. "Programming languages -- C." A public draft of this standard is available at http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf\ . diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -180,7 +180,7 @@ The Tk/Tcl development is largely taking place at ActiveState. `Tcl and the Tk Toolkit `_ - The book by John Ousterhout, the inventor of Tcl . + The book by John Ousterhout, the inventor of Tcl. `Practical Programming in Tcl and Tk `_ Brent Welch's encyclopedic book. @@ -613,7 +613,7 @@ preceded with an ``@``, as in ``"@/usr/contrib/bitmap/gumby.bit"``. boolean - You can pass integers 0 or 1 or the strings ``"yes"`` or ``"no"`` . + You can pass integers 0 or 1 or the strings ``"yes"`` or ``"no"``. callback This is any Python function that takes no arguments. For example:: diff --git a/Doc/library/trace.rst b/Doc/library/trace.rst --- a/Doc/library/trace.rst +++ b/Doc/library/trace.rst @@ -41,7 +41,7 @@ At least one of the following options must be specified when invoking :mod:`trace`. The :option:`--listfuncs <-l>` option is mutually exclusive with -the :option:`--trace <-t>` and :option:`--counts <-c>` options . When +the :option:`--trace <-t>` and :option:`--counts <-c>` options. When :option:`--listfuncs <-l>` is provided, neither :option:`--counts <-c>` nor :option:`--trace <-t>` are accepted, and vice versa. diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -1055,7 +1055,7 @@ the returned bytes object to string once it determines or guesses the appropriate encoding. -The following W3C document, http://www.w3.org/International/O-charset , lists +The following W3C document, http://www.w3.org/International/O-charset\ , lists the various ways in which a (X)HTML or a XML document could have specified its encoding information. diff --git a/Doc/library/xml.dom.minidom.rst b/Doc/library/xml.dom.minidom.rst --- a/Doc/library/xml.dom.minidom.rst +++ b/Doc/library/xml.dom.minidom.rst @@ -252,4 +252,4 @@ "UTF8" is not valid in an XML document's declaration, even though Python accepts it as an encoding name. See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl - and http://www.iana.org/assignments/character-sets . + and http://www.iana.org/assignments/character-sets\ . diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -626,7 +626,7 @@ As mentioned previously, Python comes with several default meta path finders. One of these, called the :term:`path based finder` -(:class:`~importlib.machinery.PathFinder`) , searches an :term:`import path`, +(:class:`~importlib.machinery.PathFinder`), searches an :term:`import path`, which contains a list of :term:`path entries `. Each path entry names a location to search for modules. diff --git a/Doc/whatsnew/3.2.rst b/Doc/whatsnew/3.2.rst --- a/Doc/whatsnew/3.2.rst +++ b/Doc/whatsnew/3.2.rst @@ -2469,7 +2469,7 @@ In addition to the existing Subversion code repository at http://svn.python.org there is now a `Mercurial `_ repository at -http://hg.python.org/ . +http://hg.python.org/\ . After the 3.2 release, there are plans to switch to Mercurial as the primary repository. This distributed version control system should make it easier for diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -194,7 +194,7 @@ are still permitted, but will always compare as unequal, regardless of view contents. -* For further changes see `Build and C API Changes`_ and `Porting C code`_ . +* For further changes see `Build and C API Changes`_ and `Porting C code`_. (Contributed by Stefan Krah in :issue:`10181`) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -275,7 +275,7 @@ The binary and text transforms provided in the standard library are detailed in :ref:`binary-transforms` and :ref:`text-transforms`. -(Contributed by Nick Coghlan in :issue:`7475`, , :issue:`17827`, +(Contributed by Nick Coghlan in :issue:`7475`, :issue:`17827`, :issue:`17828` and :issue:`19619`) .. _whatsnew-pep-451: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 17:26:11 2013 From: python-checkins at python.org (r.david.murray) Date: Mon, 23 Dec 2013 17:26:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Make_summary_entry_format_?= =?utf-8?q?consistent_=28always_end_with_pep_or_issue_in_parens=29=2E?= Message-ID: <3dp5Xg65ylz7Ljg@mail.python.org> http://hg.python.org/cpython/rev/054f7d1979d8 changeset: 88153:054f7d1979d8 user: R David Murray date: Mon Dec 23 10:28:57 2013 -0500 summary: Make summary entry format consistent (always end with pep or issue in parens). files: Doc/whatsnew/3.4.rst | 20 +++++++++++--------- 1 files changed, 11 insertions(+), 9 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -94,35 +94,37 @@ (:pep:`446`). * command line option for :ref:`isolated mode `, (:issue:`16499`). -* :ref:`improvements ` in the handling of - codecs that are not text encodings +* :ref:`improvements in the handling of codecs ` + that are not text encodings (multiple issues). * :ref:`A ModuleSpec Type ` for the Import System (:pep:`451`). (Affects importer authors.) New library modules: -* :mod:`asyncio`: New provisonal API for asynchronous IO (:pep:`3156`). +* :mod:`asyncio`: New provisional API for asynchronous IO (:pep:`3156`). * :mod:`enum`: Support for enumeration types (:pep:`435`). * :mod:`ensurepip`: Bootstrapping the pip installer (:pep:`453`). * :mod:`pathlib`: Object-oriented filesystem paths (:pep:`428`). * :mod:`selectors`: High-level and efficient I/O multiplexing, built upon the - :mod:`select` module primitives. + :mod:`select` module primitives (part of :pep:`3156`). * :mod:`statistics`: A basic numerically stable statistics library (:pep:`450`). * :mod:`tracemalloc`: Trace Python memory allocations (:pep:`454`). Significantly Improved Library Modules: * :ref:`Single-dispatch generic functions ` in - :mod:`functools` (:pep:`443`) -* New :mod:`pickle` :ref:`protocol 4 ` (:pep:`3154`) -* :ref:`SHA-3 (Keccak) support ` for :mod:`hashlib`. -* :ref:`TLSv1.1 and TLSv1.2 support ` for :mod:`ssl`. + :mod:`functools` (:pep:`443`). +* New :mod:`pickle` :ref:`protocol 4 ` (:pep:`3154`). +* :ref:`SHA-3 (Keccak) support ` for :mod:`hashlib` + (:issue:`16113`). +* :ref:`TLSv1.1 and TLSv1.2 support ` for :mod:`ssl` + (:issue:`16692`). * :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork on Unix ` (:issue:`8713`). * :mod:`email` has a new submodule, :mod:`~email.contentmanager`, and a new :mod:`~email.message.Message` subclass (:class:`~email.contentmanager.EmailMessage`) that :ref:`simplify MIME - handling `. + handling ` (:issue:`18891`). CPython implementation improvements: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 17:26:13 2013 From: python-checkins at python.org (r.david.murray) Date: Mon, 23 Dec 2013 17:26:13 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_link_to_selectors_desc?= =?utf-8?q?ription_from_its_summary_line=2E?= Message-ID: <3dp5Xj1C6sz7Lm4@mail.python.org> http://hg.python.org/cpython/rev/f83e8cf96c0a changeset: 88154:f83e8cf96c0a user: R David Murray date: Mon Dec 23 10:32:02 2013 -0500 summary: Add link to selectors description from its summary line. files: Doc/whatsnew/3.4.rst | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -105,8 +105,9 @@ * :mod:`enum`: Support for enumeration types (:pep:`435`). * :mod:`ensurepip`: Bootstrapping the pip installer (:pep:`453`). * :mod:`pathlib`: Object-oriented filesystem paths (:pep:`428`). -* :mod:`selectors`: High-level and efficient I/O multiplexing, built upon the - :mod:`select` module primitives (part of :pep:`3156`). +* :mod:`selectors`: :ref:`High-level and efficient I/O multiplexing + `, built upon the :mod:`select` module primitives (part + of :pep:`3156`). * :mod:`statistics`: A basic numerically stable statistics library (:pep:`450`). * :mod:`tracemalloc`: Trace Python memory allocations (:pep:`454`). @@ -378,6 +379,8 @@ PEP written and implemented by Antoine Pitrou. +.. _whatsnew-selectors: + selectors --------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 23 17:26:14 2013 From: python-checkins at python.org (r.david.murray) Date: Mon, 23 Dec 2013 17:26:14 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_pep_435_summary_entry_?= =?utf-8?q?in_new_required_features=2C_and_ensurepip_description=2E?= Message-ID: <3dp5Xk3FYkz7Llb@mail.python.org> http://hg.python.org/cpython/rev/40388ebb5aab changeset: 88155:40388ebb5aab user: R David Murray date: Mon Dec 23 11:17:51 2013 -0500 summary: Add pep 435 summary entry in new required features, and ensurepip description. Also added a note about platform packagers not being required to install pip by default, if they provide an appropriate pip hook, to the PEP description section. files: Doc/whatsnew/3.4.rst | 34 ++++++++++++++++++++++++++++++++ 1 files changed, 34 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -90,6 +90,7 @@ New expected features for Python implementations: +* :ref:`pip should always be "available" ` (:pep:`453`). * :ref:`Make newly created file descriptors non-inheritable ` (:pep:`446`). * command line option for :ref:`isolated mode `, @@ -146,6 +147,8 @@ New expected features for Python implementations ================================================ +.. _whatsnew-pep-453: + PEP 453: Explicit bootstrapping of pip in Python installations -------------------------------------------------------------- @@ -166,6 +169,12 @@ On Windows and Mac OS X, the CPython installers now offer the option to install ``pip`` along with CPython itself. +As `discussed in the PEP`__, platform packagers may choose not to install +``pip`` by default, as long as the command ``pip``, when invoked, provides +clear and simple directions on how to install ``pip`` on the platform. + +__ http://www.python.org/dev/peps/pep-0453/#recommendations-for-downstream-distributors + .. note:: The implementation of PEP 453 is still a work in progress. Refer to @@ -346,6 +355,31 @@ :pep:`3156` - Asynchronous IO Support Rebooted: the "asyncio" Module PEP written and implementation led by Guido van Rossum. + +.. _whatsnew-ensurepip + +ensurepip +--------- + +The new :mod:`ensurepip` module is the primary infrastructure for the +:pep:`453` implementation. In the normal course of events end users will not +need to interact with this module, but it can be used to manually bootstrap +``pip`` if the automated bootstrapping into an installation or virtual +environment was declined. + +:mod:`ensurepip` includes a bundled copy of ``pip``, up-to-date as of the first +release candidate of the release of CPython with which it ships (this applies +to both maintenance releases and feature releases). ``ensurepip`` does not +access the internet. (If the installation has Internet access, it is of course +possible to upgrade ``pip`` to a release more recent than the bundled ``pip`` +by using the bundled ``pip`` command itself once it is installed.) + +The module is named *ensure*\ pip because if called when ``pip`` is already +installed, it does nothing. It also has an ``--upgrade`` option that will +cause it to install the bundled copy of ``pip`` if the existing installed +version of ``pip`` is older than the bundled copy. + + enum ---- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 08:03:18 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 24 Dec 2013 08:03:18 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzIwMDMz?= =?utf-8?q?=3A_makelocalealias=2Epy_now_works_with_non-ASCII_locales_and_p?= =?utf-8?q?roduces?= Message-ID: <3dpT0k2ttsz7Ljv@mail.python.org> http://hg.python.org/cpython/rev/22c59ddba494 changeset: 88156:22c59ddba494 branch: 3.3 parent: 88151:86e54df4312a user: Serhiy Storchaka date: Mon Dec 23 18:56:08 2013 +0200 summary: Issue #20033: makelocalealias.py now works with non-ASCII locales and produces the same result as in 2.x. files: Tools/i18n/makelocalealias.py | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Tools/i18n/makelocalealias.py b/Tools/i18n/makelocalealias.py --- a/Tools/i18n/makelocalealias.py +++ b/Tools/i18n/makelocalealias.py @@ -13,8 +13,8 @@ def parse(filename): - f = open(filename) - lines = f.read().splitlines() + with open(filename, encoding='latin1') as f: + lines = list(f) data = {} for line in lines: line = line.strip() @@ -47,15 +47,15 @@ def pprint(data): items = sorted(data.items()) for k, v in items: - print(' %-40s%r,' % ('%r:' % k, v)) + print(' %-40s%a,' % ('%a:' % k, v)) def print_differences(data, olddata): items = sorted(olddata.items()) for k, v in items: if k not in data: - print('# removed %r' % k) + print('# removed %a' % k) elif olddata[k] != data[k]: - print('# updated %r -> %r to %r' % \ + print('# updated %a -> %a to %a' % \ (k, olddata[k], data[k])) # Additions are not mentioned -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 08:03:19 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 24 Dec 2013 08:03:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2320033=3A_makelocalealias=2Epy_now_works_with_no?= =?utf-8?q?n-ASCII_locales_and_produces?= Message-ID: <3dpT0l4bMkz7Ljv@mail.python.org> http://hg.python.org/cpython/rev/1287c570176b changeset: 88157:1287c570176b parent: 88152:26c938374e2e parent: 88156:22c59ddba494 user: Serhiy Storchaka date: Mon Dec 23 18:56:31 2013 +0200 summary: Issue #20033: makelocalealias.py now works with non-ASCII locales and produces the same result as in 2.x. files: Tools/i18n/makelocalealias.py | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Tools/i18n/makelocalealias.py b/Tools/i18n/makelocalealias.py --- a/Tools/i18n/makelocalealias.py +++ b/Tools/i18n/makelocalealias.py @@ -13,8 +13,8 @@ def parse(filename): - f = open(filename) - lines = f.read().splitlines() + with open(filename, encoding='latin1') as f: + lines = list(f) data = {} for line in lines: line = line.strip() @@ -47,15 +47,15 @@ def pprint(data): items = sorted(data.items()) for k, v in items: - print(' %-40s%r,' % ('%r:' % k, v)) + print(' %-40s%a,' % ('%a:' % k, v)) def print_differences(data, olddata): items = sorted(olddata.items()) for k, v in items: if k not in data: - print('# removed %r' % k) + print('# removed %a' % k) elif olddata[k] != data[k]: - print('# updated %r -> %r to %r' % \ + print('# updated %a -> %a to %a' % \ (k, olddata[k], data[k])) # Additions are not mentioned -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 08:03:21 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 24 Dec 2013 08:03:21 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_default_-=3E_default?= =?utf-8?q?=29=3A_Merge_heads?= Message-ID: <3dpT0n0SrWz7Lkl@mail.python.org> http://hg.python.org/cpython/rev/fee3dd2c5b7e changeset: 88158:fee3dd2c5b7e parent: 88157:1287c570176b parent: 88155:40388ebb5aab user: Serhiy Storchaka date: Tue Dec 24 09:02:24 2013 +0200 summary: Merge heads files: Doc/whatsnew/3.4.rst | 59 ++++++++++++++++++++++++++----- 1 files changed, 49 insertions(+), 10 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -90,39 +90,43 @@ New expected features for Python implementations: +* :ref:`pip should always be "available" ` (:pep:`453`). * :ref:`Make newly created file descriptors non-inheritable ` (:pep:`446`). * command line option for :ref:`isolated mode `, (:issue:`16499`). -* :ref:`improvements ` in the handling of - codecs that are not text encodings +* :ref:`improvements in the handling of codecs ` + that are not text encodings (multiple issues). * :ref:`A ModuleSpec Type ` for the Import System (:pep:`451`). (Affects importer authors.) New library modules: -* :mod:`asyncio`: New provisonal API for asynchronous IO (:pep:`3156`). +* :mod:`asyncio`: New provisional API for asynchronous IO (:pep:`3156`). * :mod:`enum`: Support for enumeration types (:pep:`435`). * :mod:`ensurepip`: Bootstrapping the pip installer (:pep:`453`). * :mod:`pathlib`: Object-oriented filesystem paths (:pep:`428`). -* :mod:`selectors`: High-level and efficient I/O multiplexing, built upon the - :mod:`select` module primitives. +* :mod:`selectors`: :ref:`High-level and efficient I/O multiplexing + `, built upon the :mod:`select` module primitives (part + of :pep:`3156`). * :mod:`statistics`: A basic numerically stable statistics library (:pep:`450`). * :mod:`tracemalloc`: Trace Python memory allocations (:pep:`454`). Significantly Improved Library Modules: * :ref:`Single-dispatch generic functions ` in - :mod:`functools` (:pep:`443`) -* New :mod:`pickle` :ref:`protocol 4 ` (:pep:`3154`) -* :ref:`SHA-3 (Keccak) support ` for :mod:`hashlib`. -* :ref:`TLSv1.1 and TLSv1.2 support ` for :mod:`ssl`. + :mod:`functools` (:pep:`443`). +* New :mod:`pickle` :ref:`protocol 4 ` (:pep:`3154`). +* :ref:`SHA-3 (Keccak) support ` for :mod:`hashlib` + (:issue:`16113`). +* :ref:`TLSv1.1 and TLSv1.2 support ` for :mod:`ssl` + (:issue:`16692`). * :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork on Unix ` (:issue:`8713`). * :mod:`email` has a new submodule, :mod:`~email.contentmanager`, and a new :mod:`~email.message.Message` subclass (:class:`~email.contentmanager.EmailMessage`) that :ref:`simplify MIME - handling `. + handling ` (:issue:`18891`). CPython implementation improvements: @@ -143,6 +147,8 @@ New expected features for Python implementations ================================================ +.. _whatsnew-pep-453: + PEP 453: Explicit bootstrapping of pip in Python installations -------------------------------------------------------------- @@ -163,6 +169,12 @@ On Windows and Mac OS X, the CPython installers now offer the option to install ``pip`` along with CPython itself. +As `discussed in the PEP`__, platform packagers may choose not to install +``pip`` by default, as long as the command ``pip``, when invoked, provides +clear and simple directions on how to install ``pip`` on the platform. + +__ http://www.python.org/dev/peps/pep-0453/#recommendations-for-downstream-distributors + .. note:: The implementation of PEP 453 is still a work in progress. Refer to @@ -343,6 +355,31 @@ :pep:`3156` - Asynchronous IO Support Rebooted: the "asyncio" Module PEP written and implementation led by Guido van Rossum. + +.. _whatsnew-ensurepip + +ensurepip +--------- + +The new :mod:`ensurepip` module is the primary infrastructure for the +:pep:`453` implementation. In the normal course of events end users will not +need to interact with this module, but it can be used to manually bootstrap +``pip`` if the automated bootstrapping into an installation or virtual +environment was declined. + +:mod:`ensurepip` includes a bundled copy of ``pip``, up-to-date as of the first +release candidate of the release of CPython with which it ships (this applies +to both maintenance releases and feature releases). ``ensurepip`` does not +access the internet. (If the installation has Internet access, it is of course +possible to upgrade ``pip`` to a release more recent than the bundled ``pip`` +by using the bundled ``pip`` command itself once it is installed.) + +The module is named *ensure*\ pip because if called when ``pip`` is already +installed, it does nothing. It also has an ``--upgrade`` option that will +cause it to install the bundled copy of ``pip`` if the existing installed +version of ``pip`` is older than the bundled copy. + + enum ---- @@ -376,6 +413,8 @@ PEP written and implemented by Antoine Pitrou. +.. _whatsnew-selectors: + selectors --------- -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Tue Dec 24 09:45:15 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 24 Dec 2013 09:45:15 +0100 Subject: [Python-checkins] Daily reference leaks (40388ebb5aab): sum=0 Message-ID: results for 40388ebb5aab on branch "default" -------------------------------------------- test_site leaked [-2, 0, 2] references, sum=0 test_site leaked [-2, 0, 2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/refloga8AxfF', '-x'] From python-checkins at python.org Tue Dec 24 10:08:46 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 24 Dec 2013 10:08:46 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Removed_spaces?= =?utf-8?q?_before_colons_and_semicolons=2E?= Message-ID: <3dpWnV6W95z7LjW@mail.python.org> http://hg.python.org/cpython/rev/ee924673d83a changeset: 88159:ee924673d83a branch: 2.7 parent: 88150:6aeef7be5752 user: Serhiy Storchaka date: Tue Dec 24 11:04:06 2013 +0200 summary: Removed spaces before colons and semicolons. files: Doc/faq/programming.rst | 2 +- Doc/faq/windows.rst | 2 +- Doc/howto/logging-cookbook.rst | 1 - Doc/howto/urllib2.rst | 6 +++--- Doc/library/ctypes.rst | 2 +- Doc/library/rexec.rst | 2 +- Doc/library/simplexmlrpcserver.rst | 2 +- Doc/library/telnetlib.rst | 2 +- Doc/library/tkinter.rst | 2 +- Doc/whatsnew/2.4.rst | 2 +- Doc/whatsnew/2.5.rst | 2 +- Doc/whatsnew/2.6.rst | 6 +++--- 12 files changed, 15 insertions(+), 16 deletions(-) diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -910,7 +910,7 @@ >>> a = array.array('c', s) >>> print a array('c', 'Hello, world') - >>> a[0] = 'y' ; print a + >>> a[0] = 'y'; print a array('c', 'yello, world') >>> a.tostring() 'yello, world' diff --git a/Doc/faq/windows.rst b/Doc/faq/windows.rst --- a/Doc/faq/windows.rst +++ b/Doc/faq/windows.rst @@ -243,7 +243,7 @@ ... Py_Initialize(); // Initialize Python. initmyAppc(); // Initialize (import) the helper class. - PyRun_SimpleString("import myApp") ; // Import the shadow class. + PyRun_SimpleString("import myApp"); // Import the shadow class. 5. There are two problems with Python's C API which will become apparent if you use a compiler other than MSVC, the compiler used to build pythonNN.dll. diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -834,4 +834,3 @@ Note that the order of items might be different according to the version of Python used. - diff --git a/Doc/howto/urllib2.rst b/Doc/howto/urllib2.rst --- a/Doc/howto/urllib2.rst +++ b/Doc/howto/urllib2.rst @@ -18,7 +18,7 @@ .. sidebar:: Related Articles You may also find useful the following article on fetching web resources - with Python : + with Python: * `Basic Authentication `_ @@ -439,7 +439,7 @@ When authentication is required, the server sends a header (as well as the 401 error code) requesting authentication. This specifies the authentication scheme -and a 'realm'. The header looks like : ``WWW-Authenticate: SCHEME +and a 'realm'. The header looks like: ``WWW-Authenticate: SCHEME realm="REALM"``. e.g. :: @@ -511,7 +511,7 @@ setting is detected. Normally that's a good thing, but there are occasions when it may not be helpful [#]_. One way to do this is to setup our own ``ProxyHandler``, with no proxies defined. This is done using similar steps to -setting up a `Basic Authentication`_ handler : :: +setting up a `Basic Authentication`_ handler: :: >>> proxy_support = urllib2.ProxyHandler({}) >>> opener = urllib2.build_opener(proxy_support) diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -215,7 +215,7 @@ Fundamental data types ^^^^^^^^^^^^^^^^^^^^^^ -:mod:`ctypes` defines a number of primitive C compatible data types : +:mod:`ctypes` defines a number of primitive C compatible data types: +----------------------+------------------------------------------+----------------------------+ | ctypes type | C type | Python type | diff --git a/Doc/library/rexec.rst b/Doc/library/rexec.rst --- a/Doc/library/rexec.rst +++ b/Doc/library/rexec.rst @@ -270,7 +270,7 @@ if mode in ('r', 'rb'): pass elif mode in ('w', 'wb', 'a', 'ab'): - # check filename : must begin with /tmp/ + # check filename: must begin with /tmp/ if file[:5]!='/tmp/': raise IOError("can't write outside /tmp") elif (string.find(file, '/../') >= 0 or diff --git a/Doc/library/simplexmlrpcserver.rst b/Doc/library/simplexmlrpcserver.rst --- a/Doc/library/simplexmlrpcserver.rst +++ b/Doc/library/simplexmlrpcserver.rst @@ -247,7 +247,7 @@ Example:: class MyFuncs: - def div(self, x, y) : return x // y + def div(self, x, y): return x // y handler = CGIXMLRPCRequestHandler() diff --git a/Doc/library/telnetlib.rst b/Doc/library/telnetlib.rst --- a/Doc/library/telnetlib.rst +++ b/Doc/library/telnetlib.rst @@ -208,7 +208,7 @@ .. method:: Telnet.set_option_negotiation_callback(callback) Each time a telnet option is read on the input flow, this *callback* (if set) is - called with the following parameters : callback(telnet socket, command + called with the following parameters: callback(telnet socket, command (DO/DONT/WILL/WONT), option). No other action is done afterwards by telnetlib. diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -452,7 +452,7 @@ Example:: >>> print fred.config() - {'relief' : ('relief', 'relief', 'Relief', 'raised', 'groove')} + {'relief': ('relief', 'relief', 'Relief', 'raised', 'groove')} Of course, the dictionary printed will include all the options available and their values. This is meant only as an example. diff --git a/Doc/whatsnew/2.4.rst b/Doc/whatsnew/2.4.rst --- a/Doc/whatsnew/2.4.rst +++ b/Doc/whatsnew/2.4.rst @@ -846,7 +846,7 @@ ['A', 'b', 'c', 'D'] Finally, the *reverse* parameter takes a Boolean value. If the value is true, - the list will be sorted into reverse order. Instead of ``L.sort() ; + the list will be sorted into reverse order. Instead of ``L.sort(); L.reverse()``, you can now write ``L.sort(reverse=True)``. The results of sorting are now guaranteed to be stable. This means that two diff --git a/Doc/whatsnew/2.5.rst b/Doc/whatsnew/2.5.rst --- a/Doc/whatsnew/2.5.rst +++ b/Doc/whatsnew/2.5.rst @@ -286,7 +286,7 @@ :mod:`pkg.string` and look for the standard module; generally you had to look at the contents of ``sys.modules``, which is slightly unclean. Holger Krekel's :mod:`py.std` package provides a tidier way to perform imports from the standard -library, ``import py ; py.std.string.join()``, but that package isn't available +library, ``import py; py.std.string.join()``, but that package isn't available on all Python installations. Reading code which relies on relative imports is also less clear, because a diff --git a/Doc/whatsnew/2.6.rst b/Doc/whatsnew/2.6.rst --- a/Doc/whatsnew/2.6.rst +++ b/Doc/whatsnew/2.6.rst @@ -1887,7 +1887,7 @@ >>> dq=deque(maxlen=3) >>> dq deque([], maxlen=3) - >>> dq.append(1) ; dq.append(2) ; dq.append(3) + >>> dq.append(1); dq.append(2); dq.append(3) >>> dq deque([1, 2, 3], maxlen=3) >>> dq.append(4) @@ -2779,12 +2779,12 @@ types. The following example encodes and decodes a dictionary:: >>> import json - >>> data = {"spam" : "foo", "parrot" : 42} + >>> data = {"spam": "foo", "parrot": 42} >>> in_json = json.dumps(data) # Encode the data >>> in_json '{"parrot": 42, "spam": "foo"}' >>> json.loads(in_json) # Decode into a Python object - {"spam" : "foo", "parrot" : 42} + {"spam": "foo", "parrot": 42} It's also possible to write your own decoders and encoders to support more types. Pretty-printing of the JSON strings is also supported. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 10:08:48 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 24 Dec 2013 10:08:48 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Removed_spaces?= =?utf-8?q?_before_colons_and_semicolons=2E?= Message-ID: <3dpWnX2mplz7Lk7@mail.python.org> http://hg.python.org/cpython/rev/99190ebd4935 changeset: 88160:99190ebd4935 branch: 3.3 parent: 88156:22c59ddba494 user: Serhiy Storchaka date: Tue Dec 24 11:04:36 2013 +0200 summary: Removed spaces before colons and semicolons. files: Doc/faq/windows.rst | 2 +- Doc/howto/logging-cookbook.rst | 4 ++-- Doc/howto/urllib2.rst | 4 ++-- Doc/library/collections.rst | 2 +- Doc/library/ctypes.rst | 2 +- Doc/library/telnetlib.rst | 2 +- Doc/library/tkinter.rst | 2 +- Doc/library/unittest.mock.rst | 22 +++++++++++----------- Doc/whatsnew/2.4.rst | 2 +- Doc/whatsnew/2.5.rst | 2 +- Doc/whatsnew/2.6.rst | 6 +++--- Doc/whatsnew/3.3.rst | 10 +++++----- 12 files changed, 30 insertions(+), 30 deletions(-) diff --git a/Doc/faq/windows.rst b/Doc/faq/windows.rst --- a/Doc/faq/windows.rst +++ b/Doc/faq/windows.rst @@ -247,7 +247,7 @@ ... Py_Initialize(); // Initialize Python. initmyAppc(); // Initialize (import) the helper class. - PyRun_SimpleString("import myApp") ; // Import the shadow class. + PyRun_SimpleString("import myApp"); // Import the shadow class. 5. There are two problems with Python's C API which will become apparent if you use a compiler other than MSVC, the compiler used to build pythonNN.dll. diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -841,7 +841,7 @@ }, 'loggers': { 'foo': { - 'handlers' : ['foofile'] + 'handlers': ['foofile'] } }, 'root': { @@ -1529,7 +1529,7 @@ }, 'loggers': { 'foo': { - 'handlers' : ['foofile'] + 'handlers': ['foofile'] } }, 'root': { diff --git a/Doc/howto/urllib2.rst b/Doc/howto/urllib2.rst --- a/Doc/howto/urllib2.rst +++ b/Doc/howto/urllib2.rst @@ -454,7 +454,7 @@ When authentication is required, the server sends a header (as well as the 401 error code) requesting authentication. This specifies the authentication scheme -and a 'realm'. The header looks like : ``WWW-Authenticate: SCHEME +and a 'realm'. The header looks like: ``WWW-Authenticate: SCHEME realm="REALM"``. e.g. :: @@ -526,7 +526,7 @@ setting is detected. Normally that's a good thing, but there are occasions when it may not be helpful [#]_. One way to do this is to setup our own ``ProxyHandler``, with no proxies defined. This is done using similar steps to -setting up a `Basic Authentication`_ handler : :: +setting up a `Basic Authentication`_ handler: :: >>> proxy_support = urllib.request.ProxyHandler({}) >>> opener = urllib.request.build_opener(proxy_support) diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -188,7 +188,7 @@ return raise KeyError(key) - >>> d = DeepChainMap({'zebra': 'black'}, {'elephant' : 'blue'}, {'lion' : 'yellow'}) + >>> d = DeepChainMap({'zebra': 'black'}, {'elephant': 'blue'}, {'lion': 'yellow'}) >>> d['lion'] = 'orange' # update an existing key two levels down >>> d['snake'] = 'red' # new keys get added to the topmost dict >>> del d['elephant'] # remove an existing key one level down diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -218,7 +218,7 @@ Fundamental data types ^^^^^^^^^^^^^^^^^^^^^^ -:mod:`ctypes` defines a number of primitive C compatible data types : +:mod:`ctypes` defines a number of primitive C compatible data types: +----------------------+------------------------------------------+----------------------------+ | ctypes type | C type | Python type | diff --git a/Doc/library/telnetlib.rst b/Doc/library/telnetlib.rst --- a/Doc/library/telnetlib.rst +++ b/Doc/library/telnetlib.rst @@ -205,7 +205,7 @@ .. method:: Telnet.set_option_negotiation_callback(callback) Each time a telnet option is read on the input flow, this *callback* (if set) is - called with the following parameters : callback(telnet socket, command + called with the following parameters: callback(telnet socket, command (DO/DONT/WILL/WONT), option). No other action is done afterwards by telnetlib. diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -440,7 +440,7 @@ Example:: >>> print(fred.config()) - {'relief' : ('relief', 'relief', 'Relief', 'raised', 'groove')} + {'relief': ('relief', 'relief', 'Relief', 'raised', 'groove')} Of course, the dictionary printed will include all the options available and their values. This is meant only as an example. diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -1632,17 +1632,17 @@ * ``__gt__``: NotImplemented * ``__le__``: NotImplemented * ``__ge__``: NotImplemented -* ``__int__`` : 1 -* ``__contains__`` : False -* ``__len__`` : 1 -* ``__iter__`` : iter([]) -* ``__exit__`` : False -* ``__complex__`` : 1j -* ``__float__`` : 1.0 -* ``__bool__`` : True -* ``__index__`` : 1 -* ``__hash__`` : default hash for the mock -* ``__str__`` : default str for the mock +* ``__int__``: 1 +* ``__contains__``: False +* ``__len__``: 1 +* ``__iter__``: iter([]) +* ``__exit__``: False +* ``__complex__``: 1j +* ``__float__``: 1.0 +* ``__bool__``: True +* ``__index__``: 1 +* ``__hash__``: default hash for the mock +* ``__str__``: default str for the mock * ``__sizeof__``: default sizeof for the mock For example: diff --git a/Doc/whatsnew/2.4.rst b/Doc/whatsnew/2.4.rst --- a/Doc/whatsnew/2.4.rst +++ b/Doc/whatsnew/2.4.rst @@ -846,7 +846,7 @@ ['A', 'b', 'c', 'D'] Finally, the *reverse* parameter takes a Boolean value. If the value is true, - the list will be sorted into reverse order. Instead of ``L.sort() ; + the list will be sorted into reverse order. Instead of ``L.sort(); L.reverse()``, you can now write ``L.sort(reverse=True)``. The results of sorting are now guaranteed to be stable. This means that two diff --git a/Doc/whatsnew/2.5.rst b/Doc/whatsnew/2.5.rst --- a/Doc/whatsnew/2.5.rst +++ b/Doc/whatsnew/2.5.rst @@ -286,7 +286,7 @@ :mod:`pkg.string` and look for the standard module; generally you had to look at the contents of ``sys.modules``, which is slightly unclean. Holger Krekel's :mod:`py.std` package provides a tidier way to perform imports from the standard -library, ``import py ; py.std.string.join()``, but that package isn't available +library, ``import py; py.std.string.join()``, but that package isn't available on all Python installations. Reading code which relies on relative imports is also less clear, because a diff --git a/Doc/whatsnew/2.6.rst b/Doc/whatsnew/2.6.rst --- a/Doc/whatsnew/2.6.rst +++ b/Doc/whatsnew/2.6.rst @@ -1891,7 +1891,7 @@ >>> dq=deque(maxlen=3) >>> dq deque([], maxlen=3) - >>> dq.append(1) ; dq.append(2) ; dq.append(3) + >>> dq.append(1); dq.append(2); dq.append(3) >>> dq deque([1, 2, 3], maxlen=3) >>> dq.append(4) @@ -2783,12 +2783,12 @@ types. The following example encodes and decodes a dictionary:: >>> import json - >>> data = {"spam" : "foo", "parrot" : 42} + >>> data = {"spam": "foo", "parrot": 42} >>> in_json = json.dumps(data) # Encode the data >>> in_json '{"parrot": 42, "spam": "foo"}' >>> json.loads(in_json) # Decode into a Python object - {"spam" : "foo", "parrot" : 42} + {"spam": "foo", "parrot": 42} It's also possible to write your own decoders and encoders to support more types. Pretty-printing of the JSON strings is also supported. diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -1823,12 +1823,12 @@ * The :mod:`signal` module has new functions: * :func:`~signal.pthread_sigmask`: fetch and/or change the signal mask of the - calling thread (Contributed by Jean-Paul Calderone in :issue:`8407`) ; - * :func:`~signal.pthread_kill`: send a signal to a thread ; - * :func:`~signal.sigpending`: examine pending functions ; - * :func:`~signal.sigwait`: wait a signal. + calling thread (Contributed by Jean-Paul Calderone in :issue:`8407`); + * :func:`~signal.pthread_kill`: send a signal to a thread; + * :func:`~signal.sigpending`: examine pending functions; + * :func:`~signal.sigwait`: wait a signal; * :func:`~signal.sigwaitinfo`: wait for a signal, returning detailed - information about it. + information about it; * :func:`~signal.sigtimedwait`: like :func:`~signal.sigwaitinfo` but with a timeout. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 10:08:49 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Tue, 24 Dec 2013 10:08:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Removed_spaces_before_colons_and_semicolons=2E?= Message-ID: <3dpWnY602zz7Ll4@mail.python.org> http://hg.python.org/cpython/rev/4de75eee6c4e changeset: 88161:4de75eee6c4e parent: 88158:fee3dd2c5b7e parent: 88160:99190ebd4935 user: Serhiy Storchaka date: Tue Dec 24 11:05:24 2013 +0200 summary: Removed spaces before colons and semicolons. files: Doc/faq/windows.rst | 2 +- Doc/howto/logging-cookbook.rst | 4 ++-- Doc/howto/urllib2.rst | 4 ++-- Doc/library/collections.rst | 2 +- Doc/library/ctypes.rst | 2 +- Doc/library/telnetlib.rst | 2 +- Doc/library/tkinter.rst | 2 +- Doc/library/unittest.mock.rst | 22 +++++++++++----------- Doc/whatsnew/2.4.rst | 2 +- Doc/whatsnew/2.5.rst | 2 +- Doc/whatsnew/2.6.rst | 6 +++--- Doc/whatsnew/3.3.rst | 10 +++++----- 12 files changed, 30 insertions(+), 30 deletions(-) diff --git a/Doc/faq/windows.rst b/Doc/faq/windows.rst --- a/Doc/faq/windows.rst +++ b/Doc/faq/windows.rst @@ -247,7 +247,7 @@ ... Py_Initialize(); // Initialize Python. initmyAppc(); // Initialize (import) the helper class. - PyRun_SimpleString("import myApp") ; // Import the shadow class. + PyRun_SimpleString("import myApp"); // Import the shadow class. 5. There are two problems with Python's C API which will become apparent if you use a compiler other than MSVC, the compiler used to build pythonNN.dll. diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -839,7 +839,7 @@ }, 'loggers': { 'foo': { - 'handlers' : ['foofile'] + 'handlers': ['foofile'] } }, 'root': { @@ -1527,7 +1527,7 @@ }, 'loggers': { 'foo': { - 'handlers' : ['foofile'] + 'handlers': ['foofile'] } }, 'root': { diff --git a/Doc/howto/urllib2.rst b/Doc/howto/urllib2.rst --- a/Doc/howto/urllib2.rst +++ b/Doc/howto/urllib2.rst @@ -454,7 +454,7 @@ When authentication is required, the server sends a header (as well as the 401 error code) requesting authentication. This specifies the authentication scheme -and a 'realm'. The header looks like : ``WWW-Authenticate: SCHEME +and a 'realm'. The header looks like: ``WWW-Authenticate: SCHEME realm="REALM"``. e.g. :: @@ -526,7 +526,7 @@ setting is detected. Normally that's a good thing, but there are occasions when it may not be helpful [#]_. One way to do this is to setup our own ``ProxyHandler``, with no proxies defined. This is done using similar steps to -setting up a `Basic Authentication`_ handler : :: +setting up a `Basic Authentication`_ handler: :: >>> proxy_support = urllib.request.ProxyHandler({}) >>> opener = urllib.request.build_opener(proxy_support) diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -193,7 +193,7 @@ return raise KeyError(key) - >>> d = DeepChainMap({'zebra': 'black'}, {'elephant' : 'blue'}, {'lion' : 'yellow'}) + >>> d = DeepChainMap({'zebra': 'black'}, {'elephant': 'blue'}, {'lion': 'yellow'}) >>> d['lion'] = 'orange' # update an existing key two levels down >>> d['snake'] = 'red' # new keys get added to the topmost dict >>> del d['elephant'] # remove an existing key one level down diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -218,7 +218,7 @@ Fundamental data types ^^^^^^^^^^^^^^^^^^^^^^ -:mod:`ctypes` defines a number of primitive C compatible data types : +:mod:`ctypes` defines a number of primitive C compatible data types: +----------------------+------------------------------------------+----------------------------+ | ctypes type | C type | Python type | diff --git a/Doc/library/telnetlib.rst b/Doc/library/telnetlib.rst --- a/Doc/library/telnetlib.rst +++ b/Doc/library/telnetlib.rst @@ -205,7 +205,7 @@ .. method:: Telnet.set_option_negotiation_callback(callback) Each time a telnet option is read on the input flow, this *callback* (if set) is - called with the following parameters : callback(telnet socket, command + called with the following parameters: callback(telnet socket, command (DO/DONT/WILL/WONT), option). No other action is done afterwards by telnetlib. diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -440,7 +440,7 @@ Example:: >>> print(fred.config()) - {'relief' : ('relief', 'relief', 'Relief', 'raised', 'groove')} + {'relief': ('relief', 'relief', 'Relief', 'raised', 'groove')} Of course, the dictionary printed will include all the options available and their values. This is meant only as an example. diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -1652,17 +1652,17 @@ * ``__gt__``: NotImplemented * ``__le__``: NotImplemented * ``__ge__``: NotImplemented -* ``__int__`` : 1 -* ``__contains__`` : False -* ``__len__`` : 1 -* ``__iter__`` : iter([]) -* ``__exit__`` : False -* ``__complex__`` : 1j -* ``__float__`` : 1.0 -* ``__bool__`` : True -* ``__index__`` : 1 -* ``__hash__`` : default hash for the mock -* ``__str__`` : default str for the mock +* ``__int__``: 1 +* ``__contains__``: False +* ``__len__``: 1 +* ``__iter__``: iter([]) +* ``__exit__``: False +* ``__complex__``: 1j +* ``__float__``: 1.0 +* ``__bool__``: True +* ``__index__``: 1 +* ``__hash__``: default hash for the mock +* ``__str__``: default str for the mock * ``__sizeof__``: default sizeof for the mock For example: diff --git a/Doc/whatsnew/2.4.rst b/Doc/whatsnew/2.4.rst --- a/Doc/whatsnew/2.4.rst +++ b/Doc/whatsnew/2.4.rst @@ -846,7 +846,7 @@ ['A', 'b', 'c', 'D'] Finally, the *reverse* parameter takes a Boolean value. If the value is true, - the list will be sorted into reverse order. Instead of ``L.sort() ; + the list will be sorted into reverse order. Instead of ``L.sort(); L.reverse()``, you can now write ``L.sort(reverse=True)``. The results of sorting are now guaranteed to be stable. This means that two diff --git a/Doc/whatsnew/2.5.rst b/Doc/whatsnew/2.5.rst --- a/Doc/whatsnew/2.5.rst +++ b/Doc/whatsnew/2.5.rst @@ -286,7 +286,7 @@ :mod:`pkg.string` and look for the standard module; generally you had to look at the contents of ``sys.modules``, which is slightly unclean. Holger Krekel's :mod:`py.std` package provides a tidier way to perform imports from the standard -library, ``import py ; py.std.string.join()``, but that package isn't available +library, ``import py; py.std.string.join()``, but that package isn't available on all Python installations. Reading code which relies on relative imports is also less clear, because a diff --git a/Doc/whatsnew/2.6.rst b/Doc/whatsnew/2.6.rst --- a/Doc/whatsnew/2.6.rst +++ b/Doc/whatsnew/2.6.rst @@ -1891,7 +1891,7 @@ >>> dq=deque(maxlen=3) >>> dq deque([], maxlen=3) - >>> dq.append(1) ; dq.append(2) ; dq.append(3) + >>> dq.append(1); dq.append(2); dq.append(3) >>> dq deque([1, 2, 3], maxlen=3) >>> dq.append(4) @@ -2783,12 +2783,12 @@ types. The following example encodes and decodes a dictionary:: >>> import json - >>> data = {"spam" : "foo", "parrot" : 42} + >>> data = {"spam": "foo", "parrot": 42} >>> in_json = json.dumps(data) # Encode the data >>> in_json '{"parrot": 42, "spam": "foo"}' >>> json.loads(in_json) # Decode into a Python object - {"spam" : "foo", "parrot" : 42} + {"spam": "foo", "parrot": 42} It's also possible to write your own decoders and encoders to support more types. Pretty-printing of the JSON strings is also supported. diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -1823,12 +1823,12 @@ * The :mod:`signal` module has new functions: * :func:`~signal.pthread_sigmask`: fetch and/or change the signal mask of the - calling thread (Contributed by Jean-Paul Calderone in :issue:`8407`) ; - * :func:`~signal.pthread_kill`: send a signal to a thread ; - * :func:`~signal.sigpending`: examine pending functions ; - * :func:`~signal.sigwait`: wait a signal. + calling thread (Contributed by Jean-Paul Calderone in :issue:`8407`); + * :func:`~signal.pthread_kill`: send a signal to a thread; + * :func:`~signal.sigpending`: examine pending functions; + * :func:`~signal.sigwait`: wait a signal; * :func:`~signal.sigwaitinfo`: wait for a signal, returning detailed - information about it. + information about it; * :func:`~signal.sigtimedwait`: like :func:`~signal.sigwaitinfo` but with a timeout. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 17:13:05 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 24 Dec 2013 17:13:05 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Link_new_module_summary_en?= =?utf-8?q?tries_to_their_description_sections=2E?= Message-ID: <3dpjC50B5XzMtV@mail.python.org> http://hg.python.org/cpython/rev/f8414b280cba changeset: 88162:f8414b280cba user: R David Murray date: Mon Dec 23 21:08:28 2013 -0500 summary: Link new module summary entries to their description sections. files: Doc/whatsnew/3.4.rst | 31 +++++++++++++++++++++++-------- 1 files changed, 23 insertions(+), 8 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -102,15 +102,21 @@ New library modules: -* :mod:`asyncio`: New provisional API for asynchronous IO (:pep:`3156`). -* :mod:`enum`: Support for enumeration types (:pep:`435`). -* :mod:`ensurepip`: Bootstrapping the pip installer (:pep:`453`). -* :mod:`pathlib`: Object-oriented filesystem paths (:pep:`428`). +* :mod:`asyncio`: :ref:`New provisional API for asynchronous IO + ` (:pep:`3156`). +* :mod:`ensurepip`: :ref:`Bootstrapping the pip installer ` + (:pep:`453`). +* :mod:`enum`: :ref:`Support for enumeration types ` + (:pep:`435`). +* :mod:`pathlib`: :ref:`Object-oriented filesystem paths ` + (:pep:`428`). * :mod:`selectors`: :ref:`High-level and efficient I/O multiplexing `, built upon the :mod:`select` module primitives (part of :pep:`3156`). -* :mod:`statistics`: A basic numerically stable statistics library (:pep:`450`). -* :mod:`tracemalloc`: Trace Python memory allocations (:pep:`454`). +* :mod:`statistics`: A basic :ref:`numerically stable statistics library + ` (:pep:`450`). +* :mod:`tracemalloc`: :ref:`Trace Python memory allocations + ` (:pep:`454`). Significantly Improved Library Modules: @@ -287,7 +293,7 @@ The binary and text transforms provided in the standard library are detailed in :ref:`binary-transforms` and :ref:`text-transforms`. -(Contributed by Nick Coghlan in :issue:`7475`, :issue:`17827`, +(Contributed by Nick Coghlan in :issue:`7475`, , :issue:`17827`, :issue:`17828` and :issue:`19619`) .. _whatsnew-pep-451: @@ -340,6 +346,8 @@ =========== +.. _whatsnew-asyncio: + asyncio ------- @@ -356,7 +364,7 @@ PEP written and implementation led by Guido van Rossum. -.. _whatsnew-ensurepip +.. _whatsnew-ensurepip: ensurepip --------- @@ -380,6 +388,8 @@ version of ``pip`` is older than the bundled copy. +.. _whatsnew-enum: + enum ---- @@ -396,6 +406,8 @@ implemented by Ethan Furman. +.. _whatsnew-pathlib: + pathlib ------- @@ -423,6 +435,8 @@ :mod:`select` module primitives. +.. _whatsnew-statistics: + statistics ---------- @@ -436,6 +450,7 @@ :pep:`450` - Adding A Statistics Module To The Standard Library PEP written and implemented by Steven D'Aprano +.. _whatsnew-tracemalloc: tracemalloc ----------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 17:13:06 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 24 Dec 2013 17:13:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Collapse_=27python_invocat?= =?utf-8?q?ion_changes=27_into_=27other_improvements=27=2E?= Message-ID: <3dpjC61wPQzSHG@mail.python.org> http://hg.python.org/cpython/rev/0dac94122619 changeset: 88163:0dac94122619 user: R David Murray date: Mon Dec 23 21:21:09 2013 -0500 summary: Collapse 'python invocation changes' into 'other improvements'. files: Doc/whatsnew/3.4.rst | 15 ++++++--------- 1 files changed, 6 insertions(+), 9 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -940,16 +940,13 @@ Other improvements ================== -Tab-completion is now enabled by default in the interactive interpreter. -(Contributed by Antoine Pitrou and ?ric Araujo in :issue:`5845`.) +* Tab-completion is now enabled by default in the interactive interpreter. + (Contributed by Antoine Pitrou and ?ric Araujo in :issue:`5845`.) -Python invocation changes -========================= - -Invoking the Python interpreter with ``--version`` now outputs the version to -standard output instead of standard error (:issue:`18338`). Similar changes -were made to :mod:`argparse` (:issue:`18920`) and other modules that have -script-like invocation capabilities (:issue:`18922`). +* Invoking the Python interpreter with ``--version`` now outputs the version to + standard output instead of standard error (:issue:`18338`). Similar changes + were made to :mod:`argparse` (:issue:`18920`) and other modules that have + script-like invocation capabilities (:issue:`18922`). Optimizations ============= -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 17:13:07 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 24 Dec 2013 17:13:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Move_other_improvements_an?= =?utf-8?q?d_optimization_sections_above_deprecated=2E?= Message-ID: <3dpjC74mqvzRX7@mail.python.org> http://hg.python.org/cpython/rev/4e257ae4dbe6 changeset: 88164:4e257ae4dbe6 user: R David Murray date: Mon Dec 23 21:23:36 2013 -0500 summary: Move other improvements and optimization sections above deprecated. This arrangement means the sections that have summaries come first, and everything else comes after those. files: Doc/whatsnew/3.4.rst | 177 +++++++++++++++--------------- 1 files changed, 89 insertions(+), 88 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -937,6 +937,94 @@ (Contributed by Christian Tismer in :issue:`19274`.) +CPython Implementation Changes +============================== + + +.. _whatsnew-pep-445: + +PEP 445: Customization of CPython memory allocators +--------------------------------------------------- + +:pep:`445` adds new C level interfaces to customize memory allocation in +the CPython interpreter. + +.. seealso:: + + :pep:`445` - Add new APIs to customize Python memory allocators + PEP written and implemented by Victor Stinner. + + +.. _whatsnew-pep-442: + +PEP 442: Safe object finalization +--------------------------------- + +:pep:`442` removes the current limitations and quirks of object finalization +in CPython. With it, objects with :meth:`__del__` methods, as well as +generators with :keyword:`finally` clauses, can be finalized when they are +part of a reference cycle. + +As part of this change, module globals are no longer forcibly set to +:const:`None` during interpreter shutdown in most cases, instead relying +on the normal operation of the cyclic garbage collector. This avoids a +whole class of interpreter-shutdown-time errors, usually involving +``__del__`` methods, that have plagued Python since the cyclic GC +was first introduced. + +.. seealso:: + + :pep:`442` - Safe object finalization + PEP written and implemented by Antoine Pitrou. + + +.. _whatsnew-pep-456: + +PEP 456: Secure and Interchangeable Hash Algorithm +-------------------------------------------------- + +:pep:`456` follows up on earlier security fix work done on Python's hash +algorithm to address certain DOS attacks to which public facing APIs backed by +dictionary lookups may be subject. (See :issue:`14621` for the start of the +current round of improvements.) The PEP unifies CPython's hash code to make it +easier for a packager to substitute a different hash algorithm, and switches +Python's default implementation to a SipHash implementation on platforms that +have a 64 bit data type. Any performance differences in comparison with the +older FNV algorithm are trivial. + +The PEP adds additional fields to the :func:`sys.hash_info` struct sequence to +describe the hash algorithm in use by the currently executing binary. Otherwise, +the PEP does not alter any existing CPython APIs. + + +Other build and C API changes +----------------------------- + +Changes to Python's build process and to the C API include: + +* The new :c:func:`Py_SetStandardStreamEncoding` pre-initialization API + allows applications embedding the CPython interpreter to reliably force + a particular encoding and error handler for the standard streams + (Contributed by Bastien Montagne and Nick Coghlan in :issue:`16129`) + +* Most Python C APIs that don't mutate string arguments are now correctly + marked as accepting ``const char *`` rather than ``char *`` (Contributed + by Serhiy Storchaka in :issue:`1772673`). + +.. _whatsnew-pep-436: + +* "Argument Clinic" (:pep:`436`) is now part of the CPython build process + and can be used to simplify the process of defining and maintaining + accurate signatures for builtins and standard library extension modules + implemented in C. + + .. note:: + The Argument Clinic PEP is not fully up to date with the state of the + implementation. This has been deemed acceptable by the release manager + and core development team in this case, as Argument Clinic will not + be made available as a public API for third party use in Python 3.4. + + Other improvements ================== @@ -948,6 +1036,7 @@ were made to :mod:`argparse` (:issue:`18920`) and other modules that have script-like invocation capabilities (:issue:`18922`). + Optimizations ============= @@ -979,94 +1068,6 @@ :issue:`9548`) -CPython Implementation Changes -============================== - - -.. _whatsnew-pep-445: - -PEP 445: Customization of CPython memory allocators ---------------------------------------------------- - -:pep:`445` adds new C level interfaces to customize memory allocation in -the CPython interpreter. - -.. seealso:: - - :pep:`445` - Add new APIs to customize Python memory allocators - PEP written and implemented by Victor Stinner. - - -.. _whatsnew-pep-442: - -PEP 442: Safe object finalization ---------------------------------- - -:pep:`442` removes the current limitations and quirks of object finalization -in CPython. With it, objects with :meth:`__del__` methods, as well as -generators with :keyword:`finally` clauses, can be finalized when they are -part of a reference cycle. - -As part of this change, module globals are no longer forcibly set to -:const:`None` during interpreter shutdown in most cases, instead relying -on the normal operation of the cyclic garbage collector. This avoids a -whole class of interpreter-shutdown-time errors, usually involving -``__del__`` methods, that have plagued Python since the cyclic GC -was first introduced. - -.. seealso:: - - :pep:`442` - Safe object finalization - PEP written and implemented by Antoine Pitrou. - - -.. _whatsnew-pep-456: - -PEP 456: Secure and Interchangeable Hash Algorithm --------------------------------------------------- - -:pep:`456` follows up on earlier security fix work done on Python's hash -algorithm to address certain DOS attacks to which public facing APIs backed by -dictionary lookups may be subject. (See :issue:`14621` for the start of the -current round of improvements.) The PEP unifies CPython's hash code to make it -easier for a packager to substitute a different hash algorithm, and switches -Python's default implementation to a SipHash implementation on platforms that -have a 64 bit data type. Any performance differences in comparison with the -older FNV algorithm are trivial. - -The PEP adds additional fields to the :func:`sys.hash_info` struct sequence to -describe the hash algorithm in use by the currently executing binary. Otherwise, -the PEP does not alter any existing CPython APIs. - - -Other build and C API changes ------------------------------ - -Changes to Python's build process and to the C API include: - -* The new :c:func:`Py_SetStandardStreamEncoding` pre-initialization API - allows applications embedding the CPython interpreter to reliably force - a particular encoding and error handler for the standard streams - (Contributed by Bastien Montagne and Nick Coghlan in :issue:`16129`) - -* Most Python C APIs that don't mutate string arguments are now correctly - marked as accepting ``const char *`` rather than ``char *`` (Contributed - by Serhiy Storchaka in :issue:`1772673`). - -.. _whatsnew-pep-436: - -* "Argument Clinic" (:pep:`436`) is now part of the CPython build process - and can be used to simplify the process of defining and maintaining - accurate signatures for builtins and standard library extension modules - implemented in C. - - .. note:: - The Argument Clinic PEP is not fully up to date with the state of the - implementation. This has been deemed acceptable by the release manager - and core development team in this case, as Argument Clinic will not - be made available as a public API for third party use in Python 3.4. - - Deprecated ========== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 17:13:08 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 24 Dec 2013 17:13:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Make_the_=27read_on=27_sen?= =?utf-8?q?tence_cover_more_ground=2E?= Message-ID: <3dpjC86Y71z7LjS@mail.python.org> http://hg.python.org/cpython/rev/c690f13fdd1e changeset: 88165:c690f13fdd1e user: R David Murray date: Mon Dec 23 21:30:06 2013 -0500 summary: Make the 'read on' sentence cover more ground. files: Doc/whatsnew/3.4.rst | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -146,8 +146,9 @@ * The :mod:`marshal` format has been made :ref:`more compact and efficient ` (:issue:`16475`). -Please read on for a comprehensive list of user-facing changes, including -sections on deprecations and porting issues. +Please read on for a comprehensive list of user-facing changes, including many +other smaller improvements, CPython optimizations, deprecations, and potential +porting issues. New expected features for Python implementations -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 17:13:10 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 24 Dec 2013 17:13:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Mention_PATHEXT_and_remove?= =?utf-8?q?d_Misc_dirs=3B_clarify_wording_of_a_NEWS_entry=2E?= Message-ID: <3dpjCB1bY7z7LjN@mail.python.org> http://hg.python.org/cpython/rev/488bea77396f changeset: 88166:488bea77396f user: R David Murray date: Tue Dec 24 10:46:44 2013 -0500 summary: Mention PATHEXT and removed Misc dirs; clarify wording of a NEWS entry. files: Doc/whatsnew/3.4.rst | 15 +++++++++++++++ Misc/NEWS | 4 ++-- 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -473,6 +473,7 @@ Improved Modules ================ + aifc ---- @@ -1037,6 +1038,11 @@ were made to :mod:`argparse` (:issue:`18920`) and other modules that have script-like invocation capabilities (:issue:`18922`). +* The CPython Windows installer now adds ``.py`` to the :envvar:`PATHEXT` + variable when extensions are registered, allowing users to run a python + script at the windows command prompt by just typing its name without the + ``.py`` extension. (Contributed by Paul Moore in :issue:`18569`.) + Optimizations ============= @@ -1116,6 +1122,15 @@ exists, is deprecated (:issue:`19375`). +Removed +======= + +* The unmaintained ``Misc/TextMate`` and ``Misc/vim`` directories have been + removed (see the devguide__ for what to use instead). + +__ http://docs.python.org/devguide + + Porting to Python 3.4 ===================== diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -3500,8 +3500,8 @@ - Issue #17028: Allowed Python arguments to be supplied to the Windows launcher. -- Issue #17156: pygettext.py now uses an encoding of source file and correctly - writes and escapes non-ascii characters. +- Issue #17156: pygettext.py now detects the encoding of source files and + correctly writes and escapes non-ascii characters. - Issue #15539: Fix a number of bugs in Tools/scripts/pindent.py. Now pindent.py works with a "with" statement. pindent.py no longer produces -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 18:37:51 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 24 Dec 2013 18:37:51 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Mention_Windows_2000_and_O?= =?utf-8?q?S/2_drops=2C_python-config_as_shell_script=2E?= Message-ID: <3dpl4v4x1Jz7Ljb@mail.python.org> http://hg.python.org/cpython/rev/39aa0e5edeba changeset: 88167:39aa0e5edeba user: R David Murray date: Tue Dec 24 12:13:44 2013 -0500 summary: Mention Windows 2000 and OS/2 drops, python-config as shell script. files: Doc/whatsnew/3.4.rst | 15 ++++++++++++--- Misc/NEWS | 2 +- 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1002,7 +1002,7 @@ Other build and C API changes ----------------------------- -Changes to Python's build process and to the C API include: +Other changes to Python's build process and to the C API include: * The new :c:func:`Py_SetStandardStreamEncoding` pre-initialization API allows applications embedding the CPython interpreter to reliably force @@ -1026,6 +1026,9 @@ and core development team in this case, as Argument Clinic will not be made available as a public API for third party use in Python 3.4. +* New shell version of ``python-config``; can be used even when a python + interpreter is not available (for example, in cross compilation scenarios). + Other improvements ================== @@ -1126,9 +1129,15 @@ ======= * The unmaintained ``Misc/TextMate`` and ``Misc/vim`` directories have been - removed (see the devguide__ for what to use instead). + removed (see the `devguide `_ + for what to use instead). -__ http://docs.python.org/devguide + +* OS/2 support code has been removed from the source tree and build tools + (:issue:`16135`). + +* Windows 2000 support code has been removed from the source tree and build + tools (changeset e52df05b496a). Porting to Python 3.4 diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -3297,7 +3297,7 @@ - Issue #15484: Fix _PYTHON_PROJECT_BASE for srcdir != builddir builds; use _PYTHON_PROJECT_BASE in distutils/sysconfig.py. -- Drop support for Windows 2000. +- Drop support for Windows 2000 (changeset e52df05b496a). - Issue #17029: Let h2py search the multiarch system include directory. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 18:37:52 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 24 Dec 2013 18:37:52 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Move_argument_clinic_into_?= =?utf-8?q?its_own_section=2C_like_the_other_PEPs=2E?= Message-ID: <3dpl4w6k04z7Ljr@mail.python.org> http://hg.python.org/cpython/rev/9dd2c6d9872a changeset: 88168:9dd2c6d9872a user: R David Murray date: Tue Dec 24 12:23:56 2013 -0500 summary: Move argument clinic into its own section, like the other PEPs. files: Doc/whatsnew/3.4.rst | 35 ++++++++++++++++++++----------- 1 files changed, 22 insertions(+), 13 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -999,6 +999,28 @@ the PEP does not alter any existing CPython APIs. +.. _whatsnew-pep-436: + +PEP 436: Argument Clinic +------------------------ + +"Argument Clinic" (:pep:`436`) is now part of the CPython build process +and can be used to simplify the process of defining and maintaining +accurate signatures for builtins and standard library extension modules +implemented in C. + +.. note:: + The Argument Clinic PEP is not fully up to date with the state of the + implementation. This has been deemed acceptable by the release manager + and core development team in this case, as Argument Clinic will not + be made available as a public API for third party use in Python 3.4. + +.. seealso:: + + :pep:`436` - The Argument Clinic DSL + PEP written and implemented by Larry Hastings. + + Other build and C API changes ----------------------------- @@ -1013,19 +1035,6 @@ marked as accepting ``const char *`` rather than ``char *`` (Contributed by Serhiy Storchaka in :issue:`1772673`). -.. _whatsnew-pep-436: - -* "Argument Clinic" (:pep:`436`) is now part of the CPython build process - and can be used to simplify the process of defining and maintaining - accurate signatures for builtins and standard library extension modules - implemented in C. - - .. note:: - The Argument Clinic PEP is not fully up to date with the state of the - implementation. This has been deemed acceptable by the release manager - and core development team in this case, as Argument Clinic will not - be made available as a public API for third party use in Python 3.4. - * New shell version of ``python-config``; can be used even when a python interpreter is not available (for example, in cross compilation scenarios). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 18:37:54 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 24 Dec 2013 18:37:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Use_endash_in_PEP_callouts?= =?utf-8?q?=2E?= Message-ID: <3dpl4y2PWkz7Ljl@mail.python.org> http://hg.python.org/cpython/rev/01d2c25a6804 changeset: 88169:01d2c25a6804 user: R David Murray date: Tue Dec 24 12:35:59 2013 -0500 summary: Use endash in PEP callouts. Despite Serhiy's suggestion in issue18529, the emdash just looks too long to me, at least in my browser. This usage case is sort of halfway between endash (connective) and emdash (pause/separator). files: Doc/whatsnew/3.4.rst | 28 ++++++++++++++-------------- 1 files changed, 14 insertions(+), 14 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -18,7 +18,7 @@ time of the original release will remain largely unknown to the community for years, even if they're added later. We also know from experience that other priorities can arise, and the maintainer will run out of time to do - updates - in such cases, end users will be much better served by partial + updates -- in such cases, end users will be much better served by partial notifications that at least give a hint about new features to investigate. @@ -75,7 +75,7 @@ .. seealso:: - :pep:`429` - Python 3.4 Release Schedule + :pep:`429` -- Python 3.4 Release Schedule Summary -- Release highlights @@ -193,7 +193,7 @@ .. seealso:: - :pep:`453` - Explicit bootstrapping of pip in Python installations + :pep:`453` -- Explicit bootstrapping of pip in Python installations PEP written by Donald Stufft and Nick Coghlan, implemented by Donald Stufft, Nick Coghlan, Martin von L?wis and Ned Deily. @@ -212,7 +212,7 @@ .. seealso:: - :pep:`446` - Make newly created file descriptors non-inheritable + :pep:`446` -- Make newly created file descriptors non-inheritable PEP written and implemented by Victor Stinner. @@ -361,7 +361,7 @@ .. seealso:: - :pep:`3156` - Asynchronous IO Support Rebooted: the "asyncio" Module + :pep:`3156` -- Asynchronous IO Support Rebooted: the "asyncio" Module PEP written and implementation led by Guido van Rossum. @@ -402,7 +402,7 @@ .. seealso:: - :pep:`435` - Adding an Enum type to the Python standard library + :pep:`435` -- Adding an Enum type to the Python standard library PEP written by Barry Warsaw, Eli Bendersky and Ethan Furman, implemented by Ethan Furman. @@ -422,7 +422,7 @@ .. seealso:: - :pep:`428` - The pathlib module -- object-oriented filesystem paths + :pep:`428` -- The pathlib module -- object-oriented filesystem paths PEP written and implemented by Antoine Pitrou. @@ -448,7 +448,7 @@ .. seealso:: - :pep:`450` - Adding A Statistics Module To The Standard Library + :pep:`450` -- Adding A Statistics Module To The Standard Library PEP written and implemented by Steven D'Aprano .. _whatsnew-tracemalloc: @@ -466,7 +466,7 @@ .. seealso:: - :pep:`454` - Add a new tracemalloc module to trace Python memory allocations + :pep:`454` -- Add a new tracemalloc module to trace Python memory allocations PEP written and implemented by Victor Stinner @@ -616,7 +616,7 @@ .. seealso:: - :pep:`443` - Single-dispatch generic functions + :pep:`443` -- Single-dispatch generic functions PEP written and implemented by ?ukasz Langa. @@ -757,7 +757,7 @@ .. seealso:: - :pep:`3154` - Pickle protocol 4 + :pep:`3154` -- Pickle protocol 4 PEP written by Antoine Pitrou and implemented by Alexandre Vassalotti. @@ -953,7 +953,7 @@ .. seealso:: - :pep:`445` - Add new APIs to customize Python memory allocators + :pep:`445` -- Add new APIs to customize Python memory allocators PEP written and implemented by Victor Stinner. @@ -976,7 +976,7 @@ .. seealso:: - :pep:`442` - Safe object finalization + :pep:`442` -- Safe object finalization PEP written and implemented by Antoine Pitrou. @@ -1017,7 +1017,7 @@ .. seealso:: - :pep:`436` - The Argument Clinic DSL + :pep:`436` -- The Argument Clinic DSL PEP written and implemented by Larry Hastings. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 21:43:33 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 24 Dec 2013 21:43:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_sys=2Egetallocatedblocks_+?= =?utf-8?q?_regrtest_-R=2C_make_coverage-report=2C_SO_macro_goes_away=2E?= Message-ID: <3dpqC94LrGz7LjP@mail.python.org> http://hg.python.org/cpython/rev/85b41e83f810 changeset: 88170:85b41e83f810 user: R David Murray date: Tue Dec 24 14:46:23 2013 -0500 summary: sys.getallocatedblocks + regrtest -R, make coverage-report, SO macro goes away. files: Doc/whatsnew/3.4.rst | 26 +++++++++++++++++++++++++- 1 files changed, 25 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -878,6 +878,17 @@ :meth:`sunau.open` now supports the context manager protocol (:issue:`18878`). +sys +--- + +New function :func:`sys.getallocatedblocks` returns the current number of +blocks allocated by the interpreter (in CPython with the default +``--with-pymalloc`` setting, this is allocations made through the +:c:func:`PyObject_Malloc` API). This can be useful for tracking memory leaks, +especially if automated via a test suite. (Contributed by Antoine Pitrou +in :issue:`13390`.) + + traceback --------- @@ -1055,6 +1066,17 @@ script at the windows command prompt by just typing its name without the ``.py`` extension. (Contributed by Paul Moore in :issue:`18569`.) +* A new ``make`` target `coverage-report + `_ + will build python, run the test suite, and generate an HTML coverage report + for the C codebase using ``gcov`` and `lcov + `_. + +* The ``-R`` option to the :ref:`python regression test suite ` now + also checks for memory allocation leaks, using + :func:`sys.getallocatedblocks()`. (Contributed by Antoine Pitrou in + :issue:`13390`). + Optimizations ============= @@ -1141,13 +1163,15 @@ removed (see the `devguide `_ for what to use instead). - * OS/2 support code has been removed from the source tree and build tools (:issue:`16135`). * Windows 2000 support code has been removed from the source tree and build tools (changeset e52df05b496a). +* The ``SO`` makefile macro is removed (it was replaced by the + ``SHLIB_SUFFIX`` and ``EXT_SUFFIX`` macros) (:issue:`16754`). + Porting to Python 3.4 ===================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 21:43:34 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 24 Dec 2013 21:43:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Use_title_case_for_all_sec?= =?utf-8?q?tion_titles_except_module_names=2E?= Message-ID: <3dpqCB6xtLz7LjW@mail.python.org> http://hg.python.org/cpython/rev/f395d0d176e0 changeset: 88171:f395d0d176e0 user: R David Murray date: Tue Dec 24 14:51:25 2013 -0500 summary: Use title case for all section titles except module names. files: Doc/whatsnew/3.4.rst | 24 ++++++++++++------------ 1 files changed, 12 insertions(+), 12 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -78,7 +78,7 @@ :pep:`429` -- Python 3.4 Release Schedule -Summary -- Release highlights +Summary -- Release Highlights ============================= .. This section singles out the most important changes in Python 3.4. @@ -151,12 +151,12 @@ porting issues. -New expected features for Python implementations +New Expected Features for Python Implementations ================================================ .. _whatsnew-pep-453: -PEP 453: Explicit bootstrapping of pip in Python installations +PEP 453: Explicit Bootstrapping of PIP in Python Installations -------------------------------------------------------------- The new :mod:`ensurepip` module (defined in :pep:`453`) provides a standard @@ -200,7 +200,7 @@ .. _whatsnew-pep-446: -PEP 446: Make newly created file descriptors non-inheritable +PEP 446: Make Newly Created File Descriptors Non-Inheritable ------------------------------------------------------------ :pep:`446` makes newly created file descriptors :ref:`non-inheritable @@ -218,7 +218,7 @@ .. _codec-handling-improvements: -Improvements to codec handling +Improvements to Codec Handling ------------------------------ Since it was first introduced, the :mod:`codecs` module has always been @@ -956,7 +956,7 @@ .. _whatsnew-pep-445: -PEP 445: Customization of CPython memory allocators +PEP 445: Customization of CPython Memory Allocators --------------------------------------------------- :pep:`445` adds new C level interfaces to customize memory allocation in @@ -970,7 +970,7 @@ .. _whatsnew-pep-442: -PEP 442: Safe object finalization +PEP 442: Safe Object Finalization --------------------------------- :pep:`442` removes the current limitations and quirks of object finalization @@ -1032,7 +1032,7 @@ PEP written and implemented by Larry Hastings. -Other build and C API changes +Other Build and C API Changes ----------------------------- Other changes to Python's build process and to the C API include: @@ -1050,7 +1050,7 @@ interpreter is not available (for example, in cross compilation scenarios). -Other improvements +Other Improvements ================== * Tab-completion is now enabled by default in the interactive interpreter. @@ -1120,7 +1120,7 @@ * VMS -Deprecated Python modules, functions and methods +Deprecated Python Modules, Functions and Methods ------------------------------------------------ * :meth:`difflib.SequenceMatcher.isbjunk` and @@ -1142,14 +1142,14 @@ require an explicit digest name or constructor as *digestmod* argument. -Deprecated functions and types of the C API +Deprecated Functions and Types in the C API ------------------------------------------- * The ``PyThreadState.tick_counter`` field has been removed: its value was meaningless since Python 3.2 ("new GIL"). -Deprecated features +Deprecated Features ------------------- * The site module adding a "site-python" directory to sys.path, if it -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 21:43:36 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 24 Dec 2013 21:43:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Move_no-longer-supported_o?= =?utf-8?q?perating_systems_into_=27removed=27_section=2E?= Message-ID: <3dpqCD1nmtz7Ljf@mail.python.org> http://hg.python.org/cpython/rev/b959bf4a4c7b changeset: 88172:b959bf4a4c7b user: R David Murray date: Tue Dec 24 14:59:50 2013 -0500 summary: Move no-longer-supported operating systems into 'removed' section. files: Doc/whatsnew/3.4.rst | 20 +++++++------------- 1 files changed, 7 insertions(+), 13 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1112,13 +1112,6 @@ Deprecated ========== -Unsupported Operating Systems ------------------------------ - -* OS/2 -* Windows 2000 -* VMS - Deprecated Python Modules, Functions and Methods ------------------------------------------------ @@ -1159,16 +1152,17 @@ Removed ======= +* Support for the following previously deprecated operating systems has been + removed from the source and build tools: + + * OS/2 (:issue:`16135`). + * Windows 2000 (changeset e52df05b496a). + * VMS (:issue:`16136`). + * The unmaintained ``Misc/TextMate`` and ``Misc/vim`` directories have been removed (see the `devguide `_ for what to use instead). -* OS/2 support code has been removed from the source tree and build tools - (:issue:`16135`). - -* Windows 2000 support code has been removed from the source tree and build - tools (changeset e52df05b496a). - * The ``SO`` makefile macro is removed (it was replaced by the ``SHLIB_SUFFIX`` and ``EXT_SUFFIX`` macros) (:issue:`16754`). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 21:43:37 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 24 Dec 2013 21:43:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Neaten_up_structure_of_maj?= =?utf-8?q?or_sections=2E?= Message-ID: <3dpqCF3TgGz7LjX@mail.python.org> http://hg.python.org/cpython/rev/36e6fb43b611 changeset: 88173:36e6fb43b611 user: R David Murray date: Tue Dec 24 15:33:02 2013 -0500 summary: Neaten up structure of major sections. Also move tick_counter from deprecated section to removed section. files: Doc/whatsnew/3.4.rst | 33 +++++++++++++++++++------------ 1 files changed, 20 insertions(+), 13 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1035,8 +1035,6 @@ Other Build and C API Changes ----------------------------- -Other changes to Python's build process and to the C API include: - * The new :c:func:`Py_SetStandardStreamEncoding` pre-initialization API allows applications embedding the CPython interpreter to reliably force a particular encoding and error handler for the standard streams @@ -1078,10 +1076,8 @@ :issue:`13390`). -Optimizations -============= - -Major performance enhancements have been added: +Significant Optimizations +========================= * The UTF-32 decoder is now 3x to 4x faster. @@ -1112,6 +1108,12 @@ Deprecated ========== +This section covers various APIs and other features that have been deprecated +in Python 3.4, and will be removed in Python 3.5 or later. In most (but not +all) cases, using the deprecated APIs will produce a :exc:`DeprecationWarning` +when the interpreter is run with deprecation warnings enabled (for example, by +using ``-Wd``). + Deprecated Python Modules, Functions and Methods ------------------------------------------------ @@ -1138,8 +1140,7 @@ Deprecated Functions and Types in the C API ------------------------------------------- -* The ``PyThreadState.tick_counter`` field has been removed: its value was - meaningless since Python 3.2 ("new GIL"). +XXX: None so far Deprecated Features @@ -1152,8 +1153,11 @@ Removed ======= -* Support for the following previously deprecated operating systems has been - removed from the source and build tools: +The following previously deprecated APIs and features have been removed +in Python 3.4: + +* Support for the following operating systems has been removed from the source + and build tools: * OS/2 (:issue:`16135`). * Windows 2000 (changeset e52df05b496a). @@ -1166,16 +1170,19 @@ * The ``SO`` makefile macro is removed (it was replaced by the ``SHLIB_SUFFIX`` and ``EXT_SUFFIX`` macros) (:issue:`16754`). +* The ``PyThreadState.tick_counter`` field has been removed; its value has + been meaningless since Python 3.2, when the "new GIL" was introduced. + Porting to Python 3.4 ===================== +This section lists previously described changes and other bugfixes +that may require changes to your code. + Changes in the Python API ------------------------- -This section lists previously described changes and other bugfixes -that may require changes to your code. - * The ABCs defined in :mod:`importlib.abc` now either raise the appropriate exception or return a default value instead of raising :exc:`NotImplementedError` blindly. This will only affect code calling -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 21:43:38 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 24 Dec 2013 21:43:38 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Move_IPv6_NEWS_entry_into_?= =?utf-8?q?correct_section=2E?= Message-ID: <3dpqCG5T3Gz7Ljf@mail.python.org> http://hg.python.org/cpython/rev/c3e7409f967f changeset: 88174:c3e7409f967f user: R David Murray date: Tue Dec 24 15:37:58 2013 -0500 summary: Move IPv6 NEWS entry into correct section. files: Misc/NEWS | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -3142,8 +3142,6 @@ - Issue #15539: Added regression tests for Tools/scripts/pindent.py. -- Issue #16836: Enable IPv6 support even if IPv6 is disabled on the build host. - - Issue #17479: test_io now works with unittest test discovery. Patch by Zachary Ware. @@ -3312,6 +3310,8 @@ - Fix cross compiling issue in setup.py, ensure that lib_dirs and inc_dirs are defined in cross compiling mode, too. +- Issue #16836: Enable IPv6 support even if IPv6 is disabled on the build host. + - Issue #16593: Have BSD 'make -s' do the right thing, thanks to Daniel Shahaf - Issue #16262: fix out-of-src-tree builds, if mercurial is not installed. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 21:43:40 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 24 Dec 2013 21:43:40 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_English_in_a_NEWS_entr?= =?utf-8?q?y=2E?= Message-ID: <3dpqCJ08Ksz7Ljg@mail.python.org> http://hg.python.org/cpython/rev/f2025547409d changeset: 88175:f2025547409d user: R David Murray date: Tue Dec 24 15:41:45 2013 -0500 summary: Fix English in a NEWS entry. files: Misc/NEWS | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -3061,7 +3061,7 @@ - Issue #18207: Fix test_ssl for some versions of OpenSSL that ignore seconds in ASN1_TIME fields. -- Issue #18094: test_uuid no more reports skipped tests as passed. +- Issue #18094: test_uuid no longer reports skipped tests as passed. - Issue #17992: Add timeouts to asyncore and asynchat tests so that they won't accidentally hang. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 22:14:09 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 24 Dec 2013 22:14:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_idle_-n_deprecation=2C_fix?= =?utf-8?q?_case_of_pickle_section=2E?= Message-ID: <3dpqtT5lTNz7LjP@mail.python.org> http://hg.python.org/cpython/rev/e847cb9055ab changeset: 88176:e847cb9055ab user: R David Murray date: Tue Dec 24 15:59:57 2013 -0500 summary: idle -n deprecation, fix case of pickle section. files: Doc/whatsnew/3.4.rst | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -744,7 +744,7 @@ .. _whatsnew-protocol-4: -Pickle +pickle ------ protocol 4 @@ -1146,6 +1146,9 @@ Deprecated Features ------------------- +* Running :ref:`idle` with the ``-n`` flag (no subprocess) is deprecated. + However, the feature will not be removed until :issue:`18823` is resolved. + * The site module adding a "site-python" directory to sys.path, if it exists, is deprecated (:issue:`19375`). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 24 22:14:11 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 24 Dec 2013 22:14:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_=2316832=3A_s/integer/obje?= =?utf-8?q?ct/_in_docs/docstring=2C_and_add_whatsnew_entry=2E?= Message-ID: <3dpqtW0d3nz7LjW@mail.python.org> http://hg.python.org/cpython/rev/a9f73b44ea0e changeset: 88177:a9f73b44ea0e user: R David Murray date: Tue Dec 24 16:13:32 2013 -0500 summary: #16832: s/integer/object/ in docs/docstring, and add whatsnew entry. files: Doc/library/abc.rst | 6 +++--- Doc/whatsnew/3.4.rst | 6 ++++++ Lib/abc.py | 6 +++--- 3 files changed, 12 insertions(+), 6 deletions(-) diff --git a/Doc/library/abc.rst b/Doc/library/abc.rst --- a/Doc/library/abc.rst +++ b/Doc/library/abc.rst @@ -318,9 +318,9 @@ Returns the current abstract base class cache token. - The token is an opaque integer identifying the current version of the - abstract base class cache for virtual subclasses. This number changes - with every call to :meth:`ABCMeta.register` on any ABC. + The token is an opaque object (that supports equality testing) identifying + the current version of the abstract base class cache for virtual subclasses. + The token changes with every call to :meth:`ABCMeta.register` on any ABC. .. versionadded:: 3.4 diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -473,6 +473,12 @@ Improved Modules ================ +abc +--- + +New function :func:`abc.get_cache_token` can be used to know when to invalidate +caches that are affected by changes in the object graph. (Contributed +by ?ukasz Langa in :issue:`16832`.) aifc ---- diff --git a/Lib/abc.py b/Lib/abc.py --- a/Lib/abc.py +++ b/Lib/abc.py @@ -241,8 +241,8 @@ def get_cache_token(): """Returns the current ABC cache token. - The token is an opaque integer identifying the current version of - the ABC cache for virtual subclasses. This number changes with - every call to ``register()`` on any ABC. + The token is an opaque object (supporting equality testing) identifying the + current version of the ABC cache for virtual subclasses. The token changes + with every call to ``register()`` on any ABC. """ return ABCMeta._abc_invalidation_counter -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 25 04:31:54 2013 From: python-checkins at python.org (r.david.murray) Date: Wed, 25 Dec 2013 04:31:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_pty=2Espawn_returns_os=2Ew?= =?utf-8?q?aitpid=3B_optimization_of_BZ2File_and_LZMAFile=2E?= Message-ID: <3dq0GL2SkFzRX7@mail.python.org> http://hg.python.org/cpython/rev/8a334a509dc2 changeset: 88178:8a334a509dc2 user: R David Murray date: Tue Dec 24 22:28:04 2013 -0500 summary: pty.spawn returns os.waitpid; optimization of BZ2File and LZMAFile. files: Doc/whatsnew/3.4.rst | 13 +++++++++++++ Misc/NEWS | 2 +- 2 files changed, 14 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -473,6 +473,7 @@ Improved Modules ================ + abc --- @@ -480,6 +481,7 @@ caches that are affected by changes in the object graph. (Contributed by ?ukasz Langa in :issue:`16832`.) + aifc ---- @@ -786,6 +788,13 @@ sequences (:issue:`19132`). +pty +--- + +:func:`pty.spawn` now returns the status value from :func:`os.waitpid` on +the child process, instead of ``None``. (Contributed by Gregory P. Smith.) + + pydoc ----- @@ -1110,6 +1119,10 @@ :issue:`19219`, :issue:`19218`, :issue:`19209`, :issue:`19205` and :issue:`9548`) +* :class:`bz2.BZ2File` is now as fast or faster than the Python2 version for + most cases. :class:`lzma.LZMAFile` has also been optimized. (Contributed by + Serhiy Storchaka and Nadeem Vawda in :issue:`16034`.) + Deprecated ========== diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2817,7 +2817,7 @@ - Issue #12034: Fix bogus caching of result in check_GetFinalPathNameByHandle. Patch by Atsuo Ishimoto. -- Improve performance of `lzma.LZMAFile`. +- Improve performance of `lzma.LZMAFile` (see also issue #16034). - Issue #16220: wsgiref now always calls close() on an iterable response. Patch by Brent Tubbs. -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Wed Dec 25 09:44:16 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 25 Dec 2013 09:44:16 +0100 Subject: [Python-checkins] Daily reference leaks (a9f73b44ea0e): sum=0 Message-ID: results for a9f73b44ea0e on branch "default" -------------------------------------------- test_site leaked [2, 0, -2] references, sum=0 test_site leaked [2, 0, -2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog93lSSo', '-x'] From python-checkins at python.org Wed Dec 25 13:25:34 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 25 Dec 2013 13:25:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIwMDU4?= =?utf-8?q?=3A_sys=2Estdin=2Ereadline=28=29_in_IDLE_now_always_returns_onl?= =?utf-8?q?y_one_line=2E?= Message-ID: <3dqD661lT4z7LjQ@mail.python.org> http://hg.python.org/cpython/rev/526dcd51e425 changeset: 88179:526dcd51e425 branch: 2.7 parent: 88159:ee924673d83a user: Serhiy Storchaka date: Wed Dec 25 14:24:17 2013 +0200 summary: Issue #20058: sys.stdin.readline() in IDLE now always returns only one line. files: Lib/idlelib/PyShell.py | 3 +++ Misc/NEWS | 2 ++ 2 files changed, 5 insertions(+), 0 deletions(-) diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -1393,6 +1393,9 @@ line = self._line_buffer or self.shell.readline() if size < 0: size = len(line) + eol = line.find('\n', 0, size) + if eol >= 0: + size = eol + 1 self._line_buffer = line[size:] return line[:size] diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -118,6 +118,8 @@ IDLE ---- +- Issue #20058: sys.stdin.readline() in IDLE now always returns only one line. + - Issue #19481: print() of unicode, str or bytearray subclass instance in IDLE no more hangs. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 25 13:25:35 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 25 Dec 2013 13:25:35 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzIwMDU4?= =?utf-8?q?=3A_sys=2Estdin=2Ereadline=28=29_in_IDLE_now_always_returns_onl?= =?utf-8?q?y_one_line=2E?= Message-ID: <3dqD673vFnz7LjW@mail.python.org> http://hg.python.org/cpython/rev/8f75d8ddc95b changeset: 88180:8f75d8ddc95b branch: 3.3 parent: 88160:99190ebd4935 user: Serhiy Storchaka date: Wed Dec 25 14:24:35 2013 +0200 summary: Issue #20058: sys.stdin.readline() in IDLE now always returns only one line. files: Lib/idlelib/PyShell.py | 3 +++ Misc/NEWS | 2 ++ 2 files changed, 5 insertions(+), 0 deletions(-) diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -1381,6 +1381,9 @@ line = self._line_buffer or self.shell.readline() if size < 0: size = len(line) + eol = line.find('\n', 0, size) + if eol >= 0: + size = eol + 1 self._line_buffer = line[size:] return line[:size] diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -169,6 +169,8 @@ IDLE ---- +- Issue #20058: sys.stdin.readline() in IDLE now always returns only one line. + - Issue #19481: print() of string subclass instance in IDLE no longer hangs. - Issue #18270: Prevent possible IDLE AttributeError on OS X when no initial -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 25 13:27:23 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 25 Dec 2013 13:27:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2320058=3A_sys=2Estdin=2Ereadline=28=29_in_IDLE_n?= =?utf-8?q?ow_always_returns_only_one_line=2E?= Message-ID: <3dqD8C3YfPz7Ljr@mail.python.org> http://hg.python.org/cpython/rev/2a4c083f8f6b changeset: 88181:2a4c083f8f6b parent: 88178:8a334a509dc2 parent: 88180:8f75d8ddc95b user: Serhiy Storchaka date: Wed Dec 25 14:27:16 2013 +0200 summary: Issue #20058: sys.stdin.readline() in IDLE now always returns only one line. files: Lib/idlelib/PyShell.py | 3 +++ Misc/NEWS | 2 ++ 2 files changed, 5 insertions(+), 0 deletions(-) diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -1384,6 +1384,9 @@ line = self._line_buffer or self.shell.readline() if size < 0: size = len(line) + eol = line.find('\n', 0, size) + if eol >= 0: + size = eol + 1 self._line_buffer = line[size:] return line[:size] diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -240,6 +240,8 @@ IDLE ---- +- Issue #20058: sys.stdin.readline() in IDLE now always returns only one line. + - Issue #19481: print() of string subclass instance in IDLE no longer hangs. - Issue #18270: Prevent possible IDLE AttributeError on OS X when no initial -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 25 15:43:24 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 25 Dec 2013 15:43:24 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5MDIw?= =?utf-8?q?=3A_Tkinter_now_uses_splitlist=28=29_instead_of_split=28=29_in_?= =?utf-8?q?configure?= Message-ID: <3dqH980Yhdz7LjN@mail.python.org> http://hg.python.org/cpython/rev/ff70c298dd60 changeset: 88182:ff70c298dd60 branch: 2.7 parent: 88179:526dcd51e425 user: Serhiy Storchaka date: Wed Dec 25 16:35:20 2013 +0200 summary: Issue #19020: Tkinter now uses splitlist() instead of split() in configure methods. files: Lib/lib-tk/Tix.py | 44 +++++--------------- Lib/lib-tk/Tkinter.py | 39 +++++++++--------- Lib/lib-tk/test/widget_tests.py | 4 +- Misc/NEWS | 3 + 4 files changed, 35 insertions(+), 55 deletions(-) diff --git a/Lib/lib-tk/Tix.py b/Lib/lib-tk/Tix.py --- a/Lib/lib-tk/Tix.py +++ b/Lib/lib-tk/Tix.py @@ -122,13 +122,9 @@ elif cnf: cnf = _cnfmerge(cnf) if cnf is None: - cnf = {} - for x in self.tk.split(self.tk.call('tix', 'configure')): - cnf[x[0][1:]] = (x[0][1:],) + x[1:] - return cnf + return self._getconfigure('tix', 'configure') if isinstance(cnf, StringType): - x = self.tk.split(self.tk.call('tix', 'configure', '-'+cnf)) - return (x[0][1:],) + x[1:] + return self._getconfigure1('tix', 'configure', '-'+cnf) return self.tk.call(('tix', 'configure') + self._options(cnf)) def tix_filedialog(self, dlgclass=None): @@ -380,7 +376,7 @@ """Return the name of all subwidgets.""" try: x = self.tk.call(self._w, 'subwidgets', '-all') - return self.tk.split(x) + return self.tk.splitlist(x) except TclError: return None @@ -473,13 +469,6 @@ self.tk.call('destroy', self._w) -# Useful func. to split Tcl lists and return as a dict. From Tkinter.py -def _lst2dict(lst): - dict = {} - for x in lst: - dict[x[0][1:]] = (x[0][1:],) + x[1:] - return dict - # Useful class to create a display style - later shared by many items. # Contributed by Steffen Kremser class DisplayStyle: @@ -515,10 +504,8 @@ self.tk.call(self.stylename, 'configure', '-%s'%key, value) def config(self, cnf={}, **kw): - return _lst2dict( - self.tk.split( - self.tk.call( - self.stylename, 'configure', *self._options(cnf,kw)))) + return self._getconfigure( + self.stylename, 'configure', *self._options(cnf,kw)) def __getitem__(self,key): return self.tk.call(self.stylename, 'cget', '-%s'%key) @@ -928,9 +915,7 @@ def header_configure(self, col, cnf={}, **kw): if cnf is None: - return _lst2dict( - self.tk.split( - self.tk.call(self._w, 'header', 'configure', col))) + return self._getconfigure(self._w, 'header', 'configure', col) self.tk.call(self._w, 'header', 'configure', col, *self._options(cnf, kw)) @@ -955,9 +940,8 @@ def indicator_configure(self, entry, cnf={}, **kw): if cnf is None: - return _lst2dict( - self.tk.split( - self.tk.call(self._w, 'indicator', 'configure', entry))) + return self._getconfigure( + self._w, 'indicator', 'configure', entry) self.tk.call( self._w, 'indicator', 'configure', entry, *self._options(cnf, kw)) @@ -1017,9 +1001,7 @@ def item_configure(self, entry, col, cnf={}, **kw): if cnf is None: - return _lst2dict( - self.tk.split( - self.tk.call(self._w, 'item', 'configure', entry, col))) + return self._getconfigure(self._w, 'item', 'configure', entry, col) self.tk.call(self._w, 'item', 'configure', entry, col, *self._options(cnf, kw)) @@ -1038,9 +1020,7 @@ def entryconfigure(self, entry, cnf={}, **kw): if cnf is None: - return _lst2dict( - self.tk.split( - self.tk.call(self._w, 'entryconfigure', entry))) + return self._getconfigure(self._w, 'entryconfigure', entry) self.tk.call(self._w, 'entryconfigure', entry, *self._options(cnf, kw)) @@ -1255,9 +1235,7 @@ def paneconfigure(self, entry, cnf={}, **kw): if cnf is None: - return _lst2dict( - self.tk.split( - self.tk.call(self._w, 'paneconfigure', entry))) + return self._getconfigure(self._w, 'paneconfigure', entry) self.tk.call(self._w, 'paneconfigure', entry, *self._options(cnf, kw)) def panes(self): diff --git a/Lib/lib-tk/Tkinter.py b/Lib/lib-tk/Tkinter.py --- a/Lib/lib-tk/Tkinter.py +++ b/Lib/lib-tk/Tkinter.py @@ -1234,6 +1234,19 @@ exc, val, tb = sys.exc_type, sys.exc_value, sys.exc_traceback root = self._root() root.report_callback_exception(exc, val, tb) + + def _getconfigure(self, *args): + """Call Tcl configure command and return the result as a dict.""" + cnf = {} + for x in self.tk.splitlist(self.tk.call(*args)): + x = self.tk.splitlist(x) + cnf[x[0][1:]] = (x[0][1:],) + x[1:] + return cnf + + def _getconfigure1(self, *args): + x = self.tk.splitlist(self.tk.call(*args)) + return (x[0][1:],) + x[1:] + def _configure(self, cmd, cnf, kw): """Internal function.""" if kw: @@ -1241,15 +1254,9 @@ elif cnf: cnf = _cnfmerge(cnf) if cnf is None: - cnf = {} - for x in self.tk.split( - self.tk.call(_flatten((self._w, cmd)))): - cnf[x[0][1:]] = (x[0][1:],) + x[1:] - return cnf + return self._getconfigure(_flatten((self._w, cmd))) if type(cnf) is StringType: - x = self.tk.split( - self.tk.call(_flatten((self._w, cmd, '-'+cnf)))) - return (x[0][1:],) + x[1:] + return self._getconfigure1(_flatten((self._w, cmd, '-'+cnf))) self.tk.call(_flatten((self._w, cmd)) + self._options(cnf)) # These used to be defined in Widget: def configure(self, cnf=None, **kw): @@ -1271,8 +1278,8 @@ raise TypeError("Tkinter objects don't support 'in' tests.") def keys(self): """Return a list of all resource names of this widget.""" - return map(lambda x: x[0][1:], - self.tk.split(self.tk.call(self._w, 'configure'))) + return [x[0][1:] for x in + self.tk.splitlist(self.tk.call(self._w, 'configure'))] def __str__(self): """Return the window path name of this widget.""" return self._w @@ -3725,16 +3732,10 @@ """ if cnf is None and not kw: - cnf = {} - for x in self.tk.split( - self.tk.call(self._w, - 'paneconfigure', tagOrId)): - cnf[x[0][1:]] = (x[0][1:],) + x[1:] - return cnf + return self._getconfigure(self._w, 'paneconfigure', tagOrId) if type(cnf) == StringType and not kw: - x = self.tk.split(self.tk.call( - self._w, 'paneconfigure', tagOrId, '-'+cnf)) - return (x[0][1:],) + x[1:] + return self._getconfigure1( + self._w, 'paneconfigure', tagOrId, '-'+cnf) self.tk.call((self._w, 'paneconfigure', tagOrId) + self._options(cnf, kw)) paneconfig = paneconfigure diff --git a/Lib/lib-tk/test/widget_tests.py b/Lib/lib-tk/test/widget_tests.py --- a/Lib/lib-tk/test/widget_tests.py +++ b/Lib/lib-tk/test/widget_tests.py @@ -66,9 +66,7 @@ if not isinstance(widget, Scale): t = widget.configure(name) self.assertEqual(len(t), 5) - ## XXX - if not isinstance(t[4], tuple): - self.assertEqual2(t[4], expected, eq=eq) + self.assertEqual2(t[4], expected, eq=eq) def checkInvalidParam(self, widget, name, value, errmsg=None, keep_orig=True): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -27,6 +27,9 @@ Library ------- +- Issue #19020: Tkinter now uses splitlist() instead of split() in configure + methods. + - Issue #12226: HTTPS is now used by default when connecting to PyPI. - Issue #20048: Fixed ZipExtFile.peek() when it is called on the boundary of -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 25 15:43:25 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 25 Dec 2013 15:43:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5MDIw?= =?utf-8?q?=3A_Tkinter_now_uses_splitlist=28=29_instead_of_split=28=29_in_?= =?utf-8?q?configure?= Message-ID: <3dqH993cxBz7Ljh@mail.python.org> http://hg.python.org/cpython/rev/a8f5f8c44dc8 changeset: 88183:a8f5f8c44dc8 branch: 3.3 parent: 88180:8f75d8ddc95b user: Serhiy Storchaka date: Wed Dec 25 16:35:38 2013 +0200 summary: Issue #19020: Tkinter now uses splitlist() instead of split() in configure methods. files: Lib/tkinter/__init__.py | 37 ++++++++-------- Lib/tkinter/test/widget_tests.py | 4 +- Lib/tkinter/tix.py | 44 +++++--------------- Misc/NEWS | 3 + 4 files changed, 34 insertions(+), 54 deletions(-) diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -1235,6 +1235,19 @@ exc, val, tb = sys.exc_info() root = self._root() root.report_callback_exception(exc, val, tb) + + def _getconfigure(self, *args): + """Call Tcl configure command and return the result as a dict.""" + cnf = {} + for x in self.tk.splitlist(self.tk.call(*args)): + x = self.tk.splitlist(x) + cnf[x[0][1:]] = (x[0][1:],) + x[1:] + return cnf + + def _getconfigure1(self, *args): + x = self.tk.splitlist(self.tk.call(*args)) + return (x[0][1:],) + x[1:] + def _configure(self, cmd, cnf, kw): """Internal function.""" if kw: @@ -1242,15 +1255,9 @@ elif cnf: cnf = _cnfmerge(cnf) if cnf is None: - cnf = {} - for x in self.tk.split( - self.tk.call(_flatten((self._w, cmd)))): - cnf[x[0][1:]] = (x[0][1:],) + x[1:] - return cnf + return self._getconfigure(_flatten((self._w, cmd))) if isinstance(cnf, str): - x = self.tk.split( - self.tk.call(_flatten((self._w, cmd, '-'+cnf)))) - return (x[0][1:],) + x[1:] + return self._getconfigure1(_flatten((self._w, cmd, '-'+cnf))) self.tk.call(_flatten((self._w, cmd)) + self._options(cnf)) # These used to be defined in Widget: def configure(self, cnf=None, **kw): @@ -1271,7 +1278,7 @@ def keys(self): """Return a list of all resource names of this widget.""" return [x[0][1:] for x in - self.tk.split(self.tk.call(self._w, 'configure'))] + self.tk.splitlist(self.tk.call(self._w, 'configure'))] def __str__(self): """Return the window path name of this widget.""" return self._w @@ -3825,16 +3832,10 @@ """ if cnf is None and not kw: - cnf = {} - for x in self.tk.split( - self.tk.call(self._w, - 'paneconfigure', tagOrId)): - cnf[x[0][1:]] = (x[0][1:],) + x[1:] - return cnf + return self._getconfigure(self._w, 'paneconfigure', tagOrId) if isinstance(cnf, str) and not kw: - x = self.tk.split(self.tk.call( - self._w, 'paneconfigure', tagOrId, '-'+cnf)) - return (x[0][1:],) + x[1:] + return self._getconfigure1( + self._w, 'paneconfigure', tagOrId, '-'+cnf) self.tk.call((self._w, 'paneconfigure', tagOrId) + self._options(cnf, kw)) paneconfig = paneconfigure diff --git a/Lib/tkinter/test/widget_tests.py b/Lib/tkinter/test/widget_tests.py --- a/Lib/tkinter/test/widget_tests.py +++ b/Lib/tkinter/test/widget_tests.py @@ -62,9 +62,7 @@ if not isinstance(widget, Scale): t = widget.configure(name) self.assertEqual(len(t), 5) - ## XXX - if not isinstance(t[4], tuple): - self.assertEqual2(t[4], expected, eq=eq) + self.assertEqual2(t[4], expected, eq=eq) def checkInvalidParam(self, widget, name, value, errmsg=None, *, keep_orig=True): diff --git a/Lib/tkinter/tix.py b/Lib/tkinter/tix.py --- a/Lib/tkinter/tix.py +++ b/Lib/tkinter/tix.py @@ -122,13 +122,9 @@ elif cnf: cnf = _cnfmerge(cnf) if cnf is None: - cnf = {} - for x in self.tk.split(self.tk.call('tix', 'configure')): - cnf[x[0][1:]] = (x[0][1:],) + x[1:] - return cnf + return self._getconfigure('tix', 'configure') if isinstance(cnf, str): - x = self.tk.split(self.tk.call('tix', 'configure', '-'+cnf)) - return (x[0][1:],) + x[1:] + return self._getconfigure1('tix', 'configure', '-'+cnf) return self.tk.call(('tix', 'configure') + self._options(cnf)) def tix_filedialog(self, dlgclass=None): @@ -380,7 +376,7 @@ """Return the name of all subwidgets.""" try: x = self.tk.call(self._w, 'subwidgets', '-all') - return self.tk.split(x) + return self.tk.splitlist(x) except TclError: return None @@ -473,13 +469,6 @@ self.tk.call('destroy', self._w) -# Useful func. to split Tcl lists and return as a dict. From Tkinter.py -def _lst2dict(lst): - dict = {} - for x in lst: - dict[x[0][1:]] = (x[0][1:],) + x[1:] - return dict - # Useful class to create a display style - later shared by many items. # Contributed by Steffen Kremser class DisplayStyle: @@ -515,10 +504,8 @@ self.tk.call(self.stylename, 'configure', '-%s'%key, value) def config(self, cnf={}, **kw): - return _lst2dict( - self.tk.split( - self.tk.call( - self.stylename, 'configure', *self._options(cnf,kw)))) + return self._getconfigure( + self.stylename, 'configure', *self._options(cnf,kw)) def __getitem__(self,key): return self.tk.call(self.stylename, 'cget', '-%s'%key) @@ -928,9 +915,7 @@ def header_configure(self, col, cnf={}, **kw): if cnf is None: - return _lst2dict( - self.tk.split( - self.tk.call(self._w, 'header', 'configure', col))) + return self._getconfigure(self._w, 'header', 'configure', col) self.tk.call(self._w, 'header', 'configure', col, *self._options(cnf, kw)) @@ -955,9 +940,8 @@ def indicator_configure(self, entry, cnf={}, **kw): if cnf is None: - return _lst2dict( - self.tk.split( - self.tk.call(self._w, 'indicator', 'configure', entry))) + return self._getconfigure( + self._w, 'indicator', 'configure', entry) self.tk.call( self._w, 'indicator', 'configure', entry, *self._options(cnf, kw)) @@ -1017,9 +1001,7 @@ def item_configure(self, entry, col, cnf={}, **kw): if cnf is None: - return _lst2dict( - self.tk.split( - self.tk.call(self._w, 'item', 'configure', entry, col))) + return self._getconfigure(self._w, 'item', 'configure', entry, col) self.tk.call(self._w, 'item', 'configure', entry, col, *self._options(cnf, kw)) @@ -1038,9 +1020,7 @@ def entryconfigure(self, entry, cnf={}, **kw): if cnf is None: - return _lst2dict( - self.tk.split( - self.tk.call(self._w, 'entryconfigure', entry))) + return self._getconfigure(self._w, 'entryconfigure', entry) self.tk.call(self._w, 'entryconfigure', entry, *self._options(cnf, kw)) @@ -1254,9 +1234,7 @@ def paneconfigure(self, entry, cnf={}, **kw): if cnf is None: - return _lst2dict( - self.tk.split( - self.tk.call(self._w, 'paneconfigure', entry))) + return self._getconfigure(self._w, 'paneconfigure', entry) self.tk.call(self._w, 'paneconfigure', entry, *self._options(cnf, kw)) def panes(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,9 @@ Library ------- +- Issue #19020: Tkinter now uses splitlist() instead of split() in configure + methods. + - Fix TypeError on "setup.py upload --show-response". - Issue #12226: HTTPS is now used by default when connecting to PyPI. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 25 15:43:26 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 25 Dec 2013 15:43:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319020=3A_Tkinter_now_uses_splitlist=28=29_inste?= =?utf-8?q?ad_of_split=28=29_in_configure?= Message-ID: <3dqH9B6fF2z7Ljv@mail.python.org> http://hg.python.org/cpython/rev/c6ba24ffa4ba changeset: 88184:c6ba24ffa4ba parent: 88181:2a4c083f8f6b parent: 88183:a8f5f8c44dc8 user: Serhiy Storchaka date: Wed Dec 25 16:36:43 2013 +0200 summary: Issue #19020: Tkinter now uses splitlist() instead of split() in configure methods. files: Lib/tkinter/__init__.py | 37 ++++++++-------- Lib/tkinter/test/widget_tests.py | 4 +- Lib/tkinter/tix.py | 44 +++++--------------- Misc/NEWS | 3 + 4 files changed, 34 insertions(+), 54 deletions(-) diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -1235,6 +1235,19 @@ exc, val, tb = sys.exc_info() root = self._root() root.report_callback_exception(exc, val, tb) + + def _getconfigure(self, *args): + """Call Tcl configure command and return the result as a dict.""" + cnf = {} + for x in self.tk.splitlist(self.tk.call(*args)): + x = self.tk.splitlist(x) + cnf[x[0][1:]] = (x[0][1:],) + x[1:] + return cnf + + def _getconfigure1(self, *args): + x = self.tk.splitlist(self.tk.call(*args)) + return (x[0][1:],) + x[1:] + def _configure(self, cmd, cnf, kw): """Internal function.""" if kw: @@ -1242,15 +1255,9 @@ elif cnf: cnf = _cnfmerge(cnf) if cnf is None: - cnf = {} - for x in self.tk.split( - self.tk.call(_flatten((self._w, cmd)))): - cnf[x[0][1:]] = (x[0][1:],) + x[1:] - return cnf + return self._getconfigure(_flatten((self._w, cmd))) if isinstance(cnf, str): - x = self.tk.split( - self.tk.call(_flatten((self._w, cmd, '-'+cnf)))) - return (x[0][1:],) + x[1:] + return self._getconfigure1(_flatten((self._w, cmd, '-'+cnf))) self.tk.call(_flatten((self._w, cmd)) + self._options(cnf)) # These used to be defined in Widget: def configure(self, cnf=None, **kw): @@ -1271,7 +1278,7 @@ def keys(self): """Return a list of all resource names of this widget.""" return [x[0][1:] for x in - self.tk.split(self.tk.call(self._w, 'configure'))] + self.tk.splitlist(self.tk.call(self._w, 'configure'))] def __str__(self): """Return the window path name of this widget.""" return self._w @@ -3786,16 +3793,10 @@ """ if cnf is None and not kw: - cnf = {} - for x in self.tk.split( - self.tk.call(self._w, - 'paneconfigure', tagOrId)): - cnf[x[0][1:]] = (x[0][1:],) + x[1:] - return cnf + return self._getconfigure(self._w, 'paneconfigure', tagOrId) if isinstance(cnf, str) and not kw: - x = self.tk.split(self.tk.call( - self._w, 'paneconfigure', tagOrId, '-'+cnf)) - return (x[0][1:],) + x[1:] + return self._getconfigure1( + self._w, 'paneconfigure', tagOrId, '-'+cnf) self.tk.call((self._w, 'paneconfigure', tagOrId) + self._options(cnf, kw)) paneconfig = paneconfigure diff --git a/Lib/tkinter/test/widget_tests.py b/Lib/tkinter/test/widget_tests.py --- a/Lib/tkinter/test/widget_tests.py +++ b/Lib/tkinter/test/widget_tests.py @@ -62,9 +62,7 @@ if not isinstance(widget, Scale): t = widget.configure(name) self.assertEqual(len(t), 5) - ## XXX - if not isinstance(t[4], tuple): - self.assertEqual2(t[4], expected, eq=eq) + self.assertEqual2(t[4], expected, eq=eq) def checkInvalidParam(self, widget, name, value, errmsg=None, *, keep_orig=True): diff --git a/Lib/tkinter/tix.py b/Lib/tkinter/tix.py --- a/Lib/tkinter/tix.py +++ b/Lib/tkinter/tix.py @@ -122,13 +122,9 @@ elif cnf: cnf = _cnfmerge(cnf) if cnf is None: - cnf = {} - for x in self.tk.split(self.tk.call('tix', 'configure')): - cnf[x[0][1:]] = (x[0][1:],) + x[1:] - return cnf + return self._getconfigure('tix', 'configure') if isinstance(cnf, str): - x = self.tk.split(self.tk.call('tix', 'configure', '-'+cnf)) - return (x[0][1:],) + x[1:] + return self._getconfigure1('tix', 'configure', '-'+cnf) return self.tk.call(('tix', 'configure') + self._options(cnf)) def tix_filedialog(self, dlgclass=None): @@ -380,7 +376,7 @@ """Return the name of all subwidgets.""" try: x = self.tk.call(self._w, 'subwidgets', '-all') - return self.tk.split(x) + return self.tk.splitlist(x) except TclError: return None @@ -473,13 +469,6 @@ self.tk.call('destroy', self._w) -# Useful func. to split Tcl lists and return as a dict. From Tkinter.py -def _lst2dict(lst): - dict = {} - for x in lst: - dict[x[0][1:]] = (x[0][1:],) + x[1:] - return dict - # Useful class to create a display style - later shared by many items. # Contributed by Steffen Kremser class DisplayStyle: @@ -515,10 +504,8 @@ self.tk.call(self.stylename, 'configure', '-%s'%key, value) def config(self, cnf={}, **kw): - return _lst2dict( - self.tk.split( - self.tk.call( - self.stylename, 'configure', *self._options(cnf,kw)))) + return self._getconfigure( + self.stylename, 'configure', *self._options(cnf,kw)) def __getitem__(self,key): return self.tk.call(self.stylename, 'cget', '-%s'%key) @@ -928,9 +915,7 @@ def header_configure(self, col, cnf={}, **kw): if cnf is None: - return _lst2dict( - self.tk.split( - self.tk.call(self._w, 'header', 'configure', col))) + return self._getconfigure(self._w, 'header', 'configure', col) self.tk.call(self._w, 'header', 'configure', col, *self._options(cnf, kw)) @@ -955,9 +940,8 @@ def indicator_configure(self, entry, cnf={}, **kw): if cnf is None: - return _lst2dict( - self.tk.split( - self.tk.call(self._w, 'indicator', 'configure', entry))) + return self._getconfigure( + self._w, 'indicator', 'configure', entry) self.tk.call( self._w, 'indicator', 'configure', entry, *self._options(cnf, kw)) @@ -1017,9 +1001,7 @@ def item_configure(self, entry, col, cnf={}, **kw): if cnf is None: - return _lst2dict( - self.tk.split( - self.tk.call(self._w, 'item', 'configure', entry, col))) + return self._getconfigure(self._w, 'item', 'configure', entry, col) self.tk.call(self._w, 'item', 'configure', entry, col, *self._options(cnf, kw)) @@ -1038,9 +1020,7 @@ def entryconfigure(self, entry, cnf={}, **kw): if cnf is None: - return _lst2dict( - self.tk.split( - self.tk.call(self._w, 'entryconfigure', entry))) + return self._getconfigure(self._w, 'entryconfigure', entry) self.tk.call(self._w, 'entryconfigure', entry, *self._options(cnf, kw)) @@ -1254,9 +1234,7 @@ def paneconfigure(self, entry, cnf={}, **kw): if cnf is None: - return _lst2dict( - self.tk.split( - self.tk.call(self._w, 'paneconfigure', entry))) + return self._getconfigure(self._w, 'paneconfigure', entry) self.tk.call(self._w, 'paneconfigure', entry, *self._options(cnf, kw)) def panes(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #19020: Tkinter now uses splitlist() instead of split() in configure + methods. + - Issue #19744: ensurepip now provides a better error message when Python is built without SSL/TLS support (pip currently requires that support to run, even if only operating with local wheel files) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 25 16:31:08 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 25 Dec 2013 16:31:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5MzIw?= =?utf-8?q?=3A_test=5Ftcl_no_longer_fails_when_wantobjects_is_false=2E?= Message-ID: <3dqJDD15Jdz7Lk9@mail.python.org> http://hg.python.org/cpython/rev/6fe3e855a276 changeset: 88185:6fe3e855a276 branch: 2.7 parent: 88182:ff70c298dd60 user: Serhiy Storchaka date: Wed Dec 25 17:28:50 2013 +0200 summary: Issue #19320: test_tcl no longer fails when wantobjects is false. files: Lib/test/test_tcl.py | 45 +++++++++++++++++++++---------- Misc/NEWS | 2 + 2 files changed, 32 insertions(+), 15 deletions(-) diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py --- a/Lib/test/test_tcl.py +++ b/Lib/test/test_tcl.py @@ -33,6 +33,7 @@ def setUp(self): self.interp = Tcl() + self.wantobjects = self.interp.tk.wantobjects() def testEval(self): tcl = self.interp @@ -178,24 +179,32 @@ def test_passing_values(self): def passValue(value): return self.interp.call('set', '_', value) - self.assertEqual(passValue(True), True) - self.assertEqual(passValue(False), False) - self.assertEqual(passValue('string'), 'string') - self.assertEqual(passValue('string\u20ac'), 'string\u20ac') + + self.assertEqual(passValue(True), True if self.wantobjects else '1') + self.assertEqual(passValue(False), False if self.wantobjects else '0') self.assertEqual(passValue(u'string'), u'string') self.assertEqual(passValue(u'string\u20ac'), u'string\u20ac') for i in (0, 1, -1, int(2**31-1), int(-2**31)): - self.assertEqual(passValue(i), i) + self.assertEqual(passValue(i), i if self.wantobjects else str(i)) for f in (0.0, 1.0, -1.0, 1//3, 1/3.0, sys.float_info.min, sys.float_info.max, -sys.float_info.min, -sys.float_info.max): - self.assertEqual(passValue(f), f) - for f in float('nan'), float('inf'), -float('inf'): - if f != f: # NaN - self.assertNotEqual(passValue(f), f) + if self.wantobjects: + self.assertEqual(passValue(f), f) else: - self.assertEqual(passValue(f), f) - self.assertEqual(passValue((1, '2', (3.4,))), (1, '2', (3.4,))) + self.assertEqual(float(passValue(f)), f) + if self.wantobjects: + f = passValue(float('nan')) + self.assertNotEqual(f, f) + self.assertEqual(passValue(float('inf')), float('inf')) + self.assertEqual(passValue(-float('inf')), -float('inf')) + else: + f = float(passValue(float('nan'))) + self.assertNotEqual(f, f) + self.assertEqual(float(passValue(float('inf'))), float('inf')) + self.assertEqual(float(passValue(-float('inf'))), -float('inf')) + self.assertEqual(passValue((1, '2', (3.4,))), + (1, '2', (3.4,)) if self.wantobjects else '1 2 3.4') def test_splitlist(self): splitlist = self.interp.tk.splitlist @@ -220,12 +229,15 @@ ('a 3.4', ('a', '3.4')), (('a', 3.4), ('a', 3.4)), ((), ()), - (call('list', 1, '2', (3.4,)), (1, '2', (3.4,))), + (call('list', 1, '2', (3.4,)), + (1, '2', (3.4,)) if self.wantobjects else + ('1', '2', '3.4')), ] if tcl_version >= (8, 5): testcases += [ (call('dict', 'create', 1, u'\u20ac', '\xe2\x82\xac', (3.4,)), - (1, u'\u20ac', u'\u20ac', (3.4,))), + (1, u'\u20ac', u'\u20ac', (3.4,)) if self.wantobjects else + ('1', '\xe2\x82\xac', '\xe2\x82\xac', '3.4')), ] for arg, res in testcases: self.assertEqual(splitlist(arg), res) @@ -257,12 +269,15 @@ (('a', 3.4), ('a', 3.4)), (('a', (2, 3.4)), ('a', (2, 3.4))), ((), ()), - (call('list', 1, '2', (3.4,)), (1, '2', (3.4,))), + (call('list', 1, '2', (3.4,)), + (1, '2', (3.4,)) if self.wantobjects else + ('1', '2', '3.4')), ] if tcl_version >= (8, 5): testcases += [ (call('dict', 'create', 12, u'\u20ac', '\xe2\x82\xac', (3.4,)), - (12, u'\u20ac', u'\u20ac', (3.4,))), + (12, u'\u20ac', u'\u20ac', (3.4,)) if self.wantobjects else + ('12', '\xe2\x82\xac', '\xe2\x82\xac', '3.4')), ] for arg, res in testcases: self.assertEqual(split(arg), res) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -132,6 +132,8 @@ Tests ----- +- Issue #19320: test_tcl no longer fails when wantobjects is false. + - Issue #19683: Removed empty tests from test_minidom. Initial patch by Ajitesh Gupta. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 25 16:31:09 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 25 Dec 2013 16:31:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5MzIw?= =?utf-8?q?=3A_test=5Ftcl_no_longer_fails_when_wantobjects_is_false=2E?= Message-ID: <3dqJDF4Cp9z7LkC@mail.python.org> http://hg.python.org/cpython/rev/6781a03d90c1 changeset: 88186:6781a03d90c1 branch: 3.3 parent: 88183:a8f5f8c44dc8 user: Serhiy Storchaka date: Wed Dec 25 17:29:01 2013 +0200 summary: Issue #19320: test_tcl no longer fails when wantobjects is false. files: Lib/test/test_tcl.py | 42 ++++++++++++++++++++++--------- Misc/NEWS | 2 + 2 files changed, 31 insertions(+), 13 deletions(-) diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py --- a/Lib/test/test_tcl.py +++ b/Lib/test/test_tcl.py @@ -35,6 +35,7 @@ def setUp(self): self.interp = Tcl() + self.wantobjects = self.interp.tk.wantobjects() def testEval(self): tcl = self.interp @@ -167,22 +168,31 @@ def passValue(value): return self.interp.call('set', '_', value) - self.assertEqual(passValue(True), True) - self.assertEqual(passValue(False), False) + self.assertEqual(passValue(True), True if self.wantobjects else '1') + self.assertEqual(passValue(False), False if self.wantobjects else '0') self.assertEqual(passValue('string'), 'string') self.assertEqual(passValue('string\u20ac'), 'string\u20ac') for i in (0, 1, -1, 2**31-1, -2**31): - self.assertEqual(passValue(i), i) + self.assertEqual(passValue(i), i if self.wantobjects else str(i)) for f in (0.0, 1.0, -1.0, 1/3, sys.float_info.min, sys.float_info.max, -sys.float_info.min, -sys.float_info.max): - self.assertEqual(passValue(f), f) - for f in float('nan'), float('inf'), -float('inf'): - if f != f: # NaN - self.assertNotEqual(passValue(f), f) + if self.wantobjects: + self.assertEqual(passValue(f), f) else: - self.assertEqual(passValue(f), f) - self.assertEqual(passValue((1, '2', (3.4,))), (1, '2', (3.4,))) + self.assertEqual(float(passValue(f)), f) + if self.wantobjects: + f = passValue(float('nan')) + self.assertNotEqual(f, f) + self.assertEqual(passValue(float('inf')), float('inf')) + self.assertEqual(passValue(-float('inf')), -float('inf')) + else: + f = float(passValue(float('nan'))) + self.assertNotEqual(f, f) + self.assertEqual(float(passValue(float('inf'))), float('inf')) + self.assertEqual(float(passValue(-float('inf'))), -float('inf')) + self.assertEqual(passValue((1, '2', (3.4,))), + (1, '2', (3.4,)) if self.wantobjects else '1 2 3.4') def test_splitlist(self): splitlist = self.interp.tk.splitlist @@ -207,12 +217,15 @@ ('a 3.4', ('a', '3.4')), (('a', 3.4), ('a', 3.4)), ((), ()), - (call('list', 1, '2', (3.4,)), (1, '2', (3.4,))), + (call('list', 1, '2', (3.4,)), + (1, '2', (3.4,)) if self.wantobjects else + ('1', '2', '3.4')), ] if tcl_version >= (8, 5): testcases += [ (call('dict', 'create', 1, '\u20ac', b'\xe2\x82\xac', (3.4,)), - (1, '\u20ac', '\u20ac', (3.4,))), + (1, '\u20ac', '\u20ac', (3.4,)) if self.wantobjects else + ('1', '\u20ac', '\u20ac', '3.4')), ] for arg, res in testcases: self.assertEqual(splitlist(arg), res, msg=arg) @@ -244,12 +257,15 @@ (('a', 3.4), ('a', 3.4)), (('a', (2, 3.4)), ('a', (2, 3.4))), ((), ()), - (call('list', 1, '2', (3.4,)), (1, '2', (3.4,))), + (call('list', 1, '2', (3.4,)), + (1, '2', (3.4,)) if self.wantobjects else + ('1', '2', '3.4')), ] if tcl_version >= (8, 5): testcases += [ (call('dict', 'create', 12, '\u20ac', b'\xe2\x82\xac', (3.4,)), - (12, '\u20ac', '\u20ac', (3.4,))), + (12, '\u20ac', '\u20ac', (3.4,)) if self.wantobjects else + ('12', '\u20ac', '\u20ac', '3.4')), ] for arg, res in testcases: self.assertEqual(split(arg), res, msg=arg) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -182,6 +182,8 @@ Tests ----- +- Issue #19320: test_tcl no longer fails when wantobjects is false. + - Issue #19683: Removed empty tests from test_minidom. Patch by Ajitesh Gupta. - Issue #19919: Fix flacky SSL test. connect_ex() sometimes returns -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 25 16:31:11 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 25 Dec 2013 16:31:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319320=3A_test=5Ftcl_no_longer_fails_when_wantob?= =?utf-8?q?jects_is_false=2E?= Message-ID: <3dqJDH0Cf2z7LkD@mail.python.org> http://hg.python.org/cpython/rev/78fa6dc5cc21 changeset: 88187:78fa6dc5cc21 parent: 88184:c6ba24ffa4ba parent: 88186:6781a03d90c1 user: Serhiy Storchaka date: Wed Dec 25 17:29:41 2013 +0200 summary: Issue #19320: test_tcl no longer fails when wantobjects is false. files: Lib/test/test_tcl.py | 42 ++++++++++++++++++++++--------- Misc/NEWS | 2 + 2 files changed, 31 insertions(+), 13 deletions(-) diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py --- a/Lib/test/test_tcl.py +++ b/Lib/test/test_tcl.py @@ -35,6 +35,7 @@ def setUp(self): self.interp = Tcl() + self.wantobjects = self.interp.tk.wantobjects() def testEval(self): tcl = self.interp @@ -167,22 +168,31 @@ def passValue(value): return self.interp.call('set', '_', value) - self.assertEqual(passValue(True), True) - self.assertEqual(passValue(False), False) + self.assertEqual(passValue(True), True if self.wantobjects else '1') + self.assertEqual(passValue(False), False if self.wantobjects else '0') self.assertEqual(passValue('string'), 'string') self.assertEqual(passValue('string\u20ac'), 'string\u20ac') for i in (0, 1, -1, 2**31-1, -2**31): - self.assertEqual(passValue(i), i) + self.assertEqual(passValue(i), i if self.wantobjects else str(i)) for f in (0.0, 1.0, -1.0, 1/3, sys.float_info.min, sys.float_info.max, -sys.float_info.min, -sys.float_info.max): - self.assertEqual(passValue(f), f) - for f in float('nan'), float('inf'), -float('inf'): - if f != f: # NaN - self.assertNotEqual(passValue(f), f) + if self.wantobjects: + self.assertEqual(passValue(f), f) else: - self.assertEqual(passValue(f), f) - self.assertEqual(passValue((1, '2', (3.4,))), (1, '2', (3.4,))) + self.assertEqual(float(passValue(f)), f) + if self.wantobjects: + f = passValue(float('nan')) + self.assertNotEqual(f, f) + self.assertEqual(passValue(float('inf')), float('inf')) + self.assertEqual(passValue(-float('inf')), -float('inf')) + else: + f = float(passValue(float('nan'))) + self.assertNotEqual(f, f) + self.assertEqual(float(passValue(float('inf'))), float('inf')) + self.assertEqual(float(passValue(-float('inf'))), -float('inf')) + self.assertEqual(passValue((1, '2', (3.4,))), + (1, '2', (3.4,)) if self.wantobjects else '1 2 3.4') def test_splitlist(self): splitlist = self.interp.tk.splitlist @@ -207,12 +217,15 @@ ('a 3.4', ('a', '3.4')), (('a', 3.4), ('a', 3.4)), ((), ()), - (call('list', 1, '2', (3.4,)), (1, '2', (3.4,))), + (call('list', 1, '2', (3.4,)), + (1, '2', (3.4,)) if self.wantobjects else + ('1', '2', '3.4')), ] if tcl_version >= (8, 5): testcases += [ (call('dict', 'create', 1, '\u20ac', b'\xe2\x82\xac', (3.4,)), - (1, '\u20ac', '\u20ac', (3.4,))), + (1, '\u20ac', '\u20ac', (3.4,)) if self.wantobjects else + ('1', '\u20ac', '\u20ac', '3.4')), ] for arg, res in testcases: self.assertEqual(splitlist(arg), res, msg=arg) @@ -244,12 +257,15 @@ (('a', 3.4), ('a', 3.4)), (('a', (2, 3.4)), ('a', (2, 3.4))), ((), ()), - (call('list', 1, '2', (3.4,)), (1, '2', (3.4,))), + (call('list', 1, '2', (3.4,)), + (1, '2', (3.4,)) if self.wantobjects else + ('1', '2', '3.4')), ] if tcl_version >= (8, 5): testcases += [ (call('dict', 'create', 12, '\u20ac', b'\xe2\x82\xac', (3.4,)), - (12, '\u20ac', '\u20ac', (3.4,))), + (12, '\u20ac', '\u20ac', (3.4,)) if self.wantobjects else + ('12', '\u20ac', '\u20ac', '3.4')), ] for arg, res in testcases: self.assertEqual(split(arg), res, msg=arg) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -253,6 +253,8 @@ Tests ----- +- Issue #19320: test_tcl no longer fails when wantobjects is false. + - Issue #19919: Fix flacky SSL test. connect_ex() sometimes returns EWOULDBLOCK on Windows or VMs hosted on Windows. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 25 16:36:25 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 25 Dec 2013 16:36:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogdGVzdF9kZWJ1ZyBp?= =?utf-8?q?n_test=5Ftkinter/test=5Ftext_no_longer_fails_when_wantobjects_i?= =?utf-8?q?s_false=2E?= Message-ID: <3dqJLK3SdTz7LjW@mail.python.org> http://hg.python.org/cpython/rev/50d8e0edfc3f changeset: 88188:50d8e0edfc3f branch: 2.7 parent: 88185:6fe3e855a276 user: Serhiy Storchaka date: Wed Dec 25 17:35:11 2013 +0200 summary: test_debug in test_tkinter/test_text no longer fails when wantobjects is false. files: Lib/lib-tk/test/test_tkinter/test_text.py | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Lib/lib-tk/test/test_tkinter/test_text.py b/Lib/lib-tk/test/test_tkinter/test_text.py --- a/Lib/lib-tk/test/test_tkinter/test_text.py +++ b/Lib/lib-tk/test/test_tkinter/test_text.py @@ -16,12 +16,13 @@ def test_debug(self): text = self.text + wantobjects = self.root.wantobjects() olddebug = text.debug() try: text.debug(0) - self.assertEqual(text.debug(), 0) + self.assertEqual(text.debug(), 0 if wantobjects else '0') text.debug(1) - self.assertEqual(text.debug(), 1) + self.assertEqual(text.debug(), 1 if wantobjects else '1') finally: text.debug(olddebug) self.assertEqual(text.debug(), olddebug) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 25 16:36:26 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 25 Dec 2013 16:36:26 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogdGVzdF9kZWJ1ZyBp?= =?utf-8?q?n_test=5Ftkinter/test=5Ftext_no_longer_fails_when_wantobjects_i?= =?utf-8?q?s_false=2E?= Message-ID: <3dqJLL56ybz7LjW@mail.python.org> http://hg.python.org/cpython/rev/07f4c2c8badf changeset: 88189:07f4c2c8badf branch: 3.3 parent: 88186:6781a03d90c1 user: Serhiy Storchaka date: Wed Dec 25 17:35:24 2013 +0200 summary: test_debug in test_tkinter/test_text no longer fails when wantobjects is false. files: Lib/tkinter/test/test_tkinter/test_text.py | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Lib/tkinter/test/test_tkinter/test_text.py b/Lib/tkinter/test/test_tkinter/test_text.py --- a/Lib/tkinter/test/test_tkinter/test_text.py +++ b/Lib/tkinter/test/test_tkinter/test_text.py @@ -16,12 +16,13 @@ def test_debug(self): text = self.text + wantobjects = self.root.wantobjects() olddebug = text.debug() try: text.debug(0) - self.assertEqual(text.debug(), 0) + self.assertEqual(text.debug(), 0 if wantobjects else '0') text.debug(1) - self.assertEqual(text.debug(), 1) + self.assertEqual(text.debug(), 1 if wantobjects else '1') finally: text.debug(olddebug) self.assertEqual(text.debug(), olddebug) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed Dec 25 16:36:27 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Wed, 25 Dec 2013 16:36:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_test=5Fdebug_in_test=5Ftkinter/test=5Ftext_no_longer_fai?= =?utf-8?q?ls_when_wantobjects_is_false=2E?= Message-ID: <3dqJLM6shmz7Ljp@mail.python.org> http://hg.python.org/cpython/rev/4e2b69138ab9 changeset: 88190:4e2b69138ab9 parent: 88187:78fa6dc5cc21 parent: 88189:07f4c2c8badf user: Serhiy Storchaka date: Wed Dec 25 17:36:20 2013 +0200 summary: test_debug in test_tkinter/test_text no longer fails when wantobjects is false. files: Lib/tkinter/test/test_tkinter/test_text.py | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Lib/tkinter/test/test_tkinter/test_text.py b/Lib/tkinter/test/test_tkinter/test_text.py --- a/Lib/tkinter/test/test_tkinter/test_text.py +++ b/Lib/tkinter/test/test_tkinter/test_text.py @@ -16,12 +16,13 @@ def test_debug(self): text = self.text + wantobjects = self.root.wantobjects() olddebug = text.debug() try: text.debug(0) - self.assertEqual(text.debug(), 0) + self.assertEqual(text.debug(), 0 if wantobjects else '0') text.debug(1) - self.assertEqual(text.debug(), 1) + self.assertEqual(text.debug(), 1 if wantobjects else '1') finally: text.debug(olddebug) self.assertEqual(text.debug(), olddebug) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 26 04:27:15 2013 From: python-checkins at python.org (r.david.murray) Date: Thu, 26 Dec 2013 04:27:15 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogIzIwMDYzOiBSZW1v?= =?utf-8?q?ve_inaccurate/confusing_statement_about_support_of_=27pop=27_me?= =?utf-8?q?thod=2E?= Message-ID: <3dqc6W6rWQz7LjM@mail.python.org> http://hg.python.org/cpython/rev/3598805d7636 changeset: 88191:3598805d7636 branch: 2.7 parent: 88188:50d8e0edfc3f user: R David Murray date: Wed Dec 25 22:26:59 2013 -0500 summary: #20063: Remove inaccurate/confusing statement about support of 'pop' method. Patch by Gennadiy Zlobin. files: Doc/library/stdtypes.rst | 5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1668,9 +1668,8 @@ Previously, all negative indices were truncated to zero. (6) - The :meth:`pop` method is only supported by the list and array types. The - optional argument *i* defaults to ``-1``, so that by default the last item is - removed and returned. + The :meth:`pop` method's optional argument *i* defaults to ``-1``, so that + by default the last item is removed and returned. (7) The :meth:`sort` and :meth:`reverse` methods modify the list in place for -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 26 05:21:57 2013 From: python-checkins at python.org (r.david.murray) Date: Thu, 26 Dec 2013 05:21:57 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_shelf_in_with_stmt=2C_fish?= =?utf-8?q?/csh_in_venv=2C_ElementTree_short=5Fempty=5Felements_keyword=2E?= Message-ID: <3dqdKd55Pdz7LjM@mail.python.org> http://hg.python.org/cpython/rev/94a04b8b3a12 changeset: 88192:94a04b8b3a12 parent: 88190:4e2b69138ab9 user: R David Murray date: Wed Dec 25 23:21:03 2013 -0500 summary: shelf in with stmt, fish/csh in venv, ElementTree short_empty_elements keyword. Also added versionadded for for fish/csh, fixed indentation of versionadded for ElementTree.write, and make the behavior of shelf as a context manager explicit in the docs. files: Doc/library/shelve.rst | 3 +- Doc/library/xml.etree.elementtree.rst | 4 +- Doc/using/venv-create.inc | 2 + Doc/whatsnew/3.4.rst | 25 +++++++++++++++ 4 files changed, 31 insertions(+), 3 deletions(-) diff --git a/Doc/library/shelve.rst b/Doc/library/shelve.rst --- a/Doc/library/shelve.rst +++ b/Doc/library/shelve.rst @@ -121,7 +121,8 @@ The *keyencoding* parameter is the encoding used to encode keys before they are used with the underlying dict. - :class:`Shelf` objects can also be used as context managers. + A :class:`Shelf` object can also be used as a context manager, in which + case it will be automatically closed when the :keyword:`with` block ends. .. versionchanged:: 3.2 Added the *keyencoding* parameter; previously, keys were always encoded in diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -831,8 +831,8 @@ :term:`file object`; make sure you do not try to write a string to a binary stream and vice versa. - .. versionadded:: 3.4 - The *short_empty_elements* parameter. + .. versionadded:: 3.4 + The *short_empty_elements* parameter. This is the XML file that is going to be manipulated:: diff --git a/Doc/using/venv-create.inc b/Doc/using/venv-create.inc --- a/Doc/using/venv-create.inc +++ b/Doc/using/venv-create.inc @@ -92,3 +92,5 @@ ``deactivate.bat`` and ``Deactivate.ps1`` which are installed when the venv is created. +.. versionadded:: 3.4 + ``fish`` and ``csh`` activation scripts. diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -823,6 +823,15 @@ New :func:`resource.prlimit` function and Linux specific constants. (Contributed by Christian Heimes in :issue:`16595` and :issue:`19324`.) + +shelve +------ + +:class:`~shelve.Shelf` instances may now be used in :keyword:`with` statements, +and will be automatically closed at the end of the :keyword:`with` block. +(Contributed by Filip Gruszczy?ski in :issue:`13896`.) + + smtplib ------- @@ -927,6 +936,13 @@ (Contributed by Antoine Pitrou in :issue:`16997`.) +venv +---- + +:mod:`venv` now includes activation scripts for the ``csh`` and ``fish`` +shells (Contributed by Andrew Svetlov in :issue:`15417`.) + + wave ---- @@ -956,6 +972,15 @@ :class:`~xml.etree.ElementTree.XMLPullParser`. (Contributed by Antoine Pitrou in :issue:`17741`.) +The :mod:`xml.etree.ElementTree` :func:`~xml.etree.ElementTree.tostring` and +:func:`~xml.etree.ElementTree.tostringlist` functions, and the +:class:`~xml.etree.ElementTree.ElementTree` +:meth:`~xml.etree.ElementTree.ElementTree.write` method, now have a +*short_empty_elements* :ref:`keyword-only parameter ` +providing control over whether elements with no content are written in +abbreviated (````) or expanded (````) form. (Contributed by +Ariel Poliak and Serhiy Storchaka in :issue:`14377`.) + zipfile.PyZipfile ----------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 26 16:55:43 2013 From: python-checkins at python.org (zach.ware) Date: Thu, 26 Dec 2013 16:55:43 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5OTM4?= =?utf-8?q?=3A_Re-enabled_test=5Fbug=5F1333982_in_test=5Fdis=2C_which_had_?= =?utf-8?q?been?= Message-ID: <3dqwk718rqz7LjT@mail.python.org> http://hg.python.org/cpython/rev/e04fc45b7555 changeset: 88193:e04fc45b7555 branch: 3.3 parent: 88189:07f4c2c8badf user: Zachary Ware date: Thu Dec 26 09:53:49 2013 -0600 summary: Issue #19938: Re-enabled test_bug_1333982 in test_dis, which had been disabled since 3.0 due to the changes in listcomp handling. files: Lib/test/test_dis.py | 59 +++++++++++++++---------------- Misc/NEWS | 3 + 2 files changed, 31 insertions(+), 31 deletions(-) diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py --- a/Lib/test/test_dis.py +++ b/Lib/test/test_dis.py @@ -6,6 +6,7 @@ import sys import dis import io +import re class _C: def __init__(self, x): @@ -89,26 +90,25 @@ dis_bug1333982 = """\ %-4d 0 LOAD_CONST 1 (0) - 3 JUMP_IF_TRUE 33 (to 39) - 6 POP_TOP - 7 LOAD_GLOBAL 0 (AssertionError) - 10 BUILD_LIST 0 - 13 LOAD_FAST 0 (x) - 16 GET_ITER - >> 17 FOR_ITER 12 (to 32) - 20 STORE_FAST 1 (s) - 23 LOAD_FAST 1 (s) - 26 LIST_APPEND 2 - 29 JUMP_ABSOLUTE 17 + 3 POP_JUMP_IF_TRUE 35 + 6 LOAD_GLOBAL 0 (AssertionError) + 9 LOAD_CONST 2 ( at 0x..., file "%s", line %d>) + 12 LOAD_CONST 3 ('bug1333982..') + 15 MAKE_FUNCTION 0 + 18 LOAD_FAST 0 (x) + 21 GET_ITER + 22 CALL_FUNCTION 1 (1 positional, 0 keyword pair) - %-4d >> 32 LOAD_CONST 2 (1) - 35 BINARY_ADD - 36 RAISE_VARARGS 2 - >> 39 POP_TOP + %-4d 25 LOAD_CONST 4 (1) + 28 BINARY_ADD + 29 CALL_FUNCTION 1 (1 positional, 0 keyword pair) + 32 RAISE_VARARGS 1 - %-4d 40 LOAD_CONST 0 (None) - 43 RETURN_VALUE + %-4d >> 35 LOAD_CONST 0 (None) + 38 RETURN_VALUE """ % (bug1333982.__code__.co_firstlineno + 1, + __file__, + bug1333982.__code__.co_firstlineno + 1, bug1333982.__code__.co_firstlineno + 2, bug1333982.__code__.co_firstlineno + 3) @@ -193,8 +193,13 @@ def do_disassembly_test(self, func, expected): lines = self.get_disassembly(func) expected = expected.splitlines() - if expected != lines: - self.fail( + if expected == lines: + return + else: + lines = [re.sub('0x[0-9A-Fa-f]+', '0x...', l) for l in lines] + if expected == lines: + return + self.fail( "events did not match expectation:\n" + "\n".join(difflib.ndiff(expected, lines))) @@ -217,18 +222,13 @@ def test_bug_708901(self): self.do_disassembly_test(bug708901, dis_bug708901) - # Test has been disabled due to change in the way - # list comps are handled. The byte code now includes - # a memory address and a file location, so they change from - # run to run. - @unittest.skip('disabled due to a change in the way list comps are handled') def test_bug_1333982(self): - # XXX: re-enable this test! # This one is checking bytecodes generated for an `assert` statement, # so fails if the tests are run with -O. Skip this test then. + if not __debug__: + self.skipTest('need asserts, run without -O') - if __debug__: - self.do_disassembly_test(bug1333982, dis_bug1333982) + self.do_disassembly_test(bug1333982, dis_bug1333982) def test_big_linenos(self): def func(count): @@ -451,8 +451,5 @@ self.assertEqual(dis.pretty_flags(0), '0x0') -def test_main(): - run_unittest(DisTests, CodeInfoTests) - if __name__ == "__main__": - test_main() + unittest.main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -182,6 +182,9 @@ Tests ----- +- Issue #19938: Re-enabled test_bug_1333982 in test_dis, which had been + disabled since 3.0 due to the changes in listcomp handling. + - Issue #19320: test_tcl no longer fails when wantobjects is false. - Issue #19683: Removed empty tests from test_minidom. Patch by Ajitesh Gupta. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 26 16:55:44 2013 From: python-checkins at python.org (zach.ware) Date: Thu, 26 Dec 2013 16:55:44 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319938=3A_Re-enabled_test=5Fbug=5F1333982_in_tes?= =?utf-8?q?t=5Fdis=2C_which_had_been?= Message-ID: <3dqwk84J9wz7LjZ@mail.python.org> http://hg.python.org/cpython/rev/285313c95e37 changeset: 88194:285313c95e37 parent: 88192:94a04b8b3a12 parent: 88193:e04fc45b7555 user: Zachary Ware date: Thu Dec 26 09:55:24 2013 -0600 summary: Issue #19938: Re-enabled test_bug_1333982 in test_dis, which had been disabled since 3.0 due to the changes in listcomp handling. files: Lib/test/test_dis.py | 61 +++++++++++++++---------------- Misc/NEWS | 3 + 2 files changed, 32 insertions(+), 32 deletions(-) diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py --- a/Lib/test/test_dis.py +++ b/Lib/test/test_dis.py @@ -7,6 +7,7 @@ import sys import dis import io +import re import types import contextlib @@ -106,27 +107,26 @@ pass dis_bug1333982 = """\ - %-4d 0 LOAD_CONST 1 (0) - 3 JUMP_IF_TRUE 33 (to 39) - 6 POP_TOP - 7 LOAD_GLOBAL 0 (AssertionError) - 10 BUILD_LIST 0 - 13 LOAD_FAST 0 (x) - 16 GET_ITER - >> 17 FOR_ITER 12 (to 32) - 20 STORE_FAST 1 (s) - 23 LOAD_FAST 1 (s) - 26 LIST_APPEND 2 - 29 JUMP_ABSOLUTE 17 +%3d 0 LOAD_CONST 1 (0) + 3 POP_JUMP_IF_TRUE 35 + 6 LOAD_GLOBAL 0 (AssertionError) + 9 LOAD_CONST 2 ( at 0x..., file "%s", line %d>) + 12 LOAD_CONST 3 ('bug1333982..') + 15 MAKE_FUNCTION 0 + 18 LOAD_FAST 0 (x) + 21 GET_ITER + 22 CALL_FUNCTION 1 (1 positional, 0 keyword pair) - %-4d >> 32 LOAD_CONST 2 (1) - 35 BINARY_ADD - 36 RAISE_VARARGS 2 - >> 39 POP_TOP +%3d 25 LOAD_CONST 4 (1) + 28 BINARY_ADD + 29 CALL_FUNCTION 1 (1 positional, 0 keyword pair) + 32 RAISE_VARARGS 1 - %-4d 40 LOAD_CONST 0 (None) - 43 RETURN_VALUE +%3d >> 35 LOAD_CONST 0 (None) + 38 RETURN_VALUE """ % (bug1333982.__code__.co_firstlineno + 1, + __file__, + bug1333982.__code__.co_firstlineno + 1, bug1333982.__code__.co_firstlineno + 2, bug1333982.__code__.co_firstlineno + 3) @@ -244,8 +244,14 @@ def get_disassemble_as_string(self, func, lasti=-1): return self.get_disassembly(func, lasti, False) + def strip_addresses(self, text): + return re.sub(r'\b0x[0-9A-Fa-f]+\b', '0x...', text) + def do_disassembly_test(self, func, expected): - self.assertEqual(self.get_disassembly(func), expected) + got = self.get_disassembly(func) + if got != expected: + got = self.strip_addresses(got) + self.assertEqual(got, expected) def test_opmap(self): self.assertEqual(dis.opmap["NOP"], 9) @@ -265,18 +271,13 @@ def test_bug_708901(self): self.do_disassembly_test(bug708901, dis_bug708901) - # Test has been disabled due to change in the way - # list comps are handled. The byte code now includes - # a memory address and a file location, so they change from - # run to run. - @unittest.skip('disabled due to a change in the way list comps are handled') def test_bug_1333982(self): - # XXX: re-enable this test! # This one is checking bytecodes generated for an `assert` statement, # so fails if the tests are run with -O. Skip this test then. + if not __debug__: + self.skipTest('need asserts, run without -O') - if __debug__: - self.do_disassembly_test(bug1333982, dis_bug1333982) + self.do_disassembly_test(bug1333982, dis_bug1333982) def test_big_linenos(self): def func(count): @@ -827,9 +828,5 @@ b = dis.Bytecode.from_traceback(tb) self.assertEqual(b.dis(), dis_traceback) -def test_main(): - run_unittest(DisTests, DisWithFileTests, CodeInfoTests, - InstructionTests, BytecodeTests) - if __name__ == "__main__": - test_main() + unittest.main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -253,6 +253,9 @@ Tests ----- +- Issue #19938: Re-enabled test_bug_1333982 in test_dis, which had been + disabled since 3.0 due to the changes in listcomp handling. + - Issue #19320: test_tcl no longer fails when wantobjects is false. - Issue #19919: Fix flacky SSL test. connect_ex() sometimes returns -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 26 17:37:33 2013 From: python-checkins at python.org (matthias.klose) Date: Thu, 26 Dec 2013 17:37:33 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_-_Issue_=2320070=3A_Don=27?= =?utf-8?q?t_run_test=5Furllib2net_when_network_resources_are_not?= Message-ID: <3dqxfP32y7z7LjS@mail.python.org> http://hg.python.org/cpython/rev/e00bfb70a0c0 changeset: 88195:e00bfb70a0c0 user: doko at ubuntu.com date: Thu Dec 26 17:37:11 2013 +0100 summary: - Issue #20070: Don't run test_urllib2net when network resources are not enabled. files: Lib/test/test_urllib2net.py | 3 ++- Misc/NEWS | 3 +++ 2 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_urllib2net.py b/Lib/test/test_urllib2net.py --- a/Lib/test/test_urllib2net.py +++ b/Lib/test/test_urllib2net.py @@ -14,6 +14,8 @@ except ImportError: ssl = None +support.requires("network") + TIMEOUT = 60 # seconds @@ -339,5 +341,4 @@ if __name__ == "__main__": - support.requires("network") unittest.main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -253,6 +253,9 @@ Tests ----- +- Issue #20070: Don't run test_urllib2net when network resources are not + enabled. + - Issue #19938: Re-enabled test_bug_1333982 in test_dis, which had been disabled since 3.0 due to the changes in listcomp handling. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 26 19:09:02 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 26 Dec 2013 19:09:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIwMDY3?= =?utf-8?q?=3A_Tkinter_variables_now_work_when_wantobjects_is_false=2E?= Message-ID: <3dqzgy0JCTz7Ljj@mail.python.org> http://hg.python.org/cpython/rev/b13c15a9ae54 changeset: 88196:b13c15a9ae54 branch: 2.7 parent: 88191:3598805d7636 user: Serhiy Storchaka date: Thu Dec 26 20:05:53 2013 +0200 summary: Issue #20067: Tkinter variables now work when wantobjects is false. files: Lib/lib-tk/Tkinter.py | 6 ++++-- Misc/NEWS | 2 ++ 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/Lib/lib-tk/Tkinter.py b/Lib/lib-tk/Tkinter.py --- a/Lib/lib-tk/Tkinter.py +++ b/Lib/lib-tk/Tkinter.py @@ -223,11 +223,13 @@ _varnum += 1 if value is not None: self.set(value) - elif not self._tk.call("info", "exists", self._name): + elif not self._tk.getboolean(self._tk.call("info", "exists", self._name)): self.set(self._default) def __del__(self): """Unset the variable in Tcl.""" - self._tk.globalunsetvar(self._name) + if (self._tk is not None and + self._tk.getboolean(self._tk.call("info", "exists", self._name))): + self._tk.globalunsetvar(self._name) def __str__(self): """Return the name of the variable in Tcl.""" return self._name diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -27,6 +27,8 @@ Library ------- +- Issue #20067: Tkinter variables now work when wantobjects is false. + - Issue #19020: Tkinter now uses splitlist() instead of split() in configure methods. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 26 19:09:03 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 26 Dec 2013 19:09:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzIwMDY3?= =?utf-8?q?=3A_Tkinter_variables_now_work_when_wantobjects_is_false=2E?= Message-ID: <3dqzgz2D29z7Ljn@mail.python.org> http://hg.python.org/cpython/rev/fbc1a39b68e4 changeset: 88197:fbc1a39b68e4 branch: 3.3 parent: 88193:e04fc45b7555 user: Serhiy Storchaka date: Thu Dec 26 20:06:05 2013 +0200 summary: Issue #20067: Tkinter variables now work when wantobjects is false. files: Lib/tkinter/__init__.py | 6 ++-- Lib/tkinter/test/test_tkinter/test_variables.py | 15 ++++++---- Misc/NEWS | 2 + 3 files changed, 14 insertions(+), 9 deletions(-) diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -220,12 +220,12 @@ _varnum += 1 if value is not None: self.initialize(value) - elif not self._tk.call("info", "exists", self._name): + elif not self._tk.getboolean(self._tk.call("info", "exists", self._name)): self.initialize(self._default) def __del__(self): """Unset the variable in Tcl.""" - if (self._tk is not None and self._tk.call("info", "exists", - self._name)): + if (self._tk is not None and + self._tk.getboolean(self._tk.call("info", "exists", self._name))): self._tk.globalunsetvar(self._name) def __str__(self): """Return the name of the variable in Tcl.""" diff --git a/Lib/tkinter/test/test_tkinter/test_variables.py b/Lib/tkinter/test/test_tkinter/test_variables.py --- a/Lib/tkinter/test/test_tkinter/test_variables.py +++ b/Lib/tkinter/test/test_tkinter/test_variables.py @@ -24,6 +24,9 @@ class TestVariable(TestBase): + def info_exists(self, *args): + return self.root.getboolean(self.root.call("info", "exists", *args)) + def test_default(self): v = Variable(self.root) self.assertEqual("", v.get()) @@ -35,21 +38,21 @@ self.assertEqual("varname", str(v)) def test___del__(self): - self.assertFalse(self.root.call("info", "exists", "varname")) + self.assertFalse(self.info_exists("varname")) v = Variable(self.root, "sample string", "varname") - self.assertTrue(self.root.call("info", "exists", "varname")) + self.assertTrue(self.info_exists("varname")) del v - self.assertFalse(self.root.call("info", "exists", "varname")) + self.assertFalse(self.info_exists("varname")) def test_dont_unset_not_existing(self): - self.assertFalse(self.root.call("info", "exists", "varname")) + self.assertFalse(self.info_exists("varname")) v1 = Variable(self.root, name="name") v2 = Variable(self.root, name="name") del v1 - self.assertFalse(self.root.call("info", "exists", "name")) + self.assertFalse(self.info_exists("name")) # shouldn't raise exception del v2 - self.assertFalse(self.root.call("info", "exists", "name")) + self.assertFalse(self.info_exists("name")) def test___eq__(self): # values doesn't matter, only class and name are checked diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,8 @@ Library ------- +- Issue #20067: Tkinter variables now work when wantobjects is false. + - Issue #19020: Tkinter now uses splitlist() instead of split() in configure methods. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 26 19:09:05 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 26 Dec 2013 19:09:05 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2320067=3A_Tkinter_variables_now_work_when_wantob?= =?utf-8?q?jects_is_false=2E?= Message-ID: <3dqzh11WjNz7Lk2@mail.python.org> http://hg.python.org/cpython/rev/99df5128d978 changeset: 88198:99df5128d978 parent: 88195:e00bfb70a0c0 parent: 88197:fbc1a39b68e4 user: Serhiy Storchaka date: Thu Dec 26 20:06:43 2013 +0200 summary: Issue #20067: Tkinter variables now work when wantobjects is false. files: Lib/tkinter/__init__.py | 6 ++-- Lib/tkinter/test/test_tkinter/test_variables.py | 15 ++++++---- Misc/NEWS | 2 + 3 files changed, 14 insertions(+), 9 deletions(-) diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -220,12 +220,12 @@ _varnum += 1 if value is not None: self.initialize(value) - elif not self._tk.call("info", "exists", self._name): + elif not self._tk.getboolean(self._tk.call("info", "exists", self._name)): self.initialize(self._default) def __del__(self): """Unset the variable in Tcl.""" - if (self._tk is not None and self._tk.call("info", "exists", - self._name)): + if (self._tk is not None and + self._tk.getboolean(self._tk.call("info", "exists", self._name))): self._tk.globalunsetvar(self._name) def __str__(self): """Return the name of the variable in Tcl.""" diff --git a/Lib/tkinter/test/test_tkinter/test_variables.py b/Lib/tkinter/test/test_tkinter/test_variables.py --- a/Lib/tkinter/test/test_tkinter/test_variables.py +++ b/Lib/tkinter/test/test_tkinter/test_variables.py @@ -24,6 +24,9 @@ class TestVariable(TestBase): + def info_exists(self, *args): + return self.root.getboolean(self.root.call("info", "exists", *args)) + def test_default(self): v = Variable(self.root) self.assertEqual("", v.get()) @@ -35,21 +38,21 @@ self.assertEqual("varname", str(v)) def test___del__(self): - self.assertFalse(self.root.call("info", "exists", "varname")) + self.assertFalse(self.info_exists("varname")) v = Variable(self.root, "sample string", "varname") - self.assertTrue(self.root.call("info", "exists", "varname")) + self.assertTrue(self.info_exists("varname")) del v - self.assertFalse(self.root.call("info", "exists", "varname")) + self.assertFalse(self.info_exists("varname")) def test_dont_unset_not_existing(self): - self.assertFalse(self.root.call("info", "exists", "varname")) + self.assertFalse(self.info_exists("varname")) v1 = Variable(self.root, name="name") v2 = Variable(self.root, name="name") del v1 - self.assertFalse(self.root.call("info", "exists", "name")) + self.assertFalse(self.info_exists("name")) # shouldn't raise exception del v2 - self.assertFalse(self.root.call("info", "exists", "name")) + self.assertFalse(self.info_exists("name")) def test___eq__(self): # values doesn't matter, only class and name are checked diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,8 @@ Library ------- +- Issue #20067: Tkinter variables now work when wantobjects is false. + - Issue #19020: Tkinter now uses splitlist() instead of split() in configure methods. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 26 19:09:06 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 26 Dec 2013 19:09:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Backported_tes?= =?utf-8?q?ts_for_Tkinter_variables=2E?= Message-ID: <3dqzh24QbPz7Lk0@mail.python.org> http://hg.python.org/cpython/rev/9f2ace2bbb13 changeset: 88199:9f2ace2bbb13 branch: 2.7 parent: 88196:b13c15a9ae54 user: Serhiy Storchaka date: Thu Dec 26 20:08:34 2013 +0200 summary: Backported tests for Tkinter variables. files: Lib/lib-tk/test/test_tkinter/test_variables.py | 152 ++++++++++ Misc/NEWS | 2 + 2 files changed, 154 insertions(+), 0 deletions(-) diff --git a/Lib/lib-tk/test/test_tkinter/test_variables.py b/Lib/lib-tk/test/test_tkinter/test_variables.py new file mode 100644 --- /dev/null +++ b/Lib/lib-tk/test/test_tkinter/test_variables.py @@ -0,0 +1,152 @@ +import unittest + +from Tkinter import Variable, StringVar, IntVar, DoubleVar, BooleanVar, Tk, TclError + + +class TestBase(unittest.TestCase): + + def setUp(self): + self.root = Tk() + + def tearDown(self): + self.root.destroy() + + +class TestVariable(TestBase): + + def info_exists(self, *args): + return self.root.getboolean(self.root.call("info", "exists", *args)) + + def test_default(self): + v = Variable(self.root) + self.assertEqual("", v.get()) + self.assertRegexpMatches(str(v), r"^PY_VAR(\d+)$") + + def test_name_and_value(self): + v = Variable(self.root, "sample string", "varname") + self.assertEqual("sample string", v.get()) + self.assertEqual("varname", str(v)) + + def test___del__(self): + self.assertFalse(self.info_exists("varname")) + v = Variable(self.root, "sample string", "varname") + self.assertTrue(self.info_exists("varname")) + del v + self.assertFalse(self.info_exists("varname")) + + def test_dont_unset_not_existing(self): + self.assertFalse(self.info_exists("varname")) + v1 = Variable(self.root, name="name") + v2 = Variable(self.root, name="name") + del v1 + self.assertFalse(self.info_exists("name")) + # shouldn't raise exception + del v2 + self.assertFalse(self.info_exists("name")) + + def test___eq__(self): + # values doesn't matter, only class and name are checked + v1 = Variable(self.root, name="abc") + v2 = Variable(self.root, name="abc") + self.assertEqual(v1, v2) + + v3 = Variable(self.root, name="abc") + v4 = StringVar(self.root, name="abc") + self.assertNotEqual(v3, v4) + + def test_invalid_name(self): + with self.assertRaises(TypeError): + Variable(self.root, name=123) + + +class TestStringVar(TestBase): + + def test_default(self): + v = StringVar(self.root) + self.assertEqual("", v.get()) + + def test_get(self): + v = StringVar(self.root, "abc", "name") + self.assertEqual("abc", v.get()) + self.root.globalsetvar("name", "value") + self.assertEqual("value", v.get()) + + +class TestIntVar(TestBase): + + def test_default(self): + v = IntVar(self.root) + self.assertEqual(0, v.get()) + + def test_get(self): + v = IntVar(self.root, 123, "name") + self.assertEqual(123, v.get()) + self.root.globalsetvar("name", "345") + self.assertEqual(345, v.get()) + + def test_invalid_value(self): + v = IntVar(self.root, name="name") + self.root.globalsetvar("name", "value") + with self.assertRaises(ValueError): + v.get() + self.root.globalsetvar("name", "345.0") + with self.assertRaises(ValueError): + v.get() + + +class TestDoubleVar(TestBase): + + def test_default(self): + v = DoubleVar(self.root) + self.assertEqual(0.0, v.get()) + + def test_get(self): + v = DoubleVar(self.root, 1.23, "name") + self.assertAlmostEqual(1.23, v.get()) + self.root.globalsetvar("name", "3.45") + self.assertAlmostEqual(3.45, v.get()) + + def test_get_from_int(self): + v = DoubleVar(self.root, 1.23, "name") + self.assertAlmostEqual(1.23, v.get()) + self.root.globalsetvar("name", "3.45") + self.assertAlmostEqual(3.45, v.get()) + self.root.globalsetvar("name", "456") + self.assertAlmostEqual(456, v.get()) + + def test_invalid_value(self): + v = DoubleVar(self.root, name="name") + self.root.globalsetvar("name", "value") + with self.assertRaises(ValueError): + v.get() + + +class TestBooleanVar(TestBase): + + def test_default(self): + v = BooleanVar(self.root) + self.assertEqual(False, v.get()) + + def test_get(self): + v = BooleanVar(self.root, True, "name") + self.assertAlmostEqual(True, v.get()) + self.root.globalsetvar("name", "0") + self.assertAlmostEqual(False, v.get()) + + def test_invalid_value_domain(self): + v = BooleanVar(self.root, name="name") + self.root.globalsetvar("name", "value") + with self.assertRaises(TclError): + v.get() + self.root.globalsetvar("name", "1.0") + with self.assertRaises(TclError): + v.get() + + +tests_gui = (TestVariable, TestStringVar, TestIntVar, + TestDoubleVar, TestBooleanVar) + + +if __name__ == "__main__": + from test.support import run_unittest + run_unittest(*tests_gui) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -134,6 +134,8 @@ Tests ----- +- Backported tests for Tkinter variables. + - Issue #19320: test_tcl no longer fails when wantobjects is false. - Issue #19683: Removed empty tests from test_minidom. Initial patch by -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 26 20:22:17 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 26 Dec 2013 20:22:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIwMDI3?= =?utf-8?q?=3A_Fixed_locale_aliases_for_devanagari_locales=2E?= Message-ID: <3dr1JT51PQz7LkJ@mail.python.org> http://hg.python.org/cpython/rev/aad582f717da changeset: 88200:aad582f717da branch: 2.7 user: Serhiy Storchaka date: Thu Dec 26 21:20:46 2013 +0200 summary: Issue #20027: Fixed locale aliases for devanagari locales. files: Lib/locale.py | 7 ++++--- Lib/test/test_locale.py | 14 ++++++++++++++ Misc/NEWS | 2 ++ Tools/i18n/makelocalealias.py | 6 ++++++ 4 files changed, 26 insertions(+), 3 deletions(-) diff --git a/Lib/locale.py b/Lib/locale.py --- a/Lib/locale.py +++ b/Lib/locale.py @@ -1311,7 +1311,7 @@ 'korean.euc': 'ko_KR.eucKR', 'ks': 'ks_IN.UTF-8', 'ks_in': 'ks_IN.UTF-8', - 'ks_in at devanagari': 'ks_IN at devanagari.UTF-8', + 'ks_in at devanagari': 'ks_IN.UTF-8 at devanagari', 'kw': 'kw_GB.ISO8859-1', 'kw_gb': 'kw_GB.ISO8859-1', 'kw_gb.iso88591': 'kw_GB.ISO8859-1', @@ -1476,8 +1476,9 @@ 'rw_rw': 'rw_RW.ISO8859-1', 'rw_rw.iso88591': 'rw_RW.ISO8859-1', 'sd': 'sd_IN.UTF-8', - 'sd at devanagari': 'sd_IN at devanagari.UTF-8', - 'sd_in at devanagari': 'sd_IN at devanagari.UTF-8', + 'sd at devanagari': 'sd_IN.UTF-8 at devanagari', + 'sd_in': 'sd_IN.UTF-8', + 'sd_in at devanagari': 'sd_IN.UTF-8 at devanagari', 'se_no': 'se_NO.UTF-8', 'serbocroatian': 'sr_RS.UTF-8 at latin', 'sh': 'sr_RS.UTF-8 at latin', diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py --- a/Lib/test/test_locale.py +++ b/Lib/test/test_locale.py @@ -428,6 +428,20 @@ self.check('ca_ES at valencia', 'ca_ES.ISO8859-1 at valencia') self.check('ca at valencia', 'ca_ES.ISO8859-1 at valencia') + def test_devanagari_modifier(self): + self.check('ks_IN.UTF-8 at devanagari', 'ks_IN.UTF-8 at devanagari') + self.check('ks_IN at devanagari', 'ks_IN.UTF-8 at devanagari') + self.check('ks at devanagari', 'ks_IN.UTF-8 at devanagari') + self.check('ks_IN.UTF-8', 'ks_IN.UTF-8') + self.check('ks_IN', 'ks_IN.UTF-8') + self.check('ks', 'ks_IN.UTF-8') + self.check('sd_IN.UTF-8 at devanagari', 'sd_IN.UTF-8 at devanagari') + self.check('sd_IN at devanagari', 'sd_IN.UTF-8 at devanagari') + self.check('sd at devanagari', 'sd_IN.UTF-8 at devanagari') + self.check('sd_IN.UTF-8', 'sd_IN.UTF-8') + self.check('sd_IN', 'sd_IN.UTF-8') + self.check('sd', 'sd_IN.UTF-8') + class TestMiscellaneous(unittest.TestCase): def test_getpreferredencoding(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -27,6 +27,8 @@ Library ------- +- Issue #20027: Fixed locale aliases for devanagari locales. + - Issue #20067: Tkinter variables now work when wantobjects is false. - Issue #19020: Tkinter now uses splitlist() instead of split() in configure diff --git a/Tools/i18n/makelocalealias.py b/Tools/i18n/makelocalealias.py --- a/Tools/i18n/makelocalealias.py +++ b/Tools/i18n/makelocalealias.py @@ -23,6 +23,12 @@ if line[:1] == '#': continue locale, alias = line.split() + # Fix non-standard locale names, e.g. ks_IN at devanagari.UTF-8 + if '@' in alias: + alias_lang, _, alias_mod = alias.partition('@') + if '.' in alias_mod: + alias_mod, _, alias_enc = alias_mod.partition('.') + alias = alias_lang + '.' + alias_enc + '@' + alias_mod # Strip ':' if locale[-1] == ':': locale = locale[:-1] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 26 20:22:19 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 26 Dec 2013 20:22:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzIwMDI3?= =?utf-8?q?=3A_Fixed_locale_aliases_for_devanagari_locales=2E?= Message-ID: <3dr1JW1FzYz7LkM@mail.python.org> http://hg.python.org/cpython/rev/7615c009e925 changeset: 88201:7615c009e925 branch: 3.3 parent: 88197:fbc1a39b68e4 user: Serhiy Storchaka date: Thu Dec 26 21:20:59 2013 +0200 summary: Issue #20027: Fixed locale aliases for devanagari locales. files: Lib/locale.py | 7 ++++--- Lib/test/test_locale.py | 14 ++++++++++++++ Misc/NEWS | 2 ++ Tools/i18n/makelocalealias.py | 6 ++++++ 4 files changed, 26 insertions(+), 3 deletions(-) diff --git a/Lib/locale.py b/Lib/locale.py --- a/Lib/locale.py +++ b/Lib/locale.py @@ -1320,7 +1320,7 @@ 'korean.euc': 'ko_KR.eucKR', 'ks': 'ks_IN.UTF-8', 'ks_in': 'ks_IN.UTF-8', - 'ks_in at devanagari': 'ks_IN at devanagari.UTF-8', + 'ks_in at devanagari': 'ks_IN.UTF-8 at devanagari', 'kw': 'kw_GB.ISO8859-1', 'kw_gb': 'kw_GB.ISO8859-1', 'kw_gb.iso88591': 'kw_GB.ISO8859-1', @@ -1485,8 +1485,9 @@ 'rw_rw': 'rw_RW.ISO8859-1', 'rw_rw.iso88591': 'rw_RW.ISO8859-1', 'sd': 'sd_IN.UTF-8', - 'sd at devanagari': 'sd_IN at devanagari.UTF-8', - 'sd_in at devanagari': 'sd_IN at devanagari.UTF-8', + 'sd at devanagari': 'sd_IN.UTF-8 at devanagari', + 'sd_in': 'sd_IN.UTF-8', + 'sd_in at devanagari': 'sd_IN.UTF-8 at devanagari', 'se_no': 'se_NO.UTF-8', 'serbocroatian': 'sr_RS.UTF-8 at latin', 'sh': 'sr_RS.UTF-8 at latin', diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py --- a/Lib/test/test_locale.py +++ b/Lib/test/test_locale.py @@ -421,6 +421,20 @@ self.check('ca_ES at valencia', 'ca_ES.ISO8859-1 at valencia') self.check('ca at valencia', 'ca_ES.ISO8859-1 at valencia') + def test_devanagari_modifier(self): + self.check('ks_IN.UTF-8 at devanagari', 'ks_IN.UTF-8 at devanagari') + self.check('ks_IN at devanagari', 'ks_IN.UTF-8 at devanagari') + self.check('ks at devanagari', 'ks_IN.UTF-8 at devanagari') + self.check('ks_IN.UTF-8', 'ks_IN.UTF-8') + self.check('ks_IN', 'ks_IN.UTF-8') + self.check('ks', 'ks_IN.UTF-8') + self.check('sd_IN.UTF-8 at devanagari', 'sd_IN.UTF-8 at devanagari') + self.check('sd_IN at devanagari', 'sd_IN.UTF-8 at devanagari') + self.check('sd at devanagari', 'sd_IN.UTF-8 at devanagari') + self.check('sd_IN.UTF-8', 'sd_IN.UTF-8') + self.check('sd_IN', 'sd_IN.UTF-8') + self.check('sd', 'sd_IN.UTF-8') + class TestMiscellaneous(unittest.TestCase): def test_getpreferredencoding(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,8 @@ Library ------- +- Issue #20027: Fixed locale aliases for devanagari locales. + - Issue #20067: Tkinter variables now work when wantobjects is false. - Issue #19020: Tkinter now uses splitlist() instead of split() in configure diff --git a/Tools/i18n/makelocalealias.py b/Tools/i18n/makelocalealias.py --- a/Tools/i18n/makelocalealias.py +++ b/Tools/i18n/makelocalealias.py @@ -23,6 +23,12 @@ if line[:1] == '#': continue locale, alias = line.split() + # Fix non-standard locale names, e.g. ks_IN at devanagari.UTF-8 + if '@' in alias: + alias_lang, _, alias_mod = alias.partition('@') + if '.' in alias_mod: + alias_mod, _, alias_enc = alias_mod.partition('.') + alias = alias_lang + '.' + alias_enc + '@' + alias_mod # Strip ':' if locale[-1] == ':': locale = locale[:-1] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 26 20:22:20 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 26 Dec 2013 20:22:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2320027=3A_Fixed_locale_aliases_for_devanagari_lo?= =?utf-8?q?cales=2E?= Message-ID: <3dr1JX4X9sz7Lkb@mail.python.org> http://hg.python.org/cpython/rev/fff3f28733b4 changeset: 88202:fff3f28733b4 parent: 88198:99df5128d978 parent: 88201:7615c009e925 user: Serhiy Storchaka date: Thu Dec 26 21:21:52 2013 +0200 summary: Issue #20027: Fixed locale aliases for devanagari locales. files: Lib/locale.py | 7 ++++--- Lib/test/test_locale.py | 14 ++++++++++++++ Misc/NEWS | 2 ++ Tools/i18n/makelocalealias.py | 6 ++++++ 4 files changed, 26 insertions(+), 3 deletions(-) diff --git a/Lib/locale.py b/Lib/locale.py --- a/Lib/locale.py +++ b/Lib/locale.py @@ -1309,7 +1309,7 @@ 'korean.euc': 'ko_KR.eucKR', 'ks': 'ks_IN.UTF-8', 'ks_in': 'ks_IN.UTF-8', - 'ks_in at devanagari': 'ks_IN at devanagari.UTF-8', + 'ks_in at devanagari': 'ks_IN.UTF-8 at devanagari', 'kw': 'kw_GB.ISO8859-1', 'kw_gb': 'kw_GB.ISO8859-1', 'kw_gb.iso88591': 'kw_GB.ISO8859-1', @@ -1474,8 +1474,9 @@ 'rw_rw': 'rw_RW.ISO8859-1', 'rw_rw.iso88591': 'rw_RW.ISO8859-1', 'sd': 'sd_IN.UTF-8', - 'sd at devanagari': 'sd_IN at devanagari.UTF-8', - 'sd_in at devanagari': 'sd_IN at devanagari.UTF-8', + 'sd at devanagari': 'sd_IN.UTF-8 at devanagari', + 'sd_in': 'sd_IN.UTF-8', + 'sd_in at devanagari': 'sd_IN.UTF-8 at devanagari', 'se_no': 'se_NO.UTF-8', 'serbocroatian': 'sr_RS.UTF-8 at latin', 'sh': 'sr_RS.UTF-8 at latin', diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py --- a/Lib/test/test_locale.py +++ b/Lib/test/test_locale.py @@ -422,6 +422,20 @@ self.check('ca_ES at valencia', 'ca_ES.ISO8859-1 at valencia') self.check('ca at valencia', 'ca_ES.ISO8859-1 at valencia') + def test_devanagari_modifier(self): + self.check('ks_IN.UTF-8 at devanagari', 'ks_IN.UTF-8 at devanagari') + self.check('ks_IN at devanagari', 'ks_IN.UTF-8 at devanagari') + self.check('ks at devanagari', 'ks_IN.UTF-8 at devanagari') + self.check('ks_IN.UTF-8', 'ks_IN.UTF-8') + self.check('ks_IN', 'ks_IN.UTF-8') + self.check('ks', 'ks_IN.UTF-8') + self.check('sd_IN.UTF-8 at devanagari', 'sd_IN.UTF-8 at devanagari') + self.check('sd_IN at devanagari', 'sd_IN.UTF-8 at devanagari') + self.check('sd at devanagari', 'sd_IN.UTF-8 at devanagari') + self.check('sd_IN.UTF-8', 'sd_IN.UTF-8') + self.check('sd_IN', 'sd_IN.UTF-8') + self.check('sd', 'sd_IN.UTF-8') + class TestMiscellaneous(unittest.TestCase): def test_getpreferredencoding(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,8 @@ Library ------- +- Issue #20027: Fixed locale aliases for devanagari locales. + - Issue #20067: Tkinter variables now work when wantobjects is false. - Issue #19020: Tkinter now uses splitlist() instead of split() in configure diff --git a/Tools/i18n/makelocalealias.py b/Tools/i18n/makelocalealias.py --- a/Tools/i18n/makelocalealias.py +++ b/Tools/i18n/makelocalealias.py @@ -23,6 +23,12 @@ if line[:1] == '#': continue locale, alias = line.split() + # Fix non-standard locale names, e.g. ks_IN at devanagari.UTF-8 + if '@' in alias: + alias_lang, _, alias_mod = alias.partition('@') + if '.' in alias_mod: + alias_mod, _, alias_enc = alias_mod.partition('.') + alias = alias_lang + '.' + alias_enc + '@' + alias_mod # Strip ':' if locale[-1] == ':': locale = locale[:-1] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 26 21:12:54 2013 From: python-checkins at python.org (r.david.murray) Date: Thu, 26 Dec 2013 21:12:54 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew_for_gc=2Eget=5Fst?= =?utf-8?q?ats=2C_plus_doc_tweaks=2E?= Message-ID: <3dr2Qt65WYz7LjT@mail.python.org> http://hg.python.org/cpython/rev/17bd04fbf3d3 changeset: 88203:17bd04fbf3d3 user: R David Murray date: Thu Dec 26 15:11:28 2013 -0500 summary: whatsnew for gc.get_stats, plus doc tweaks. Clarified the "At the moment" wording, and added the get_stats entry in the module summary that Serhiy noted was missing at the end of issue 16351. Given that pydoc lists all the function docstrings, I'm not sure that module summary section is actually needed; but, it is probably better to address that when the module is converted to use Argument Clinic. In the meantime we should keep the list complete. files: Doc/library/gc.rst | 7 ++++--- Doc/whatsnew/3.4.rst | 8 ++++++++ Modules/gcmodule.c | 1 + 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst --- a/Doc/library/gc.rst +++ b/Doc/library/gc.rst @@ -69,9 +69,10 @@ .. function:: get_stats() - Return a list of 3 per-generation dictionaries containing collection - statistics since interpreter start. At this moment, each dictionary will - contain the following items: + Return a list of three per-generation dictionaries containing collection + statistics since interpreter start. The number of keys may change + in the future, but currently each dictionary will contain the following + items: * ``collections`` is the number of times this generation was collected; diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -628,6 +628,14 @@ PEP written and implemented by ?ukasz Langa. +gc +-- + +New :func:`~gc.get_stats` returns a list of three per-generation dictionaries +containing the collections statistics since interpreter startup. (Contributed +by Antoine Pitrou in :issue:`16351`.) + + hashlib ------- diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c --- a/Modules/gcmodule.c +++ b/Modules/gcmodule.c @@ -1498,6 +1498,7 @@ "isenabled() -- Returns true if automatic collection is enabled.\n" "collect() -- Do a full collection right now.\n" "get_count() -- Return the current collection counts.\n" +"get_stats() -- Return list of dictionaries containing per-generation stats.\n" "set_debug() -- Set debugging flags.\n" "get_debug() -- Get debugging flags.\n" "set_threshold() -- Set the collection thresholds.\n" -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu Dec 26 23:57:06 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Thu, 26 Dec 2013 23:57:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2320046=3A_Locale_a?= =?utf-8?q?lias_table_no_longer_contains_entities_which_can_be?= Message-ID: <3dr64L6VrGz7Ljv@mail.python.org> http://hg.python.org/cpython/rev/63bc68d7f449 changeset: 88204:63bc68d7f449 user: Serhiy Storchaka date: Fri Dec 27 00:56:53 2013 +0200 summary: Issue #20046: Locale alias table no longer contains entities which can be calculated. Generalized support of the euro modifier. files: Lib/locale.py | 466 +-------------------- Lib/test/test_locale.py | 39 + Misc/NEWS | 3 + Tools/i18n/makelocalealias.py | 27 + 4 files changed, 109 insertions(+), 426 deletions(-) diff --git a/Lib/locale.py b/Lib/locale.py --- a/Lib/locale.py +++ b/Lib/locale.py @@ -344,14 +344,32 @@ # Convert the encoding to a C lib compatible encoding string norm_encoding = encodings.normalize_encoding(encoding) #print('norm encoding: %r' % norm_encoding) - norm_encoding = encodings.aliases.aliases.get(norm_encoding, + norm_encoding = encodings.aliases.aliases.get(norm_encoding.lower(), norm_encoding) #print('aliased encoding: %r' % norm_encoding) - encoding = locale_encoding_alias.get(norm_encoding, - norm_encoding) + encoding = norm_encoding + norm_encoding = norm_encoding.lower() + if norm_encoding in locale_encoding_alias: + encoding = locale_encoding_alias[norm_encoding] + else: + norm_encoding = norm_encoding.replace('_', '') + norm_encoding = norm_encoding.replace('-', '') + if norm_encoding in locale_encoding_alias: + encoding = locale_encoding_alias[norm_encoding] #print('found encoding %r' % encoding) return langname + '.' + encoding +def _append_modifier(code, modifier): + if modifier == 'euro': + if '.' not in code: + return code + '.ISO8859-15' + _, _, encoding = code.partition('.') + if encoding in ('ISO8859-15', 'UTF-8'): + return code + if encoding == 'ISO8859-1': + return _replace_encoding(code, 'ISO8859-15') + return code + '@' + modifier + def normalize(localename): """ Returns a normalized locale code for the given locale @@ -403,7 +421,7 @@ if code is not None: #print('lookup without modifier succeeded') if '@' not in code: - return code + '@' + modifier + return _append_modifier(code, modifier) if code.split('@', 1)[1].lower() == modifier: return code #print('second lookup failed') @@ -427,7 +445,8 @@ if code is not None: #print('lookup without modifier and encoding succeeded') if '@' not in code: - return _replace_encoding(code, encoding) + '@' + modifier + code = _replace_encoding(code, encoding) + return _append_modifier(code, modifier) code, defmod = code.split('@', 1) if defmod.lower() == modifier: return _replace_encoding(code, encoding) + '@' + defmod @@ -643,6 +662,14 @@ 'jis': 'JIS7', 'jis7': 'JIS7', 'ajec': 'eucJP', + 'koi8c': 'KOI8-C', + 'microsoftcp1251': 'CP1251', + 'microsoftcp1255': 'CP1255', + 'microsoftcp1256': 'CP1256', + '88591': 'ISO8859-1', + '88592': 'ISO8859-2', + '88595': 'ISO8859-5', + '885915': 'ISO8859-15', # Mappings from Python codec names to C lib encoding names 'ascii': 'ISO8859-1', @@ -670,10 +697,18 @@ 'utf_8': 'UTF-8', 'koi8_r': 'KOI8-R', 'koi8_u': 'KOI8-U', + 'cp1251': 'CP1251', + 'cp1255': 'CP1255', + 'cp1256': 'CP1256', + # XXX This list is still incomplete. If you know more # mappings, please file a bug report. Thanks. } +for k, v in sorted(locale_encoding_alias.items()): + k = k.replace('_', '') + locale_encoding_alias.setdefault(k, v) + # # The locale_alias table maps lowercase alias names to C locale names # (case-sensitive). Encodings are always separated from the locale @@ -786,55 +821,33 @@ locale_alias = { 'a3': 'az_AZ.KOI8-C', 'a3_az': 'az_AZ.KOI8-C', - 'a3_az.koi8c': 'az_AZ.KOI8-C', 'a3_az.koic': 'az_AZ.KOI8-C', 'af': 'af_ZA.ISO8859-1', 'af_za': 'af_ZA.ISO8859-1', - 'af_za.iso88591': 'af_ZA.ISO8859-1', 'am': 'am_ET.UTF-8', 'am_et': 'am_ET.UTF-8', 'american': 'en_US.ISO8859-1', - 'american.iso88591': 'en_US.ISO8859-1', 'ar': 'ar_AA.ISO8859-6', 'ar_aa': 'ar_AA.ISO8859-6', - 'ar_aa.iso88596': 'ar_AA.ISO8859-6', 'ar_ae': 'ar_AE.ISO8859-6', - 'ar_ae.iso88596': 'ar_AE.ISO8859-6', 'ar_bh': 'ar_BH.ISO8859-6', - 'ar_bh.iso88596': 'ar_BH.ISO8859-6', 'ar_dz': 'ar_DZ.ISO8859-6', - 'ar_dz.iso88596': 'ar_DZ.ISO8859-6', 'ar_eg': 'ar_EG.ISO8859-6', - 'ar_eg.iso88596': 'ar_EG.ISO8859-6', 'ar_in': 'ar_IN.UTF-8', 'ar_iq': 'ar_IQ.ISO8859-6', - 'ar_iq.iso88596': 'ar_IQ.ISO8859-6', 'ar_jo': 'ar_JO.ISO8859-6', - 'ar_jo.iso88596': 'ar_JO.ISO8859-6', 'ar_kw': 'ar_KW.ISO8859-6', - 'ar_kw.iso88596': 'ar_KW.ISO8859-6', 'ar_lb': 'ar_LB.ISO8859-6', - 'ar_lb.iso88596': 'ar_LB.ISO8859-6', 'ar_ly': 'ar_LY.ISO8859-6', - 'ar_ly.iso88596': 'ar_LY.ISO8859-6', 'ar_ma': 'ar_MA.ISO8859-6', - 'ar_ma.iso88596': 'ar_MA.ISO8859-6', 'ar_om': 'ar_OM.ISO8859-6', - 'ar_om.iso88596': 'ar_OM.ISO8859-6', 'ar_qa': 'ar_QA.ISO8859-6', - 'ar_qa.iso88596': 'ar_QA.ISO8859-6', 'ar_sa': 'ar_SA.ISO8859-6', - 'ar_sa.iso88596': 'ar_SA.ISO8859-6', 'ar_sd': 'ar_SD.ISO8859-6', - 'ar_sd.iso88596': 'ar_SD.ISO8859-6', 'ar_sy': 'ar_SY.ISO8859-6', - 'ar_sy.iso88596': 'ar_SY.ISO8859-6', 'ar_tn': 'ar_TN.ISO8859-6', - 'ar_tn.iso88596': 'ar_TN.ISO8859-6', 'ar_ye': 'ar_YE.ISO8859-6', - 'ar_ye.iso88596': 'ar_YE.ISO8859-6', 'arabic': 'ar_AA.ISO8859-6', - 'arabic.iso88596': 'ar_AA.ISO8859-6', 'as': 'as_IN.UTF-8', 'as_in': 'as_IN.UTF-8', 'az': 'az_AZ.ISO8859-9E', @@ -843,35 +856,20 @@ 'be': 'be_BY.CP1251', 'be at latin': 'be_BY.UTF-8 at latin', 'be_by': 'be_BY.CP1251', - 'be_by.cp1251': 'be_BY.CP1251', - 'be_by.microsoftcp1251': 'be_BY.CP1251', - 'be_by.utf8 at latin': 'be_BY.UTF-8 at latin', 'be_by at latin': 'be_BY.UTF-8 at latin', 'bg': 'bg_BG.CP1251', 'bg_bg': 'bg_BG.CP1251', - 'bg_bg.cp1251': 'bg_BG.CP1251', - 'bg_bg.iso88595': 'bg_BG.ISO8859-5', - 'bg_bg.koi8r': 'bg_BG.KOI8-R', - 'bg_bg.microsoftcp1251': 'bg_BG.CP1251', 'bn_in': 'bn_IN.UTF-8', 'bo_in': 'bo_IN.UTF-8', 'bokmal': 'nb_NO.ISO8859-1', 'bokm\xe5l': 'nb_NO.ISO8859-1', 'br': 'br_FR.ISO8859-1', 'br_fr': 'br_FR.ISO8859-1', - 'br_fr.iso88591': 'br_FR.ISO8859-1', - 'br_fr.iso885914': 'br_FR.ISO8859-14', - 'br_fr.iso885915': 'br_FR.ISO8859-15', - 'br_fr.iso885915 at euro': 'br_FR.ISO8859-15', - 'br_fr.utf8 at euro': 'br_FR.UTF-8', - 'br_fr at euro': 'br_FR.ISO8859-15', 'bs': 'bs_BA.ISO8859-2', 'bs_ba': 'bs_BA.ISO8859-2', - 'bs_ba.iso88592': 'bs_BA.ISO8859-2', 'bulgarian': 'bg_BG.CP1251', 'c': 'C', 'c-french': 'fr_CA.ISO8859-1', - 'c-french.iso88591': 'fr_CA.ISO8859-1', 'c.ascii': 'C', 'c.en': 'C', 'c.iso88591': 'en_US.ISO8859-1', @@ -879,343 +877,131 @@ 'c_c.c': 'C', 'ca': 'ca_ES.ISO8859-1', 'ca_ad': 'ca_AD.ISO8859-1', - 'ca_ad.iso88591': 'ca_AD.ISO8859-1', - 'ca_ad.iso885915': 'ca_AD.ISO8859-15', - 'ca_ad.iso885915 at euro': 'ca_AD.ISO8859-15', - 'ca_ad.utf8 at euro': 'ca_AD.UTF-8', - 'ca_ad at euro': 'ca_AD.ISO8859-15', 'ca_es': 'ca_ES.ISO8859-1', - 'ca_es.iso88591': 'ca_ES.ISO8859-1', - 'ca_es.iso885915': 'ca_ES.ISO8859-15', - 'ca_es.iso885915 at euro': 'ca_ES.ISO8859-15', - 'ca_es.utf8 at euro': 'ca_ES.UTF-8', - 'ca_es at euro': 'ca_ES.ISO8859-15', 'ca_fr': 'ca_FR.ISO8859-1', - 'ca_fr.iso88591': 'ca_FR.ISO8859-1', - 'ca_fr.iso885915': 'ca_FR.ISO8859-15', - 'ca_fr.iso885915 at euro': 'ca_FR.ISO8859-15', - 'ca_fr.utf8 at euro': 'ca_FR.UTF-8', - 'ca_fr at euro': 'ca_FR.ISO8859-15', 'ca_it': 'ca_IT.ISO8859-1', - 'ca_it.iso88591': 'ca_IT.ISO8859-1', - 'ca_it.iso885915': 'ca_IT.ISO8859-15', - 'ca_it.iso885915 at euro': 'ca_IT.ISO8859-15', - 'ca_it.utf8 at euro': 'ca_IT.UTF-8', - 'ca_it at euro': 'ca_IT.ISO8859-15', 'catalan': 'ca_ES.ISO8859-1', 'cextend': 'en_US.ISO8859-1', - 'cextend.en': 'en_US.ISO8859-1', 'chinese-s': 'zh_CN.eucCN', 'chinese-t': 'zh_TW.eucTW', 'croatian': 'hr_HR.ISO8859-2', 'cs': 'cs_CZ.ISO8859-2', 'cs_cs': 'cs_CZ.ISO8859-2', - 'cs_cs.iso88592': 'cs_CZ.ISO8859-2', 'cs_cz': 'cs_CZ.ISO8859-2', - 'cs_cz.iso88592': 'cs_CZ.ISO8859-2', 'cy': 'cy_GB.ISO8859-1', 'cy_gb': 'cy_GB.ISO8859-1', - 'cy_gb.iso88591': 'cy_GB.ISO8859-1', - 'cy_gb.iso885914': 'cy_GB.ISO8859-14', - 'cy_gb.iso885915': 'cy_GB.ISO8859-15', - 'cy_gb at euro': 'cy_GB.ISO8859-15', 'cz': 'cs_CZ.ISO8859-2', 'cz_cz': 'cs_CZ.ISO8859-2', 'czech': 'cs_CZ.ISO8859-2', 'da': 'da_DK.ISO8859-1', - 'da.iso885915': 'da_DK.ISO8859-15', 'da_dk': 'da_DK.ISO8859-1', - 'da_dk.88591': 'da_DK.ISO8859-1', - 'da_dk.885915': 'da_DK.ISO8859-15', - 'da_dk.iso88591': 'da_DK.ISO8859-1', - 'da_dk.iso885915': 'da_DK.ISO8859-15', - 'da_dk at euro': 'da_DK.ISO8859-15', 'danish': 'da_DK.ISO8859-1', - 'danish.iso88591': 'da_DK.ISO8859-1', 'dansk': 'da_DK.ISO8859-1', 'de': 'de_DE.ISO8859-1', - 'de.iso885915': 'de_DE.ISO8859-15', 'de_at': 'de_AT.ISO8859-1', - 'de_at.iso88591': 'de_AT.ISO8859-1', - 'de_at.iso885915': 'de_AT.ISO8859-15', - 'de_at.iso885915 at euro': 'de_AT.ISO8859-15', - 'de_at.utf8 at euro': 'de_AT.UTF-8', - 'de_at at euro': 'de_AT.ISO8859-15', 'de_be': 'de_BE.ISO8859-1', - 'de_be.iso88591': 'de_BE.ISO8859-1', - 'de_be.iso885915': 'de_BE.ISO8859-15', - 'de_be.iso885915 at euro': 'de_BE.ISO8859-15', - 'de_be.utf8 at euro': 'de_BE.UTF-8', - 'de_be at euro': 'de_BE.ISO8859-15', 'de_ch': 'de_CH.ISO8859-1', - 'de_ch.iso88591': 'de_CH.ISO8859-1', - 'de_ch.iso885915': 'de_CH.ISO8859-15', - 'de_ch at euro': 'de_CH.ISO8859-15', 'de_de': 'de_DE.ISO8859-1', - 'de_de.88591': 'de_DE.ISO8859-1', - 'de_de.885915': 'de_DE.ISO8859-15', - 'de_de.885915 at euro': 'de_DE.ISO8859-15', - 'de_de.iso88591': 'de_DE.ISO8859-1', - 'de_de.iso885915': 'de_DE.ISO8859-15', - 'de_de.iso885915 at euro': 'de_DE.ISO8859-15', - 'de_de.utf8 at euro': 'de_DE.UTF-8', - 'de_de at euro': 'de_DE.ISO8859-15', 'de_lu': 'de_LU.ISO8859-1', - 'de_lu.iso88591': 'de_LU.ISO8859-1', - 'de_lu.iso885915': 'de_LU.ISO8859-15', - 'de_lu.iso885915 at euro': 'de_LU.ISO8859-15', - 'de_lu.utf8 at euro': 'de_LU.UTF-8', - 'de_lu at euro': 'de_LU.ISO8859-15', 'deutsch': 'de_DE.ISO8859-1', 'dutch': 'nl_NL.ISO8859-1', 'dutch.iso88591': 'nl_BE.ISO8859-1', 'ee': 'ee_EE.ISO8859-4', 'ee_ee': 'ee_EE.ISO8859-4', - 'ee_ee.iso88594': 'ee_EE.ISO8859-4', 'eesti': 'et_EE.ISO8859-1', 'el': 'el_GR.ISO8859-7', 'el_gr': 'el_GR.ISO8859-7', - 'el_gr.iso88597': 'el_GR.ISO8859-7', 'el_gr at euro': 'el_GR.ISO8859-15', 'en': 'en_US.ISO8859-1', - 'en.iso88591': 'en_US.ISO8859-1', 'en_au': 'en_AU.ISO8859-1', - 'en_au.iso88591': 'en_AU.ISO8859-1', 'en_be': 'en_BE.ISO8859-1', - 'en_be at euro': 'en_BE.ISO8859-15', 'en_bw': 'en_BW.ISO8859-1', - 'en_bw.iso88591': 'en_BW.ISO8859-1', 'en_ca': 'en_CA.ISO8859-1', - 'en_ca.iso88591': 'en_CA.ISO8859-1', 'en_gb': 'en_GB.ISO8859-1', - 'en_gb.88591': 'en_GB.ISO8859-1', - 'en_gb.iso88591': 'en_GB.ISO8859-1', - 'en_gb.iso885915': 'en_GB.ISO8859-15', - 'en_gb at euro': 'en_GB.ISO8859-15', 'en_hk': 'en_HK.ISO8859-1', - 'en_hk.iso88591': 'en_HK.ISO8859-1', 'en_ie': 'en_IE.ISO8859-1', - 'en_ie.iso88591': 'en_IE.ISO8859-1', - 'en_ie.iso885915': 'en_IE.ISO8859-15', - 'en_ie.iso885915 at euro': 'en_IE.ISO8859-15', - 'en_ie.utf8 at euro': 'en_IE.UTF-8', - 'en_ie at euro': 'en_IE.ISO8859-15', 'en_in': 'en_IN.ISO8859-1', 'en_nz': 'en_NZ.ISO8859-1', - 'en_nz.iso88591': 'en_NZ.ISO8859-1', 'en_ph': 'en_PH.ISO8859-1', - 'en_ph.iso88591': 'en_PH.ISO8859-1', 'en_sg': 'en_SG.ISO8859-1', - 'en_sg.iso88591': 'en_SG.ISO8859-1', 'en_uk': 'en_GB.ISO8859-1', 'en_us': 'en_US.ISO8859-1', - 'en_us.88591': 'en_US.ISO8859-1', - 'en_us.885915': 'en_US.ISO8859-15', - 'en_us.iso88591': 'en_US.ISO8859-1', - 'en_us.iso885915': 'en_US.ISO8859-15', - 'en_us.iso885915 at euro': 'en_US.ISO8859-15', - 'en_us at euro': 'en_US.ISO8859-15', 'en_us at euro@euro': 'en_US.ISO8859-15', 'en_za': 'en_ZA.ISO8859-1', - 'en_za.88591': 'en_ZA.ISO8859-1', - 'en_za.iso88591': 'en_ZA.ISO8859-1', - 'en_za.iso885915': 'en_ZA.ISO8859-15', - 'en_za at euro': 'en_ZA.ISO8859-15', 'en_zw': 'en_ZW.ISO8859-1', - 'en_zw.iso88591': 'en_ZW.ISO8859-1', 'eng_gb': 'en_GB.ISO8859-1', - 'eng_gb.8859': 'en_GB.ISO8859-1', 'english': 'en_EN.ISO8859-1', - 'english.iso88591': 'en_EN.ISO8859-1', 'english_uk': 'en_GB.ISO8859-1', - 'english_uk.8859': 'en_GB.ISO8859-1', 'english_united-states': 'en_US.ISO8859-1', 'english_united-states.437': 'C', 'english_us': 'en_US.ISO8859-1', - 'english_us.8859': 'en_US.ISO8859-1', - 'english_us.ascii': 'en_US.ISO8859-1', 'eo': 'eo_XX.ISO8859-3', 'eo_eo': 'eo_EO.ISO8859-3', - 'eo_eo.iso88593': 'eo_EO.ISO8859-3', 'eo_xx': 'eo_XX.ISO8859-3', - 'eo_xx.iso88593': 'eo_XX.ISO8859-3', 'es': 'es_ES.ISO8859-1', 'es_ar': 'es_AR.ISO8859-1', - 'es_ar.iso88591': 'es_AR.ISO8859-1', 'es_bo': 'es_BO.ISO8859-1', - 'es_bo.iso88591': 'es_BO.ISO8859-1', 'es_cl': 'es_CL.ISO8859-1', - 'es_cl.iso88591': 'es_CL.ISO8859-1', 'es_co': 'es_CO.ISO8859-1', - 'es_co.iso88591': 'es_CO.ISO8859-1', 'es_cr': 'es_CR.ISO8859-1', - 'es_cr.iso88591': 'es_CR.ISO8859-1', 'es_do': 'es_DO.ISO8859-1', - 'es_do.iso88591': 'es_DO.ISO8859-1', 'es_ec': 'es_EC.ISO8859-1', - 'es_ec.iso88591': 'es_EC.ISO8859-1', 'es_es': 'es_ES.ISO8859-1', - 'es_es.88591': 'es_ES.ISO8859-1', - 'es_es.iso88591': 'es_ES.ISO8859-1', - 'es_es.iso885915': 'es_ES.ISO8859-15', - 'es_es.iso885915 at euro': 'es_ES.ISO8859-15', - 'es_es.utf8 at euro': 'es_ES.UTF-8', - 'es_es at euro': 'es_ES.ISO8859-15', 'es_gt': 'es_GT.ISO8859-1', - 'es_gt.iso88591': 'es_GT.ISO8859-1', 'es_hn': 'es_HN.ISO8859-1', - 'es_hn.iso88591': 'es_HN.ISO8859-1', 'es_mx': 'es_MX.ISO8859-1', - 'es_mx.iso88591': 'es_MX.ISO8859-1', 'es_ni': 'es_NI.ISO8859-1', - 'es_ni.iso88591': 'es_NI.ISO8859-1', 'es_pa': 'es_PA.ISO8859-1', - 'es_pa.iso88591': 'es_PA.ISO8859-1', - 'es_pa.iso885915': 'es_PA.ISO8859-15', - 'es_pa at euro': 'es_PA.ISO8859-15', 'es_pe': 'es_PE.ISO8859-1', - 'es_pe.iso88591': 'es_PE.ISO8859-1', - 'es_pe.iso885915': 'es_PE.ISO8859-15', - 'es_pe at euro': 'es_PE.ISO8859-15', 'es_pr': 'es_PR.ISO8859-1', - 'es_pr.iso88591': 'es_PR.ISO8859-1', 'es_py': 'es_PY.ISO8859-1', - 'es_py.iso88591': 'es_PY.ISO8859-1', - 'es_py.iso885915': 'es_PY.ISO8859-15', - 'es_py at euro': 'es_PY.ISO8859-15', 'es_sv': 'es_SV.ISO8859-1', - 'es_sv.iso88591': 'es_SV.ISO8859-1', - 'es_sv.iso885915': 'es_SV.ISO8859-15', - 'es_sv at euro': 'es_SV.ISO8859-15', 'es_us': 'es_US.ISO8859-1', - 'es_us.iso88591': 'es_US.ISO8859-1', 'es_uy': 'es_UY.ISO8859-1', - 'es_uy.iso88591': 'es_UY.ISO8859-1', - 'es_uy.iso885915': 'es_UY.ISO8859-15', - 'es_uy at euro': 'es_UY.ISO8859-15', 'es_ve': 'es_VE.ISO8859-1', - 'es_ve.iso88591': 'es_VE.ISO8859-1', - 'es_ve.iso885915': 'es_VE.ISO8859-15', - 'es_ve at euro': 'es_VE.ISO8859-15', 'estonian': 'et_EE.ISO8859-1', 'et': 'et_EE.ISO8859-15', 'et_ee': 'et_EE.ISO8859-15', - 'et_ee.iso88591': 'et_EE.ISO8859-1', - 'et_ee.iso885913': 'et_EE.ISO8859-13', - 'et_ee.iso885915': 'et_EE.ISO8859-15', - 'et_ee.iso88594': 'et_EE.ISO8859-4', - 'et_ee at euro': 'et_EE.ISO8859-15', 'eu': 'eu_ES.ISO8859-1', 'eu_es': 'eu_ES.ISO8859-1', - 'eu_es.iso88591': 'eu_ES.ISO8859-1', - 'eu_es.iso885915': 'eu_ES.ISO8859-15', - 'eu_es.iso885915 at euro': 'eu_ES.ISO8859-15', - 'eu_es.utf8 at euro': 'eu_ES.UTF-8', - 'eu_es at euro': 'eu_ES.ISO8859-15', 'fa': 'fa_IR.UTF-8', 'fa_ir': 'fa_IR.UTF-8', 'fa_ir.isiri3342': 'fa_IR.ISIRI-3342', 'fi': 'fi_FI.ISO8859-15', - 'fi.iso885915': 'fi_FI.ISO8859-15', 'fi_fi': 'fi_FI.ISO8859-15', - 'fi_fi.88591': 'fi_FI.ISO8859-1', - 'fi_fi.iso88591': 'fi_FI.ISO8859-1', - 'fi_fi.iso885915': 'fi_FI.ISO8859-15', - 'fi_fi.iso885915 at euro': 'fi_FI.ISO8859-15', - 'fi_fi.utf8 at euro': 'fi_FI.UTF-8', - 'fi_fi at euro': 'fi_FI.ISO8859-15', 'finnish': 'fi_FI.ISO8859-1', - 'finnish.iso88591': 'fi_FI.ISO8859-1', 'fo': 'fo_FO.ISO8859-1', 'fo_fo': 'fo_FO.ISO8859-1', - 'fo_fo.iso88591': 'fo_FO.ISO8859-1', - 'fo_fo.iso885915': 'fo_FO.ISO8859-15', - 'fo_fo at euro': 'fo_FO.ISO8859-15', 'fr': 'fr_FR.ISO8859-1', - 'fr.iso885915': 'fr_FR.ISO8859-15', 'fr_be': 'fr_BE.ISO8859-1', - 'fr_be.88591': 'fr_BE.ISO8859-1', - 'fr_be.iso88591': 'fr_BE.ISO8859-1', - 'fr_be.iso885915': 'fr_BE.ISO8859-15', - 'fr_be.iso885915 at euro': 'fr_BE.ISO8859-15', - 'fr_be.utf8 at euro': 'fr_BE.UTF-8', - 'fr_be at euro': 'fr_BE.ISO8859-15', 'fr_ca': 'fr_CA.ISO8859-1', - 'fr_ca.88591': 'fr_CA.ISO8859-1', - 'fr_ca.iso88591': 'fr_CA.ISO8859-1', - 'fr_ca.iso885915': 'fr_CA.ISO8859-15', - 'fr_ca at euro': 'fr_CA.ISO8859-15', 'fr_ch': 'fr_CH.ISO8859-1', - 'fr_ch.88591': 'fr_CH.ISO8859-1', - 'fr_ch.iso88591': 'fr_CH.ISO8859-1', - 'fr_ch.iso885915': 'fr_CH.ISO8859-15', - 'fr_ch at euro': 'fr_CH.ISO8859-15', 'fr_fr': 'fr_FR.ISO8859-1', - 'fr_fr.88591': 'fr_FR.ISO8859-1', - 'fr_fr.iso88591': 'fr_FR.ISO8859-1', - 'fr_fr.iso885915': 'fr_FR.ISO8859-15', - 'fr_fr.iso885915 at euro': 'fr_FR.ISO8859-15', - 'fr_fr.utf8 at euro': 'fr_FR.UTF-8', - 'fr_fr at euro': 'fr_FR.ISO8859-15', 'fr_lu': 'fr_LU.ISO8859-1', - 'fr_lu.88591': 'fr_LU.ISO8859-1', - 'fr_lu.iso88591': 'fr_LU.ISO8859-1', - 'fr_lu.iso885915': 'fr_LU.ISO8859-15', - 'fr_lu.iso885915 at euro': 'fr_LU.ISO8859-15', - 'fr_lu.utf8 at euro': 'fr_LU.UTF-8', - 'fr_lu at euro': 'fr_LU.ISO8859-15', 'fran\xe7ais': 'fr_FR.ISO8859-1', 'fre_fr': 'fr_FR.ISO8859-1', - 'fre_fr.8859': 'fr_FR.ISO8859-1', 'french': 'fr_FR.ISO8859-1', 'french.iso88591': 'fr_CH.ISO8859-1', 'french_france': 'fr_FR.ISO8859-1', - 'french_france.8859': 'fr_FR.ISO8859-1', 'ga': 'ga_IE.ISO8859-1', 'ga_ie': 'ga_IE.ISO8859-1', - 'ga_ie.iso88591': 'ga_IE.ISO8859-1', - 'ga_ie.iso885914': 'ga_IE.ISO8859-14', - 'ga_ie.iso885915': 'ga_IE.ISO8859-15', - 'ga_ie.iso885915 at euro': 'ga_IE.ISO8859-15', - 'ga_ie.utf8 at euro': 'ga_IE.UTF-8', - 'ga_ie at euro': 'ga_IE.ISO8859-15', 'galego': 'gl_ES.ISO8859-1', 'galician': 'gl_ES.ISO8859-1', 'gd': 'gd_GB.ISO8859-1', 'gd_gb': 'gd_GB.ISO8859-1', - 'gd_gb.iso88591': 'gd_GB.ISO8859-1', - 'gd_gb.iso885914': 'gd_GB.ISO8859-14', - 'gd_gb.iso885915': 'gd_GB.ISO8859-15', - 'gd_gb at euro': 'gd_GB.ISO8859-15', 'ger_de': 'de_DE.ISO8859-1', - 'ger_de.8859': 'de_DE.ISO8859-1', 'german': 'de_DE.ISO8859-1', 'german.iso88591': 'de_CH.ISO8859-1', 'german_germany': 'de_DE.ISO8859-1', - 'german_germany.8859': 'de_DE.ISO8859-1', 'gl': 'gl_ES.ISO8859-1', 'gl_es': 'gl_ES.ISO8859-1', - 'gl_es.iso88591': 'gl_ES.ISO8859-1', - 'gl_es.iso885915': 'gl_ES.ISO8859-15', - 'gl_es.iso885915 at euro': 'gl_ES.ISO8859-15', - 'gl_es.utf8 at euro': 'gl_ES.UTF-8', - 'gl_es at euro': 'gl_ES.ISO8859-15', 'greek': 'el_GR.ISO8859-7', - 'greek.iso88597': 'el_GR.ISO8859-7', 'gu_in': 'gu_IN.UTF-8', 'gv': 'gv_GB.ISO8859-1', 'gv_gb': 'gv_GB.ISO8859-1', - 'gv_gb.iso88591': 'gv_GB.ISO8859-1', - 'gv_gb.iso885914': 'gv_GB.ISO8859-14', - 'gv_gb.iso885915': 'gv_GB.ISO8859-15', - 'gv_gb at euro': 'gv_GB.ISO8859-15', 'he': 'he_IL.ISO8859-8', 'he_il': 'he_IL.ISO8859-8', - 'he_il.cp1255': 'he_IL.CP1255', - 'he_il.iso88598': 'he_IL.ISO8859-8', - 'he_il.microsoftcp1255': 'he_IL.CP1255', 'hebrew': 'he_IL.ISO8859-8', - 'hebrew.iso88598': 'he_IL.ISO8859-8', 'hi': 'hi_IN.ISCII-DEV', 'hi_in': 'hi_IN.ISCII-DEV', 'hi_in.isciidev': 'hi_IN.ISCII-DEV', @@ -1223,23 +1009,17 @@ 'hne_in': 'hne_IN.UTF-8', 'hr': 'hr_HR.ISO8859-2', 'hr_hr': 'hr_HR.ISO8859-2', - 'hr_hr.iso88592': 'hr_HR.ISO8859-2', 'hrvatski': 'hr_HR.ISO8859-2', 'hu': 'hu_HU.ISO8859-2', 'hu_hu': 'hu_HU.ISO8859-2', - 'hu_hu.iso88592': 'hu_HU.ISO8859-2', 'hungarian': 'hu_HU.ISO8859-2', 'icelandic': 'is_IS.ISO8859-1', - 'icelandic.iso88591': 'is_IS.ISO8859-1', 'id': 'id_ID.ISO8859-1', 'id_id': 'id_ID.ISO8859-1', 'in': 'id_ID.ISO8859-1', 'in_id': 'id_ID.ISO8859-1', 'is': 'is_IS.ISO8859-1', 'is_is': 'is_IS.ISO8859-1', - 'is_is.iso88591': 'is_IS.ISO8859-1', - 'is_is.iso885915': 'is_IS.ISO8859-15', - 'is_is at euro': 'is_IS.ISO8859-15', 'iso-8859-1': 'en_US.ISO8859-1', 'iso-8859-15': 'en_US.ISO8859-15', 'iso8859-1': 'en_US.ISO8859-1', @@ -1247,46 +1027,23 @@ 'iso_8859_1': 'en_US.ISO8859-1', 'iso_8859_15': 'en_US.ISO8859-15', 'it': 'it_IT.ISO8859-1', - 'it.iso885915': 'it_IT.ISO8859-15', 'it_ch': 'it_CH.ISO8859-1', - 'it_ch.iso88591': 'it_CH.ISO8859-1', - 'it_ch.iso885915': 'it_CH.ISO8859-15', - 'it_ch at euro': 'it_CH.ISO8859-15', 'it_it': 'it_IT.ISO8859-1', - 'it_it.88591': 'it_IT.ISO8859-1', - 'it_it.iso88591': 'it_IT.ISO8859-1', - 'it_it.iso885915': 'it_IT.ISO8859-15', - 'it_it.iso885915 at euro': 'it_IT.ISO8859-15', - 'it_it.utf8 at euro': 'it_IT.UTF-8', - 'it_it at euro': 'it_IT.ISO8859-15', 'italian': 'it_IT.ISO8859-1', - 'italian.iso88591': 'it_IT.ISO8859-1', 'iu': 'iu_CA.NUNACOM-8', 'iu_ca': 'iu_CA.NUNACOM-8', 'iu_ca.nunacom8': 'iu_CA.NUNACOM-8', 'iw': 'he_IL.ISO8859-8', 'iw_il': 'he_IL.ISO8859-8', - 'iw_il.iso88598': 'he_IL.ISO8859-8', 'ja': 'ja_JP.eucJP', - 'ja.jis': 'ja_JP.JIS7', - 'ja.sjis': 'ja_JP.SJIS', 'ja_jp': 'ja_JP.eucJP', - 'ja_jp.ajec': 'ja_JP.eucJP', 'ja_jp.euc': 'ja_JP.eucJP', - 'ja_jp.eucjp': 'ja_JP.eucJP', - 'ja_jp.iso-2022-jp': 'ja_JP.JIS7', - 'ja_jp.iso2022jp': 'ja_JP.JIS7', - 'ja_jp.jis': 'ja_JP.JIS7', - 'ja_jp.jis7': 'ja_JP.JIS7', 'ja_jp.mscode': 'ja_JP.SJIS', 'ja_jp.pck': 'ja_JP.SJIS', - 'ja_jp.sjis': 'ja_JP.SJIS', - 'ja_jp.ujis': 'ja_JP.eucJP', 'japan': 'ja_JP.eucJP', 'japanese': 'ja_JP.eucJP', 'japanese-euc': 'ja_JP.eucJP', 'japanese.euc': 'ja_JP.eucJP', - 'japanese.sjis': 'ja_JP.SJIS', 'jp_jp': 'ja_JP.eucJP', 'ka': 'ka_GE.GEORGIAN-ACADEMY', 'ka_ge': 'ka_GE.GEORGIAN-ACADEMY', @@ -1295,27 +1052,18 @@ 'ka_ge.georgianrs': 'ka_GE.GEORGIAN-ACADEMY', 'kl': 'kl_GL.ISO8859-1', 'kl_gl': 'kl_GL.ISO8859-1', - 'kl_gl.iso88591': 'kl_GL.ISO8859-1', - 'kl_gl.iso885915': 'kl_GL.ISO8859-15', - 'kl_gl at euro': 'kl_GL.ISO8859-15', 'km_kh': 'km_KH.UTF-8', 'kn': 'kn_IN.UTF-8', 'kn_in': 'kn_IN.UTF-8', 'ko': 'ko_KR.eucKR', 'ko_kr': 'ko_KR.eucKR', 'ko_kr.euc': 'ko_KR.eucKR', - 'ko_kr.euckr': 'ko_KR.eucKR', 'korean': 'ko_KR.eucKR', 'korean.euc': 'ko_KR.eucKR', 'ks': 'ks_IN.UTF-8', 'ks_in': 'ks_IN.UTF-8', - 'ks_in at devanagari': 'ks_IN.UTF-8 at devanagari', 'kw': 'kw_GB.ISO8859-1', 'kw_gb': 'kw_GB.ISO8859-1', - 'kw_gb.iso88591': 'kw_GB.ISO8859-1', - 'kw_gb.iso885914': 'kw_GB.ISO8859-14', - 'kw_gb.iso885915': 'kw_GB.ISO8859-15', - 'kw_gb at euro': 'kw_GB.ISO8859-15', 'ky': 'ky_KG.UTF-8', 'ky_kg': 'ky_KG.UTF-8', 'lithuanian': 'lt_LT.ISO8859-13', @@ -1326,157 +1074,78 @@ 'lo_la.mulelao1': 'lo_LA.MULELAO-1', 'lt': 'lt_LT.ISO8859-13', 'lt_lt': 'lt_LT.ISO8859-13', - 'lt_lt.iso885913': 'lt_LT.ISO8859-13', - 'lt_lt.iso88594': 'lt_LT.ISO8859-4', 'lv': 'lv_LV.ISO8859-13', 'lv_lv': 'lv_LV.ISO8859-13', - 'lv_lv.iso885913': 'lv_LV.ISO8859-13', - 'lv_lv.iso88594': 'lv_LV.ISO8859-4', 'mai': 'mai_IN.UTF-8', 'mai_in': 'mai_IN.UTF-8', 'mi': 'mi_NZ.ISO8859-1', 'mi_nz': 'mi_NZ.ISO8859-1', - 'mi_nz.iso88591': 'mi_NZ.ISO8859-1', 'mk': 'mk_MK.ISO8859-5', 'mk_mk': 'mk_MK.ISO8859-5', - 'mk_mk.cp1251': 'mk_MK.CP1251', - 'mk_mk.iso88595': 'mk_MK.ISO8859-5', - 'mk_mk.microsoftcp1251': 'mk_MK.CP1251', 'ml': 'ml_IN.UTF-8', 'ml_in': 'ml_IN.UTF-8', 'mr': 'mr_IN.UTF-8', 'mr_in': 'mr_IN.UTF-8', 'ms': 'ms_MY.ISO8859-1', 'ms_my': 'ms_MY.ISO8859-1', - 'ms_my.iso88591': 'ms_MY.ISO8859-1', 'mt': 'mt_MT.ISO8859-3', 'mt_mt': 'mt_MT.ISO8859-3', - 'mt_mt.iso88593': 'mt_MT.ISO8859-3', 'nb': 'nb_NO.ISO8859-1', 'nb_no': 'nb_NO.ISO8859-1', - 'nb_no.88591': 'nb_NO.ISO8859-1', - 'nb_no.iso88591': 'nb_NO.ISO8859-1', - 'nb_no.iso885915': 'nb_NO.ISO8859-15', - 'nb_no at euro': 'nb_NO.ISO8859-15', 'ne_np': 'ne_NP.UTF-8', 'nl': 'nl_NL.ISO8859-1', - 'nl.iso885915': 'nl_NL.ISO8859-15', 'nl_be': 'nl_BE.ISO8859-1', - 'nl_be.88591': 'nl_BE.ISO8859-1', - 'nl_be.iso88591': 'nl_BE.ISO8859-1', - 'nl_be.iso885915': 'nl_BE.ISO8859-15', - 'nl_be.iso885915 at euro': 'nl_BE.ISO8859-15', - 'nl_be.utf8 at euro': 'nl_BE.UTF-8', - 'nl_be at euro': 'nl_BE.ISO8859-15', 'nl_nl': 'nl_NL.ISO8859-1', - 'nl_nl.88591': 'nl_NL.ISO8859-1', - 'nl_nl.iso88591': 'nl_NL.ISO8859-1', - 'nl_nl.iso885915': 'nl_NL.ISO8859-15', - 'nl_nl.iso885915 at euro': 'nl_NL.ISO8859-15', - 'nl_nl.utf8 at euro': 'nl_NL.UTF-8', - 'nl_nl at euro': 'nl_NL.ISO8859-15', 'nn': 'nn_NO.ISO8859-1', 'nn_no': 'nn_NO.ISO8859-1', - 'nn_no.88591': 'nn_NO.ISO8859-1', - 'nn_no.iso88591': 'nn_NO.ISO8859-1', - 'nn_no.iso885915': 'nn_NO.ISO8859-15', - 'nn_no at euro': 'nn_NO.ISO8859-15', 'no': 'no_NO.ISO8859-1', 'no at nynorsk': 'ny_NO.ISO8859-1', 'no_no': 'no_NO.ISO8859-1', - 'no_no.88591': 'no_NO.ISO8859-1', - 'no_no.iso88591': 'no_NO.ISO8859-1', - 'no_no.iso885915': 'no_NO.ISO8859-15', 'no_no.iso88591 at bokmal': 'no_NO.ISO8859-1', 'no_no.iso88591 at nynorsk': 'no_NO.ISO8859-1', - 'no_no at euro': 'no_NO.ISO8859-15', 'norwegian': 'no_NO.ISO8859-1', - 'norwegian.iso88591': 'no_NO.ISO8859-1', 'nr': 'nr_ZA.ISO8859-1', 'nr_za': 'nr_ZA.ISO8859-1', - 'nr_za.iso88591': 'nr_ZA.ISO8859-1', 'nso': 'nso_ZA.ISO8859-15', 'nso_za': 'nso_ZA.ISO8859-15', - 'nso_za.iso885915': 'nso_ZA.ISO8859-15', 'ny': 'ny_NO.ISO8859-1', 'ny_no': 'ny_NO.ISO8859-1', - 'ny_no.88591': 'ny_NO.ISO8859-1', - 'ny_no.iso88591': 'ny_NO.ISO8859-1', - 'ny_no.iso885915': 'ny_NO.ISO8859-15', - 'ny_no at euro': 'ny_NO.ISO8859-15', 'nynorsk': 'nn_NO.ISO8859-1', 'oc': 'oc_FR.ISO8859-1', 'oc_fr': 'oc_FR.ISO8859-1', - 'oc_fr.iso88591': 'oc_FR.ISO8859-1', - 'oc_fr.iso885915': 'oc_FR.ISO8859-15', - 'oc_fr at euro': 'oc_FR.ISO8859-15', 'or': 'or_IN.UTF-8', 'or_in': 'or_IN.UTF-8', 'pa': 'pa_IN.UTF-8', 'pa_in': 'pa_IN.UTF-8', 'pd': 'pd_US.ISO8859-1', 'pd_de': 'pd_DE.ISO8859-1', - 'pd_de.iso88591': 'pd_DE.ISO8859-1', - 'pd_de.iso885915': 'pd_DE.ISO8859-15', - 'pd_de at euro': 'pd_DE.ISO8859-15', 'pd_us': 'pd_US.ISO8859-1', - 'pd_us.iso88591': 'pd_US.ISO8859-1', - 'pd_us.iso885915': 'pd_US.ISO8859-15', - 'pd_us at euro': 'pd_US.ISO8859-15', 'ph': 'ph_PH.ISO8859-1', 'ph_ph': 'ph_PH.ISO8859-1', - 'ph_ph.iso88591': 'ph_PH.ISO8859-1', 'pl': 'pl_PL.ISO8859-2', 'pl_pl': 'pl_PL.ISO8859-2', - 'pl_pl.iso88592': 'pl_PL.ISO8859-2', 'polish': 'pl_PL.ISO8859-2', 'portuguese': 'pt_PT.ISO8859-1', - 'portuguese.iso88591': 'pt_PT.ISO8859-1', 'portuguese_brazil': 'pt_BR.ISO8859-1', - 'portuguese_brazil.8859': 'pt_BR.ISO8859-1', 'posix': 'C', 'posix-utf2': 'C', 'pp': 'pp_AN.ISO8859-1', 'pp_an': 'pp_AN.ISO8859-1', - 'pp_an.iso88591': 'pp_AN.ISO8859-1', 'pt': 'pt_PT.ISO8859-1', - 'pt.iso885915': 'pt_PT.ISO8859-15', 'pt_br': 'pt_BR.ISO8859-1', - 'pt_br.88591': 'pt_BR.ISO8859-1', - 'pt_br.iso88591': 'pt_BR.ISO8859-1', - 'pt_br.iso885915': 'pt_BR.ISO8859-15', - 'pt_br at euro': 'pt_BR.ISO8859-15', 'pt_pt': 'pt_PT.ISO8859-1', - 'pt_pt.88591': 'pt_PT.ISO8859-1', - 'pt_pt.iso88591': 'pt_PT.ISO8859-1', - 'pt_pt.iso885915': 'pt_PT.ISO8859-15', - 'pt_pt.iso885915 at euro': 'pt_PT.ISO8859-15', - 'pt_pt.utf8 at euro': 'pt_PT.UTF-8', - 'pt_pt at euro': 'pt_PT.ISO8859-15', 'ro': 'ro_RO.ISO8859-2', 'ro_ro': 'ro_RO.ISO8859-2', - 'ro_ro.iso88592': 'ro_RO.ISO8859-2', 'romanian': 'ro_RO.ISO8859-2', 'ru': 'ru_RU.UTF-8', - 'ru.koi8r': 'ru_RU.KOI8-R', 'ru_ru': 'ru_RU.UTF-8', - 'ru_ru.cp1251': 'ru_RU.CP1251', - 'ru_ru.iso88595': 'ru_RU.ISO8859-5', - 'ru_ru.koi8r': 'ru_RU.KOI8-R', - 'ru_ru.microsoftcp1251': 'ru_RU.CP1251', 'ru_ua': 'ru_UA.KOI8-U', - 'ru_ua.cp1251': 'ru_UA.CP1251', - 'ru_ua.koi8u': 'ru_UA.KOI8-U', - 'ru_ua.microsoftcp1251': 'ru_UA.CP1251', 'rumanian': 'ro_RO.ISO8859-2', 'russian': 'ru_RU.ISO8859-5', 'rw': 'rw_RW.ISO8859-1', 'rw_rw': 'rw_RW.ISO8859-1', - 'rw_rw.iso88591': 'rw_RW.ISO8859-1', 'sd': 'sd_IN.UTF-8', - 'sd at devanagari': 'sd_IN.UTF-8 at devanagari', 'sd_in': 'sd_IN.UTF-8', - 'sd_in at devanagari': 'sd_IN.UTF-8 at devanagari', 'se_no': 'se_NO.UTF-8', 'serbocroatian': 'sr_RS.UTF-8 at latin', 'sh': 'sr_RS.UTF-8 at latin', @@ -1490,37 +1159,26 @@ 'sinhala': 'si_LK.UTF-8', 'sk': 'sk_SK.ISO8859-2', 'sk_sk': 'sk_SK.ISO8859-2', - 'sk_sk.iso88592': 'sk_SK.ISO8859-2', 'sl': 'sl_SI.ISO8859-2', 'sl_cs': 'sl_CS.ISO8859-2', 'sl_si': 'sl_SI.ISO8859-2', - 'sl_si.iso88592': 'sl_SI.ISO8859-2', 'slovak': 'sk_SK.ISO8859-2', 'slovene': 'sl_SI.ISO8859-2', 'slovenian': 'sl_SI.ISO8859-2', 'sp': 'sr_CS.ISO8859-5', 'sp_yu': 'sr_CS.ISO8859-5', 'spanish': 'es_ES.ISO8859-1', - 'spanish.iso88591': 'es_ES.ISO8859-1', 'spanish_spain': 'es_ES.ISO8859-1', - 'spanish_spain.8859': 'es_ES.ISO8859-1', 'sq': 'sq_AL.ISO8859-2', 'sq_al': 'sq_AL.ISO8859-2', - 'sq_al.iso88592': 'sq_AL.ISO8859-2', 'sr': 'sr_RS.UTF-8', 'sr at cyrillic': 'sr_RS.UTF-8', - 'sr at latin': 'sr_RS.UTF-8 at latin', 'sr at latn': 'sr_CS.UTF-8 at latin', 'sr_cs': 'sr_CS.UTF-8', - 'sr_cs.iso88592': 'sr_CS.ISO8859-2', 'sr_cs.iso88592 at latn': 'sr_CS.ISO8859-2', - 'sr_cs.iso88595': 'sr_CS.ISO8859-5', - 'sr_cs.utf8 at latn': 'sr_CS.UTF-8 at latin', 'sr_cs at latn': 'sr_CS.UTF-8 at latin', 'sr_me': 'sr_ME.UTF-8', 'sr_rs': 'sr_RS.UTF-8', - 'sr_rs.utf8 at latn': 'sr_RS.UTF-8 at latin', - 'sr_rs at latin': 'sr_RS.UTF-8 at latin', 'sr_rs at latn': 'sr_RS.UTF-8 at latin', 'sr_sp': 'sr_CS.ISO8859-2', 'sr_yu': 'sr_RS.UTF-8 at latin', @@ -1529,29 +1187,15 @@ 'sr_yu.iso88595': 'sr_CS.ISO8859-5', 'sr_yu.iso88595 at cyrillic': 'sr_CS.ISO8859-5', 'sr_yu.microsoftcp1251 at cyrillic': 'sr_CS.CP1251', - 'sr_yu.utf8 at cyrillic': 'sr_RS.UTF-8', 'sr_yu at cyrillic': 'sr_RS.UTF-8', 'ss': 'ss_ZA.ISO8859-1', 'ss_za': 'ss_ZA.ISO8859-1', - 'ss_za.iso88591': 'ss_ZA.ISO8859-1', 'st': 'st_ZA.ISO8859-1', 'st_za': 'st_ZA.ISO8859-1', - 'st_za.iso88591': 'st_ZA.ISO8859-1', 'sv': 'sv_SE.ISO8859-1', - 'sv.iso885915': 'sv_SE.ISO8859-15', 'sv_fi': 'sv_FI.ISO8859-1', - 'sv_fi.iso88591': 'sv_FI.ISO8859-1', - 'sv_fi.iso885915': 'sv_FI.ISO8859-15', - 'sv_fi.iso885915 at euro': 'sv_FI.ISO8859-15', - 'sv_fi.utf8 at euro': 'sv_FI.UTF-8', - 'sv_fi at euro': 'sv_FI.ISO8859-15', 'sv_se': 'sv_SE.ISO8859-1', - 'sv_se.88591': 'sv_SE.ISO8859-1', - 'sv_se.iso88591': 'sv_SE.ISO8859-1', - 'sv_se.iso885915': 'sv_SE.ISO8859-15', - 'sv_se at euro': 'sv_SE.ISO8859-15', 'swedish': 'sv_SE.ISO8859-1', - 'swedish.iso88591': 'sv_SE.ISO8859-1', 'ta': 'ta_IN.TSCII-0', 'ta_in': 'ta_IN.TSCII-0', 'ta_in.tscii': 'ta_IN.TSCII-0', @@ -1559,49 +1203,33 @@ 'te': 'te_IN.UTF-8', 'tg': 'tg_TJ.KOI8-C', 'tg_tj': 'tg_TJ.KOI8-C', - 'tg_tj.koi8c': 'tg_TJ.KOI8-C', 'th': 'th_TH.ISO8859-11', 'th_th': 'th_TH.ISO8859-11', - 'th_th.iso885911': 'th_TH.ISO8859-11', 'th_th.tactis': 'th_TH.TIS620', 'th_th.tis620': 'th_TH.TIS620', 'thai': 'th_TH.ISO8859-11', 'tl': 'tl_PH.ISO8859-1', 'tl_ph': 'tl_PH.ISO8859-1', - 'tl_ph.iso88591': 'tl_PH.ISO8859-1', 'tn': 'tn_ZA.ISO8859-15', 'tn_za': 'tn_ZA.ISO8859-15', - 'tn_za.iso885915': 'tn_ZA.ISO8859-15', 'tr': 'tr_TR.ISO8859-9', 'tr_tr': 'tr_TR.ISO8859-9', - 'tr_tr.iso88599': 'tr_TR.ISO8859-9', 'ts': 'ts_ZA.ISO8859-1', 'ts_za': 'ts_ZA.ISO8859-1', - 'ts_za.iso88591': 'ts_ZA.ISO8859-1', 'tt': 'tt_RU.TATAR-CYR', 'tt_ru': 'tt_RU.TATAR-CYR', - 'tt_ru.koi8c': 'tt_RU.KOI8-C', 'tt_ru.tatarcyr': 'tt_RU.TATAR-CYR', 'turkish': 'tr_TR.ISO8859-9', - 'turkish.iso88599': 'tr_TR.ISO8859-9', 'uk': 'uk_UA.KOI8-U', 'uk_ua': 'uk_UA.KOI8-U', - 'uk_ua.cp1251': 'uk_UA.CP1251', - 'uk_ua.iso88595': 'uk_UA.ISO8859-5', - 'uk_ua.koi8u': 'uk_UA.KOI8-U', - 'uk_ua.microsoftcp1251': 'uk_UA.CP1251', 'univ': 'en_US.utf', 'universal': 'en_US.utf', 'universal.utf8 at ucs4': 'en_US.UTF-8', 'ur': 'ur_PK.CP1256', 'ur_in': 'ur_IN.UTF-8', 'ur_pk': 'ur_PK.CP1256', - 'ur_pk.cp1256': 'ur_PK.CP1256', - 'ur_pk.microsoftcp1256': 'ur_PK.CP1256', 'uz': 'uz_UZ.UTF-8', 'uz_uz': 'uz_UZ.UTF-8', - 'uz_uz.iso88591': 'uz_UZ.ISO8859-1', - 'uz_uz.utf8 at cyrillic': 'uz_UZ.UTF-8', 'uz_uz at cyrillic': 'uz_UZ.UTF-8', 've': 've_ZA.UTF-8', 've_za': 've_ZA.UTF-8', @@ -1613,35 +1241,21 @@ 'vi_vn.viscii111': 'vi_VN.VISCII', 'wa': 'wa_BE.ISO8859-1', 'wa_be': 'wa_BE.ISO8859-1', - 'wa_be.iso88591': 'wa_BE.ISO8859-1', - 'wa_be.iso885915': 'wa_BE.ISO8859-15', - 'wa_be.iso885915 at euro': 'wa_BE.ISO8859-15', - 'wa_be at euro': 'wa_BE.ISO8859-15', 'xh': 'xh_ZA.ISO8859-1', 'xh_za': 'xh_ZA.ISO8859-1', - 'xh_za.iso88591': 'xh_ZA.ISO8859-1', 'yi': 'yi_US.CP1255', 'yi_us': 'yi_US.CP1255', - 'yi_us.cp1255': 'yi_US.CP1255', - 'yi_us.microsoftcp1255': 'yi_US.CP1255', 'zh': 'zh_CN.eucCN', 'zh_cn': 'zh_CN.gb2312', 'zh_cn.big5': 'zh_TW.big5', 'zh_cn.euc': 'zh_CN.eucCN', - 'zh_cn.gb18030': 'zh_CN.gb18030', - 'zh_cn.gb2312': 'zh_CN.gb2312', - 'zh_cn.gbk': 'zh_CN.gbk', 'zh_hk': 'zh_HK.big5hkscs', - 'zh_hk.big5': 'zh_HK.big5', 'zh_hk.big5hk': 'zh_HK.big5hkscs', - 'zh_hk.big5hkscs': 'zh_HK.big5hkscs', 'zh_tw': 'zh_TW.big5', - 'zh_tw.big5': 'zh_TW.big5', 'zh_tw.euc': 'zh_TW.eucTW', 'zh_tw.euctw': 'zh_TW.eucTW', 'zu': 'zu_ZA.ISO8859-1', 'zu_za': 'zu_ZA.ISO8859-1', - 'zu_za.iso88591': 'zu_ZA.ISO8859-1', } # diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py --- a/Lib/test/test_locale.py +++ b/Lib/test/test_locale.py @@ -384,6 +384,7 @@ def test_english(self): self.check('en', 'en_US.ISO8859-1') self.check('EN', 'en_US.ISO8859-1') + self.check('en.iso88591', 'en_US.ISO8859-1') self.check('en_US', 'en_US.ISO8859-1') self.check('en_us', 'en_US.ISO8859-1') self.check('en_GB', 'en_GB.ISO8859-1') @@ -392,7 +393,10 @@ self.check('en_US:UTF-8', 'en_US.UTF-8') self.check('en_US.ISO8859-1', 'en_US.ISO8859-1') self.check('en_US.US-ASCII', 'en_US.ISO8859-1') + self.check('en_US.88591', 'en_US.ISO8859-1') + self.check('en_US.885915', 'en_US.ISO8859-15') self.check('english', 'en_EN.ISO8859-1') + self.check('english_uk.ascii', 'en_GB.ISO8859-1') def test_hyphenated_encoding(self): self.check('az_AZ.iso88599e', 'az_AZ.ISO8859-9E') @@ -412,10 +416,12 @@ def test_euro_modifier(self): self.check('de_DE at euro', 'de_DE.ISO8859-15') self.check('en_US.ISO8859-15 at euro', 'en_US.ISO8859-15') + self.check('de_DE.utf8 at euro', 'de_DE.UTF-8') def test_latin_modifier(self): self.check('be_BY.UTF-8 at latin', 'be_BY.UTF-8 at latin') self.check('sr_RS.UTF-8 at latin', 'sr_RS.UTF-8 at latin') + self.check('sr_RS.UTF-8 at latn', 'sr_RS.UTF-8 at latin') def test_valencia_modifier(self): self.check('ca_ES.UTF-8 at valencia', 'ca_ES.UTF-8 at valencia') @@ -436,6 +442,39 @@ self.check('sd_IN', 'sd_IN.UTF-8') self.check('sd', 'sd_IN.UTF-8') + def test_euc_encoding(self): + self.check('ja_jp.euc', 'ja_JP.eucJP') + self.check('ja_jp.eucjp', 'ja_JP.eucJP') + self.check('ko_kr.euc', 'ko_KR.eucKR') + self.check('ko_kr.euckr', 'ko_KR.eucKR') + self.check('zh_cn.euc', 'zh_CN.eucCN') + self.check('zh_tw.euc', 'zh_TW.eucTW') + self.check('zh_tw.euctw', 'zh_TW.eucTW') + + def test_japanese(self): + self.check('ja', 'ja_JP.eucJP') + self.check('ja.jis', 'ja_JP.JIS7') + self.check('ja.sjis', 'ja_JP.SJIS') + self.check('ja_jp', 'ja_JP.eucJP') + self.check('ja_jp.ajec', 'ja_JP.eucJP') + self.check('ja_jp.euc', 'ja_JP.eucJP') + self.check('ja_jp.eucjp', 'ja_JP.eucJP') + self.check('ja_jp.iso-2022-jp', 'ja_JP.JIS7') + self.check('ja_jp.iso2022jp', 'ja_JP.JIS7') + self.check('ja_jp.jis', 'ja_JP.JIS7') + self.check('ja_jp.jis7', 'ja_JP.JIS7') + self.check('ja_jp.mscode', 'ja_JP.SJIS') + self.check('ja_jp.pck', 'ja_JP.SJIS') + self.check('ja_jp.sjis', 'ja_JP.SJIS') + self.check('ja_jp.ujis', 'ja_JP.eucJP') + self.check('ja_jp.utf8', 'ja_JP.UTF-8') + self.check('japan', 'ja_JP.eucJP') + self.check('japanese', 'ja_JP.eucJP') + self.check('japanese-euc', 'ja_JP.eucJP') + self.check('japanese.euc', 'ja_JP.eucJP') + self.check('japanese.sjis', 'ja_JP.SJIS') + self.check('jp_jp', 'ja_JP.eucJP') + class TestMiscellaneous(unittest.TestCase): def test_getpreferredencoding(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #20046: Locale alias table no longer contains entities which can be + calculated. Generalized support of the euro modifier. + - Issue #20027: Fixed locale aliases for devanagari locales. - Issue #20067: Tkinter variables now work when wantobjects is false. diff --git a/Tools/i18n/makelocalealias.py b/Tools/i18n/makelocalealias.py --- a/Tools/i18n/makelocalealias.py +++ b/Tools/i18n/makelocalealias.py @@ -7,6 +7,7 @@ """ import locale +import sys # Location of the alias file LOCALE_ALIAS = '/usr/share/X11/locale/locale.alias' @@ -65,9 +66,35 @@ (k, olddata[k], data[k])) # Additions are not mentioned +def optimize(data): + locale_alias = locale.locale_alias + locale.locale_alias = data.copy() + for k, v in data.items(): + del locale.locale_alias[k] + if locale.normalize(k) != v: + locale.locale_alias[k] = v + newdata = locale.locale_alias + errors = check(data) + locale.locale_alias = locale_alias + if errors: + sys.exit(1) + return newdata + +def check(data): + # Check that all alias definitions from the X11 file + # are actually mapped to the correct alias locales. + errors = 0 + for k, v in data.items(): + if locale.normalize(k) != v: + print('ERROR: %a -> %a != %a' % (k, locale.normalize(k), v), + file=sys.stderr) + errors += 1 + return errors + if __name__ == '__main__': data = locale.locale_alias.copy() data.update(parse(LOCALE_ALIAS)) + data = optimize(data) print_differences(data, locale.locale_alias) print() print('locale_alias = {') -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 27 00:13:02 2013 From: python-checkins at python.org (guido.van.rossum) Date: Fri, 27 Dec 2013 00:13:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?peps=3A_Typo_fix=2C_patch_by_Brett_Sl?= =?utf-8?q?atkin=2E?= Message-ID: <3dr6Qk68nsz7Ljn@mail.python.org> http://hg.python.org/peps/rev/02847b00c293 changeset: 5334:02847b00c293 user: Guido van Rossum date: Thu Dec 26 15:12:56 2013 -0800 summary: Typo fix, patch by Brett Slatkin. files: pep-3156.txt | 14 +++++++------- 1 files changed, 7 insertions(+), 7 deletions(-) diff --git a/pep-3156.txt b/pep-3156.txt --- a/pep-3156.txt +++ b/pep-3156.txt @@ -444,7 +444,7 @@ Callbacks associated with the same event loop are strictly serialized: one callback must finish before the next one will be called. This is an important guarantee: when two or more callbacks use or modify -shared state, each callback is guaranteed that while is running, the +shared state, each callback is guaranteed that while it is running, the shared state isn't changed by another callback. - ``call_soon(callback, *args)``. This schedules a callback to be @@ -864,17 +864,17 @@ - ``stdin``: Either a file-like object representing the pipe to be connected to the subprocess's standard input stream using - ``create_write_pipe()``, or the constant ``subprocess.PIPE`` (the + ``connect_write_pipe()``, or the constant ``subprocess.PIPE`` (the default). By default a new pipe will be created and connected. - ``stdout``: Either a file-like object representing the pipe to be connected to the subprocess's standard output stream using - ``create_read_pipe()``, or the constant ``subprocess.PIPE`` (the + ``connect_read_pipe()``, or the constant ``subprocess.PIPE`` (the default). By default a new pipe will be created and connected. - ``stderr``: Either a file-like object representing the pipe to be connected to the subprocess's standard error stream using - ``create_read_pipe()``, or one of the constants ``subprocess.PIPE`` + ``connect_read_pipe()``, or one of the constants ``subprocess.PIPE`` (the default) or ``subprocess.STDOUT``. By default a new pipe will be created and connected. When ``subprocess.STDOUT`` is specified, the subprocess's standard error stream will be connected to the same @@ -1925,7 +1925,7 @@ References ========== -- PEP 380 describes the semantics of ``yield from``. +- PEP 380 describes the semantics of ``yield from``. - Greg Ewing's ``yield from`` tutorials: http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yield_from.html @@ -1967,14 +1967,14 @@ as frequent email exchanges) with several Twisted developers, including Glyph, Brian Warner, David Reid, and Duncan McGreggor. -Contributors to the implementation include +Contributors to the implementation include Eli Bendersky, Gustavo Carneiro (Gambit Research), Sa?l Ibarra Corretg?, Geert Jansen, A. Jesse Jiryu Davis, Nikolay Kim, -Charles-Fran?ois Natali, +Charles-Fran?ois Natali, Richard Oudkerk, Antoine Pitrou, Giampaolo Rodol?, -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri Dec 27 08:34:42 2013 From: python-checkins at python.org (donald.stufft) Date: Fri, 27 Dec 2013 08:34:42 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Update_bundled_pip_to_1=2E?= =?utf-8?q?5rc3?= Message-ID: <3drKYZ4b7Wz7LjR@mail.python.org> http://hg.python.org/cpython/rev/1fb9383baa5f changeset: 88205:1fb9383baa5f user: Donald Stufft date: Fri Dec 27 02:07:49 2013 -0500 summary: Update bundled pip to 1.5rc3 files: Lib/ensurepip/__init__.py | 2 +- Lib/ensurepip/_bundled/pip-1.5rc2-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/pip-1.5rc3-py2.py3-none-any.whl | Bin 3 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -12,7 +12,7 @@ _SETUPTOOLS_VERSION = "2.0.1" -_PIP_VERSION = "1.5rc2" +_PIP_VERSION = "1.5rc3" # pip currently requires ssl support, so we try to provide a nicer # error message when that is missing (http://bugs.python.org/issue19744) diff --git a/Lib/ensurepip/_bundled/pip-1.5rc2-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-1.5rc2-py2.py3-none-any.whl deleted file mode 100644 index 719bd84505dbdcbf13516df0b9ab979afa178d78..0000000000000000000000000000000000000000 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/pip-1.5rc3-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-1.5rc3-py2.py3-none-any.whl new file mode 100644 index 0000000000000000000000000000000000000000..58443638a6a26bc0e34b3cfe44d6f4a716776f62 GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Fri Dec 27 09:44:56 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 27 Dec 2013 09:44:56 +0100 Subject: [Python-checkins] Daily reference leaks (63bc68d7f449): sum=-4 Message-ID: results for 63bc68d7f449 on branch "default" -------------------------------------------- test_site leaked [0, -2, 0] references, sum=-2 test_site leaked [0, -2, 0] memory blocks, sum=-2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog0dzieA', '-x'] From python-checkins at python.org Fri Dec 27 17:48:25 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 27 Dec 2013 17:48:25 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogIzE4MTE2OiBiYWNr?= =?utf-8?q?port_fix_to_3=2E3_since_real-world_failure_mode_demonstrated=2E?= Message-ID: <3drYrT0QNCz7Lky@mail.python.org> http://hg.python.org/cpython/rev/100f632d4306 changeset: 88206:100f632d4306 branch: 3.3 parent: 88201:7615c009e925 user: R David Murray date: Fri Dec 27 11:24:32 2013 -0500 summary: #18116: backport fix to 3.3 since real-world failure mode demonstrated. In issue 20074 it was pointed out that getpass would fail with a traceback if stdin was, for example /dev/null, which is a non-unlikely scenario. Also backported the tests from issue 17484 as modified by issue 18116. (What I really did was copy getpass.py and test_getpass.py from their state on tip as of 17bd04fbf3d3). files: Lib/getpass.py | 95 +++++++++------- Lib/test/test_getpass.py | 155 +++++++++++++++++++++++++++ Lib/test/test_sundry.py | 1 - Misc/NEWS | 5 + 4 files changed, 212 insertions(+), 44 deletions(-) diff --git a/Lib/getpass.py b/Lib/getpass.py --- a/Lib/getpass.py +++ b/Lib/getpass.py @@ -15,7 +15,11 @@ # Guido van Rossum (Windows support and cleanup) # Gregory P. Smith (tty support & GetPassWarning) -import os, sys, warnings +import contextlib +import io +import os +import sys +import warnings __all__ = ["getpass","getuser","GetPassWarning"] @@ -38,52 +42,57 @@ Always restores terminal settings before returning. """ - fd = None - tty = None - try: - # Always try reading and writing directly on the tty first. - fd = os.open('/dev/tty', os.O_RDWR|os.O_NOCTTY) - tty = os.fdopen(fd, 'w+', 1) - input = tty - if not stream: - stream = tty - except EnvironmentError as e: - # If that fails, see if stdin can be controlled. + passwd = None + with contextlib.ExitStack() as stack: try: - fd = sys.stdin.fileno() - except (AttributeError, ValueError): - passwd = fallback_getpass(prompt, stream) - input = sys.stdin - if not stream: - stream = sys.stderr + # Always try reading and writing directly on the tty first. + fd = os.open('/dev/tty', os.O_RDWR|os.O_NOCTTY) + tty = io.FileIO(fd, 'w+') + stack.enter_context(tty) + input = io.TextIOWrapper(tty) + stack.enter_context(input) + if not stream: + stream = input + except OSError as e: + # If that fails, see if stdin can be controlled. + stack.close() + try: + fd = sys.stdin.fileno() + except (AttributeError, ValueError): + fd = None + passwd = fallback_getpass(prompt, stream) + input = sys.stdin + if not stream: + stream = sys.stderr - if fd is not None: - passwd = None - try: - old = termios.tcgetattr(fd) # a copy to save - new = old[:] - new[3] &= ~termios.ECHO # 3 == 'lflags' - tcsetattr_flags = termios.TCSAFLUSH - if hasattr(termios, 'TCSASOFT'): - tcsetattr_flags |= termios.TCSASOFT + if fd is not None: try: - termios.tcsetattr(fd, tcsetattr_flags, new) - passwd = _raw_input(prompt, stream, input=input) - finally: - termios.tcsetattr(fd, tcsetattr_flags, old) - stream.flush() # issue7208 - except termios.error: - if passwd is not None: - # _raw_input succeeded. The final tcsetattr failed. Reraise - # instead of leaving the terminal in an unknown state. - raise - # We can't control the tty or stdin. Give up and use normal IO. - # fallback_getpass() raises an appropriate warning. - del input, tty # clean up unused file objects before blocking - passwd = fallback_getpass(prompt, stream) + old = termios.tcgetattr(fd) # a copy to save + new = old[:] + new[3] &= ~termios.ECHO # 3 == 'lflags' + tcsetattr_flags = termios.TCSAFLUSH + if hasattr(termios, 'TCSASOFT'): + tcsetattr_flags |= termios.TCSASOFT + try: + termios.tcsetattr(fd, tcsetattr_flags, new) + passwd = _raw_input(prompt, stream, input=input) + finally: + termios.tcsetattr(fd, tcsetattr_flags, old) + stream.flush() # issue7208 + except termios.error: + if passwd is not None: + # _raw_input succeeded. The final tcsetattr failed. Reraise + # instead of leaving the terminal in an unknown state. + raise + # We can't control the tty or stdin. Give up and use normal IO. + # fallback_getpass() raises an appropriate warning. + if stream is not input: + # clean up unused file objects before blocking + stack.close() + passwd = fallback_getpass(prompt, stream) - stream.write('\n') - return passwd + stream.write('\n') + return passwd def win_getpass(prompt='Password: ', stream=None): diff --git a/Lib/test/test_getpass.py b/Lib/test/test_getpass.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_getpass.py @@ -0,0 +1,155 @@ +import getpass +import os +import unittest +from io import BytesIO, StringIO +from unittest import mock +from test import support + +try: + import termios +except ImportError: + termios = None +try: + import pwd +except ImportError: + pwd = None + + at mock.patch('os.environ') +class GetpassGetuserTest(unittest.TestCase): + + def test_username_takes_username_from_env(self, environ): + expected_name = 'some_name' + environ.get.return_value = expected_name + self.assertEqual(expected_name, getpass.getuser()) + + def test_username_priorities_of_env_values(self, environ): + environ.get.return_value = None + try: + getpass.getuser() + except ImportError: # in case there's no pwd module + pass + self.assertEqual( + environ.get.call_args_list, + [mock.call(x) for x in ('LOGNAME', 'USER', 'LNAME', 'USERNAME')]) + + def test_username_falls_back_to_pwd(self, environ): + expected_name = 'some_name' + environ.get.return_value = None + if pwd: + with mock.patch('os.getuid') as uid, \ + mock.patch('pwd.getpwuid') as getpw: + uid.return_value = 42 + getpw.return_value = [expected_name] + self.assertEqual(expected_name, + getpass.getuser()) + getpw.assert_called_once_with(42) + else: + self.assertRaises(ImportError, getpass.getuser) + + +class GetpassRawinputTest(unittest.TestCase): + + def test_flushes_stream_after_prompt(self): + # see issue 1703 + stream = mock.Mock(spec=StringIO) + input = StringIO('input_string') + getpass._raw_input('some_prompt', stream, input=input) + stream.flush.assert_called_once_with() + + def test_uses_stderr_as_default(self): + input = StringIO('input_string') + prompt = 'some_prompt' + with mock.patch('sys.stderr') as stderr: + getpass._raw_input(prompt, input=input) + stderr.write.assert_called_once_with(prompt) + + @mock.patch('sys.stdin') + def test_uses_stdin_as_default_input(self, mock_input): + mock_input.readline.return_value = 'input_string' + getpass._raw_input(stream=StringIO()) + mock_input.readline.assert_called_once_with() + + def test_raises_on_empty_input(self): + input = StringIO('') + self.assertRaises(EOFError, getpass._raw_input, input=input) + + def test_trims_trailing_newline(self): + input = StringIO('test\n') + self.assertEqual('test', getpass._raw_input(input=input)) + + +# Some of these tests are a bit white-box. The functional requirement is that +# the password input be taken directly from the tty, and that it not be echoed +# on the screen, unless we are falling back to stderr/stdin. + +# Some of these might run on platforms without termios, but play it safe. + at unittest.skipUnless(termios, 'tests require system with termios') +class UnixGetpassTest(unittest.TestCase): + + def test_uses_tty_directly(self): + with mock.patch('os.open') as open, \ + mock.patch('io.FileIO') as fileio, \ + mock.patch('io.TextIOWrapper') as textio: + # By setting open's return value to None the implementation will + # skip code we don't care about in this test. We can mock this out + # fully if an alternate implementation works differently. + open.return_value = None + getpass.unix_getpass() + open.assert_called_once_with('/dev/tty', + os.O_RDWR | os.O_NOCTTY) + fileio.assert_called_once_with(open.return_value, 'w+') + textio.assert_called_once_with(fileio.return_value) + + def test_resets_termios(self): + with mock.patch('os.open') as open, \ + mock.patch('io.FileIO'), \ + mock.patch('io.TextIOWrapper'), \ + mock.patch('termios.tcgetattr') as tcgetattr, \ + mock.patch('termios.tcsetattr') as tcsetattr: + open.return_value = 3 + fake_attrs = [255, 255, 255, 255, 255] + tcgetattr.return_value = list(fake_attrs) + getpass.unix_getpass() + tcsetattr.assert_called_with(3, mock.ANY, fake_attrs) + + def test_falls_back_to_fallback_if_termios_raises(self): + with mock.patch('os.open') as open, \ + mock.patch('io.FileIO') as fileio, \ + mock.patch('io.TextIOWrapper') as textio, \ + mock.patch('termios.tcgetattr'), \ + mock.patch('termios.tcsetattr') as tcsetattr, \ + mock.patch('getpass.fallback_getpass') as fallback: + open.return_value = 3 + fileio.return_value = BytesIO() + tcsetattr.side_effect = termios.error + getpass.unix_getpass() + fallback.assert_called_once_with('Password: ', + textio.return_value) + + def test_flushes_stream_after_input(self): + # issue 7208 + with mock.patch('os.open') as open, \ + mock.patch('io.FileIO'), \ + mock.patch('io.TextIOWrapper'), \ + mock.patch('termios.tcgetattr'), \ + mock.patch('termios.tcsetattr'): + open.return_value = 3 + mock_stream = mock.Mock(spec=StringIO) + getpass.unix_getpass(stream=mock_stream) + mock_stream.flush.assert_called_with() + + def test_falls_back_to_stdin(self): + with mock.patch('os.open') as os_open, \ + mock.patch('sys.stdin', spec=StringIO) as stdin: + os_open.side_effect = IOError + stdin.fileno.side_effect = AttributeError + with support.captured_stderr() as stderr: + with self.assertWarns(getpass.GetPassWarning): + getpass.unix_getpass() + stdin.readline.assert_called_once_with() + self.assertIn('Warning', stderr.getvalue()) + self.assertIn('Password:', stderr.getvalue()) + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_sundry.py b/Lib/test/test_sundry.py --- a/Lib/test/test_sundry.py +++ b/Lib/test/test_sundry.py @@ -41,7 +41,6 @@ import encodings import formatter - import getpass import html.entities import imghdr import keyword diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,11 @@ Library ------- +- Issue #18116: getpass was always getting an error when testing /dev/tty, + and thus was always falling back to stdin, and would then raise an exception + if stdin could not be used (such as /dev/null). It also leaked an open file. + All of these issues are now fixed. + - Issue #20027: Fixed locale aliases for devanagari locales. - Issue #20067: Tkinter variables now work when wantobjects is false. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 27 17:48:27 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 27 Dec 2013 17:48:27 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Mostly-null_merge_of_=2318116_backport_=28updated_NEWS_e?= =?utf-8?b?bnRyeSku?= Message-ID: <3drYrW0VL1z7LlM@mail.python.org> http://hg.python.org/cpython/rev/29a5a5b39dd6 changeset: 88207:29a5a5b39dd6 parent: 88205:1fb9383baa5f parent: 88206:100f632d4306 user: R David Murray date: Fri Dec 27 11:43:42 2013 -0500 summary: Mostly-null merge of #18116 backport (updated NEWS entry). files: Misc/NEWS | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1898,8 +1898,9 @@ do with byte strings. - Issue #18116: getpass was always getting an error when testing /dev/tty, - and thus was always falling back to stdin. It also leaked an open file - when it did so. Both of these issues are now fixed. + and thus was always falling back to stdin, and would then raise an exception + if stdin could not be used (such as /dev/null). It also leaked an open file. + All of these issues are now fixed. - Issue #17198: Fix a NameError in the dbm module. Patch by Valentina Mukhamedzhanova. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 27 20:07:23 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 27 Dec 2013 20:07:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_importlib_clea?= =?utf-8?q?nup_and_source=5Fto=5Fcode_add=3B_python_partial_impl=2E?= Message-ID: <3drcwq5prtz7Ljv@mail.python.org> http://hg.python.org/cpython/rev/9d9c2b4850fc changeset: 88208:9d9c2b4850fc user: R David Murray date: Fri Dec 27 14:06:15 2013 -0500 summary: whatsnew: importlib cleanup and source_to_code add; python partial impl. files: Doc/whatsnew/3.4.rst | 23 +++++++++++++++++++++-- 1 files changed, 21 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -627,6 +627,11 @@ :pep:`443` -- Single-dispatch generic functions PEP written and implemented by ?ukasz Langa. +A pure-python version of the :func:`~functools.partial` function is now in the +stdlib; in CPython it is overridden by the C accelerated version, but it is +available for other implementations to use. (Contributed by Brian Thorne in +:issue:`12428`.) + gc -- @@ -667,6 +672,16 @@ (Contributed by Ezio Melotti in :issue:`15114`) +importlib +--------- + +The :class:`~importlib.abc.InspectLoader` ABC defines a new method, +:meth:`~importlib.abc.InspectLoader.source_to_code` that accepts source +data and a path and returns a code object. The default implementation +is equivalent to ``compile(data, path, 'exec', dont_inherit=True)``. +(Contributed by Eric Snow and Brett Cannon in :issue:`15627`.) + + inspect ------- @@ -1208,8 +1223,8 @@ Removed ======= -The following previously deprecated APIs and features have been removed -in Python 3.4: +The following obsolete and previously deprecated APIs and features have been +removed in Python 3.4: * Support for the following operating systems has been removed from the source and build tools: @@ -1228,6 +1243,9 @@ * The ``PyThreadState.tick_counter`` field has been removed; its value has been meaningless since Python 3.2, when the "new GIL" was introduced. +* ``PyLoader`` and ``PyPycLoader`` have been removed from :mod:`importlib`. + (Contributed by Taras Lyapun in :issue:`15641`.) + Porting to Python 3.4 ===================== @@ -1294,6 +1312,7 @@ ``-m`` with the interpreter (this does not influence when the path to a file is specified on the command-line). + Changes in the C API -------------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri Dec 27 23:02:09 2013 From: python-checkins at python.org (r.david.murray) Date: Fri, 27 Dec 2013 23:02:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_rewrite_urllib?= =?utf-8?q?=2C_doctest=2C_and_poplib_sections=2E?= Message-ID: <3drhpT5gbTz7LjW@mail.python.org> http://hg.python.org/cpython/rev/d766589ed5a2 changeset: 88209:d766589ed5a2 user: R David Murray date: Fri Dec 27 17:01:16 2013 -0500 summary: whatsnew: rewrite urllib, doctest, and poplib sections. Also collapse redundant versionadded/versionchanged markup in poplib.stls entry. files: Doc/library/poplib.rst | 10 +++----- Doc/whatsnew/3.4.rst | 33 ++++++++++++++--------------- 2 files changed, 20 insertions(+), 23 deletions(-) diff --git a/Doc/library/poplib.rst b/Doc/library/poplib.rst --- a/Doc/library/poplib.rst +++ b/Doc/library/poplib.rst @@ -198,15 +198,13 @@ *context* parameter is a :class:`ssl.SSLContext` object which allows bundling SSL configuration options, certificates and private keys into - a single (potentially long-lived) structure. + a single (potentially long-lived) structure. This method supports + hostname checking via :attr:`SSLContext.check_hostname` + :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see + :data:`~ssl.HAS_SNI`). .. versionadded:: 3.4 - .. versionchanged:: 3.4 - The method now supports hostname check with - :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see - :data:`~ssl.HAS_SNI`). - Instances of :class:`POP3_SSL` have no additional methods. The interface of this subclass is identical to its parent. diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -554,15 +554,15 @@ doctest ------- -Added :data:`~doctest.FAIL_FAST` flag to halt test running as soon as the first -failure is detected. (Contributed by R. David Murray and Daniel Urban in -:issue:`16522`.) +A new :ref:`option flag `, :data:`~doctest.FAIL_FAST`, halts +test running as soon as the first failure is detected. (Contributed by R. +David Murray and Daniel Urban in :issue:`16522`.) -Updated the doctest command line interface to use :mod:`argparse`, and added -``-o`` and ``-f`` options to the interface. ``-o`` allows doctest options to -be specified on the command line, and ``-f`` is a shorthand for ``-o -FAIL_FAST`` (to parallel the similar option supported by the :mod:`unittest` -CLI). (Contributed by R. David Murray in :issue:`11390`.) +The :mod:`doctest` command line interface now uses :mod:`argparse`, and has two +new options, ``-o`` and ``-f``. ``-o`` allows :ref:`doctest options +` to be specified on the command line, and ``-f`` is a +shorthand for ``-o FAIL_FAST`` (to parallel the similar option supported by the +:mod:`unittest` CLI). (Contributed by R. David Murray in :issue:`11390`.) email @@ -795,13 +795,11 @@ poplib ------ -New :meth:`~poplib.POP3.stls` method to switch a clear-text POP3 session into -an encrypted POP3 session. - -New :meth:`~poplib.POP3.capa` method to query the capabilities advertised by the -POP3 server. - -(Contributed by Lorenzo Catucci in :issue:`4473`.) +Two new methods have been added to :mod:`poplib`: :meth:`~poplib.POP3.capa`, +which returns the list of capabilities advertised by the POP server, and +:meth:`~poplib.POP3.stls`, which switches a clear-text POP3 session into an +encrypted POP3 session if the POP server supports it. (Contributed by Lorenzo +Catucci in :issue:`4473`.) pprint @@ -947,8 +945,9 @@ urllib ------ -Add support.for ``data:`` URLs in :mod:`urllib.request`. -(Contributed by Mathias Panzenb?ck in :issue:`16423`.) +:mod:`urllib.request` now supports ``data:`` URLs via the +:class:`~urllib.request.DataHandler` class. (Contributed by Mathias Panzenb?ck +in :issue:`16423`.) unittest -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 28 09:19:17 2013 From: python-checkins at python.org (serhiy.storchaka) Date: Sat, 28 Dec 2013 09:19:17 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Fixed_the_wave?= =?utf-8?q?_module_testing_on_big-endian_platforms=2E?= Message-ID: <3dryVY6zCzz7LjW@mail.python.org> http://hg.python.org/cpython/rev/293e4edb0592 changeset: 88210:293e4edb0592 branch: 2.7 parent: 88200:aad582f717da user: Serhiy Storchaka date: Sat Dec 28 10:18:44 2013 +0200 summary: Fixed the wave module testing on big-endian platforms. array.fromfile() works only with file objects, not io.FileIO instances. files: Lib/test/audiotests.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/audiotests.py b/Lib/test/audiotests.py --- a/Lib/test/audiotests.py +++ b/Lib/test/audiotests.py @@ -6,7 +6,7 @@ import sys import base64 -class UnseekableIO(io.FileIO): +class UnseekableIO(file): def tell(self): raise io.UnsupportedOperation -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sat Dec 28 09:45:26 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 28 Dec 2013 09:45:26 +0100 Subject: [Python-checkins] Daily reference leaks (d766589ed5a2): sum=0 Message-ID: results for d766589ed5a2 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog1BMjv8', '-x'] From python-checkins at python.org Sat Dec 28 17:00:23 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 28 Dec 2013 17:00:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319648=3A_implemen?= =?utf-8?q?t_empty_tests_in_pickletester=2E__Patch_by_Gennadiy_Zlobin=2E?= Message-ID: <3ds8kb4gXyz7LkM@mail.python.org> http://hg.python.org/cpython/rev/fef075ddaec9 changeset: 88211:fef075ddaec9 parent: 88209:d766589ed5a2 user: Antoine Pitrou date: Sat Dec 28 16:57:37 2013 +0100 summary: Issue #19648: implement empty tests in pickletester. Patch by Gennadiy Zlobin. files: Lib/test/pickletester.py | 12 ++++++++++-- Misc/ACKS | 1 + 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -812,10 +812,18 @@ self.assertEqual(self.dumps(1.2, 0)[0:3], b'F1.') def test_reduce(self): - pass + for proto in protocols: + inst = AAA() + dumped = self.dumps(inst, proto) + loaded = self.loads(dumped) + self.assertEqual(loaded, REDUCE_A) def test_getinitargs(self): - pass + for proto in protocols: + inst = initarg(1, 2) + dumped = self.dumps(inst, proto) + loaded = self.loads(dumped) + self.assert_is_copy(inst, loaded) def test_pop_empty_stack(self): # Test issue7455 diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1456,4 +1456,5 @@ Cheng Zhang Kai Zhu Tarek Ziad? +Gennadiy Zlobin Peter ?strand -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 28 17:31:01 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 28 Dec 2013 17:31:01 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWUgIzE5NDIy?= =?utf-8?q?=3A_Explicitly_disallow_non-SOCK=5FSTREAM_sockets_in_the_ssl_mo?= =?utf-8?q?dule=2C?= Message-ID: <3ds9Px5bHMz7LkK@mail.python.org> http://hg.python.org/cpython/rev/a00842b783cf changeset: 88212:a00842b783cf branch: 3.3 parent: 88206:100f632d4306 user: Antoine Pitrou date: Sat Dec 28 17:26:33 2013 +0100 summary: Issue #19422: Explicitly disallow non-SOCK_STREAM sockets in the ssl module, rather than silently let them emit clear text data. files: Doc/library/ssl.rst | 22 ++++++++++++++-------- Lib/ssl.py | 5 +++++ Lib/test/test_ssl.py | 12 ++++++++++++ Misc/NEWS | 3 +++ 4 files changed, 34 insertions(+), 8 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -141,13 +141,16 @@ Takes an instance ``sock`` of :class:`socket.socket`, and returns an instance of :class:`ssl.SSLSocket`, a subtype of :class:`socket.socket`, which wraps - the underlying socket in an SSL context. For client-side sockets, the - context construction is lazy; if the underlying socket isn't connected yet, - the context construction will be performed after :meth:`connect` is called on - the socket. For server-side sockets, if the socket has no remote peer, it is - assumed to be a listening socket, and the server-side SSL wrapping is - automatically performed on client connections accepted via the :meth:`accept` - method. :func:`wrap_socket` may raise :exc:`SSLError`. + the underlying socket in an SSL context. ``sock`` must be a + :data:`~socket.SOCK_STREAM` socket; other socket types are unsupported. + + For client-side sockets, the context construction is lazy; if the + underlying socket isn't connected yet, the context construction will be + performed after :meth:`connect` is called on the socket. For + server-side sockets, if the socket has no remote peer, it is assumed + to be a listening socket, and the server-side SSL wrapping is + automatically performed on client connections accepted via the + :meth:`accept` method. :func:`wrap_socket` may raise :exc:`SSLError`. The ``keyfile`` and ``certfile`` parameters specify optional files which contain a certificate to be used to identify the local side of the @@ -836,7 +839,10 @@ server_hostname=None) Wrap an existing Python socket *sock* and return an :class:`SSLSocket` - object. The SSL socket is tied to the context, its settings and + object. *sock* must be a :data:`~socket.SOCK_STREAM` socket; other socket + types are unsupported. + + The returned SSL socket is tied to the context, its settings and certificates. The parameters *server_side*, *do_handshake_on_connect* and *suppress_ragged_eofs* have the same meaning as in the top-level :func:`wrap_socket` function. diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -111,6 +111,7 @@ from socket import getnameinfo as _getnameinfo from socket import error as socket_error from socket import socket, AF_INET, SOCK_STREAM, create_connection +from socket import SOL_SOCKET, SO_TYPE import base64 # for DER-to-PEM translation import traceback import errno @@ -296,6 +297,10 @@ self.ssl_version = ssl_version self.ca_certs = ca_certs self.ciphers = ciphers + # Can't use sock.type as other flags (such as SOCK_NONBLOCK) get + # mixed in. + if sock.getsockopt(SOL_SOCKET, SO_TYPE) != SOCK_STREAM: + raise NotImplementedError("only stream sockets are supported") if server_side and server_hostname: raise ValueError("server_hostname can only be specified " "in client mode") diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -493,6 +493,18 @@ support.gc_collect() self.assertIn(r, str(cm.warning.args[0])) + def test_unsupported_dtls(self): + s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + self.addCleanup(s.close) + with self.assertRaises(NotImplementedError) as cx: + ssl.wrap_socket(s, cert_reqs=ssl.CERT_NONE) + self.assertEqual(str(cx.exception), "only stream sockets are supported") + ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + with self.assertRaises(NotImplementedError) as cx: + ctx.wrap_socket(s) + self.assertEqual(str(cx.exception), "only stream sockets are supported") + + class ContextTests(unittest.TestCase): @skip_if_broken_ubuntu_ssl diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,9 @@ Library ------- +- Issue #19422: Explicitly disallow non-SOCK_STREAM sockets in the ssl + module, rather than silently let them emit clear text data. + - Issue #18116: getpass was always getting an error when testing /dev/tty, and thus was always falling back to stdin, and would then raise an exception if stdin could not be used (such as /dev/null). It also leaked an open file. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 28 17:31:03 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 28 Dec 2013 17:31:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Issue_=2319422=3A_Explicitly_disallow_non-SOCK=5FSTREAM_?= =?utf-8?q?sockets_in_the_ssl_module=2C?= Message-ID: <3ds9Pz1g82z7Lkl@mail.python.org> http://hg.python.org/cpython/rev/f7dc02e6987a changeset: 88213:f7dc02e6987a parent: 88211:fef075ddaec9 parent: 88212:a00842b783cf user: Antoine Pitrou date: Sat Dec 28 17:30:51 2013 +0100 summary: Issue #19422: Explicitly disallow non-SOCK_STREAM sockets in the ssl module, rather than silently let them emit clear text data. files: Doc/library/ssl.rst | 22 ++++++++++++++-------- Lib/ssl.py | 5 +++++ Lib/test/test_ssl.py | 11 +++++++++++ Misc/NEWS | 3 +++ 4 files changed, 33 insertions(+), 8 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -142,13 +142,16 @@ Takes an instance ``sock`` of :class:`socket.socket`, and returns an instance of :class:`ssl.SSLSocket`, a subtype of :class:`socket.socket`, which wraps - the underlying socket in an SSL context. For client-side sockets, the - context construction is lazy; if the underlying socket isn't connected yet, - the context construction will be performed after :meth:`connect` is called on - the socket. For server-side sockets, if the socket has no remote peer, it is - assumed to be a listening socket, and the server-side SSL wrapping is - automatically performed on client connections accepted via the :meth:`accept` - method. :func:`wrap_socket` may raise :exc:`SSLError`. + the underlying socket in an SSL context. ``sock`` must be a + :data:`~socket.SOCK_STREAM` socket; other socket types are unsupported. + + For client-side sockets, the context construction is lazy; if the + underlying socket isn't connected yet, the context construction will be + performed after :meth:`connect` is called on the socket. For + server-side sockets, if the socket has no remote peer, it is assumed + to be a listening socket, and the server-side SSL wrapping is + automatically performed on client connections accepted via the + :meth:`accept` method. :func:`wrap_socket` may raise :exc:`SSLError`. The ``keyfile`` and ``certfile`` parameters specify optional files which contain a certificate to be used to identify the local side of the @@ -1146,7 +1149,10 @@ server_hostname=None) Wrap an existing Python socket *sock* and return an :class:`SSLSocket` - object. The SSL socket is tied to the context, its settings and + object. *sock* must be a :data:`~socket.SOCK_STREAM` socket; other socket + types are unsupported. + + The returned SSL socket is tied to the context, its settings and certificates. The parameters *server_side*, *do_handshake_on_connect* and *suppress_ragged_eofs* have the same meaning as in the top-level :func:`wrap_socket` function. diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -150,6 +150,7 @@ from socket import getnameinfo as _getnameinfo from socket import SHUT_RDWR as _SHUT_RDWR from socket import socket, AF_INET, SOCK_STREAM, create_connection +from socket import SOL_SOCKET, SO_TYPE import base64 # for DER-to-PEM translation import traceback import errno @@ -482,6 +483,10 @@ self.ssl_version = ssl_version self.ca_certs = ca_certs self.ciphers = ciphers + # Can't use sock.type as other flags (such as SOCK_NONBLOCK) get + # mixed in. + if sock.getsockopt(SOL_SOCKET, SO_TYPE) != SOCK_STREAM: + raise NotImplementedError("only stream sockets are supported") if server_side and server_hostname: raise ValueError("server_hostname can only be specified " "in client mode") diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -632,6 +632,17 @@ self.assertEqual(ssl.Purpose.CLIENT_AUTH.oid, '1.3.6.1.5.5.7.3.2') + def test_unsupported_dtls(self): + s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + self.addCleanup(s.close) + with self.assertRaises(NotImplementedError) as cx: + ssl.wrap_socket(s, cert_reqs=ssl.CERT_NONE) + self.assertEqual(str(cx.exception), "only stream sockets are supported") + ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv23) + with self.assertRaises(NotImplementedError) as cx: + ctx.wrap_socket(s) + self.assertEqual(str(cx.exception), "only stream sockets are supported") + class ContextTests(unittest.TestCase): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #19422: Explicitly disallow non-SOCK_STREAM sockets in the ssl + module, rather than silently let them emit clear text data. + - Issue #20046: Locale alias table no longer contains entities which can be calculated. Generalized support of the euro modifier. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 28 17:35:23 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 28 Dec 2013 17:35:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE5NDIy?= =?utf-8?q?=3A_Explicitly_disallow_non-SOCK=5FSTREAM_sockets_in_the_ssl_mo?= =?utf-8?q?dule=2C?= Message-ID: <3ds9Vz0N2gz7LkZ@mail.python.org> http://hg.python.org/cpython/rev/44841d81bf14 changeset: 88214:44841d81bf14 branch: 2.7 parent: 88210:293e4edb0592 user: Antoine Pitrou date: Sat Dec 28 17:26:33 2013 +0100 summary: Issue #19422: Explicitly disallow non-SOCK_STREAM sockets in the ssl module, rather than silently let them emit clear text data. files: Doc/library/ssl.rst | 17 ++++++++++------- Lib/ssl.py | 5 +++++ Lib/test/test_ssl.py | 7 +++++++ Misc/NEWS | 3 +++ 4 files changed, 25 insertions(+), 7 deletions(-) diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -69,13 +69,16 @@ Takes an instance ``sock`` of :class:`socket.socket`, and returns an instance of :class:`ssl.SSLSocket`, a subtype of :class:`socket.socket`, which wraps - the underlying socket in an SSL context. For client-side sockets, the - context construction is lazy; if the underlying socket isn't connected yet, - the context construction will be performed after :meth:`connect` is called on - the socket. For server-side sockets, if the socket has no remote peer, it is - assumed to be a listening socket, and the server-side SSL wrapping is - automatically performed on client connections accepted via the :meth:`accept` - method. :func:`wrap_socket` may raise :exc:`SSLError`. + the underlying socket in an SSL context. ``sock`` must be a + :data:`~socket.SOCK_STREAM` socket; other socket types are unsupported. + + For client-side sockets, the context construction is lazy; if the + underlying socket isn't connected yet, the context construction will be + performed after :meth:`connect` is called on the socket. For + server-side sockets, if the socket has no remote peer, it is assumed + to be a listening socket, and the server-side SSL wrapping is + automatically performed on client connections accepted via the + :meth:`accept` method. :func:`wrap_socket` may raise :exc:`SSLError`. The ``keyfile`` and ``certfile`` parameters specify optional files which contain a certificate to be used to identify the local side of the diff --git a/Lib/ssl.py b/Lib/ssl.py --- a/Lib/ssl.py +++ b/Lib/ssl.py @@ -89,6 +89,7 @@ from socket import socket, _fileobject, _delegate_methods, error as socket_error from socket import getnameinfo as _getnameinfo +from socket import SOL_SOCKET, SO_TYPE, SOCK_STREAM import base64 # for DER-to-PEM translation import errno @@ -108,6 +109,10 @@ ssl_version=PROTOCOL_SSLv23, ca_certs=None, do_handshake_on_connect=True, suppress_ragged_eofs=True, ciphers=None): + # Can't use sock.type as other flags (such as SOCK_NONBLOCK) get + # mixed in. + if sock.getsockopt(SOL_SOCKET, SO_TYPE) != SOCK_STREAM: + raise NotImplementedError("only stream sockets are supported") socket.__init__(self, _sock=sock._sock) # The initializer for socket overrides the methods send(), recv(), etc. # in the instancce, which we don't need -- but we want to provide the diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -232,6 +232,13 @@ self.assertRaises(socket.error, ss.send, b'x') self.assertRaises(socket.error, ss.sendto, b'x', ('0.0.0.0', 0)) + def test_unsupported_dtls(self): + s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + self.addCleanup(s.close) + with self.assertRaises(NotImplementedError) as cx: + ssl.wrap_socket(s, cert_reqs=ssl.CERT_NONE) + self.assertEqual(str(cx.exception), "only stream sockets are supported") + class NetworkedTests(unittest.TestCase): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -27,6 +27,9 @@ Library ------- +- Issue #19422: Explicitly disallow non-SOCK_STREAM sockets in the ssl + module, rather than silently let them emit clear text data. + - Issue #20027: Fixed locale aliases for devanagari locales. - Issue #20067: Tkinter variables now work when wantobjects is false. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 28 17:37:34 2013 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 28 Dec 2013 17:37:34 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_complain_if_th?= =?utf-8?q?e_codec_doesn=27t_return_unicode?= Message-ID: <3ds9YV2LG0z7LjR@mail.python.org> http://hg.python.org/cpython/rev/990d7647ea51 changeset: 88215:990d7647ea51 branch: 2.7 user: Benjamin Peterson date: Sat Dec 28 10:33:58 2013 -0600 summary: complain if the codec doesn't return unicode files: Lib/test/bad_coding3.py | 2 ++ Lib/test/test_pep263.py | 5 +++++ Misc/NEWS | 3 +++ Parser/tokenizer.c | 6 ++++++ 4 files changed, 16 insertions(+), 0 deletions(-) diff --git a/Lib/test/bad_coding3.py b/Lib/test/bad_coding3.py new file mode 100644 --- /dev/null +++ b/Lib/test/bad_coding3.py @@ -0,0 +1,2 @@ +# coding: string-escape +\x70\x72\x69\x6e\x74\x20\x32\x2b\x32\x0a diff --git a/Lib/test/test_pep263.py b/Lib/test/test_pep263.py --- a/Lib/test/test_pep263.py +++ b/Lib/test/test_pep263.py @@ -58,6 +58,11 @@ with self.assertRaisesRegexp(SyntaxError, 'BOM'): compile('\xef\xbb\xbf# -*- coding: fake -*-\n', 'dummy', 'exec') + def test_non_unicode_codec(self): + with self.assertRaisesRegexp(SyntaxError, + 'codec did not return a unicode'): + from test import bad_coding3 + def test_main(): test_support.run_unittest(PEP263Test) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -9,6 +9,9 @@ Core and Builtins ----------------- +- Raise a better error when non-unicode codecs are used for a file's coding + cookie. + - Issue #17976: Fixed potential problem with file.write() not detecting IO error by inspecting the return value of fwrite(). Based on patches by Jaakko Moisio and Victor Stinner. diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -400,6 +400,12 @@ buf = PyObject_CallObject(tok->decoding_readline, NULL); if (buf == NULL) return error_ret(tok); + if (!PyUnicode_Check(buf)) { + Py_DECREF(buf); + PyErr_SetString(PyExc_SyntaxError, + "codec did not return a unicode object"); + return error_ret(tok); + } } else { tok->decoding_buffer = NULL; if (PyString_CheckExact(buf)) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 28 19:06:49 2013 From: python-checkins at python.org (guido.van.rossum) Date: Sat, 28 Dec 2013 19:06:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Export_asyncio=2Eiscorouti?= =?utf-8?q?ne=5Bfunction=5D=2E?= Message-ID: <3dsCXT6V3Jz7LkK@mail.python.org> http://hg.python.org/cpython/rev/232d4af24c7c changeset: 88216:232d4af24c7c parent: 88213:f7dc02e6987a user: Guido van Rossum date: Sat Dec 28 08:06:40 2013 -1000 summary: Export asyncio.iscoroutine[function]. files: Lib/asyncio/tasks.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/asyncio/tasks.py b/Lib/asyncio/tasks.py --- a/Lib/asyncio/tasks.py +++ b/Lib/asyncio/tasks.py @@ -1,6 +1,7 @@ """Support for tasks, coroutines and the scheduler.""" __all__ = ['coroutine', 'Task', + 'iscoroutinefunction', 'iscoroutine', 'FIRST_COMPLETED', 'FIRST_EXCEPTION', 'ALL_COMPLETED', 'wait', 'wait_for', 'as_completed', 'sleep', 'async', 'gather', 'shield', -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 28 19:49:11 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 28 Dec 2013 19:49:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319918=3A_Fix_Pure?= =?utf-8?q?Path=2Erelative=5Fto=28=29_under_Windows=2E?= Message-ID: <3dsDTM6mhLz7LkK@mail.python.org> http://hg.python.org/cpython/rev/763f4416c4fd changeset: 88217:763f4416c4fd user: Antoine Pitrou date: Sat Dec 28 19:49:04 2013 +0100 summary: Issue #19918: Fix PurePath.relative_to() under Windows. files: Lib/pathlib.py | 20 ++--- Lib/test/test_pathlib.py | 85 ++++++++++++++++++--------- Misc/NEWS | 2 + 3 files changed, 67 insertions(+), 40 deletions(-) diff --git a/Lib/pathlib.py b/Lib/pathlib.py --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -780,27 +780,23 @@ parts = self._parts drv = self._drv root = self._root - if drv or root: - if root: - abs_parts = [drv, root] + parts[1:] - else: - abs_parts = [drv] + parts[1:] + if root: + abs_parts = [drv, root] + parts[1:] else: abs_parts = parts to_drv, to_root, to_parts = self._parse_args(other) - if to_drv or to_root: - if to_root: - to_abs_parts = [to_drv, to_root] + to_parts[1:] - else: - to_abs_parts = [to_drv] + to_parts[1:] + if to_root: + to_abs_parts = [to_drv, to_root] + to_parts[1:] else: to_abs_parts = to_parts n = len(to_abs_parts) - if n == 0 and (drv or root) or abs_parts[:n] != to_abs_parts: + cf = self._flavour.casefold_parts + if (root or drv) if n == 0 else cf(abs_parts[:n]) != cf(to_abs_parts): formatted = self._format_parsed_parts(to_drv, to_root, to_parts) raise ValueError("{!r} does not start with {!r}" .format(str(self), str(formatted))) - return self._from_parsed_parts('', '', abs_parts[n:]) + return self._from_parsed_parts('', root if n == 1 else '', + abs_parts[n:]) @property def parts(self): diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -536,9 +536,14 @@ P = self.cls p = P('a/b') self.assertRaises(TypeError, p.relative_to) + self.assertRaises(TypeError, p.relative_to, b'a') self.assertEqual(p.relative_to(P()), P('a/b')) + self.assertEqual(p.relative_to(''), P('a/b')) self.assertEqual(p.relative_to(P('a')), P('b')) + self.assertEqual(p.relative_to('a'), P('b')) + self.assertEqual(p.relative_to('a/'), P('b')) self.assertEqual(p.relative_to(P('a/b')), P()) + self.assertEqual(p.relative_to('a/b'), P()) # With several args self.assertEqual(p.relative_to('a', 'b'), P()) # Unrelated paths @@ -548,13 +553,18 @@ self.assertRaises(ValueError, p.relative_to, P('/a')) p = P('/a/b') self.assertEqual(p.relative_to(P('/')), P('a/b')) + self.assertEqual(p.relative_to('/'), P('a/b')) self.assertEqual(p.relative_to(P('/a')), P('b')) + self.assertEqual(p.relative_to('/a'), P('b')) + self.assertEqual(p.relative_to('/a/'), P('b')) self.assertEqual(p.relative_to(P('/a/b')), P()) + self.assertEqual(p.relative_to('/a/b'), P()) # Unrelated paths self.assertRaises(ValueError, p.relative_to, P('/c')) self.assertRaises(ValueError, p.relative_to, P('/a/b/c')) self.assertRaises(ValueError, p.relative_to, P('/a/c')) self.assertRaises(ValueError, p.relative_to, P()) + self.assertRaises(ValueError, p.relative_to, '') self.assertRaises(ValueError, p.relative_to, P('a')) def test_pickling_common(self): @@ -917,42 +927,61 @@ def test_relative_to(self): P = self.cls - p = P('c:a/b') - self.assertEqual(p.relative_to(P('c:')), P('a/b')) - self.assertEqual(p.relative_to(P('c:a')), P('b')) - self.assertEqual(p.relative_to(P('c:a/b')), P()) + p = P('C:Foo/Bar') + self.assertEqual(p.relative_to(P('c:')), P('Foo/Bar')) + self.assertEqual(p.relative_to('c:'), P('Foo/Bar')) + self.assertEqual(p.relative_to(P('c:foO')), P('Bar')) + self.assertEqual(p.relative_to('c:foO'), P('Bar')) + self.assertEqual(p.relative_to('c:foO/'), P('Bar')) + self.assertEqual(p.relative_to(P('c:foO/baR')), P()) + self.assertEqual(p.relative_to('c:foO/baR'), P()) # Unrelated paths self.assertRaises(ValueError, p.relative_to, P()) + self.assertRaises(ValueError, p.relative_to, '') self.assertRaises(ValueError, p.relative_to, P('d:')) - self.assertRaises(ValueError, p.relative_to, P('a')) - self.assertRaises(ValueError, p.relative_to, P('/a')) - self.assertRaises(ValueError, p.relative_to, P('c:a/b/c')) - self.assertRaises(ValueError, p.relative_to, P('c:a/c')) - self.assertRaises(ValueError, p.relative_to, P('c:/a')) - p = P('c:/a/b') - self.assertEqual(p.relative_to(P('c:')), P('/a/b')) - self.assertEqual(p.relative_to(P('c:/')), P('a/b')) - self.assertEqual(p.relative_to(P('c:/a')), P('b')) - self.assertEqual(p.relative_to(P('c:/a/b')), P()) + self.assertRaises(ValueError, p.relative_to, P('/')) + self.assertRaises(ValueError, p.relative_to, P('Foo')) + self.assertRaises(ValueError, p.relative_to, P('/Foo')) + self.assertRaises(ValueError, p.relative_to, P('C:/Foo')) + self.assertRaises(ValueError, p.relative_to, P('C:Foo/Bar/Baz')) + self.assertRaises(ValueError, p.relative_to, P('C:Foo/Baz')) + p = P('C:/Foo/Bar') + self.assertEqual(p.relative_to(P('c:')), P('/Foo/Bar')) + self.assertEqual(p.relative_to('c:'), P('/Foo/Bar')) + self.assertEqual(str(p.relative_to(P('c:'))), '\\Foo\\Bar') + self.assertEqual(str(p.relative_to('c:')), '\\Foo\\Bar') + self.assertEqual(p.relative_to(P('c:/')), P('Foo/Bar')) + self.assertEqual(p.relative_to('c:/'), P('Foo/Bar')) + self.assertEqual(p.relative_to(P('c:/foO')), P('Bar')) + self.assertEqual(p.relative_to('c:/foO'), P('Bar')) + self.assertEqual(p.relative_to('c:/foO/'), P('Bar')) + self.assertEqual(p.relative_to(P('c:/foO/baR')), P()) + self.assertEqual(p.relative_to('c:/foO/baR'), P()) # Unrelated paths - self.assertRaises(ValueError, p.relative_to, P('c:/c')) - self.assertRaises(ValueError, p.relative_to, P('c:/a/b/c')) - self.assertRaises(ValueError, p.relative_to, P('c:/a/c')) - self.assertRaises(ValueError, p.relative_to, P('c:a')) + self.assertRaises(ValueError, p.relative_to, P('C:/Baz')) + self.assertRaises(ValueError, p.relative_to, P('C:/Foo/Bar/Baz')) + self.assertRaises(ValueError, p.relative_to, P('C:/Foo/Baz')) + self.assertRaises(ValueError, p.relative_to, P('C:Foo')) self.assertRaises(ValueError, p.relative_to, P('d:')) self.assertRaises(ValueError, p.relative_to, P('d:/')) - self.assertRaises(ValueError, p.relative_to, P('/a')) - self.assertRaises(ValueError, p.relative_to, P('//c/a')) + self.assertRaises(ValueError, p.relative_to, P('/')) + self.assertRaises(ValueError, p.relative_to, P('/Foo')) + self.assertRaises(ValueError, p.relative_to, P('//C/Foo')) # UNC paths - p = P('//a/b/c/d') - self.assertEqual(p.relative_to(P('//a/b')), P('c/d')) - self.assertEqual(p.relative_to(P('//a/b/c')), P('d')) - self.assertEqual(p.relative_to(P('//a/b/c/d')), P()) + p = P('//Server/Share/Foo/Bar') + self.assertEqual(p.relative_to(P('//sErver/sHare')), P('Foo/Bar')) + self.assertEqual(p.relative_to('//sErver/sHare'), P('Foo/Bar')) + self.assertEqual(p.relative_to('//sErver/sHare/'), P('Foo/Bar')) + self.assertEqual(p.relative_to(P('//sErver/sHare/Foo')), P('Bar')) + self.assertEqual(p.relative_to('//sErver/sHare/Foo'), P('Bar')) + self.assertEqual(p.relative_to('//sErver/sHare/Foo/'), P('Bar')) + self.assertEqual(p.relative_to(P('//sErver/sHare/Foo/Bar')), P()) + self.assertEqual(p.relative_to('//sErver/sHare/Foo/Bar'), P()) # Unrelated paths - self.assertRaises(ValueError, p.relative_to, P('/a/b/c')) - self.assertRaises(ValueError, p.relative_to, P('c:/a/b/c')) - self.assertRaises(ValueError, p.relative_to, P('//z/b/c')) - self.assertRaises(ValueError, p.relative_to, P('//a/z/c')) + self.assertRaises(ValueError, p.relative_to, P('/Server/Share/Foo')) + self.assertRaises(ValueError, p.relative_to, P('c:/Server/Share/Foo')) + self.assertRaises(ValueError, p.relative_to, P('//z/Share/Foo')) + self.assertRaises(ValueError, p.relative_to, P('//Server/z/Foo')) def test_is_absolute(self): P = self.cls diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,8 @@ Library ------- +- Issue #19918: Fix PurePath.relative_to() under Windows. + - Issue #19422: Explicitly disallow non-SOCK_STREAM sockets in the ssl module, rather than silently let them emit clear text data. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat Dec 28 20:38:06 2013 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 28 Dec 2013 20:38:06 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fix_breakage_in_TestSuite?= =?utf-8?q?=2EcountTestCases=28=29_introduced_by_issue_=2311798=2E?= Message-ID: <3dsFYp2XrSz7Lky@mail.python.org> http://hg.python.org/cpython/rev/b668c409c10a changeset: 88218:b668c409c10a user: Antoine Pitrou date: Sat Dec 28 20:37:58 2013 +0100 summary: Fix breakage in TestSuite.countTestCases() introduced by issue #11798. files: Lib/unittest/suite.py | 18 ++++++++--- Lib/unittest/test/test_suite.py | 31 +++++++++++++++++++++ Misc/NEWS | 2 + 3 files changed, 46 insertions(+), 5 deletions(-) diff --git a/Lib/unittest/suite.py b/Lib/unittest/suite.py --- a/Lib/unittest/suite.py +++ b/Lib/unittest/suite.py @@ -20,6 +20,7 @@ def __init__(self, tests=()): self._tests = [] + self._removed_tests = 0 self.addTests(tests) def __repr__(self): @@ -37,9 +38,10 @@ return iter(self._tests) def countTestCases(self): - cases = 0 + cases = self._removed_tests for test in self: - cases += test.countTestCases() + if test: + cases += test.countTestCases() return cases def addTest(self, test): @@ -70,10 +72,16 @@ def _removeTestAtIndex(self, index): """Stop holding a reference to the TestCase at index.""" try: + test = self._tests[index] + except TypeError: + # support for suite implementations that have overriden self._tests + pass + else: + # Some unittest tests add non TestCase/TestSuite objects to + # the suite. + if hasattr(test, 'countTestCases'): + self._removed_tests += test.countTestCases() self._tests[index] = None - except TypeError: - # support for suite implementations that have overriden self._test - pass def __call__(self, *args, **kwds): return self.run(*args, **kwds) diff --git a/Lib/unittest/test/test_suite.py b/Lib/unittest/test/test_suite.py --- a/Lib/unittest/test/test_suite.py +++ b/Lib/unittest/test/test_suite.py @@ -51,6 +51,9 @@ suite = unittest.TestSuite() self.assertEqual(suite.countTestCases(), 0) + # countTestCases() still works after tests are run + suite.run(unittest.TestResult()) + self.assertEqual(suite.countTestCases(), 0) # "class TestSuite([tests])" # ... @@ -63,6 +66,9 @@ suite = unittest.TestSuite([]) self.assertEqual(suite.countTestCases(), 0) + # countTestCases() still works after tests are run + suite.run(unittest.TestResult()) + self.assertEqual(suite.countTestCases(), 0) # "class TestSuite([tests])" # ... @@ -84,6 +90,14 @@ suite_3 = unittest.TestSuite(set(suite_1)) self.assertEqual(suite_3.countTestCases(), 2) + # countTestCases() still works after tests are run + suite_1.run(unittest.TestResult()) + self.assertEqual(suite_1.countTestCases(), 2) + suite_2.run(unittest.TestResult()) + self.assertEqual(suite_2.countTestCases(), 2) + suite_3.run(unittest.TestResult()) + self.assertEqual(suite_3.countTestCases(), 2) + # "class TestSuite([tests])" # ... # "If tests is given, it must be an iterable of individual test cases @@ -99,6 +113,9 @@ suite = unittest.TestSuite(tests()) self.assertEqual(suite.countTestCases(), 2) + # countTestCases() still works after tests are run + suite.run(unittest.TestResult()) + self.assertEqual(suite.countTestCases(), 2) ################################################################ ### /Tests for TestSuite.__init__ @@ -145,6 +162,9 @@ suite = unittest.TestSuite((test1, test2)) self.assertEqual(suite.countTestCases(), 2) + # countTestCases() still works after tests are run + suite.run(unittest.TestResult()) + self.assertEqual(suite.countTestCases(), 2) # "Return the number of tests represented by the this test object. # ...this method is also implemented by the TestSuite class, which can @@ -162,6 +182,10 @@ parent = unittest.TestSuite((test3, child, Test1('test1'))) self.assertEqual(parent.countTestCases(), 4) + # countTestCases() still works after tests are run + parent.run(unittest.TestResult()) + self.assertEqual(parent.countTestCases(), 4) + self.assertEqual(child.countTestCases(), 2) # "Run the tests associated with this suite, collecting the result into # the test result object passed as result." @@ -220,6 +244,9 @@ self.assertEqual(suite.countTestCases(), 1) self.assertEqual(list(suite), [test]) + # countTestCases() still works after tests are run + suite.run(unittest.TestResult()) + self.assertEqual(suite.countTestCases(), 1) # "Add a ... TestSuite to the suite" def test_addTest__TestSuite(self): @@ -233,6 +260,9 @@ self.assertEqual(suite.countTestCases(), 1) self.assertEqual(list(suite), [suite_2]) + # countTestCases() still works after tests are run + suite.run(unittest.TestResult()) + self.assertEqual(suite.countTestCases(), 1) # "Add all the tests from an iterable of TestCase and TestSuite # instances to this test suite." @@ -392,6 +422,7 @@ self.assertEqual(len(result.errors), 1) self.assertEqual(len(result.failures), 0) self.assertEqual(result.testsRun, 2) + self.assertEqual(suite.countTestCases(), 2) def test_overriding_call(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,8 @@ Library ------- +- Fix breakage in TestSuite.countTestCases() introduced by issue #11798. + - Issue #19918: Fix PurePath.relative_to() under Windows. - Issue #19422: Explicitly disallow non-SOCK_STREAM sockets in the ssl -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 29 02:37:47 2013 From: python-checkins at python.org (senthil.kumaran) Date: Sun, 29 Dec 2013 02:37:47 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Backporing_the?= =?utf-8?q?_fix_from_Issue_=2312692?= Message-ID: <3dsPXq5TJtz7Ljd@mail.python.org> http://hg.python.org/cpython/rev/a43e96695203 changeset: 88219:a43e96695203 branch: 3.3 parent: 88212:a00842b783cf user: Senthil Kumaran date: Sat Dec 28 17:36:18 2013 -0800 summary: Backporing the fix from Issue #12692 files: Lib/test/test_urllib2.py | 1 + Lib/urllib/request.py | 6 ++++++ Misc/NEWS | 3 +++ 3 files changed, 10 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -283,6 +283,7 @@ self.req_headers = [] self.data = None self.raise_on_endheaders = False + self.sock = None self._tunnel_headers = {} def __call__(self, host, timeout=socket._GLOBAL_DEFAULT_TIMEOUT): diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -1251,6 +1251,12 @@ raise URLError(err) else: r = h.getresponse() + # If the server does not send us a 'Connection: close' header, + # HTTPConnection assumes the socket should be left open. Manually + # mark the socket to be closed when this response object goes away. + if h.sock: + h.sock.close() + h.sock = None r.url = req.get_full_url() # This line replaces the .msg attribute of the HTTPResponse diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -29,6 +29,9 @@ Library ------- +- Issue #12692: Backport the fix for ResourceWarning in test_urllib2net. This + also helps in closing the socket when Connection Close header is not sent. + - Issue #19422: Explicitly disallow non-SOCK_STREAM sockets in the ssl module, rather than silently let them emit clear text data. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 29 02:37:49 2013 From: python-checkins at python.org (senthil.kumaran) Date: Sun, 29 Dec 2013 02:37:49 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_=2312692=3A_null_merge_with_3=2E3?= Message-ID: <3dsPXs0jWVz7Ljd@mail.python.org> http://hg.python.org/cpython/rev/031417ee8351 changeset: 88220:031417ee8351 parent: 88218:b668c409c10a parent: 88219:a43e96695203 user: Senthil Kumaran date: Sat Dec 28 17:37:38 2013 -0800 summary: #12692: null merge with 3.3 files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 29 05:16:09 2013 From: python-checkins at python.org (r.david.murray) Date: Sun, 29 Dec 2013 05:16:09 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Add_context_to_a_json_NEWS?= =?utf-8?q?_entry=2E?= Message-ID: <3dsT3Y52cfz7LkW@mail.python.org> http://hg.python.org/cpython/rev/25b7e76c8513 changeset: 88221:25b7e76c8513 user: R David Murray date: Fri Dec 27 17:08:18 2013 -0500 summary: Add context to a json NEWS entry. files: Misc/NEWS | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2710,8 +2710,8 @@ - Issue #10182: The re module doesn't truncate indices to 32 bits anymore. Patch by Serhiy Storchaka. -- Issue #16333: use (",", ": ") as default separator when indent is specified - to avoid trailing whitespace. Patch by Serhiy Storchaka. +- Issue #16333: use (",", ": ") as default separator in json when indent is + specified, to avoid trailing whitespace. Patch by Serhiy Storchaka. - Issue #16573: In 2to3, treat enumerate() like a consuming call, so superfluous list() calls aren't added to filter(), map(), and zip() which are directly -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun Dec 29 05:16:11 2013 From: python-checkins at python.org (r.david.murray) Date: Sun, 29 Dec 2013 05:16:11 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_abc=2EABC=2E__?= =?utf-8?q?Also_add_issue_number_to_news_entry_and_reword=2E?= Message-ID: <3dsT3b0DWpz7LkW@mail.python.org> http://hg.python.org/cpython/rev/08b570183af8 changeset: 88222:08b570183af8 user: R David Murray date: Sat Dec 28 23:15:12 2013 -0500 summary: whatsnew: abc.ABC. Also add issue number to news entry and reword. files: Doc/whatsnew/3.4.rst | 6 ++++++ Misc/NEWS | 5 +++-- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -481,6 +481,12 @@ caches that are affected by changes in the object graph. (Contributed by ?ukasz Langa in :issue:`16832`.) +New class :class:`~abc.ABC` has :class:`~abc.ABCMeta` as its meta class. +Using ``ABC`` as a base class has essentially the same effect as specifying +``metaclass=abc.ABCMeta``, but is simpler to type and easier to read. +(Contributed by Bruno Dupuis in :issue:`16049`.) + + aifc ---- diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2671,8 +2671,9 @@ Content-Length and the incoming stream is finished. Patch by Eran Rundstein. -- Add abc.ABC class to use inheritance rather than a direct invocation of - ABCMeta metaclass. Patch by Bruno Dupuis. +- Issue #16049: Add abc.ABC class to enable the use of inheritance to create + ABCs, rather than the more cumbersome metaclass=ABCMeta. Patch by Bruno + Dupuis. - Expose the TCP_FASTOPEN and MSG_FASTOPEN flags in socket when they're available. -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sun Dec 29 09:45:40 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 29 Dec 2013 09:45:40 +0100 Subject: [Python-checkins] Daily reference leaks (031417ee8351): sum=-4 Message-ID: results for 031417ee8351 on branch "default" -------------------------------------------- test_site leaked [0, -2, 0] references, sum=-2 test_site leaked [0, -2, 0] memory blocks, sum=-2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog8bZBBO', '-x'] From python-checkins at python.org Mon Dec 30 00:39:07 2013 From: python-checkins at python.org (michael.foord) Date: Mon, 30 Dec 2013 00:39:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Closes_issue_20031=2E_Docu?= =?utf-8?q?ment_unittest=2ETextTestRunner=2Erun_method=2E?= Message-ID: <3dsysR6tdMz7Ljm@mail.python.org> http://hg.python.org/cpython/rev/19464d77ec2e changeset: 88223:19464d77ec2e user: Michael Foord date: Sun Dec 29 23:38:55 2013 +0000 summary: Closes issue 20031. Document unittest.TextTestRunner.run method. files: Doc/library/unittest.rst | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -1939,6 +1939,14 @@ stream, descriptions, verbosity + .. method:: run(test) + + This method is the main public interface to the `TextTestRunner`. This + method takes a :class:`TestSuite` or :class:`TestCase` instance. A + :class:`TestResult` is created by calling + :func:`_makeResult` and the test(s) are run and the + results printed to stdout. + .. function:: main(module='__main__', defaultTest=None, argv=None, testRunner=None, \ testLoader=unittest.defaultTestLoader, exit=True, verbosity=1, \ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 30 09:34:02 2013 From: python-checkins at python.org (christian.heimes) Date: Mon, 30 Dec 2013 09:34:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Fixed_typo_=28thx_Arfrever?= =?utf-8?q?=29?= Message-ID: <3dtBkf2Lswz7LjN@mail.python.org> http://hg.python.org/cpython/rev/fd846837492d changeset: 88224:fd846837492d user: Christian Heimes date: Mon Dec 30 09:33:46 2013 +0100 summary: Fixed typo (thx Arfrever) files: Doc/whatsnew/3.4.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -655,7 +655,7 @@ .. _whatsnew-sha3: -New :ref:`hash algorithms ` ``sah3_224()``, ``sha3_256()``, +New :ref:`hash algorithms ` ``sha3_224()``, ``sha3_256()``, ``sha3_384()``, and ``sha3_512()``. (Contributed by Christian Heimes in :issue:`16113`.) -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Mon Dec 30 09:48:04 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 30 Dec 2013 09:48:04 +0100 Subject: [Python-checkins] Daily reference leaks (19464d77ec2e): sum=0 Message-ID: results for 19464d77ec2e on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogiBS2K9', '-x'] From python-checkins at python.org Mon Dec 30 21:40:07 2013 From: python-checkins at python.org (zach.ware) Date: Mon, 30 Dec 2013 21:40:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319544=2C_=236516?= =?utf-8?q?=3A_check_ZLIB=5FSUPPORT=2C_not_zlib_=28which_might_not_be_boun?= =?utf-8?q?d=29?= Message-ID: <3dtVrR2l0kz7Lk3@mail.python.org> http://hg.python.org/cpython/rev/83f12a9593db changeset: 88225:83f12a9593db user: Zachary Ware date: Mon Dec 30 14:39:46 2013 -0600 summary: Issue #19544, #6516: check ZLIB_SUPPORT, not zlib (which might not be bound) files: Lib/distutils/tests/test_archive_util.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/distutils/tests/test_archive_util.py b/Lib/distutils/tests/test_archive_util.py --- a/Lib/distutils/tests/test_archive_util.py +++ b/Lib/distutils/tests/test_archive_util.py @@ -308,7 +308,7 @@ owner='kjhkjhkjg', group='oihohoh') self.assertTrue(os.path.exists(res)) - @unittest.skipUnless(zlib, "Requires zlib") + @unittest.skipUnless(ZLIB_SUPPORT, "Requires zlib") @unittest.skipUnless(UID_GID_SUPPORT, "Requires grp and pwd support") def test_tarfile_root_owner(self): tmpdir, tmpdir2, base_name = self._create_files() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 30 21:54:23 2013 From: python-checkins at python.org (zach.ware) Date: Mon, 30 Dec 2013 21:54:23 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue19619=3A_skip_zlib_er?= =?utf-8?q?ror_test_when_zlib_not_available?= Message-ID: <3dtW8v4jyhz7Lpf@mail.python.org> http://hg.python.org/cpython/rev/0e10367c88ce changeset: 88226:0e10367c88ce user: Zachary Ware date: Mon Dec 30 14:54:11 2013 -0600 summary: Issue19619: skip zlib error test when zlib not available files: Lib/test/test_codecs.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -2335,7 +2335,7 @@ try: import zlib except ImportError: - pass + zlib = None else: bytes_transform_encodings.append("zlib_codec") transform_aliases["zlib_codec"] = ["zip", "zlib"] @@ -2440,6 +2440,7 @@ bad_input.decode("rot_13") self.assertIsNone(failure.exception.__cause__) + @unittest.skipUnless(zlib, "Requires zlib support") def test_custom_zlib_error_is_wrapped(self): # Check zlib codec gives a good error for malformed input msg = "^decoding with 'zlib_codec' codec failed" -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon Dec 30 22:09:28 2013 From: python-checkins at python.org (zach.ware) Date: Mon, 30 Dec 2013 22:09:28 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2319544=2C_=236516?= =?utf-8?q?=3A_check_ZLIB=5FSUPPORT=2C_not_zlib_=28which_might_not_be_boun?= =?utf-8?q?d=29?= Message-ID: <3dtWVJ2th6z7LnS@mail.python.org> http://hg.python.org/cpython/rev/2a872126f4a1 changeset: 88227:2a872126f4a1 user: Zachary Ware date: Mon Dec 30 15:09:20 2013 -0600 summary: Issue #19544, #6516: check ZLIB_SUPPORT, not zlib (which might not be bound) files: Lib/distutils/tests/test_sdist.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py --- a/Lib/distutils/tests/test_sdist.py +++ b/Lib/distutils/tests/test_sdist.py @@ -429,7 +429,7 @@ self.assertEqual(sorted(filenames), ['fake-1.0', 'fake-1.0/PKG-INFO', 'fake-1.0/README.manual']) - @unittest.skipUnless(zlib, "requires zlib") + @unittest.skipUnless(ZLIB_SUPPORT, "requires zlib") @unittest.skipUnless(UID_GID_SUPPORT, "Requires grp and pwd support") @unittest.skipIf(find_executable('tar') is None, "The tar command is not found") -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Tue Dec 31 09:47:05 2013 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 31 Dec 2013 09:47:05 +0100 Subject: [Python-checkins] Daily reference leaks (2a872126f4a1): sum=0 Message-ID: results for 2a872126f4a1 on branch "default" -------------------------------------------- test_site leaked [0, 2, -2] references, sum=0 test_site leaked [0, 2, -2] memory blocks, sum=0 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogPisg0t', '-x'] From python-checkins at python.org Tue Dec 31 13:02:19 2013 From: python-checkins at python.org (donald.stufft) Date: Tue, 31 Dec 2013 13:02:19 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Update_pip_to_1=2E5rc4?= Message-ID: <3dtvJW3vyyz7LqB@mail.python.org> http://hg.python.org/cpython/rev/b2d1e0fe73ca changeset: 88228:b2d1e0fe73ca user: Donald Stufft date: Tue Dec 31 06:49:54 2013 -0500 summary: Update pip to 1.5rc4 files: Lib/ensurepip/__init__.py | 2 +- Lib/ensurepip/_bundled/pip-1.5rc3-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/pip-1.5rc4-py2.py3-none-any.whl | Bin 3 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -12,7 +12,7 @@ _SETUPTOOLS_VERSION = "2.0.1" -_PIP_VERSION = "1.5rc3" +_PIP_VERSION = "1.5rc4" # pip currently requires ssl support, so we try to provide a nicer # error message when that is missing (http://bugs.python.org/issue19744) diff --git a/Lib/ensurepip/_bundled/pip-1.5rc3-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-1.5rc3-py2.py3-none-any.whl deleted file mode 100644 index 58443638a6a26bc0e34b3cfe44d6f4a716776f62..0000000000000000000000000000000000000000 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/pip-1.5rc4-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-1.5rc4-py2.py3-none-any.whl new file mode 100644 index 0000000000000000000000000000000000000000..2a16332814c690b3fc6f040fc79709dfed2e5095 GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 31 13:02:20 2013 From: python-checkins at python.org (donald.stufft) Date: Tue, 31 Dec 2013 13:02:20 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_Update_setuptools_to_2=2E0?= =?utf-8?q?=2E2?= Message-ID: <3dtvJX6BFvz7Lk0@mail.python.org> http://hg.python.org/cpython/rev/e5fa13aae1cc changeset: 88229:e5fa13aae1cc user: Donald Stufft date: Tue Dec 31 06:52:47 2013 -0500 summary: Update setuptools to 2.0.2 files: Lib/ensurepip/__init__.py | 2 +- Lib/ensurepip/_bundled/setuptools-2.0.1-py2.py3-none-any.whl | Bin Lib/ensurepip/_bundled/setuptools-2.0.2-py2.py3-none-any.whl | Bin 3 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -10,7 +10,7 @@ __all__ = ["version", "bootstrap"] -_SETUPTOOLS_VERSION = "2.0.1" +_SETUPTOOLS_VERSION = "2.0.2" _PIP_VERSION = "1.5rc4" diff --git a/Lib/ensurepip/_bundled/setuptools-2.0.1-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-2.0.1-py2.py3-none-any.whl deleted file mode 100644 index 67ef8a1a9757835bb1de367c0f8c3b5b579ffa15..0000000000000000000000000000000000000000 GIT binary patch [stripped] diff --git a/Lib/ensurepip/_bundled/setuptools-2.0.2-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-2.0.2-py2.py3-none-any.whl new file mode 100644 index 0000000000000000000000000000000000000000..267eeff2387c02dbf6b1450024b1c5101c307056 GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 31 17:48:03 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 31 Dec 2013 17:48:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_epoll_supports?= =?utf-8?q?_with=2E?= Message-ID: <3dv1fC186Cz7Ljk@mail.python.org> http://hg.python.org/cpython/rev/44b6b492c7e3 changeset: 88230:44b6b492c7e3 user: R David Murray date: Tue Dec 31 11:17:21 2013 -0500 summary: whatsnew: epoll supports with. Also reworded the description of the feature in the docs. files: Doc/library/select.rst | 11 ++++++++--- Doc/whatsnew/3.4.rst | 9 +++++++++ 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/Doc/library/select.rst b/Doc/library/select.rst --- a/Doc/library/select.rst +++ b/Doc/library/select.rst @@ -58,9 +58,14 @@ which can be used as Edge or Level Triggered interface for I/O events. *sizehint* is deprecated and completely ignored. *flags* can be set to :const:`EPOLL_CLOEXEC`, which causes the epoll descriptor to be closed - automatically when :func:`os.execve` is called. See section - :ref:`epoll-objects` below for the methods supported by epolling objects. - They also support the :keyword:`with` statement. + automatically when :func:`os.execve` is called. + + See the :ref:`epoll-objects` section below for the methods supported by + epolling objects. + + ``epoll`` objects support the context management protocol: when used in a + :keyword:`with` statement, the new file descriptor is automatically closed + at the end of the block. The new file descriptor is :ref:`non-inheritable `. diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -851,6 +851,15 @@ (Contributed by Christian Heimes in :issue:`16595` and :issue:`19324`.) +select +------ + +:class:`~select.epoll` objects now support the context management protocol. +When used in a :keyword:`with` statement, the :meth:`~select.epoll.close` +method will be called automatically at the end of the block. (Contributed +by Serhiy Storchaka in :issue:`16488`.) + + shelve ------ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 31 17:48:04 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 31 Dec 2013 17:48:04 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_argparse_FileT?= =?utf-8?q?ype_accepts_errors_and_encodings_args=2E?= Message-ID: <3dv1fD3hvzz7LkG@mail.python.org> http://hg.python.org/cpython/rev/751c50136ce3 changeset: 88231:751c50136ce3 user: R David Murray date: Tue Dec 31 11:18:01 2013 -0500 summary: whatsnew: argparse FileType accepts errors and encodings args. files: Doc/library/argparse.rst | 3 +++ Doc/whatsnew/3.4.rst | 8 ++++++++ 2 files changed, 11 insertions(+), 0 deletions(-) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1671,6 +1671,9 @@ >>> parser.parse_args(['-']) Namespace(infile=<_io.TextIOWrapper name='' encoding='UTF-8'>) + .. versionadded:: 3.4 + The *encodings* and *errors* keyword arguments. + Argument groups ^^^^^^^^^^^^^^^ diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -495,6 +495,14 @@ plain tuple. (Contributed by Claudiu Popa in :issue:`17818`.) +argparse +-------- + +The :class:`~argparse.FileType` class now accepts *encoding* and +*errors* arguments, which are passed through to :func:`open`. (Contributed +by Lucas Maystre in :issue:`11175`.) + + audioop ------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 31 19:10:36 2013 From: python-checkins at python.org (zach.ware) Date: Tue, 31 Dec 2013 19:10:36 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_str_subclasses?= =?utf-8?q?_may_have_non-empty_=5F=5Fslots=5F=5F=2C_bytes_subclasses_can?= =?utf-8?b?J3Qu?= Message-ID: <3dv3TS22CRz7LkX@mail.python.org> http://hg.python.org/cpython/rev/8b4d36d0c090 changeset: 88232:8b4d36d0c090 branch: 3.3 parent: 88219:a43e96695203 user: Zachary Ware date: Tue Dec 31 12:09:26 2013 -0600 summary: str subclasses may have non-empty __slots__, bytes subclasses can't. files: Doc/reference/datamodel.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1571,7 +1571,7 @@ program undefined. In the future, a check may be added to prevent this. * Nonempty *__slots__* does not work for classes derived from "variable-length" - built-in types such as :class:`int`, :class:`str` and :class:`tuple`. + built-in types such as :class:`int`, :class:`bytes` and :class:`tuple`. * Any non-string iterable may be assigned to *__slots__*. Mappings may also be used; however, in the future, special meaning may be assigned to the values -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 31 19:10:37 2013 From: python-checkins at python.org (zach.ware) Date: Tue, 31 Dec 2013 19:10:37 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge_with_3=2E3?= Message-ID: <3dv3TT4VlMz7Lq0@mail.python.org> http://hg.python.org/cpython/rev/b35b204e5d0b changeset: 88233:b35b204e5d0b parent: 88231:751c50136ce3 parent: 88232:8b4d36d0c090 user: Zachary Ware date: Tue Dec 31 12:10:24 2013 -0600 summary: Merge with 3.3 files: Doc/reference/datamodel.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1583,7 +1583,7 @@ program undefined. In the future, a check may be added to prevent this. * Nonempty *__slots__* does not work for classes derived from "variable-length" - built-in types such as :class:`int`, :class:`str` and :class:`tuple`. + built-in types such as :class:`int`, :class:`bytes` and :class:`tuple`. * Any non-string iterable may be assigned to *__slots__*. Mappings may also be used; however, in the future, special meaning may be assigned to the values -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 31 23:03:07 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 31 Dec 2013 23:03:07 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_afic=2Eopen_su?= =?utf-8?q?pports_=27with=27=2E?= Message-ID: <3dv8dl3XGtz7Ljk@mail.python.org> http://hg.python.org/cpython/rev/f6ce2284e8dd changeset: 88234:f6ce2284e8dd user: R David Murray date: Tue Dec 31 13:45:38 2013 -0500 summary: whatsnew: afic.open supports 'with'. files: Doc/library/aifc.rst | 4 ++-- Doc/whatsnew/3.4.rst | 5 +++++ 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/Doc/library/aifc.rst b/Doc/library/aifc.rst --- a/Doc/library/aifc.rst +++ b/Doc/library/aifc.rst @@ -54,8 +54,8 @@ The :func:`.open` function may be used in a :keyword:`with` statement. When the :keyword:`with` block completes, the :meth:`~aifc.close` method is called. -.. versionchanged:: 3.4 - Support for the :keyword:`with` statement was added. + .. versionchanged:: 3.4 + Support for the :keyword:`with` statement was added. Objects returned by :func:`.open` when a file is opened for reading have the following methods: diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -494,6 +494,11 @@ The :meth:`~aifc.getparams` method now returns a namedtuple rather than a plain tuple. (Contributed by Claudiu Popa in :issue:`17818`.) +:func:`aifc.open` now supports the context manager protocol: when used in a +:keyword:`with` block, the :meth:`~aifc.aifc.close` method of the returned +object will be called automatically at the end of the block. (Contributed by +Serhiy Storchacha in :issue:`16486`.) + argparse -------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 31 23:03:08 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 31 Dec 2013 23:03:08 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_random=2Egetra?= =?utf-8?q?ndbits_performance=2E?= Message-ID: <3dv8dm5Gbnz7Ljk@mail.python.org> http://hg.python.org/cpython/rev/171224d8ec68 changeset: 88235:171224d8ec68 user: R David Murray date: Tue Dec 31 15:06:05 2013 -0500 summary: whatsnew: random.getrandbits performance. Also fix a NEWS file issue number error, and some spacing fixes in whatsnew. files: Doc/whatsnew/3.4.rst | 5 +++++ Misc/NEWS | 2 +- 2 files changed, 6 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -453,6 +453,7 @@ .. _whatsnew-tracemalloc: + tracemalloc ----------- @@ -902,6 +903,7 @@ using the new :mod:`enum` module. This allows descriptive reporting during debugging, instead of seeing integer "magic numbers". + ssl --- @@ -1198,6 +1200,9 @@ most cases. :class:`lzma.LZMAFile` has also been optimized. (Contributed by Serhiy Storchaka and Nadeem Vawda in :issue:`16034`.) +* :func:`random.getrandbits` is 20%-40% faster for small integers (the most + common use case). (Contributed by Serhiy Storchaka in :issue:`16674`). + Deprecated ========== diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2518,7 +2518,7 @@ byte-compiled file is a symlink or non-regular file as a warning that import will not keep the file path type if it writes to that path. -- Issue #180022: Have site.addpackage() consider already known paths even when +- Issue #16972: Have site.addpackage() consider already known paths even when none are explicitly passed in. Bug report and fix by Kirill. - Issue #1602133: on Mac OS X a shared library build (``--enable-shared``) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 31 23:03:10 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 31 Dec 2013 23:03:10 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython=3A_whatsnew=3A_mock_called=5F?= =?utf-8?q?with_improvements=2C_socket_CAN=5FBCM_support=2E?= Message-ID: <3dv8dp06KMz7Ljk@mail.python.org> http://hg.python.org/cpython/rev/879b83024509 changeset: 88236:879b83024509 user: R David Murray date: Tue Dec 31 16:04:50 2013 -0500 summary: whatsnew: mock called_with improvements, socket CAN_BCM support. Also reworded the other entries in the socket section of whatsnew, as well as a couple of unrelated news entries. files: Doc/whatsnew/3.4.rst | 23 +++++++++++++++++------ Misc/NEWS | 8 ++++---- 2 files changed, 21 insertions(+), 10 deletions(-) diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -744,6 +744,15 @@ (Contributed by Valerie Lambert in :issue:`4885`.) +mock +---- + +:mod:`~unittest.mock` objects now inspect their specification signatures when +matching calls, which means an argument can now be matched by either position +or name, instead of only by position. (Contributed by Antoine Pitrou in +:issue:`17015`.) + + multiprocessing --------------- @@ -894,14 +903,16 @@ socket ------ +The socket module now supports the :data:`~socket.CAN_BCM` protocol on +platforms that support it. (Contributed by Brian Thorne in :issue:`15359`.) + Socket objects have new methods to get or set their :ref:`inheritable flag -`: +`, :meth:`~socket.socket.get_inheritable` and +:meth:`~socket.socket.set_inheritable`. -* :meth:`socket.socket.get_inheritable`, :meth:`socket.socket.set_inheritable` - -The ``socket.AF_*`` and ``socket.SOCK_*`` constants are enumeration values, -using the new :mod:`enum` module. This allows descriptive reporting during -debugging, instead of seeing integer "magic numbers". +The ``socket.AF_*`` and ``socket.SOCK_*`` constants are now enumeration values +using the new :mod:`enum` module. This allows meaningful names to be printed +during debugging, instead of integer "magic numbers". ssl diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2474,12 +2474,12 @@ - Issue #17132: Update symbol for "yield from" grammar changes. -- Issue #17076: Make copying of xattrs more permissive of missing FS support. +- Issue #17076: Make copying of xattrs more tolerant of missing FS support. Patch by Thomas Wouters. -- Issue #17089: Expat parser now correctly works with string input not only when - an internal XML encoding is UTF-8 or US-ASCII. It now accepts bytes and - strings larger than 2 GiB. +- Issue #17089: Expat parser now correctly works with string input when the + internal XML encoding is not UTF-8 or US-ASCII. It also now accepts bytes + and strings larger than 2 GiB. - Issue #6083: Fix multiple segmentation faults occured when PyArg_ParseTuple parses nested mutating sequence. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 31 23:35:02 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 31 Dec 2013 23:35:02 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_Clarify_the_wo?= =?utf-8?q?rding_of_a_news_entry=2E?= Message-ID: <3dv9LZ3TQ1z7Ljk@mail.python.org> http://hg.python.org/cpython/rev/9c88280245e0 changeset: 88237:9c88280245e0 branch: 3.3 parent: 88232:8b4d36d0c090 user: R David Murray date: Tue Dec 31 17:33:47 2013 -0500 summary: Clarify the wording of a news entry. files: Misc/NEWS | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1497,9 +1497,10 @@ - Issue #16564: Fixed regression relative to Python2 in the operation of email.encoders.encode_noop when used with binary data. -- Issue #10355: In SpooledTemporaryFile class mode, name, encoding and - newlines properties now work for unrolled files. Obsoleted and never - working on Python 3 xreadline method now removed. +- Issue #10355: The mode, name, encoding and newlines properties now work on + SpooledTemporaryFile objects even when they have not yet rolled over. + Obsolete method xreadline (which has never worked in Python 3) has been + removed. - Issue #16686: Fixed a lot of bugs in audioop module. Fixed crashes in avgpp(), maxpp() and ratecv(). Fixed an integer overflow in add(), bias(), -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue Dec 31 23:35:03 2013 From: python-checkins at python.org (r.david.murray) Date: Tue, 31 Dec 2013 23:35:03 +0100 (CET) Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E3_-=3E_default?= =?utf-8?q?=29=3A_Merge=3A_Clarify_the_wording_of_a_news_entry=2E?= Message-ID: <3dv9Lb5RSqz7Ljk@mail.python.org> http://hg.python.org/cpython/rev/b5f85b093252 changeset: 88238:b5f85b093252 parent: 88236:879b83024509 parent: 88237:9c88280245e0 user: R David Murray date: Tue Dec 31 17:34:40 2013 -0500 summary: Merge: Clarify the wording of a news entry. files: Misc/NEWS | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2448,9 +2448,10 @@ - Issue #16564: Fixed regression relative to Python2 in the operation of email.encoders.encode_noop when used with binary data. -- Issue #10355: In SpooledTemporaryFile class mode, name, encoding and - newlines properties now work for unrolled files. Obsoleted and never - working on Python 3 xreadline method now removed. +- Issue #10355: The mode, name, encoding and newlines properties now work on + SpooledTemporaryFile objects even when they have not yet rolled over. + Obsolete method xreadline (which has never worked in Python 3) has been + removed. - Issue #16686: Fixed a lot of bugs in audioop module. Fixed crashes in avgpp(), maxpp() and ratecv(). Fixed an integer overflow in add(), bias(), -- Repository URL: http://hg.python.org/cpython