Mailman 3 May 2012 - Python-checkins

cpython (merge 3.2 -> default): Issue #14775: Fix a potential quadratic dict build-up due to the garbage
by antoine.pitrou May 28, 2012

May 28, 2012

http://hg.python.org/cpython/rev/7951900afd00 changeset: 77211:7951900afd00 parent: 77209:7c7bbf4a70b7 parent: 77210:47e6217d0e84 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Mon May 28 22:23:42 2012 +0200 summary: Issue #14775: Fix a potential quadratic dict build-up due to the garbage collector repeatedly trying to untrack dicts. Additional comments by Tim Silk. files: Misc/ACKS | 1 + Misc/NEWS | 3 + Modules/gcmodule.c | 60 ++++++++++++++++++++++++++++++++- 3 files changed, 61 insertions(+), 3 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -957,6 +957,7 @@ Itamar Shtull-Trauring Eric Siegerman Paul Sijben +Tim Silk Kirill Simonov Nathan Paul Simons Adam Simpkins diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #14775: Fix a potential quadratic dict build-up due to the garbage + collector repeatedly trying to untrack dicts. + - Issue #14857: fix regression in references to PEP 3135 implicit __class__ closure variable (Reopens issue #12370) diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c --- a/Modules/gcmodule.c +++ b/Modules/gcmodule.c @@ -116,6 +116,46 @@ http://mail.python.org/pipermail/python-dev/2008-June/080579.html */ +/* + NOTE: about untracking of mutable objects. + + Certain types of container cannot participate in a reference cycle, and + so do not need to be tracked by the garbage collector. Untracking these + objects reduces the cost of garbage collections. However, determining + which objects may be untracked is not free, and the costs must be + weighed against the benefits for garbage collection. + + There are two possible strategies for when to untrack a container: + + i) When the container is created. + ii) When the container is examined by the garbage collector. + + Tuples containing only immutable objects (integers, strings etc, and + recursively, tuples of immutable objects) do not need to be tracked. + The interpreter creates a large number of tuples, many of which will + not survive until garbage collection. It is therefore not worthwhile + to untrack eligible tuples at creation time. + + Instead, all tuples except the empty tuple are tracked when created. + During garbage collection it is determined whether any surviving tuples + can be untracked. A tuple can be untracked if all of its contents are + already not tracked. Tuples are examined for untracking in all garbage + collection cycles. It may take more than one cycle to untrack a tuple. + + Dictionaries containing only immutable objects also do not need to be + tracked. Dictionaries are untracked when created. If a tracked item is + inserted into a dictionary (either as a key or value), the dictionary + becomes tracked. During a full garbage collection (all generations), + the collector will untrack any dictionaries whose contents are not + tracked. + + The module provides the python function is_tracked(obj), which returns + the CURRENT tracking status of the object. Subsequent garbage + collections may change the tracking status of the object. + + Untracking of certain containers was introduced in issue #4688, and + the algorithm was refined in response to issue #14775. +*/ /* set for debugging information */ #define DEBUG_STATS (1<<0) /* print collection statistics */ @@ -437,9 +477,6 @@ if (PyTuple_CheckExact(op)) { _PyTuple_MaybeUntrack(op); } - else if (PyDict_CheckExact(op)) { - _PyDict_MaybeUntrack(op); - } } else { /* This *may* be unreachable. To make progress, @@ -457,6 +494,20 @@ } } +/* Try to untrack all currently tracked dictionaries */ +static void +untrack_dicts(PyGC_Head *head) +{ + PyGC_Head *next, *gc = head->gc.gc_next; + while (gc != head) { + PyObject *op = FROM_GC(gc); + next = gc->gc.gc_next; + if (PyDict_CheckExact(op)) + _PyDict_MaybeUntrack(op); + gc = next; + } +} + /* Return true if object has a finalization method. */ static int has_finalizer(PyObject *op) @@ -856,6 +907,9 @@ gc_list_merge(young, old); } else { + /* We only untrack dicts in full collections, to avoid quadratic + dict build-up. See issue #14775. */ + untrack_dicts(young); long_lived_pending = 0; long_lived_total = gc_list_size(young); } -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #14775: Fix a potential quadratic dict build-up due to the garbage
by antoine.pitrou May 28, 2012

May 28, 2012

http://hg.python.org/cpython/rev/47e6217d0e84 changeset: 77210:47e6217d0e84 branch: 3.2 parent: 77208:a2e140b083e0 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Mon May 28 22:22:34 2012 +0200 summary: Issue #14775: Fix a potential quadratic dict build-up due to the garbage collector repeatedly trying to untrack dicts. Additional comments by Tim Silk. files: Misc/ACKS | 1 + Misc/NEWS | 3 + Modules/gcmodule.c | 60 ++++++++++++++++++++++++++++++++- 3 files changed, 61 insertions(+), 3 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -866,6 +866,7 @@ Itamar Shtull-Trauring Eric Siegerman Paul Sijben +Tim Silk Kirill Simonov Nathan Paul Simons Janne Sinkkonen diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #14775: Fix a potential quadratic dict build-up due to the garbage + collector repeatedly trying to untrack dicts. + - Issue #14494: Fix __future__.py and its documentation to note that absolute imports are the default behavior in 3.0 instead of 2.7. Patch by Sven Marnach. diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c --- a/Modules/gcmodule.c +++ b/Modules/gcmodule.c @@ -116,6 +116,46 @@ http://mail.python.org/pipermail/python-dev/2008-June/080579.html */ +/* + NOTE: about untracking of mutable objects. + + Certain types of container cannot participate in a reference cycle, and + so do not need to be tracked by the garbage collector. Untracking these + objects reduces the cost of garbage collections. However, determining + which objects may be untracked is not free, and the costs must be + weighed against the benefits for garbage collection. + + There are two possible strategies for when to untrack a container: + + i) When the container is created. + ii) When the container is examined by the garbage collector. + + Tuples containing only immutable objects (integers, strings etc, and + recursively, tuples of immutable objects) do not need to be tracked. + The interpreter creates a large number of tuples, many of which will + not survive until garbage collection. It is therefore not worthwhile + to untrack eligible tuples at creation time. + + Instead, all tuples except the empty tuple are tracked when created. + During garbage collection it is determined whether any surviving tuples + can be untracked. A tuple can be untracked if all of its contents are + already not tracked. Tuples are examined for untracking in all garbage + collection cycles. It may take more than one cycle to untrack a tuple. + + Dictionaries containing only immutable objects also do not need to be + tracked. Dictionaries are untracked when created. If a tracked item is + inserted into a dictionary (either as a key or value), the dictionary + becomes tracked. During a full garbage collection (all generations), + the collector will untrack any dictionaries whose contents are not + tracked. + + The module provides the python function is_tracked(obj), which returns + the CURRENT tracking status of the object. Subsequent garbage + collections may change the tracking status of the object. + + Untracking of certain containers was introduced in issue #4688, and + the algorithm was refined in response to issue #14775. +*/ /* set for debugging information */ #define DEBUG_STATS (1<<0) /* print collection statistics */ @@ -437,9 +477,6 @@ if (PyTuple_CheckExact(op)) { _PyTuple_MaybeUntrack(op); } - else if (PyDict_CheckExact(op)) { - _PyDict_MaybeUntrack(op); - } } else { /* This *may* be unreachable. To make progress, @@ -457,6 +494,20 @@ } } +/* Try to untrack all currently tracked dictionaries */ +static void +untrack_dicts(PyGC_Head *head) +{ + PyGC_Head *next, *gc = head->gc.gc_next; + while (gc != head) { + PyObject *op = FROM_GC(gc); + next = gc->gc.gc_next; + if (PyDict_CheckExact(op)) + _PyDict_MaybeUntrack(op); + gc = next; + } +} + /* Return true if object has a finalization method. */ static int has_finalizer(PyObject *op) @@ -857,6 +908,9 @@ gc_list_merge(young, old); } else { + /* We only untrack dicts in full collections, to avoid quadratic + dict build-up. See issue #14775. */ + untrack_dicts(young); long_lived_pending = 0; long_lived_total = gc_list_size(young); } -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Issue #9041: raised exception is misleading
by meador.inge May 28, 2012

May 28, 2012

http://hg.python.org/cpython/rev/7c7bbf4a70b7 changeset: 77209:7c7bbf4a70b7 parent: 77206:f4a7e738ad24 parent: 77208:a2e140b083e0 user: Meador Inge <meadori(a)gmail.com> date: Mon May 28 14:47:53 2012 -0500 summary: Issue #9041: raised exception is misleading An issue in ctypes.c_longdouble, ctypes.c_double, and ctypes.c_float that caused an incorrect exception to be returned in the case of overflow has been fixed. files: Lib/ctypes/test/test_numbers.py | 10 +++++++ Misc/NEWS | 4 ++ Modules/_ctypes/cfield.c | 30 +++----------------- 3 files changed, 19 insertions(+), 25 deletions(-) diff --git a/Lib/ctypes/test/test_numbers.py b/Lib/ctypes/test/test_numbers.py --- a/Lib/ctypes/test/test_numbers.py +++ b/Lib/ctypes/test/test_numbers.py @@ -217,6 +217,16 @@ # probably be changed: self.assertRaises(TypeError, c_int, c_long(42)) + def test_float_overflow(self): + import sys + big_int = int(sys.float_info.max) * 2 + for t in float_types + [c_longdouble]: + self.assertRaises(OverflowError, t, big_int) + if (hasattr(t, "__ctype_be__")): + self.assertRaises(OverflowError, t.__ctype_be__, big_int) + if (hasattr(t, "__ctype_le__")): + self.assertRaises(OverflowError, t.__ctype_le__, big_int) + ## def test_perf(self): ## check_perf() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -714,6 +714,10 @@ Extension Modules ----------------- +- Issue #9041: An issue in ctypes.c_longdouble, ctypes.c_double, and + ctypes.c_float that caused an incorrect exception to be returned in the + case of overflow has been fixed. + - Issue #14212: The re module didn't retain a reference to buffers it was scanning, resulting in segfaults. diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -999,12 +999,8 @@ long double x; x = PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } memcpy(ptr, &x, sizeof(long double)); _RET(value); } @@ -1023,12 +1019,8 @@ double x; x = PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } memcpy(ptr, &x, sizeof(double)); _RET(value); } @@ -1047,12 +1039,8 @@ double x; x = PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } #ifdef WORDS_BIGENDIAN if (_PyFloat_Pack8(x, (unsigned char *)ptr, 1)) return NULL; @@ -1079,12 +1067,8 @@ float x; x = (float)PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } memcpy(ptr, &x, sizeof(x)); _RET(value); } @@ -1103,12 +1087,8 @@ float x; x = (float)PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } #ifdef WORDS_BIGENDIAN if (_PyFloat_Pack4(x, (unsigned char *)ptr, 1)) return NULL; -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #9041: raised exception is misleading
by meador.inge May 28, 2012

May 28, 2012

http://hg.python.org/cpython/rev/a2e140b083e0 changeset: 77208:a2e140b083e0 branch: 3.2 parent: 77202:3d61e27cc570 user: Meador Inge <meadori(a)gmail.com> date: Mon May 28 14:21:16 2012 -0500 summary: Issue #9041: raised exception is misleading An issue in ctypes.c_longdouble, ctypes.c_double, and ctypes.c_float that caused an incorrect exception to be returned in the case of overflow has been fixed. files: Lib/ctypes/test/test_numbers.py | 10 +++++++ Misc/NEWS | 4 ++ Modules/_ctypes/cfield.c | 30 +++----------------- 3 files changed, 19 insertions(+), 25 deletions(-) diff --git a/Lib/ctypes/test/test_numbers.py b/Lib/ctypes/test/test_numbers.py --- a/Lib/ctypes/test/test_numbers.py +++ b/Lib/ctypes/test/test_numbers.py @@ -217,6 +217,16 @@ # probably be changed: self.assertRaises(TypeError, c_int, c_long(42)) + def test_float_overflow(self): + import sys + big_int = int(sys.float_info.max) * 2 + for t in float_types + [c_longdouble]: + self.assertRaises(OverflowError, t, big_int) + if (hasattr(t, "__ctype_be__")): + self.assertRaises(OverflowError, t.__ctype_be__, big_int) + if (hasattr(t, "__ctype_le__")): + self.assertRaises(OverflowError, t.__ctype_le__, big_int) + ## def test_perf(self): ## check_perf() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -262,6 +262,10 @@ Extension Modules ----------------- +- Issue #9041: An issue in ctypes.c_longdouble, ctypes.c_double, and + ctypes.c_float that caused an incorrect exception to be returned in the + case of overflow has been fixed. + - Issue #14212: The re module didn't retain a reference to buffers it was scanning, resulting in segfaults. diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -1002,12 +1002,8 @@ long double x; x = PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } memcpy(ptr, &x, sizeof(long double)); _RET(value); } @@ -1026,12 +1022,8 @@ double x; x = PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } memcpy(ptr, &x, sizeof(double)); _RET(value); } @@ -1050,12 +1042,8 @@ double x; x = PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } #ifdef WORDS_BIGENDIAN if (_PyFloat_Pack8(x, (unsigned char *)ptr, 1)) return NULL; @@ -1082,12 +1070,8 @@ float x; x = (float)PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } memcpy(ptr, &x, sizeof(x)); _RET(value); } @@ -1106,12 +1090,8 @@ float x; x = (float)PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } #ifdef WORDS_BIGENDIAN if (_PyFloat_Pack4(x, (unsigned char *)ptr, 1)) return NULL; -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #9041: raised exception is misleading
by meador.inge May 28, 2012

May 28, 2012

http://hg.python.org/cpython/rev/8a65eceea56c changeset: 77207:8a65eceea56c branch: 2.7 parent: 77194:938b12452af7 user: Meador Inge <meadori(a)gmail.com> date: Mon May 28 13:52:59 2012 -0500 summary: Issue #9041: raised exception is misleading An issue in ctypes.c_longdouble, ctypes.c_double, and ctypes.c_float that caused an incorrect exception to be returned in the case of overflow has been fixed. files: Lib/ctypes/test/test_numbers.py | 10 +++++++ Misc/NEWS | 4 ++ Modules/_ctypes/cfield.c | 30 +++----------------- 3 files changed, 19 insertions(+), 25 deletions(-) diff --git a/Lib/ctypes/test/test_numbers.py b/Lib/ctypes/test/test_numbers.py --- a/Lib/ctypes/test/test_numbers.py +++ b/Lib/ctypes/test/test_numbers.py @@ -216,6 +216,16 @@ # probably be changed: self.assertRaises(TypeError, c_int, c_long(42)) + def test_float_overflow(self): + import sys + big_int = int(sys.float_info.max) * 2 + for t in float_types + [c_longdouble]: + self.assertRaises(OverflowError, t, big_int) + if (hasattr(t, "__ctype_be__")): + self.assertRaises(OverflowError, t.__ctype_be__, big_int) + if (hasattr(t, "__ctype_le__")): + self.assertRaises(OverflowError, t.__ctype_le__, big_int) + ## def test_perf(self): ## check_perf() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -708,6 +708,10 @@ Extension Modules ----------------- +- Issue #9041: An issue in ctypes.c_longdouble, ctypes.c_double, and + ctypes.c_float that caused an incorrect exception to be returned in the + case of overflow has been fixed. + - bsddb module: Erratic behaviour of "DBEnv->rep_elect()" because a typo. Possible crash. diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -1003,12 +1003,8 @@ long double x; x = PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } memcpy(ptr, &x, sizeof(long double)); _RET(value); } @@ -1027,12 +1023,8 @@ double x; x = PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } memcpy(ptr, &x, sizeof(double)); _RET(value); } @@ -1051,12 +1043,8 @@ double x; x = PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } #ifdef WORDS_BIGENDIAN if (_PyFloat_Pack8(x, (unsigned char *)ptr, 1)) return NULL; @@ -1083,12 +1071,8 @@ float x; x = (float)PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } memcpy(ptr, &x, sizeof(x)); _RET(value); } @@ -1107,12 +1091,8 @@ float x; x = (float)PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } #ifdef WORDS_BIGENDIAN if (_PyFloat_Pack4(x, (unsigned char *)ptr, 1)) return NULL; -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Fixed typo.
by vinay.sajip May 28, 2012

May 28, 2012

http://hg.python.org/cpython/rev/f4a7e738ad24 changeset: 77206:f4a7e738ad24 user: Vinay Sajip <vinay_sajip(a)yahoo.co.uk> date: Mon May 28 17:01:17 2012 +0100 summary: Fixed typo. files: Lib/test/test_venv.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py --- a/Lib/test/test_venv.py +++ b/Lib/test/test_venv.py @@ -29,7 +29,7 @@ self.ps3name = 'pysetup3' self.lib = ('lib', 'python%s' % sys.version[:3]) self.include = 'include' - if sys.platform == 'darwin' and '__PYTHONV_LAUNCHER__' in env: + if sys.platform == 'darwin' and '__PYTHONV_LAUNCHER__' in os.environ: executable = os.environ['__PYTHONV_LAUNCHER__'] else: executable = sys.executable -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Tweaked tests to use launcher executable name on OS X.
by vinay.sajip May 28, 2012

May 28, 2012

http://hg.python.org/cpython/rev/ba24c81ac713 changeset: 77205:ba24c81ac713 user: Vinay Sajip <vinay_sajip(a)yahoo.co.uk> date: Mon May 28 16:34:47 2012 +0100 summary: Tweaked tests to use launcher executable name on OS X. files: Lib/test/test_venv.py | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py --- a/Lib/test/test_venv.py +++ b/Lib/test/test_venv.py @@ -29,7 +29,11 @@ self.ps3name = 'pysetup3' self.lib = ('lib', 'python%s' % sys.version[:3]) self.include = 'include' - self.exe = os.path.split(sys.executable)[-1] + if sys.platform == 'darwin' and '__PYTHONV_LAUNCHER__' in env: + executable = os.environ['__PYTHONV_LAUNCHER__'] + else: + executable = sys.executable + self.exe = os.path.split(executable)[-1] def tearDown(self): shutil.rmtree(self.env_dir) -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Added python3 symlink; removed Distribute-related code, docs and comments;
by vinay.sajip May 28, 2012

May 28, 2012

http://hg.python.org/cpython/rev/459821d578e1 changeset: 77204:459821d578e1 user: Vinay Sajip <vinay_sajip(a)yahoo.co.uk> date: Mon May 28 16:33:01 2012 +0100 summary: Added python3 symlink; removed Distribute-related code, docs and comments; changed Mac OS X computation to determine framework builds. files: Lib/venv/__init__.py | 142 +++--------------------------- 1 files changed, 16 insertions(+), 126 deletions(-) diff --git a/Lib/venv/__init__.py b/Lib/venv/__init__.py --- a/Lib/venv/__init__.py +++ b/Lib/venv/__init__.py @@ -3,8 +3,8 @@ Copyright (C) 20011-2012 Vinay Sajip. All Rights Reserved. -usage: python -m venv [-h] [--no-distribute] [--system-site-packages] - [--symlinks] [--clear] [--upgrade] +usage: python -m venv [-h] [--system-site-packages] [--symlinks] [--clear] + [--upgrade] ENV_DIR [ENV_DIR ...] Creates virtual Python environments in one or more target directories. @@ -14,7 +14,6 @@ optional arguments: -h, --help show this help message and exit - --no-distribute Don't install Distribute in the virtual environment.* --system-site-packages Give the virtual environment access to the system site-packages dir. @@ -24,10 +23,6 @@ raised. --upgrade Upgrade the environment directory to use this version of Python, assuming Python has been upgraded in-place. - -*Note: Distribute support will be available during the alpha phase to -facilitate testing third-party packages with venvs created using this package. -This support will be removed after the alpha phase. """ import base64 import io @@ -36,6 +31,7 @@ import os.path import shutil import sys +import sysconfig try: import threading except ImportError: @@ -88,11 +84,11 @@ """ if (self.symlinks and sys.platform == 'darwin' and - 'Library/Framework' in sys.base_prefix): + sysconfig.get_config_var('PYTHONFRAMEWORK')): # Symlinking the stub executable in an OSX framework build will # result in a broken virtual environment. raise ValueError( - "Symlinking is not supported on OSX framework Python.") + 'Symlinking is not supported on OSX framework Python.') env_dir = os.path.abspath(env_dir) context = self.ensure_directories(env_dir) self.create_configuration(context) @@ -206,9 +202,10 @@ if os.name != 'nt': if not os.path.islink(path): os.chmod(path, 0o755) - path = os.path.join(binpath, 'python') - if not os.path.exists(path): - os.symlink(exename, path) + for suffix in ('python', 'python3'): + path = os.path.join(binpath, suffix) + if not os.path.exists(path): + os.symlink(exename, path) else: subdir = 'DLLs' include = self.include_binary @@ -322,104 +319,6 @@ os.chmod(dstfile, 0o755) -# This class will not be included in Python core; it's here for now to -# facilitate experimentation and testing, and as proof-of-concept of what could -# be done by external extension tools. -class DistributeEnvBuilder(EnvBuilder): - """ - By default, this builder installs Distribute so that you can pip or - easy_install other packages into the created environment. - - :param nodist: If True, Distribute is not installed into the created - environment. - :param progress: If Distribute is installed, the progress of the - installation can be monitored by passing a progress - callable. If specified, it is called with two - arguments: a string indicating some progress, and a - context indicating where the string is coming from. - The context argument can have one of three values: - 'main', indicating that it is called from virtualize() - itself, and 'stdout' and 'stderr', which are obtained - by reading lines from the output streams of a subprocess - which is used to install Distribute. - - If a callable is not specified, default progress - information is output to sys.stderr. - """ - - def __init__(self, *args, **kwargs): - self.nodist = kwargs.pop("nodist", False) - self.progress = kwargs.pop("progress", None) - super().__init__(*args, **kwargs) - - def post_setup(self, context): - """ - Set up any packages which need to be pre-installed into the - environment being created. - - :param context: The information for the environment creation request - being processed. - """ - if not self.nodist: - if threading: - self.install_distribute(context) - - def reader(self, stream, context): - """ - Read lines from a subprocess' output stream and either pass to a progress - callable (if specified) or write progress information to sys.stderr. - """ - progress = self.progress - while True: - s = stream.readline() - if not s: - break - if progress is not None: - progress(s, context) - else: - sys.stderr.write('.') - #sys.stderr.write(s.decode('utf-8')) - sys.stderr.flush() - stream.close() - - def install_distribute(self, context): - """ - Install Distribute in the environment. - - :param context: The information for the environment creation request - being processed. - """ - from subprocess import Popen, PIPE - from urllib.request import urlretrieve - - url = 'http://python-distribute.org/distribute_setup.py' - binpath = context.bin_path - distpath = os.path.join(binpath, 'distribute_setup.py') - # Download Distribute in the env - urlretrieve(url, distpath) - progress = self.progress - if progress is not None: - progress('Installing distribute', 'main') - else: - sys.stderr.write('Installing distribute ') - sys.stderr.flush() - # Install Distribute in the env - args = [context.env_exe, 'distribute_setup.py'] - p = Popen(args, stdout=PIPE, stderr=PIPE, cwd=binpath) - t1 = threading.Thread(target=self.reader, args=(p.stdout, 'stdout')) - t1.start() - t2 = threading.Thread(target=self.reader, args=(p.stderr, 'stderr')) - t2.start() - p.wait() - t1.join() - t2.join() - if progress is not None: - progress('done.', 'main') - else: - sys.stderr.write('done.\n') - # Clean up - no longer needed - os.unlink(distpath) - def create(env_dir, system_site_packages=False, clear=False, symlinks=False): """ Create a virtual environment in a directory. @@ -436,8 +335,7 @@ :param symlinks: If True, attempt to symlink rather than copy files into virtual environment. """ - # XXX This should be changed to EnvBuilder. - builder = DistributeEnvBuilder(system_site_packages=system_site_packages, + builder = EnvBuilder(system_site_packages=system_site_packages, clear=clear, symlinks=symlinks) builder.create(env_dir) @@ -460,17 +358,12 @@ 'directories.') parser.add_argument('dirs', metavar='ENV_DIR', nargs='+', help='A directory to create the environment in.') - # XXX This option will be removed. - parser.add_argument('--no-distribute', default=False, - action='store_true', dest='nodist', - help="Don't install Distribute in the virtual " - "environment.") parser.add_argument('--system-site-packages', default=False, action='store_true', dest='system_site', - help="Give the virtual environment access to the " - "system site-packages dir. ") + help='Give the virtual environment access to the ' + 'system site-packages dir.') if os.name == 'nt' or (sys.platform == 'darwin' and - 'Library/Framework' in sys.base_prefix): + sysconfig.get_config_var('PYTHONFRAMEWORK')): use_symlinks = False else: use_symlinks = True @@ -491,12 +384,9 @@ options = parser.parse_args(args) if options.upgrade and options.clear: raise ValueError('you cannot supply --upgrade and --clear together.') - # XXX This will be changed to EnvBuilder - builder = DistributeEnvBuilder(system_site_packages=options.system_site, - clear=options.clear, - symlinks=options.symlinks, - upgrade=options.upgrade, - nodist=options.nodist) + builder = EnvBuilder(system_site_packages=options.system_site, + clear=options.clear, symlinks=options.symlinks, + upgrade=options.upgrade) for d in options.dirs: builder.create(d) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #14443: Tell rpmbuild to use the correct version of Python
by nick.coghlan May 28, 2012

May 28, 2012

http://hg.python.org/cpython/rev/3d61e27cc570 changeset: 77202:3d61e27cc570 branch: 3.2 parent: 77199:12454f78967b user: Nick Coghlan <ncoghlan(a)gmail.com> date: Mon May 28 22:34:46 2012 +1000 summary: Issue #14443: Tell rpmbuild to use the correct version of Python files: Lib/distutils/command/bdist_rpm.py | 3 ++- Misc/NEWS | 3 +++ 2 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Lib/distutils/command/bdist_rpm.py b/Lib/distutils/command/bdist_rpm.py --- a/Lib/distutils/command/bdist_rpm.py +++ b/Lib/distutils/command/bdist_rpm.py @@ -190,7 +190,7 @@ if self.fix_python: self.python = sys.executable else: - self.python = "python" + self.python = "python3" elif self.fix_python: raise DistutilsOptionError( "--python and --fix-python are mutually exclusive options") @@ -320,6 +320,7 @@ rpm_cmd.append('-bb') else: rpm_cmd.append('-ba') + rpm_cmd.extend(['--define', '__python %s' % self.python]) if self.rpm3_mode: rpm_cmd.extend(['--define', '_topdir %s' % os.path.abspath(self.rpm_base)]) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -67,6 +67,9 @@ Library ------- +- Issue #14443: Tell rpmbuild to use the correct version of Python in + bdist_rpm. Initial patch by Ross Lagerwall. + - Issue14929: Stop Idle 3.x from closing on Unicode decode errors when grepping. Patch by Roger Serwy. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Merge from 3.2
by nick.coghlan May 28, 2012

May 28, 2012

http://hg.python.org/cpython/rev/cb716ee277cc changeset: 77203:cb716ee277cc parent: 77201:f26721ab3476 parent: 77202:3d61e27cc570 user: Nick Coghlan <ncoghlan(a)gmail.com> date: Mon May 28 22:36:07 2012 +1000 summary: Merge from 3.2 files: Lib/distutils/command/bdist_rpm.py | 3 ++- Misc/NEWS | 3 +++ 2 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Lib/distutils/command/bdist_rpm.py b/Lib/distutils/command/bdist_rpm.py --- a/Lib/distutils/command/bdist_rpm.py +++ b/Lib/distutils/command/bdist_rpm.py @@ -190,7 +190,7 @@ if self.fix_python: self.python = sys.executable else: - self.python = "python" + self.python = "python3" elif self.fix_python: raise DistutilsOptionError( "--python and --fix-python are mutually exclusive options") @@ -320,6 +320,7 @@ rpm_cmd.append('-bb') else: rpm_cmd.append('-ba') + rpm_cmd.extend(['--define', '__python %s' % self.python]) if self.rpm3_mode: rpm_cmd.extend(['--define', '_topdir %s' % os.path.abspath(self.rpm_base)]) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -49,6 +49,9 @@ Library ------- +- Issue #14443: Tell rpmbuild to use the correct version of Python in + bdist_rpm. Initial patch by Ross Lagerwall. + - Issue14929: Stop Idle 3.x from closing on Unicode decode errors when grepping. Patch by Roger Serwy. -- Repository URL: http://hg.python.org/cpython

1 0