Mailman 3 March 2011 - Python-checkins

devguide: Add a page about continuous integration (buildbots)
by antoine.pitrou 31 Mar '11

31 Mar '11

http://hg.python.org/devguide/rev/74f6a2fea6d1 changeset: 407:74f6a2fea6d1 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Wed Mar 30 22:59:11 2011 +0200 summary: Add a page about continuous integration (buildbots) files: buildbots.rst | 177 ++++++++++++++++++++++++++++++++++++++ index.rst | 10 +- 2 files changed, 183 insertions(+), 4 deletions(-) diff --git a/buildbots.rst b/buildbots.rst new file mode 100644 --- /dev/null +++ b/buildbots.rst @@ -0,0 +1,177 @@ +.. _buildbots: + +Continuous Integration +====================== + +To assert that there are no regressions in the :doc:`development and maintenance +branches <devcycle>`, Python has a set of dedicated machines (called *buildbots* or +*build slaves*) used for continuous integration. They span a number of +hardware/operating system combinations. Furthermore, each machine hosts +several *builders*, one per active branch: when a new change is pushed +to this branch on the public Mercurial repository, all corresponding builders +will schedule a new build to be run as soon as possible. + +The build steps run by the buildbots are the following: + +* Checkout of the source tree for the changeset which triggered the build +* Compiling Python +* Running the test suite using :ref:`strenuous settings <strenuous_testing>` +* Cleaning up the build tree + +It is your responsibility, as a core developer, to check the automatic +build results after you push a change to the repository. It is therefore +important that you get acquainted with the way these results are presented, +and how various kinds of failures can be explained and diagnosed. + +Checking results of automatic builds +------------------------------------ + +There are two ways of visualizing recent build results: + +* The Web interface for each branch at http://python.org/dev/buildbot/, + where the so-called "waterfall" view presents a vertical rundown of recent + builds for each builder. When interested in one build, you'll have to + click on it to know which changesets it corresponds to. + +* The command-line ``bbreport.py`` client, which you can get from + http://code.google.com/p/bbreport/. Installing it is trivial: just add + the directory containing ``bbreport.py`` to your system path so that + you can run it from any filesystem location. For example, if you want + to display the latest build results on the development ("default") branch, + type:: + + bbreport.py -q 3.x + +Some buildbots are much faster than others. Over time, you will learn which +ones produce the quickest results after a build, and which ones take the +longest time. + +Also, when several changesets are pushed in a quick succession in the same +branch, it often happens that a single build is scheduled for all these +changesets. + +Stability +--------- + +A subset of the buildbots are marked "stable". They are taken into account +when making a new release. The rule is that no stable buildbot must witness +persistent failures when the release is cut. It is absolutely **vital** +that core developers fix any issue they introduce on the stable buildbots, +as soon as possible. + +This does not mean that other builders' test results can be taken lightly, +either. Some of them are known for having platform-specific issues that +prevent some tests from succeeding (or even terminating at all), but +introducing additional failures should generally not be an option. + +Flags-dependent failures +------------------------ + +Sometimes, while you have run the :doc:`whole test suite <runtests>` before +committing, you may witness unexpected failures on the buildbots. One source +of such discrepancies is if different flags have been passed to the test runner +or to Python itself. To reproduce, make sure you use the same flags as the +buildbots: they can be found out simply by clicking the **stdio** link for +the failing build's tests. For example:: + + ./python.exe -Wd -E -bb ./Lib/test/regrtest.py -uall -rwW + +.. note:: + Running ``Lib/test/regrtest.py`` is exactly equivalent to running + ``-m test``. + +Ordering-dependent failures +--------------------------- + +Sometimes even the failure is subtler, as it relies on the order in which +the tests are run. The buildbots *randomize* test order (by using the ``-r`` +option to the test runner) to maximize the probability that potential +interferences between library modules are exercised; the downside is that it +can make for seemingly sporadic failures. + +The ``--randseed`` option makes it easy to reproduce the exact randomization +used in a given build. Again, open the ``stdio`` link for the failing test +run, and check the beginning of the test output proper. + +Let's assume, for the sake of example, that the output starts with:: + + ./python -Wd -E -bb Lib/test/regrtest.py -uall -rwW + == CPython 3.3a0 (default:22ae2b002865, Mar 30 2011, 13:58:40) [GCC 4.4.5] + == Linux-2.6.36-gentoo-r5-x86_64-AMD_Athlon-tm-_64_X2_Dual_Core_Processor_4400+-with-gentoo-1.12.14 little-endian + == /home/buildbot/buildarea/3.x.ochtman-gentoo-amd64/build/build/test_python_29628 + Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=1, verbose=0, bytes_warning=2, quiet=0) + Using random seed 2613169 + [ 1/353] test_augassign + [ 2/353] test_functools + +You can reproduce the exact same order using:: + + ./python -Wd -E -bb -m test -uall -rwW --randseed 2613169 + +It will run the following sequence (trimmed for brevity):: + + [ 1/353] test_augassign + [ 2/353] test_functools + [ 3/353] test_bool + [ 4/353] test_contains + [ 5/353] test_compileall + [ 6/353] test_unicode + +If this is enough to reproduce the failure on your setup, you can then +bisect the test sequence to look for the specific interference causing the +failure. Copy and paste the test sequence in a text file, then use the +``--fromfile`` (or ``-f``) option of the test runner to run the exact +sequence recorded in that text file:: + + ./python -Wd -E -bb -m test -uall -rwW --fromfile mytestsequence.txt + +In the example sequence above, if ``test_unicode`` had failed, you would +first test the following sequence:: + + [ 1/353] test_augassign + [ 2/353] test_functools + [ 3/353] test_bool + [ 6/353] test_unicode + +And, if it succeeds, the following one instead (which, hopefully, shall +fail):: + + [ 4/353] test_contains + [ 5/353] test_compileall + [ 6/353] test_unicode + +Then, recursively, narrow down the search until you get a single pair of +tests which triggers the failure. It is very rare that such an interference +involves more than **two** tests. If this is the case, we can only wish you +good luck! + +.. note:: + You cannot use the ``-j`` option (for parallel testing) when diagnosing + ordering-dependent failures. Using ``-j`` isolates each test in a + pristine subprocess and, therefore, prevents you from reproducing any + interference between tests. + + +Transient failures +------------------ + +While we try to make the test suite as reliable as possible, some tests do +not reach a perfect level of reproduceability. Some of them will sometimes +display spurious failures, depending on various conditions. Here are common +offenders: + +* Network-related tests, such as ``test_poplib``, ``test_urllibnet``, etc. + Their failures can stem from adverse network conditions, or imperfect + thread synchronization in the test code, which often has to run a + server in a separate thread. + +* Tests dealing with delicate issues such as inter-thread or inter-process + synchronization, or Unix signals: ``test_multiprocessing``, + ``test_threading``, ``test_subprocess``, ``test_threadsignals``. + +When you think a failure might be transient, it is recommended you confirm by +waiting for the next build. Still, even if the failure does turn out sporadic +and unpredictable, the issue should be reported on the bug tracker; even +better if it can be diagnosed and suppressed by fixing the test's implementation, +or by making its parameters - such as a timeout - more robust. + diff --git a/index.rst b/index.rst --- a/index.rst +++ b/index.rst @@ -21,6 +21,7 @@ developers committing devcycle + buildbots stdlibchanges langchanges @@ -40,7 +41,7 @@ contributing to Python. * `Issue tracker <http://bugs.python.org/>`_ -* Buildbots_ +* `Buildbot status`_ * :doc:`faq` * PEPs_ (Python Enhancement Proposals) @@ -69,7 +70,7 @@ * :doc:`coverage` * Advanced tasks for once you are comfortable * :doc:`silencewarnings` - * Fixing issues found by the buildbots_ + * Fixing issues found by the :doc:`buildbots <buildbots>` * :doc:`fixingissues` * :doc:`helptriage` * :doc:`devrole` @@ -79,6 +80,7 @@ * :doc:`coredev` * :doc:`committing` * :doc:`devcycle` + * :doc:`buildbots` Proposing changes to Python itself @@ -102,7 +104,7 @@ tracker for the issue tracker) * :doc:`experts` * `Firefox search engine plug-in`_ -* Buildbots_ +* `Buildbot status`_ * Source code * `Browse online <http://hg.python.org/cpython/>`_ * `Snapshot of py3k <http://hg.python.org/cpython/archive/tip.tar.bz2>`_ @@ -120,7 +122,7 @@ * :doc:`developers` -.. _buildbots: http://python.org/dev/buildbot/ +.. _Buildbot status: http://python.org/dev/buildbot/ .. _Firefox search engine plug-in: http://www.python.org/dev/searchplugin/ .. _Misc directory: http://hg.python.org/cpython/file/tip/Misc .. _PEPs: http://www.python.org/dev/peps -- Repository URL: http://hg.python.org/devguide

1 0

cpython (merge 3.2 -> 3.2): Merge
by kristjan.jonsson 30 Mar '11

30 Mar '11

http://hg.python.org/cpython/rev/e74299695014 changeset: 69067:e74299695014 branch: 3.2 parent: 69063:50e9d4515ea2 parent: 69060:075637897403 user: Kristjan Valur Jonsson <sweskman(a)gmail.com> date: Wed Mar 30 11:55:52 2011 +0000 summary: Merge files: Doc/library/urllib.request.rst | 4 +++ Lib/test/test_urllib.py | 16 +++++++++++++ Lib/test/test_urllib2.py | 24 +++++++++++++++++++ Lib/urllib/request.py | 27 ++++++++++++++++++++++ Misc/NEWS | 3 ++ Modules/_collectionsmodule.c | 2 +- 6 files changed, 75 insertions(+), 1 deletions(-) diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -650,6 +650,10 @@ is the case, :exc:`HTTPError` is raised. See :rfc:`2616` for details of the precise meanings of the various redirection codes. + An :class:`HTTPError` exception raised as a security consideration if the + HTTPRedirectHandler is presented with a redirected url which is not an HTTP, + HTTPS or FTP url. + .. method:: HTTPRedirectHandler.redirect_request(req, fp, code, msg, hdrs, newurl) diff --git a/Lib/test/test_urllib.py b/Lib/test/test_urllib.py --- a/Lib/test/test_urllib.py +++ b/Lib/test/test_urllib.py @@ -2,6 +2,7 @@ import urllib.parse import urllib.request +import urllib.error import http.client import email.message import io @@ -206,6 +207,21 @@ finally: self.unfakehttp() + def test_invalid_redirect(self): + # urlopen() should raise IOError for many error codes. + self.fakehttp(b'''HTTP/1.1 302 Found +Date: Wed, 02 Jan 2008 03:03:54 GMT +Server: Apache/1.3.33 (Debian GNU/Linux) mod_ssl/2.8.22 OpenSSL/0.9.7e +Location: file://guidocomputer.athome.com:/python/license +Connection: close +Content-Type: text/html; charset=iso-8859-1 +''') + try: + self.assertRaises(urllib.error.HTTPError, urlopen, + "http://python.org/") + finally: + self.unfakehttp() + def test_empty_socket(self): # urlopen() raises IOError if the underlying socket does not send any # data. (#1680230) diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -10,6 +10,7 @@ # The proxy bypass method imported below has logic specific to the OSX # proxy config data structure but is testable on all platforms. from urllib.request import Request, OpenerDirector, _proxy_bypass_macosx_sysconf +import urllib.error # XXX # Request @@ -1031,6 +1032,29 @@ self.assertEqual(count, urllib.request.HTTPRedirectHandler.max_redirections) + + def test_invalid_redirect(self): + from_url = "http://example.com/a.html" + valid_schemes = ['http','https','ftp'] + invalid_schemes = ['file','imap','ldap'] + schemeless_url = "example.com/b.html" + h = urllib.request.HTTPRedirectHandler() + o = h.parent = MockOpener() + req = Request(from_url) + req.timeout = socket._GLOBAL_DEFAULT_TIMEOUT + + for scheme in invalid_schemes: + invalid_url = scheme + '://' + schemeless_url + self.assertRaises(urllib.error.HTTPError, h.http_error_302, + req, MockFile(), 302, "Security Loophole", + MockHeaders({"location": invalid_url})) + + for scheme in valid_schemes: + valid_url = scheme + '://' + schemeless_url + h.http_error_302(req, MockFile(), 302, "That's fine", + MockHeaders({"location": valid_url})) + self.assertEqual(o.req.get_full_url(), valid_url) + def test_cookie_redirect(self): # cookies shouldn't leak into redirected requests from http.cookiejar import CookieJar diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -545,6 +545,17 @@ # fix a possible malformed URL urlparts = urlparse(newurl) + + # For security reasons we don't allow redirection to anything other + # than http, https or ftp. + + if not urlparts.scheme in ('http', 'https', 'ftp'): + raise HTTPError(newurl, code, + msg + + " - Redirection to url '%s' is not allowed" % + newurl, + headers, fp) + if not urlparts.path: urlparts = list(urlparts) urlparts[2] = "/" @@ -1903,8 +1914,24 @@ return void = fp.read() fp.close() + # In case the server sent a relative URL, join with original: newurl = urljoin(self.type + ":" + url, newurl) + + urlparts = urlparse(newurl) + + # For security reasons, we don't allow redirection to anything other + # than http, https and ftp. + + # We are using newer HTTPError with older redirect_internal method + # This older method will get deprecated in 3.3 + + if not urlparts.scheme in ('http', 'https', 'ftp'): + raise HTTPError(newurl, errcode, + errmsg + + " Redirection to url '%s' is not allowed." % newurl, + headers, fp) + return self.open(newurl) def http_error_301(self, url, fp, errcode, errmsg, headers, data=None): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -74,6 +74,9 @@ - Issue #11666: let help() display named tuple attributes and methods that start with a leading underscore. +- Issue #11662: Make urllib and urllib2 ignore redirections if the + scheme is not HTTP, HTTPS or FTP (CVE-2011-1521). + - Issue #5537: Fix time2isoz() and time2netscape() functions of httplib.cookiejar for expiration year greater than 2038 on 32-bit systems. diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -1002,7 +1002,7 @@ PyDoc_STRVAR(deque_doc, "deque(iterable[, maxlen]) --> deque object\n\ \n\ -Build an ordered collection accessible from endpoints only."); +Build an ordered collection with optimized access from its endpoints."); static PyTypeObject deque_type = { PyVarObject_HEAD_INIT(NULL, 0) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.1 -> 3.1): Merge 3.1
by kristjan.jonsson 30 Mar '11

30 Mar '11

http://hg.python.org/cpython/rev/8e074d9b1587 changeset: 69066:8e074d9b1587 branch: 3.1 parent: 69062:9a6da9bc257d parent: 69059:36344c9505b2 user: Kristjan Valur Jonsson <sweskman(a)gmail.com> date: Wed Mar 30 11:54:13 2011 +0000 summary: Merge 3.1 files: Doc/library/urllib.request.rst | 4 +++ Lib/test/test_urllib.py | 16 +++++++++++++ Lib/test/test_urllib2.py | 24 +++++++++++++++++++ Lib/urllib/request.py | 27 ++++++++++++++++++++++ Misc/NEWS | 3 ++ Modules/_collectionsmodule.c | 2 +- 6 files changed, 75 insertions(+), 1 deletions(-) diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -783,6 +783,10 @@ is the case, :exc:`HTTPError` is raised. See :rfc:`2616` for details of the precise meanings of the various redirection codes. + An :class:`HTTPError` exception raised as a security consideration if the + HTTPRedirectHandler is presented with a redirected url which is not an HTTP, + HTTPS or FTP url. + .. method:: HTTPRedirectHandler.redirect_request(req, fp, code, msg, hdrs, newurl) diff --git a/Lib/test/test_urllib.py b/Lib/test/test_urllib.py --- a/Lib/test/test_urllib.py +++ b/Lib/test/test_urllib.py @@ -2,6 +2,7 @@ import urllib.parse import urllib.request +import urllib.error import http.client import email.message import io @@ -183,6 +184,21 @@ finally: self.unfakehttp() + def test_invalid_redirect(self): + # urlopen() should raise IOError for many error codes. + self.fakehttp(b'''HTTP/1.1 302 Found +Date: Wed, 02 Jan 2008 03:03:54 GMT +Server: Apache/1.3.33 (Debian GNU/Linux) mod_ssl/2.8.22 OpenSSL/0.9.7e +Location: file://guidocomputer.athome.com:/python/license +Connection: close +Content-Type: text/html; charset=iso-8859-1 +''') + try: + self.assertRaises(urllib.error.HTTPError, urlopen, + "http://python.org/") + finally: + self.unfakehttp() + def test_empty_socket(self): # urlopen() raises IOError if the underlying socket does not send any # data. (#1680230) diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -9,6 +9,7 @@ # The proxy bypass method imported below has logic specific to the OSX # proxy config data structure but is testable on all platforms. from urllib.request import Request, OpenerDirector, _proxy_bypass_macosx_sysconf +import urllib.error # XXX # Request @@ -985,6 +986,29 @@ self.assertEqual(count, urllib.request.HTTPRedirectHandler.max_redirections) + + def test_invalid_redirect(self): + from_url = "http://example.com/a.html" + valid_schemes = ['http','https','ftp'] + invalid_schemes = ['file','imap','ldap'] + schemeless_url = "example.com/b.html" + h = urllib.request.HTTPRedirectHandler() + o = h.parent = MockOpener() + req = Request(from_url) + req.timeout = socket._GLOBAL_DEFAULT_TIMEOUT + + for scheme in invalid_schemes: + invalid_url = scheme + '://' + schemeless_url + self.assertRaises(urllib.error.HTTPError, h.http_error_302, + req, MockFile(), 302, "Security Loophole", + MockHeaders({"location": invalid_url})) + + for scheme in valid_schemes: + valid_url = scheme + '://' + schemeless_url + h.http_error_302(req, MockFile(), 302, "That's fine", + MockHeaders({"location": valid_url})) + self.assertEqual(o.req.get_full_url(), valid_url) + def test_cookie_redirect(self): # cookies shouldn't leak into redirected requests from http.cookiejar import CookieJar diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -528,6 +528,17 @@ # fix a possible malformed URL urlparts = urlparse(newurl) + + # For security reasons we don't allow redirection to anything other + # than http, https or ftp. + + if not urlparts.scheme in ('http', 'https', 'ftp'): + raise HTTPError(newurl, code, + msg + + " - Redirection to url '%s' is not allowed" % + newurl, + headers, fp) + if not urlparts.path: urlparts = list(urlparts) urlparts[2] = "/" @@ -1864,8 +1875,24 @@ return void = fp.read() fp.close() + # In case the server sent a relative URL, join with original: newurl = urljoin(self.type + ":" + url, newurl) + + urlparts = urlparse(newurl) + + # For security reasons, we don't allow redirection to anything other + # than http, https and ftp. + + # We are using newer HTTPError with older redirect_internal method + # This older method will get deprecated in 3.3 + + if not urlparts.scheme in ('http', 'https', 'ftp'): + raise HTTPError(newurl, errcode, + errmsg + + " Redirection to url '%s' is not allowed." % newurl, + headers, fp) + return self.open(newurl) def http_error_301(self, url, fp, errcode, errmsg, headers, data=None): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -58,6 +58,9 @@ - Issue #11659: Fix ResourceWarning in test_subprocess introduced by #11459. Patch by Ben Hayden. +- Issue #11662: Make urllib and urllib2 ignore redirections if the + scheme is not HTTP, HTTPS or FTP (CVE-2011-1521). + - Issue #5537: Fix time2isoz() and time2netscape() functions of httplib.cookiejar for expiration year greater than 2038 on 32-bit systems. diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -913,7 +913,7 @@ PyDoc_STRVAR(deque_doc, "deque(iterable[, maxlen]) --> deque object\n\ \n\ -Build an ordered collection accessible from endpoints only."); +Build an ordered collection with optimized access from its endpoints."); static PyTypeObject deque_type = { PyVarObject_HEAD_INIT(NULL, 0) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge default -> default): Merge
by kristjan.jonsson 30 Mar '11

30 Mar '11

http://hg.python.org/cpython/rev/22ae2b002865 changeset: 69065:22ae2b002865 parent: 69064:b63eeb5a5c07 parent: 69061:80e71592b114 user: Kristjan Valur Jonsson <sweskman(a)gmail.com> date: Wed Mar 30 11:39:24 2011 +0000 summary: Merge files: Doc/library/email.parser.rst | 11 +- Doc/library/urllib.request.rst | 4 + Lib/email/parser.py | 24 +- Lib/logging/__init__.py | 8 +- Lib/test/test_logging.py | 537 ++++++++++++++++++++- Lib/test/test_urllib.py | 16 + Lib/test/test_urllib2.py | 24 + Lib/urllib/request.py | 27 + Misc/NEWS | 6 + Modules/_collectionsmodule.c | 2 +- 10 files changed, 601 insertions(+), 58 deletions(-) diff --git a/Doc/library/email.parser.rst b/Doc/library/email.parser.rst --- a/Doc/library/email.parser.rst +++ b/Doc/library/email.parser.rst @@ -102,7 +102,7 @@ class. -.. class:: Parser(_class=email.message.Message, strict=None) +.. class:: Parser(_class=email.message.Message) The constructor for the :class:`Parser` class takes an optional argument *_class*. This must be a callable factory (such as a function or a class), and @@ -110,13 +110,8 @@ :class:`~email.message.Message` (see :mod:`email.message`). The factory will be called without arguments. - The optional *strict* flag is ignored. - - .. deprecated:: 2.4 - Because the :class:`Parser` class is a backward compatible API wrapper - around the new-in-Python 2.4 :class:`FeedParser`, *all* parsing is - effectively non-strict. You should simply stop passing a *strict* flag to - the :class:`Parser` constructor. + .. versionchanged:: 3.2 + Removed the *strict* argument that was deprecated in 2.4. The other public :class:`Parser` methods are: diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -650,6 +650,10 @@ is the case, :exc:`HTTPError` is raised. See :rfc:`2616` for details of the precise meanings of the various redirection codes. + An :class:`HTTPError` exception raised as a security consideration if the + HTTPRedirectHandler is presented with a redirected url which is not an HTTP, + HTTPS or FTP url. + .. method:: HTTPRedirectHandler.redirect_request(req, fp, code, msg, hdrs, newurl) diff --git a/Lib/email/parser.py b/Lib/email/parser.py --- a/Lib/email/parser.py +++ b/Lib/email/parser.py @@ -15,7 +15,7 @@ class Parser: - def __init__(self, *args, **kws): + def __init__(self, _class=Message): """Parser of RFC 2822 and MIME email messages. Creates an in-memory object tree representing the email message, which @@ -31,27 +31,7 @@ must be created. This class must have a constructor that can take zero arguments. Default is Message.Message. """ - if len(args) >= 1: - if '_class' in kws: - raise TypeError("Multiple values for keyword arg '_class'") - kws['_class'] = args[0] - if len(args) == 2: - if 'strict' in kws: - raise TypeError("Multiple values for keyword arg 'strict'") - kws['strict'] = args[1] - if len(args) > 2: - raise TypeError('Too many arguments') - if '_class' in kws: - self._class = kws['_class'] - del kws['_class'] - else: - self._class = Message - if 'strict' in kws: - warnings.warn("'strict' argument is deprecated (and ignored)", - DeprecationWarning, 2) - del kws['strict'] - if kws: - raise TypeError('Unexpected keyword arguments') + self._class = _class def parse(self, fp, headersonly=False): """Create a message structure from the data in a file. diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2001-2010 by Vinay Sajip. All Rights Reserved. +# Copyright 2001-2011 by Vinay Sajip. All Rights Reserved. # # Permission to use, copy, modify, and distribute this software and its # documentation for any purpose and without fee is hereby granted, @@ -18,7 +18,7 @@ Logging package for Python. Based on PEP 282 and comments thereto in comp.lang.python, and influenced by Apache's log4j system. -Copyright (C) 2001-2010 Vinay Sajip. All Rights Reserved. +Copyright (C) 2001-2011 Vinay Sajip. All Rights Reserved. To use, simply 'import logging' and log away! """ @@ -1826,10 +1826,10 @@ package. """ def handle(self, record): - pass + """Stub.""" def emit(self, record): - pass + """Stub.""" def createLock(self): self.lock = None diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -40,7 +40,7 @@ import struct import sys import tempfile -from test.support import captured_stdout, run_with_locale, run_unittest +from test.support import captured_stdout, run_with_locale, run_unittest, patch import textwrap import unittest import warnings @@ -1082,28 +1082,39 @@ def test_warnings(self): with warnings.catch_warnings(): logging.captureWarnings(True) - try: - warnings.filterwarnings("always", category=UserWarning) - file = io.StringIO() - h = logging.StreamHandler(file) - logger = logging.getLogger("py.warnings") - logger.addHandler(h) - warnings.warn("I'm warning you...") - logger.removeHandler(h) - s = file.getvalue() - h.close() - self.assertTrue(s.find("UserWarning: I'm warning you...\n") > 0) - - #See if an explicit file uses the original implementation - file = io.StringIO() - warnings.showwarning("Explicit", UserWarning, "dummy.py", 42, - file, "Dummy line") - s = file.getvalue() - file.close() - self.assertEqual(s, - "dummy.py:42: UserWarning: Explicit\n Dummy line\n") - finally: - logging.captureWarnings(False) + self.addCleanup(lambda: logging.captureWarnings(False)) + warnings.filterwarnings("always", category=UserWarning) + stream = io.StringIO() + h = logging.StreamHandler(stream) + logger = logging.getLogger("py.warnings") + logger.addHandler(h) + warnings.warn("I'm warning you...") + logger.removeHandler(h) + s = stream.getvalue() + h.close() + self.assertTrue(s.find("UserWarning: I'm warning you...\n") > 0) + + #See if an explicit file uses the original implementation + a_file = io.StringIO() + warnings.showwarning("Explicit", UserWarning, "dummy.py", 42, + a_file, "Dummy line") + s = a_file.getvalue() + a_file.close() + self.assertEqual(s, + "dummy.py:42: UserWarning: Explicit\n Dummy line\n") + + def test_warnings_no_handlers(self): + with warnings.catch_warnings(): + logging.captureWarnings(True) + self.addCleanup(lambda: logging.captureWarnings(False)) + + # confirm our assumption: no loggers are set + logger = logging.getLogger("py.warnings") + assert logger.handlers == [] + + warnings.showwarning("Explicit", UserWarning, "dummy.py", 42) + self.assertTrue(len(logger.handlers) == 1) + self.assertIsInstance(logger.handlers[0], logging.NullHandler) def formatFunc(format, datefmt=None): @@ -2007,6 +2018,11 @@ self.assertEqual(logged, ['should appear in logged']) + def test_set_log_record_factory(self): + man = logging.Manager(None) + expected = object() + man.setLogRecordFactory(expected) + self.assertEqual(man.logRecordFactory, expected) class ChildLoggerTest(BaseTest): def test_child_loggers(self): @@ -2198,6 +2214,479 @@ logging.raiseExceptions = old_raise_exceptions +class FakeHandler: + + def __init__(self, identifier, called): + for method in ('acquire', 'flush', 'close', 'release'): + setattr(self, method, self.record_call(identifier, method, called)) + + def record_call(self, identifier, method_name, called): + def inner(): + called.append('{} - {}'.format(identifier, method_name)) + return inner + + +class RecordingHandler(logging.NullHandler): + + def __init__(self, *args, **kwargs): + super(RecordingHandler, self).__init__(*args, **kwargs) + self.records = [] + + def handle(self, record): + """Keep track of all the emitted records.""" + self.records.append(record) + + +class ShutdownTest(BaseTest): + + """Tets suite for the shutdown method.""" + + def setUp(self): + super(ShutdownTest, self).setUp() + self.called = [] + + raise_exceptions = logging.raiseExceptions + self.addCleanup(lambda: setattr(logging, 'raiseExceptions', raise_exceptions)) + + def raise_error(self, error): + def inner(): + raise error() + return inner + + def test_no_failure(self): + # create some fake handlers + handler0 = FakeHandler(0, self.called) + handler1 = FakeHandler(1, self.called) + handler2 = FakeHandler(2, self.called) + + # create live weakref to those handlers + handlers = map(logging.weakref.ref, [handler0, handler1, handler2]) + + logging.shutdown(handlerList=list(handlers)) + + expected = ['2 - acquire', '2 - flush', '2 - close', '2 - release', + '1 - acquire', '1 - flush', '1 - close', '1 - release', + '0 - acquire', '0 - flush', '0 - close', '0 - release'] + self.assertEqual(expected, self.called) + + def _test_with_failure_in_method(self, method, error): + handler = FakeHandler(0, self.called) + setattr(handler, method, self.raise_error(error)) + handlers = [logging.weakref.ref(handler)] + + logging.shutdown(handlerList=list(handlers)) + + self.assertEqual('0 - release', self.called[-1]) + + def test_with_ioerror_in_acquire(self): + self._test_with_failure_in_method('acquire', IOError) + + def test_with_ioerror_in_flush(self): + self._test_with_failure_in_method('flush', IOError) + + def test_with_ioerror_in_close(self): + self._test_with_failure_in_method('close', IOError) + + def test_with_valueerror_in_acquire(self): + self._test_with_failure_in_method('acquire', ValueError) + + def test_with_valueerror_in_flush(self): + self._test_with_failure_in_method('flush', ValueError) + + def test_with_valueerror_in_close(self): + self._test_with_failure_in_method('close', ValueError) + + def test_with_other_error_in_acquire_without_raise(self): + logging.raiseExceptions = False + self._test_with_failure_in_method('acquire', IndexError) + + def test_with_other_error_in_flush_without_raise(self): + logging.raiseExceptions = False + self._test_with_failure_in_method('flush', IndexError) + + def test_with_other_error_in_close_without_raise(self): + logging.raiseExceptions = False + self._test_with_failure_in_method('close', IndexError) + + def test_with_other_error_in_acquire_with_raise(self): + logging.raiseExceptions = True + self.assertRaises(IndexError, self._test_with_failure_in_method, + 'acquire', IndexError) + + def test_with_other_error_in_flush_with_raise(self): + logging.raiseExceptions = True + self.assertRaises(IndexError, self._test_with_failure_in_method, + 'flush', IndexError) + + def test_with_other_error_in_close_with_raise(self): + logging.raiseExceptions = True + self.assertRaises(IndexError, self._test_with_failure_in_method, + 'close', IndexError) + + +class ModuleLevelMiscTest(BaseTest): + + """Tets suite for some module level methods.""" + + def test_disable(self): + old_disable = logging.root.manager.disable + # confirm our assumptions are correct + assert old_disable == 0 + self.addCleanup(lambda: logging.disable(old_disable)) + + logging.disable(83) + self.assertEqual(logging.root.manager.disable, 83) + + def _test_log(self, method, level=None): + called = [] + patch(self, logging, 'basicConfig', + lambda *a, **kw: called.append(a, kw)) + + recording = RecordingHandler() + logging.root.addHandler(recording) + + log_method = getattr(logging, method) + if level is not None: + log_method(level, "test me: %r", recording) + else: + log_method("test me: %r", recording) + + self.assertEqual(len(recording.records), 1) + record = recording.records[0] + self.assertEqual(record.getMessage(), "test me: %r" % recording) + + expected_level = level if level is not None else getattr(logging, method.upper()) + self.assertEqual(record.levelno, expected_level) + + # basicConfig was not called! + self.assertEqual(called, []) + + def test_log(self): + self._test_log('log', logging.ERROR) + + def test_debug(self): + self._test_log('debug') + + def test_info(self): + self._test_log('info') + + def test_warning(self): + self._test_log('warning') + + def test_error(self): + self._test_log('error') + + def test_critical(self): + self._test_log('critical') + + def test_set_logger_class(self): + self.assertRaises(TypeError, logging.setLoggerClass, object) + + class MyLogger(logging.Logger): + pass + + logging.setLoggerClass(MyLogger) + self.assertEqual(logging.getLoggerClass(), MyLogger) + + logging.setLoggerClass(logging.Logger) + self.assertEqual(logging.getLoggerClass(), logging.Logger) + + +class BasicConfigTest(unittest.TestCase): + + """Tets suite for logging.basicConfig.""" + + def setUp(self): + super(BasicConfigTest, self).setUp() + handlers = logging.root.handlers + self.addCleanup(lambda: setattr(logging.root, 'handlers', handlers)) + logging.root.handlers = [] + + def tearDown(self): + logging.shutdown() + super(BasicConfigTest, self).tearDown() + + def test_no_kwargs(self): + logging.basicConfig() + + # handler defaults to a StreamHandler to sys.stderr + self.assertEqual(len(logging.root.handlers), 1) + handler = logging.root.handlers[0] + self.assertIsInstance(handler, logging.StreamHandler) + self.assertEqual(handler.stream, sys.stderr) + + formatter = handler.formatter + # format defaults to logging.BASIC_FORMAT + self.assertEqual(formatter._style._fmt, logging.BASIC_FORMAT) + # datefmt defaults to None + self.assertIsNone(formatter.datefmt) + # style defaults to % + self.assertIsInstance(formatter._style, logging.PercentStyle) + + # level is not explicitely set + self.assertEqual(logging.root.level, logging.WARNING) + + def test_filename(self): + logging.basicConfig(filename='test.log') + + self.assertEqual(len(logging.root.handlers), 1) + handler = logging.root.handlers[0] + self.assertIsInstance(handler, logging.FileHandler) + + expected = logging.FileHandler('test.log', 'a') + self.addCleanup(expected.close) + self.assertEqual(handler.stream.mode, expected.stream.mode) + self.assertEqual(handler.stream.name, expected.stream.name) + + def test_filemode(self): + logging.basicConfig(filename='test.log', filemode='wb') + + handler = logging.root.handlers[0] + expected = logging.FileHandler('test.log', 'wb') + self.addCleanup(expected.close) + self.assertEqual(handler.stream.mode, expected.stream.mode) + + def test_stream(self): + stream = io.StringIO() + self.addCleanup(stream.close) + logging.basicConfig(stream=stream) + + self.assertEqual(len(logging.root.handlers), 1) + handler = logging.root.handlers[0] + self.assertIsInstance(handler, logging.StreamHandler) + self.assertEqual(handler.stream, stream) + + def test_format(self): + logging.basicConfig(format='foo') + + formatter = logging.root.handlers[0].formatter + self.assertEqual(formatter._style._fmt, 'foo') + + def test_datefmt(self): + logging.basicConfig(datefmt='bar') + + formatter = logging.root.handlers[0].formatter + self.assertEqual(formatter.datefmt, 'bar') + + def test_style(self): + logging.basicConfig(style='$') + + formatter = logging.root.handlers[0].formatter + self.assertIsInstance(formatter._style, logging.StringTemplateStyle) + + def test_level(self): + old_level = logging.root.level + self.addCleanup(lambda: logging.root.setLevel(old_level)) + + logging.basicConfig(level=57) + self.assertEqual(logging.root.level, 57) + + def _test_log(self, method, level=None): + # logging.root has no handlers so basicConfig should be called + called = [] + + old_basic_config = logging.basicConfig + def my_basic_config(*a, **kw): + old_basic_config() + old_level = logging.root.level + logging.root.setLevel(100) # avoid having messages in stderr + self.addCleanup(lambda: logging.root.setLevel(old_level)) + called.append((a, kw)) + + patch(self, logging, 'basicConfig', my_basic_config) + + log_method = getattr(logging, method) + if level is not None: + log_method(level, "test me") + else: + log_method("test me") + + # basicConfig was called with no arguments + self.assertEqual(called, [((), {})]) + + def test_log(self): + self._test_log('log', logging.WARNING) + + def test_debug(self): + self._test_log('debug') + + def test_info(self): + self._test_log('info') + + def test_warning(self): + self._test_log('warning') + + def test_error(self): + self._test_log('error') + + def test_critical(self): + self._test_log('critical') + + +class LoggerAdapterTest(unittest.TestCase): + + def setUp(self): + super(LoggerAdapterTest, self).setUp() + old_handler_list = logging._handlerList[:] + + self.recording = RecordingHandler() + self.logger = logging.root + self.logger.addHandler(self.recording) + self.addCleanup(lambda: self.logger.removeHandler(self.recording)) + self.addCleanup(self.recording.close) + + def cleanup(): + logging._handlerList[:] = old_handler_list + + self.addCleanup(cleanup) + self.addCleanup(logging.shutdown) + self.adapter = logging.LoggerAdapter(logger=self.logger, extra=None) + + def test_exception(self): + msg = 'testing exception: %r' + exc = None + try: + assert False + except AssertionError as e: + exc = e + self.adapter.exception(msg, self.recording) + + self.assertEqual(len(self.recording.records), 1) + record = self.recording.records[0] + self.assertEqual(record.levelno, logging.ERROR) + self.assertEqual(record.msg, msg) + self.assertEqual(record.args, (self.recording,)) + self.assertEqual(record.exc_info, + (exc.__class__, exc, exc.__traceback__)) + + def test_critical(self): + msg = 'critical test! %r' + self.adapter.critical(msg, self.recording) + + self.assertEqual(len(self.recording.records), 1) + record = self.recording.records[0] + self.assertEqual(record.levelno, logging.CRITICAL) + self.assertEqual(record.msg, msg) + self.assertEqual(record.args, (self.recording,)) + + def test_is_enabled_for(self): + old_disable = self.adapter.logger.manager.disable + self.adapter.logger.manager.disable = 33 + self.addCleanup(lambda: setattr(self.adapter.logger.manager, + 'disable', old_disable)) + self.assertFalse(self.adapter.isEnabledFor(32)) + + def test_has_handlers(self): + self.assertTrue(self.adapter.hasHandlers()) + + for handler in self.logger.handlers: + self.logger.removeHandler(handler) + assert not self.logger.hasHandlers() + + self.assertFalse(self.adapter.hasHandlers()) + + +class LoggerTest(BaseTest): + + def setUp(self): + super(LoggerTest, self).setUp() + self.recording = RecordingHandler() + self.logger = logging.Logger(name='blah') + self.logger.addHandler(self.recording) + self.addCleanup(lambda: self.logger.removeHandler(self.recording)) + self.addCleanup(self.recording.close) + self.addCleanup(logging.shutdown) + + def test_set_invalid_level(self): + self.assertRaises(TypeError, self.logger.setLevel, object()) + + def test_exception(self): + msg = 'testing exception: %r' + exc = None + try: + assert False + except AssertionError as e: + exc = e + self.logger.exception(msg, self.recording) + + self.assertEqual(len(self.recording.records), 1) + record = self.recording.records[0] + self.assertEqual(record.levelno, logging.ERROR) + self.assertEqual(record.msg, msg) + self.assertEqual(record.args, (self.recording,)) + self.assertEqual(record.exc_info, + (exc.__class__, exc, exc.__traceback__)) + + def test_log_invalid_level_with_raise(self): + old_raise = logging.raiseExceptions + self.addCleanup(lambda: setattr(logging, 'raiseExecptions', old_raise)) + + logging.raiseExceptions = True + self.assertRaises(TypeError, self.logger.log, '10', 'test message') + + def test_log_invalid_level_no_raise(self): + old_raise = logging.raiseExceptions + self.addCleanup(lambda: setattr(logging, 'raiseExecptions', old_raise)) + + logging.raiseExceptions = False + self.logger.log('10', 'test message') # no exception happens + + def test_find_caller_with_stack_info(self): + called = [] + patch(self, logging.traceback, 'print_stack', + lambda f, file: called.append(file.getvalue())) + + self.logger.findCaller(stack_info=True) + + self.assertEqual(len(called), 1) + self.assertEqual('Stack (most recent call last):\n', called[0]) + + def test_make_record_with_extra_overwrite(self): + name = 'my record' + level = 13 + fn = lno = msg = args = exc_info = func = sinfo = None + rv = logging._logRecordFactory(name, level, fn, lno, msg, args, + exc_info, func, sinfo) + + for key in ('message', 'asctime') + tuple(rv.__dict__.keys()): + extra = {key: 'some value'} + self.assertRaises(KeyError, self.logger.makeRecord, name, level, + fn, lno, msg, args, exc_info, + extra=extra, sinfo=sinfo) + + def test_make_record_with_extra_no_overwrite(self): + name = 'my record' + level = 13 + fn = lno = msg = args = exc_info = func = sinfo = None + extra = {'valid_key': 'some value'} + result = self.logger.makeRecord(name, level, fn, lno, msg, args, + exc_info, extra=extra, sinfo=sinfo) + self.assertIn('valid_key', result.__dict__) + + def test_has_handlers(self): + self.assertTrue(self.logger.hasHandlers()) + + for handler in self.logger.handlers: + self.logger.removeHandler(handler) + assert not self.logger.hasHandlers() + + self.assertFalse(self.logger.hasHandlers()) + + def test_has_handlers_no_propagate(self): + child_logger = logging.getLogger('blah.child') + child_logger.propagate = False + assert child_logger.handlers == [] + + self.assertFalse(child_logger.hasHandlers()) + + def test_is_enabled_for(self): + old_disable = self.logger.manager.disable + self.logger.manager.disable = 23 + self.addCleanup(lambda: setattr(self.logger.manager, + 'disable', old_disable)) + self.assertFalse(self.logger.isEnabledFor(22)) + + class BaseFileTest(BaseTest): "Base class for handler tests that write log files" @@ -2319,6 +2808,8 @@ EncodingTest, WarningsTest, ConfigDictTest, ManagerTest, FormatterTest, LogRecordFactoryTest, ChildLoggerTest, QueueHandlerTest, + ShutdownTest, ModuleLevelMiscTest, BasicConfigTest, + LoggerAdapterTest, LoggerTest, RotatingFileHandlerTest, LastResortTest, TimedRotatingFileHandlerTest diff --git a/Lib/test/test_urllib.py b/Lib/test/test_urllib.py --- a/Lib/test/test_urllib.py +++ b/Lib/test/test_urllib.py @@ -2,6 +2,7 @@ import urllib.parse import urllib.request +import urllib.error import http.client import email.message import io @@ -206,6 +207,21 @@ finally: self.unfakehttp() + def test_invalid_redirect(self): + # urlopen() should raise IOError for many error codes. + self.fakehttp(b'''HTTP/1.1 302 Found +Date: Wed, 02 Jan 2008 03:03:54 GMT +Server: Apache/1.3.33 (Debian GNU/Linux) mod_ssl/2.8.22 OpenSSL/0.9.7e +Location: file://guidocomputer.athome.com:/python/license +Connection: close +Content-Type: text/html; charset=iso-8859-1 +''') + try: + self.assertRaises(urllib.error.HTTPError, urlopen, + "http://python.org/") + finally: + self.unfakehttp() + def test_empty_socket(self): # urlopen() raises IOError if the underlying socket does not send any # data. (#1680230) diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -10,6 +10,7 @@ # The proxy bypass method imported below has logic specific to the OSX # proxy config data structure but is testable on all platforms. from urllib.request import Request, OpenerDirector, _proxy_bypass_macosx_sysconf +import urllib.error # XXX # Request @@ -1031,6 +1032,29 @@ self.assertEqual(count, urllib.request.HTTPRedirectHandler.max_redirections) + + def test_invalid_redirect(self): + from_url = "http://example.com/a.html" + valid_schemes = ['http','https','ftp'] + invalid_schemes = ['file','imap','ldap'] + schemeless_url = "example.com/b.html" + h = urllib.request.HTTPRedirectHandler() + o = h.parent = MockOpener() + req = Request(from_url) + req.timeout = socket._GLOBAL_DEFAULT_TIMEOUT + + for scheme in invalid_schemes: + invalid_url = scheme + '://' + schemeless_url + self.assertRaises(urllib.error.HTTPError, h.http_error_302, + req, MockFile(), 302, "Security Loophole", + MockHeaders({"location": invalid_url})) + + for scheme in valid_schemes: + valid_url = scheme + '://' + schemeless_url + h.http_error_302(req, MockFile(), 302, "That's fine", + MockHeaders({"location": valid_url})) + self.assertEqual(o.req.get_full_url(), valid_url) + def test_cookie_redirect(self): # cookies shouldn't leak into redirected requests from http.cookiejar import CookieJar diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -545,6 +545,17 @@ # fix a possible malformed URL urlparts = urlparse(newurl) + + # For security reasons we don't allow redirection to anything other + # than http, https or ftp. + + if not urlparts.scheme in ('http', 'https', 'ftp'): + raise HTTPError(newurl, code, + msg + + " - Redirection to url '%s' is not allowed" % + newurl, + headers, fp) + if not urlparts.path: urlparts = list(urlparts) urlparts[2] = "/" @@ -1903,8 +1914,24 @@ return void = fp.read() fp.close() + # In case the server sent a relative URL, join with original: newurl = urljoin(self.type + ":" + url, newurl) + + urlparts = urlparse(newurl) + + # For security reasons, we don't allow redirection to anything other + # than http, https and ftp. + + # We are using newer HTTPError with older redirect_internal method + # This older method will get deprecated in 3.3 + + if not urlparts.scheme in ('http', 'https', 'ftp'): + raise HTTPError(newurl, errcode, + errmsg + + " Redirection to url '%s' is not allowed." % newurl, + headers, fp) + return self.open(newurl) def http_error_301(self, url, fp, errcode, errmsg, headers, data=None): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -87,6 +87,9 @@ Library ------- +- Removed the 'strict' argument to email.parser.Parser, which has been + deprecated since Python 2.4. + - Issue #11256: Fix inspect.getcallargs on functions that take only keyword arguments. @@ -136,6 +139,9 @@ - Issue #11666: let help() display named tuple attributes and methods that start with a leading underscore. +- Issue #11662: Make urllib and urllib2 ignore redirections if the + scheme is not HTTP, HTTPS or FTP (CVE-2011-1521). + - Issue #5537: Fix time2isoz() and time2netscape() functions of httplib.cookiejar for expiration year greater than 2038 on 32-bit systems. diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -1002,7 +1002,7 @@ PyDoc_STRVAR(deque_doc, "deque(iterable[, maxlen]) --> deque object\n\ \n\ -Build an ordered collection accessible from endpoints only."); +Build an ordered collection with optimized access from its endpoints."); static PyTypeObject deque_type = { PyVarObject_HEAD_INIT(NULL, 0) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.2 -> default): Merge 3.2
by kristjan.jonsson 30 Mar '11

30 Mar '11

http://hg.python.org/cpython/rev/b63eeb5a5c07 changeset: 69064:b63eeb5a5c07 parent: 69037:7a8ada66c07d parent: 69063:50e9d4515ea2 user: Kristjan Valur Jonsson <sweskman(a)gmail.com> date: Wed Mar 30 11:32:06 2011 +0000 summary: Merge 3.2 files: Lib/test/test_itertools.py | 5 +++++ Modules/itertoolsmodule.c | 9 ++++++--- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py --- a/Lib/test/test_itertools.py +++ b/Lib/test/test_itertools.py @@ -367,6 +367,8 @@ list(range(maxsize-5, maxsize+5))) self.assertEqual(list(islice(count(-maxsize-5), 10)), list(range(-maxsize-5, -maxsize+5))) + self.assertEqual(list(islice(count(10, maxsize+5), 3)), + list(range(10, 10+3*(maxsize+5), maxsize+5))) c = count(3) self.assertEqual(repr(c), 'count(3)') next(c) @@ -389,6 +391,9 @@ self.assertEqual(next(copy.deepcopy(c)), value) self.assertEqual(next(pickle.loads(pickle.dumps(c))), value) + #check proper internal error handling for large "step' sizes + count(1, maxsize+5); sys.exc_info() + def test_count_with_stride(self): self.assertEqual(lzip('abc',count(2,3)), [('a', 2), ('b', 5), ('c', 8)]) self.assertEqual(lzip('abc',count(start=2,step=3)), diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c --- a/Modules/itertoolsmodule.c +++ b/Modules/itertoolsmodule.c @@ -3060,6 +3060,7 @@ Py_ssize_t cnt = 0; PyObject *long_cnt = NULL; PyObject *long_step = NULL; + long step; static char *kwlist[] = {"start", "step", 0}; if (!PyArg_ParseTupleAndKeywords(args, kwds, "|OO:count", @@ -3097,9 +3098,11 @@ assert(long_cnt != NULL && long_step != NULL); /* Fast mode only works when the step is 1 */ - if (!PyLong_Check(long_step) || - PyLong_AS_LONG(long_step) != 1) { - slow_mode = 1; + step = PyLong_AsLong(long_step); + if (step != 1) { + slow_mode = 1; + if (step == -1 && PyErr_Occurred()) + PyErr_Clear(); } if (slow_mode) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.1 -> 3.2): Merge 3.1
by kristjan.jonsson 30 Mar '11

30 Mar '11

http://hg.python.org/cpython/rev/50e9d4515ea2 changeset: 69063:50e9d4515ea2 branch: 3.2 parent: 69031:57e99f5f5e8f parent: 69062:9a6da9bc257d user: Kristjan Valur Jonsson <sweskman(a)gmail.com> date: Wed Mar 30 11:24:58 2011 +0000 summary: Merge 3.1 files: Lib/test/test_itertools.py | 5 +++++ Modules/itertoolsmodule.c | 9 ++++++--- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py --- a/Lib/test/test_itertools.py +++ b/Lib/test/test_itertools.py @@ -357,6 +357,8 @@ list(range(maxsize-5, maxsize+5))) self.assertEqual(list(islice(count(-maxsize-5), 10)), list(range(-maxsize-5, -maxsize+5))) + self.assertEqual(list(islice(count(10, maxsize+5), 3)), + list(range(10, 10+3*(maxsize+5), maxsize+5))) c = count(3) self.assertEqual(repr(c), 'count(3)') next(c) @@ -379,6 +381,9 @@ self.assertEqual(next(copy.deepcopy(c)), value) self.assertEqual(next(pickle.loads(pickle.dumps(c))), value) + #check proper internal error handling for large "step' sizes + count(1, maxsize+5); sys.exc_info() + def test_count_with_stride(self): self.assertEqual(lzip('abc',count(2,3)), [('a', 2), ('b', 5), ('c', 8)]) self.assertEqual(lzip('abc',count(start=2,step=3)), diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c --- a/Modules/itertoolsmodule.c +++ b/Modules/itertoolsmodule.c @@ -3050,6 +3050,7 @@ Py_ssize_t cnt = 0; PyObject *long_cnt = NULL; PyObject *long_step = NULL; + long step; static char *kwlist[] = {"start", "step", 0}; if (!PyArg_ParseTupleAndKeywords(args, kwds, "|OO:count", @@ -3087,9 +3088,11 @@ assert(long_cnt != NULL && long_step != NULL); /* Fast mode only works when the step is 1 */ - if (!PyLong_Check(long_step) || - PyLong_AS_LONG(long_step) != 1) { - slow_mode = 1; + step = PyLong_AsLong(long_step); + if (step != 1) { + slow_mode = 1; + if (step == -1 && PyErr_Occurred()) + PyErr_Clear(); } if (slow_mode) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.1): Bugfix: Properly test for errors from PyLong_AsLong() in itertools.cycle.
by kristjan.jonsson 30 Mar '11

30 Mar '11

http://hg.python.org/cpython/rev/9a6da9bc257d changeset: 69062:9a6da9bc257d branch: 3.1 parent: 69027:8517e1f5d927 user: Kristjan Valur Jonsson <sweskman(a)gmail.com> date: Wed Mar 30 11:04:28 2011 +0000 summary: Bugfix: Properly test for errors from PyLong_AsLong() in itertools.cycle. ti can raise an exception even if PyLong_Check() has succeeded. files: Lib/test/test_itertools.py | 5 +++++ Modules/itertoolsmodule.c | 9 ++++++--- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py --- a/Lib/test/test_itertools.py +++ b/Lib/test/test_itertools.py @@ -339,6 +339,8 @@ list(range(maxsize-5, maxsize+5))) self.assertEqual(list(islice(count(-maxsize-5), 10)), list(range(-maxsize-5, -maxsize+5))) + self.assertEqual(list(islice(count(10, maxsize+5), 3)), + list(range(10, 10+3*(maxsize+5), maxsize+5))) c = count(3) self.assertEqual(repr(c), 'count(3)') next(c) @@ -361,6 +363,9 @@ self.assertEqual(next(copy.deepcopy(c)), value) self.assertEqual(next(pickle.loads(pickle.dumps(c))), value) + #check proper internal error handling for large "step' sizes + count(1, maxsize+5); sys.exc_info() + def test_count_with_stride(self): self.assertEqual(lzip('abc',count(2,3)), [('a', 2), ('b', 5), ('c', 8)]) self.assertEqual(lzip('abc',count(start=2,step=3)), diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c --- a/Modules/itertoolsmodule.c +++ b/Modules/itertoolsmodule.c @@ -2918,6 +2918,7 @@ Py_ssize_t cnt = 0; PyObject *long_cnt = NULL; PyObject *long_step = NULL; + long step; static char *kwlist[] = {"start", "step", 0}; if (!PyArg_ParseTupleAndKeywords(args, kwds, "|OO:count", @@ -2955,9 +2956,11 @@ assert(long_cnt != NULL && long_step != NULL); /* Fast mode only works when the step is 1 */ - if (!PyLong_Check(long_step) || - PyLong_AS_LONG(long_step) != 1) { - slow_mode = 1; + step = PyLong_AsLong(long_step); + if (step != 1) { + slow_mode = 1; + if (step == -1 && PyErr_Occurred()) + PyErr_Clear(); } if (slow_mode) -- Repository URL: http://hg.python.org/cpython

1 0

Daily reference leaks (80e71592b114): sum=0
by solipsis＠pitrou.net 30 Mar '11

30 Mar '11

results for 80e71592b114 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog5D_2pX', '-x']

1 0

cpython (merge 3.2 -> default): Issue 11713: clarify docstring for collections.deque()
by raymond.hettinger 30 Mar '11

30 Mar '11

http://hg.python.org/cpython/rev/80e71592b114 changeset: 69061:80e71592b114 parent: 69055:c8701b9256cf parent: 69060:075637897403 user: Raymond Hettinger <python(a)rcn.com> date: Tue Mar 29 17:38:15 2011 -0700 summary: Issue 11713: clarify docstring for collections.deque() files: Modules/_collectionsmodule.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -1002,7 +1002,7 @@ PyDoc_STRVAR(deque_doc, "deque(iterable[, maxlen]) --> deque object\n\ \n\ -Build an ordered collection accessible from endpoints only."); +Build an ordered collection with optimized access from its endpoints."); static PyTypeObject deque_type = { PyVarObject_HEAD_INIT(NULL, 0) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue 11713: clarify docstring for collections.deque()
by raymond.hettinger 30 Mar '11

30 Mar '11

http://hg.python.org/cpython/rev/075637897403 changeset: 69060:075637897403 branch: 3.2 parent: 69054:968bca2cab60 user: Raymond Hettinger <python(a)rcn.com> date: Tue Mar 29 17:36:31 2011 -0700 summary: Issue 11713: clarify docstring for collections.deque() files: Modules/_collectionsmodule.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -1002,7 +1002,7 @@ PyDoc_STRVAR(deque_doc, "deque(iterable[, maxlen]) --> deque object\n\ \n\ -Build an ordered collection accessible from endpoints only."); +Build an ordered collection with optimized access from its endpoints."); static PyTypeObject deque_type = { PyVarObject_HEAD_INIT(NULL, 0) -- Repository URL: http://hg.python.org/cpython

1 0