Mailman 3 August 2014 - Python-Dev

os.walk() is going to be *fast* with scandir
by Ben Hoyt Aug. 19, 2014

Aug. 19, 2014

Just thought I'd share some of my excitement about how fast the all-C version [1] of os.scandir() is turning out to be. Below are the results of my scandir / walk benchmark run with three different versions. I'm using an SSD, which seems to make it especially faster than listdir / walk. Note that benchmark results can vary a lot, depending on operating system, file system, hard drive type, and the OS's caching state. Anyway, os.walk() can be FIFTY times as fast using os.scandir(). # Old … [View More]

8 9

PEP 467: Minor API improvements for bytes & bytearray
by Nick Coghlan Aug. 18, 2014

Aug. 18, 2014

I just posted an updated version of PEP 467 after recently finishing the updates to the Python 3.4+ binary sequence docs to decouple them from the str docs. Key points in the proposal: * deprecate passing integers to bytes() and bytearray() * add bytes.zeros() and bytearray.zeros() as a replacement * add bytes.byte() and bytearray.byte() as counterparts to ord() for binary data * add bytes.iterbytes(), bytearray.iterbytes() and memoryview.iterbytes() As far as I am aware, that last item … [View More]poses the only open question, with the alternative being to add an "iterbytes" builtin with a definition along the lines of the following: def iterbytes(data): try: getiter = type(data).__iterbytes__ except AttributeError: iter = map(bytes.byte, data) else: iter = getiter(data) return iter Regards, Nick. PEP URL: http://www.python.org/dev/peps/pep-0467/ Full PEP text: ============================= PEP: 467 Title: Minor API improvements for bytes and bytearray Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2014-03-30 Python-Version: 3.5 Post-History: 2014-03-30 2014-08-15 Abstract ======== During the initial development of the Python 3 language specification, the core ``bytes`` type for arbitrary binary data started as the mutable type that is now referred to as ``bytearray``. Other aspects of operating in the binary domain in Python have also evolved over the course of the Python 3 series. This PEP proposes a number of small adjustments to the APIs of the ``bytes`` and ``bytearray`` types to make it easier to operate entirely in the binary domain. Background ========== To simplify the task of writing the Python 3 documentation, the ``bytes`` and ``bytearray`` types were documented primarily in terms of the way they differed from the Unicode based Python 3 ``str`` type. Even when I `heavily revised the sequence documentation <http://hg.python.org/cpython/rev/463f52d20314>`__ in 2012, I retained that simplifying shortcut. However, it turns out that this approach to the documentation of these types had a problem: it doesn't adequately introduce users to their hybrid nature, where they can be manipulated *either* as a "sequence of integers" type, *or* as ``str``-like types that assume ASCII compatible data. That oversight has now been corrected, with the binary sequence types now being documented entirely independently of the ``str`` documentation in `Python 3.4+ <https://docs.python.org/3/library/stdtypes.html#binary-sequence-types-bytes…>`__ The confusion isn't just a documentation issue, however, as there are also some lingering design quirks from an earlier pre-release design where there was *no* separate ``bytearray`` type, and instead the core ``bytes`` type was mutable (with no immutable counterpart). Finally, additional experience with using the existing Python 3 binary sequence types in real world applications has suggested it would be beneficial to make it easier to convert integers to length 1 bytes objects. Proposals ========= As a "consistency improvement" proposal, this PEP is actually about a few smaller micro-proposals, each aimed at improving the usability of the binary data model in Python 3. Proposals are motivated by one of two main factors: * removing remnants of the original design of ``bytes`` as a mutable type * allowing users to easily convert integer values to a length 1 ``bytes`` object Alternate Constructors ---------------------- The ``bytes`` and ``bytearray`` constructors currently accept an integer argument, but interpret it to mean a zero-filled object of the given length. This is a legacy of the original design of ``bytes`` as a mutable type, rather than a particularly intuitive behaviour for users. It has become especially confusing now that some other ``bytes`` interfaces treat integers and the corresponding length 1 bytes instances as equivalent input. Compare:: >>> b"\x03" in bytes([1, 2, 3]) True >>> 3 in bytes([1, 2, 3]) True >>> bytes(b"\x03") b'\x03' >>> bytes(3) b'\x00\x00\x00' This PEP proposes that the current handling of integers in the bytes and bytearray constructors by deprecated in Python 3.5 and targeted for removal in Python 3.7, being replaced by two more explicit alternate constructors provided as class methods. The initial python-ideas thread [ideas-thread1]_ that spawned this PEP was specifically aimed at deprecating this constructor behaviour. Firstly, a ``byte`` constructor is proposed that converts integers in the range 0 to 255 (inclusive) to a ``bytes`` object:: >>> bytes.byte(3) b'\x03' >>> bytearray.byte(3) bytearray(b'\x03') >>> bytes.byte(512) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: bytes must be in range(0, 256) One specific use case for this alternate constructor is to easily convert the result of indexing operations on ``bytes`` and other binary sequences from an integer to a ``bytes`` object. The documentation for this API should note that its counterpart for the reverse conversion is ``ord()``. The ``ord()`` documentation will also be updated to note that while ``chr()`` is the counterpart for ``str`` input, ``bytes.byte`` and ``bytearray.byte`` are the counterparts for binary input. Secondly, a ``zeros`` constructor is proposed that serves as a direct replacement for the current constructor behaviour, rather than having to use sequence repetition to achieve the same effect in a less intuitive way:: >>> bytes.zeros(3) b'\x00\x00\x00' >>> bytearray.zeros(3) bytearray(b'\x00\x00\x00') The chosen name here is taken from the corresponding initialisation function in NumPy (although, as these are sequence types rather than N-dimensional matrices, the constructors take a length as input rather than a shape tuple) While ``bytes.byte`` and ``bytearray.zeros`` are expected to be the more useful duo amongst the new constructors, ``bytes.zeros`` and `bytearray.byte`` are provided in order to maintain API consistency between the two types. Iteration --------- While iteration over ``bytes`` objects and other binary sequences produces integers, it is sometimes desirable to iterate over length 1 bytes objects instead. To handle this situation more obviously (and more efficiently) than would be the case with the ``map(bytes.byte, data)`` construct enabled by the above constructor changes, this PEP proposes the addition of a new ``iterbytes`` method to ``bytes``, ``bytearray`` and ``memoryview``:: for x in data.iterbytes(): # x is a length 1 ``bytes`` object, rather than an integer Third party types and arbitrary containers of integers that lack the new method can still be handled by combining ``map`` with the new ``bytes.byte()`` alternate constructor proposed above:: for x in map(bytes.byte, data): # x is a length 1 ``bytes`` object, rather than an integer # This works with *any* container of integers in the range # 0 to 255 inclusive Open questions ^^^^^^^^^^^^^^ * The fallback case above suggests that this could perhaps be better handled as an ``iterbytes(data)`` *builtin*, that used ``data.__iterbytes__()`` if defined, but otherwise fell back to ``map(bytes.byte, data)``:: for x in iterbytes(data): # x is a length 1 ``bytes`` object, rather than an integer # This works with *any* container of integers in the range # 0 to 255 inclusive References ========== .. [ideas-thread1] https://mail.python.org/pipermail/python-ideas/2014-March/027295.html .. [empty-buffer-issue] http://bugs.python.org/issue20895 .. [GvR-initial-feedback] https://mail.python.org/pipermail/python-ideas/2014-March/027376.html Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia [View Less]

12 24

Review needed for patch for issue #12067
by Andreas Maier Aug. 18, 2014

Aug. 18, 2014

Hello, a patch for issue #12067 (targeting Py 3.5) is available since a few weeks and is ready for review. From my perspective, it is ready for commit. Could the community please review the patch? https://bugs.python.org/issue12067 Thanks, Andy

1 0

"embedded NUL character" exceptions
by Serhiy Storchaka Aug. 17, 2014

Aug. 17, 2014

Currently most functions which accepts string argument which then passed to C function as NUL-terminated string, reject strings with embedded NUL character and raise TypeError. ValueError looks more appropriate here, because argument type is correct (str), only its value is wrong. But this is backward incompatible change. I think that we should get rid of this legacy inconsistency sooner or later. Why not fix it right now? I have opened an issue on the tracker [1], but this issue … [View More]

2 1

Re: [Python-Dev] [Python-checkins] cpython (merge 3.4 -> default): Issue #22165: Fixed test_undecodable_filename on non-UTF-8 locales.
by Senthil Kumaran Aug. 17, 2014

Aug. 17, 2014

This change is okay and not harmful. But I think, It might still not fix the encoding issue that we encountered on Mac. [localhost cpython]$ hg log -l 1 changeset: 92128:7cdc941d5180 tag: tip parent: 92126:3153a400b739 parent: 92127:a894b629bbea user: Serhiy Storchaka <storchaka(a)gmail.com> date: Sun Aug 17 12:21:06 2014 +0300 description: Issue #22165: Fixed test_undecodable_filename on non-UTF-8 locales. [localhost cpython]$ ./python.exe -m test.… [View More]regrtest test_httpservers [1/1] test_httpservers test test_httpservers failed -- Traceback (most recent call last): File "/Users/skumaran/python/cpython/Lib/test/test_httpservers.py", line 283, in test_undecodable_filename .encode(enc, 'surrogateescape'), body) AssertionError: b'href="%40test_5809_tmp%ED%B3%A7w%ED%B3%B0.txt"' not found in b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" " http://www.w3.org/TR/html4/strict.dtd">\n<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html; charset=utf-8">\n<title>Directory listing for tmpj54lc8m1/</title>\n</head>\n<body>\n<h1>Directory listing for tmpj54lc8m1/</h1>\n<hr>\n<ul>\n<li><a href="%40test_5809_tmp%25E7w%25F0.txt">@test_5809_tmp%E7w%F0.txt</a></li>\n<li><a href="test">test</a></li>\n</ul>\n<hr>\n</body>\n</html>\n' 1 test failed: test_httpservers The underlying problem seems to be difference in which os.listdir() which uses C-API and os.fsdecode represent the decoded chars. Ref: http://bugs.python.org/issue22165#msg225428 On Sun, Aug 17, 2014 at 2:52 PM, serhiy.storchaka < python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/7cdc941d5180 > changeset: 92128:7cdc941d5180 > parent: 92126:3153a400b739 > parent: 92127:a894b629bbea > user: Serhiy Storchaka <storchaka(a)gmail.com> > date: Sun Aug 17 12:21:06 2014 +0300 > summary: > Issue #22165: Fixed test_undecodable_filename on non-UTF-8 locales. > > files: > Lib/test/test_httpservers.py | 5 +++-- > 1 files changed, 3 insertions(+), 2 deletions(-) > > > diff --git a/Lib/test/test_httpservers.py b/Lib/test/test_httpservers.py > --- a/Lib/test/test_httpservers.py > +++ b/Lib/test/test_httpservers.py > @@ -272,6 +272,7 @@ > @unittest.skipUnless(support.TESTFN_UNDECODABLE, > 'need support.TESTFN_UNDECODABLE') > def test_undecodable_filename(self): > + enc = sys.getfilesystemencoding() > filename = os.fsdecode(support.TESTFN_UNDECODABLE) + '.txt' > with open(os.path.join(self.tempdir, filename), 'wb') as f: > f.write(support.TESTFN_UNDECODABLE) > @@ -279,9 +280,9 @@ > body = self.check_status_and_reason(response, 200) > quotedname = urllib.parse.quote(filename, errors='surrogatepass') > self.assertIn(('href="%s"' % quotedname) > - .encode('utf-8', 'surrogateescape'), body) > + .encode(enc, 'surrogateescape'), body) > self.assertIn(('>%s<' % html.escape(filename)) > - .encode('utf-8', 'surrogateescape'), body) > + .encode(enc, 'surrogateescape'), body) > response = self.request(self.tempdir_name + '/' + quotedname) > self.check_status_and_reason(response, 200, > data=support.TESTFN_UNDECODABLE) > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > https://mail.python.org/mailman/listinfo/python-checkins > > [View Less]

1 0

Multiline with statement line continuation
by Allen Li Aug. 17, 2014

Aug. 17, 2014

This is a problem I sometimes run into when working with a lot of files simultaneously, where I need three or more `with` statements: with open('foo') as foo: with open('bar') as bar: with open('baz') as baz: pass Thankfully, support for multiple items was added in 3.1: with open('foo') as foo, open('bar') as bar, open('baz') as baz: pass However, this begs the need for a multiline form, especially when working with three or more items: … [View More]

15 28

Re: [Python-Dev] [Python-checkins] cpython (2.7): Fix Issue #8797: Raise HTTPError on failed Basic Authentication immediately.
by Senthil Kumaran Aug. 16, 2014

Aug. 16, 2014

I added some extra coverage for basic auth in the tests and I notice that in buildbots, some of them are throwing "error: [Errno 32] Broken pipe" error. I am looking into this and will fix this. Thanks, Senthil On Sat, Aug 16, 2014 at 2:19 PM, senthil.kumaran <python-checkins(a)python.org > wrote: > http://hg.python.org/cpython/rev/e0510a3bdf8f > changeset: 92111:e0510a3bdf8f > branch: 2.7 > parent: 92097:6d41f139709b > user: Senthil Kumaran <… [View More]senthil(a)uthcode.com> > date: Sat Aug 16 14:16:14 2014 +0530 > summary: > Fix Issue #8797: Raise HTTPError on failed Basic Authentication > immediately. Initial patch by Sam Bull. > > files: > Lib/test/test_urllib2_localnet.py | 86 ++++++++++++++++++- > Lib/urllib2.py | 19 +--- > Misc/NEWS | 3 + > 3 files changed, 90 insertions(+), 18 deletions(-) > > > diff --git a/Lib/test/test_urllib2_localnet.py > b/Lib/test/test_urllib2_localnet.py > --- a/Lib/test/test_urllib2_localnet.py > +++ b/Lib/test/test_urllib2_localnet.py > @@ -1,6 +1,8 @@ > +import base64 > import urlparse > import urllib2 > import BaseHTTPServer > +import SimpleHTTPServer > import unittest > import hashlib > > @@ -66,6 +68,48 @@ > > # Authentication infrastructure > > + > +class BasicAuthHandler(SimpleHTTPServer.SimpleHTTPRequestHandler): > + """Handler for performing Basic Authentication.""" > + # Server side values > + USER = "testUser" > + PASSWD = "testPass" > + REALM = "Test" > + USER_PASSWD = "%s:%s" % (USER, PASSWD) > + ENCODED_AUTH = base64.b64encode(USER_PASSWD) > + > + def __init__(self, *args, **kwargs): > + SimpleHTTPServer.SimpleHTTPRequestHandler.__init__(self, *args, > + **kwargs) > + > + def log_message(self, format, *args): > + # Supress the HTTP Console log output > + pass > + > + def do_HEAD(self): > + self.send_response(200) > + self.send_header("Content-type", "text/html") > + self.end_headers() > + > + def do_AUTHHEAD(self): > + self.send_response(401) > + self.send_header("WWW-Authenticate", "Basic realm=\"%s\"" % > self.REALM) > + self.send_header("Content-type", "text/html") > + self.end_headers() > + > + def do_GET(self): > + if self.headers.getheader("Authorization") == None: > + self.do_AUTHHEAD() > + self.wfile.write("No Auth Header Received") > + elif self.headers.getheader( > + "Authorization") == "Basic " + self.ENCODED_AUTH: > + SimpleHTTPServer.SimpleHTTPRequestHandler.do_GET(self) > + else: > + self.do_AUTHHEAD() > + self.wfile.write(self.headers.getheader("Authorization")) > + self.wfile.write("Not Authenticated") > + > + > class DigestAuthHandler: > """Handler for performing digest authentication.""" > > @@ -228,6 +272,45 @@ > test_support.threading_cleanup(*self._threads) > > > +class BasicAuthTests(BaseTestCase): > + USER = "testUser" > + PASSWD = "testPass" > + INCORRECT_PASSWD = "Incorrect" > + REALM = "Test" > + > + def setUp(self): > + super(BasicAuthTests, self).setUp() > + # With Basic Authentication > + def http_server_with_basic_auth_handler(*args, **kwargs): > + return BasicAuthHandler(*args, **kwargs) > + self.server = > LoopbackHttpServerThread(http_server_with_basic_auth_handler) > + self.server_url = 'http://127.0.0.1:%s' % self.server.port > + self.server.start() > + self.server.ready.wait() > + > + def tearDown(self): > + self.server.stop() > + super(BasicAuthTests, self).tearDown() > + > + def test_basic_auth_success(self): > + ah = urllib2.HTTPBasicAuthHandler() > + ah.add_password(self.REALM, self.server_url, self.USER, > self.PASSWD) > + urllib2.install_opener(urllib2.build_opener(ah)) > + try: > + self.assertTrue(urllib2.urlopen(self.server_url)) > + except urllib2.HTTPError: > + self.fail("Basic Auth Failed for url: %s" % self.server_url) > + except Exception as e: > + raise e > + > + def test_basic_auth_httperror(self): > + ah = urllib2.HTTPBasicAuthHandler() > + ah.add_password(self.REALM, self.server_url, self.USER, > + self.INCORRECT_PASSWD) > + urllib2.install_opener(urllib2.build_opener(ah)) > + self.assertRaises(urllib2.HTTPError, urllib2.urlopen, > self.server_url) > + > + > class ProxyAuthTests(BaseTestCase): > URL = "http://localhost" > > @@ -240,6 +323,7 @@ > self.digest_auth_handler = DigestAuthHandler() > self.digest_auth_handler.set_users({self.USER: self.PASSWD}) > self.digest_auth_handler.set_realm(self.REALM) > + # With Digest Authentication > def create_fake_proxy_handler(*args, **kwargs): > return FakeProxyHandler(self.digest_auth_handler, *args, > **kwargs) > > @@ -544,7 +628,7 @@ > # the next line. > #test_support.requires("network") > > - test_support.run_unittest(ProxyAuthTests, TestUrlopen) > + test_support.run_unittest(BasicAuthTests, ProxyAuthTests, TestUrlopen) > > if __name__ == "__main__": > test_main() > diff --git a/Lib/urllib2.py b/Lib/urllib2.py > --- a/Lib/urllib2.py > +++ b/Lib/urllib2.py > @@ -843,10 +843,7 @@ > password_mgr = HTTPPasswordMgr() > self.passwd = password_mgr > self.add_password = self.passwd.add_password > - self.retried = 0 > > - def reset_retry_count(self): > - self.retried = 0 > > def http_error_auth_reqed(self, authreq, host, req, headers): > # host may be an authority (without userinfo) or a URL with an > @@ -854,13 +851,6 @@ > # XXX could be multiple headers > authreq = headers.get(authreq, None) > > - if self.retried > 5: > - # retry sending the username:password 5 times before failing. > - raise HTTPError(req.get_full_url(), 401, "basic auth failed", > - headers, None) > - else: > - self.retried += 1 > - > if authreq: > mo = AbstractBasicAuthHandler.rx.search(authreq) > if mo: > @@ -869,17 +859,14 @@ > warnings.warn("Basic Auth Realm was unquoted", > UserWarning, 2) > if scheme.lower() == 'basic': > - response = self.retry_http_basic_auth(host, req, > realm) > - if response and response.code != 401: > - self.retried = 0 > - return response > + return self.retry_http_basic_auth(host, req, realm) > > def retry_http_basic_auth(self, host, req, realm): > user, pw = self.passwd.find_user_password(realm, host) > if pw is not None: > raw = "%s:%s" % (user, pw) > auth = 'Basic %s' % base64.b64encode(raw).strip() > - if req.headers.get(self.auth_header, None) == auth: > + if req.get_header(self.auth_header, None) == auth: > return None > req.add_unredirected_header(self.auth_header, auth) > return self.parent.open(req, timeout=req.timeout) > @@ -895,7 +882,6 @@ > url = req.get_full_url() > response = self.http_error_auth_reqed('www-authenticate', > url, req, headers) > - self.reset_retry_count() > return response > > > @@ -911,7 +897,6 @@ > authority = req.get_host() > response = self.http_error_auth_reqed('proxy-authenticate', > authority, req, headers) > - self.reset_retry_count() > return response > > > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -19,6 +19,9 @@ > Library > ------- > > +- Issue #8797: Raise HTTPError on failed Basic Authentication immediately. > + Initial patch by Sam Bull. > + > - Issue #21448: Changed FeedParser feed() to avoid O(N**2) behavior when > parsing long line. Original patch by Raymond Hettinger. > > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > https://mail.python.org/mailman/listinfo/python-checkins > > [View Less]

1 0

Summary of Python tracker Issues
by Python tracker Aug. 15, 2014

Aug. 15, 2014

ACTIVITY SUMMARY (2014-08-08 - 2014-08-15) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4602 ( +0) closed 29371 (+31) total 33973 (+31) Open issues with patches: 2175 Issues opened (23) ================== #21166: Bus error in pybuilddir.txt 'python -m sysconfigure --generate http://bugs.python.org/issue21166 reopened by ned.deily #22176: update … [View More]

1 0

Documenting enum types
by Serhiy Storchaka Aug. 15, 2014

Aug. 15, 2014

Should new enum types added recently to collect module constants be documented at all? For example AddressFamily is absent in socket.__all__ [1]. [1] http://bugs.python.org/issue20689

7 6

Reviving restricted mode?
by Guido van Rossum Aug. 14, 2014

Aug. 14, 2014

I've received some enthusiastic emails from someone who wants to revive restricted mode. He started out with a bunch of patches to the CPython runtime using ctypes, which he attached to an App Engine bug: http://code.google.com/p/googleappengine/issues/detail?id=671 Based on his code (the file secure.py is all you need, included in secure.tar.gz) it seems he believes the only security leaks are __subclasses__, gi_frame and gi_code. (I have since convinced him that if we add "restricted" … [View More]

19 35