Mailman 3 October 2011 - Python-checkins

Re: [Python-checkins] cpython: pyexpat uses the new Unicode API
by Amaury Forgeot d'Arc July 29, 2012

July 29, 2012

> changeset: 72548:a1be34457ccf > user: Victor Stinner <victor.stinner at haypocalc.com> > date: Sat Oct 01 01:05:40 2011 +0200 > summary: > pyexat uses the new Unicode API > > files: > Modules/pyexpat.c | 12 +++++++----- > 1 files changed, 7 insertions(+), 5 deletions(-) > > > diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c > --- a/Modules/pyexpat.c > +++ b/Modules/pyexpat.c > @@ -1234,11 +1234,13 @@ > static PyObject * > xmlparse_getattro(xmlparseobject *self, PyObject *nameobj) > { > - const Py_UNICODE *name; > + Py_UCS4 first_char; > int handlernum = -1; > > if (!PyUnicode_Check(nameobj)) > goto generic; > + if (PyUnicode_READY(nameobj)) > + return NULL; Why is this PyUnicode_READY necessary? Can tp_getattro pass unfinished unicode objects? I hope we don't have to update all extension modules? -- Amaury Forgeot d'Arc

2 1

cpython (merge 3.2 -> default): Issue #13304: Skip test case if user site-packages disabled (-s or
by ned.deily Oct. 31, 2011

Oct. 31, 2011

http://hg.python.org/cpython/rev/c343c095d08b changeset: 73268:c343c095d08b parent: 73265:3f025427f02b parent: 73267:c497011a4769 user: Ned Deily <nad(a)acm.org> date: Mon Oct 31 16:22:53 2011 -0700 summary: Issue #13304: Skip test case if user site-packages disabled (-s or PYTHONNOUSERSITE). (Patch by Carl Meyer) files: Lib/test/test_site.py | 4 +++- Misc/NEWS | 3 +++ 2 files changed, 6 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py --- a/Lib/test/test_site.py +++ b/Lib/test/test_site.py @@ -24,7 +24,7 @@ else: raise unittest.SkipTest("importation of site.py suppressed") -if not os.path.isdir(site.USER_SITE): +if site.ENABLE_USER_SITE and not os.path.isdir(site.USER_SITE): # need to add user site directory for tests os.makedirs(site.USER_SITE) site.addsitedir(site.USER_SITE) @@ -157,6 +157,8 @@ finally: pth_file.cleanup() + @unittest.skipUnless(site.ENABLE_USER_SITE, "requires access to PEP 370 " + "user-site (site.ENABLE_USER_SITE)") def test_s_option(self): usersite = site.USER_SITE self.assertIn(usersite, sys.path) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -1523,6 +1523,9 @@ Tests ----- +- Issue #13304: Skip test case if user site-packages disabled (-s or + PYTHONNOUSERSITE). (Patch by Carl Meyer) + - Issue #5661: Add a test for ECONNRESET/EPIPE handling to test_asyncore. Patch by Xavier de Gaye. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.2): Issue #13304: Skip test case if user site-packages disabled (-s or
by ned.deily Oct. 31, 2011

Oct. 31, 2011

http://hg.python.org/cpython/rev/c497011a4769 changeset: 73267:c497011a4769 branch: 3.2 parent: 73256:2ca415cbf2ac user: Ned Deily <nad(a)acm.org> date: Mon Oct 31 16:16:35 2011 -0700 summary: Issue #13304: Skip test case if user site-packages disabled (-s or PYTHONNOUSERSITE). (Patch by Carl Meyer) files: Lib/test/test_site.py | 4 +++- Misc/ACKS | 1 + Misc/NEWS | 3 +++ 3 files changed, 7 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py --- a/Lib/test/test_site.py +++ b/Lib/test/test_site.py @@ -24,7 +24,7 @@ else: raise unittest.SkipTest("importation of site.py suppressed") -if not os.path.isdir(site.USER_SITE): +if site.ENABLE_USER_SITE and not os.path.isdir(site.USER_SITE): # need to add user site directory for tests os.makedirs(site.USER_SITE) site.addsitedir(site.USER_SITE) @@ -157,6 +157,8 @@ finally: pth_file.cleanup() + @unittest.skipUnless(site.ENABLE_USER_SITE, "requires access to PEP 370 " + "user-site (site.ENABLE_USER_SITE)") def test_s_option(self): usersite = site.USER_SITE self.assertIn(usersite, sys.path) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -601,6 +601,7 @@ Ezio Melotti Brian Merrell Luke Mewburn +Carl Meyer Mike Meyer Steven Miale Trent Mick diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -187,6 +187,9 @@ Tests ----- +- Issue #13304: Skip test case if user site-packages disabled (-s or + PYTHONNOUSERSITE). (Patch by Carl Meyer) + - Issue #13218: Fix test_ssl failures on Debian/Ubuntu. - Issue #12821: Fix test_fcntl failures on OpenBSD 5. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Issue #13304: Skip test case if user site-packages disabled (-s or
by ned.deily Oct. 31, 2011

Oct. 31, 2011

http://hg.python.org/cpython/rev/1689b9cf6b1c changeset: 73266:1689b9cf6b1c branch: 2.7 parent: 73263:848210b179ab user: Ned Deily <nad(a)acm.org> date: Mon Oct 31 16:14:52 2011 -0700 summary: Issue #13304: Skip test case if user site-packages disabled (-s or PYTHONNOUSERSITE). (Patch by Carl Meyer) files: Lib/test/test_site.py | 4 +++- Misc/ACKS | 1 + Misc/NEWS | 3 +++ 3 files changed, 7 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py --- a/Lib/test/test_site.py +++ b/Lib/test/test_site.py @@ -24,7 +24,7 @@ else: raise unittest.SkipTest("importation of site.py suppressed") -if not os.path.isdir(site.USER_SITE): +if site.ENABLE_USER_SITE and not os.path.isdir(site.USER_SITE): # need to add user site directory for tests os.makedirs(site.USER_SITE) site.addsitedir(site.USER_SITE) @@ -161,6 +161,8 @@ finally: pth_file.cleanup() + @unittest.skipUnless(site.ENABLE_USER_SITE, "requires access to PEP 370 " + "user-site (site.ENABLE_USER_SITE)") def test_s_option(self): usersite = site.USER_SITE self.assertIn(usersite, sys.path) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -556,6 +556,7 @@ Ezio Melotti Brian Merrell Luke Mewburn +Carl Meyer Mike Meyer Steven Miale Trent Mick diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -328,6 +328,9 @@ Tests ----- +- Issue #13304: Skip test case if user site-packages disabled (-s or + PYTHONNOUSERSITE). (Patch by Carl Meyer) + - Issue #13218: Fix test_ssl failures on Debian/Ubuntu. - Issue #12821: Fix test_fcntl failures on OpenBSD 5. -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Fix regression due to changeset 2096158376e5 (issue #13305).
by florent.xicluna Oct. 31, 2011

Oct. 31, 2011

http://hg.python.org/cpython/rev/3f025427f02b changeset: 73265:3f025427f02b user: Florent Xicluna <florent.xicluna(a)gmail.com> date: Tue Nov 01 00:06:58 2011 +0100 summary: Fix regression due to changeset 2096158376e5 (issue #13305). files: Lib/xmlrpc/client.py | 27 +++++++++++++++++++++------ 1 files changed, 21 insertions(+), 6 deletions(-) diff --git a/Lib/xmlrpc/client.py b/Lib/xmlrpc/client.py --- a/Lib/xmlrpc/client.py +++ b/Lib/xmlrpc/client.py @@ -250,18 +250,33 @@ # Wrapper for XML-RPC DateTime values. This converts a time value to # the format used by XML-RPC. # <p> -# The value can be given as a string in the format -# "yyyymmddThh:mm:ss", as a 9-item time tuple (as returned by +# The value can be given as a datetime object, as a string in the +# format "yyyymmddThh:mm:ss", as a 9-item time tuple (as returned by # time.localtime()), or an integer value (as returned by time.time()). # The wrapper uses time.localtime() to convert an integer to a time # tuple. # -# @param value The time, given as an ISO 8601 string, a time -# tuple, or a integer time value. +# @param value The time, given as a datetime object, an ISO 8601 string, +# a time tuple, or an integer time value. + + +# Issue #13305: different format codes across platforms +_day0 = datetime(1, 1, 1) +if _day0.strftime('%Y') == '0001': # Mac OS X + def _iso8601_format(value): + return value.strftime("%Y%m%dT%H:%M:%S") +elif _day0.strftime('%4Y') == '0001': # Linux + def _iso8601_format(value): + return value.strftime("%4Y%m%dT%H:%M:%S") +else: + def _iso8601_format(value): + return value.strftime("%Y%m%dT%H:%M:%S").zfill(17) +del _day0 + def _strftime(value): if isinstance(value, datetime): - return value.strftime("%Y%m%dT%H:%M:%S") + return _iso8601_format(value) if not isinstance(value, (tuple, time.struct_time)): if value == 0: @@ -288,7 +303,7 @@ o = other.value elif isinstance(other, datetime): s = self.value - o = other.strftime("%Y%m%dT%H:%M:%S") + o = _iso8601_format(other) elif isinstance(other, str): s = self.value o = other -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #13303: Fix a race condition in the bytecode file creation.
by charles-francois.natali Oct. 31, 2011

Oct. 31, 2011

http://hg.python.org/cpython/rev/740baff4f169 changeset: 73264:740baff4f169 parent: 73261:0d64d9ac2b78 user: Charles-François Natali <neologix(a)free.fr> date: Mon Oct 31 20:47:31 2011 +0100 summary: Issue #13303: Fix a race condition in the bytecode file creation. files: Lib/importlib/_bootstrap.py | 7 +- Python/import.c | 52 ++++-------------------- 2 files changed, 13 insertions(+), 46 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -85,10 +85,11 @@ Be prepared to handle a FileExistsError if concurrent writing of the temporary file is attempted.""" if not sys.platform.startswith('win'): - # On POSIX-like platforms, renaming is atomic - path_tmp = path + '.tmp' + # On POSIX-like platforms, renaming is atomic. id() is used to generate + # a pseudo-random filename. + path_tmp = '{}.{}'.format(path, id(path)) + fd = _os.open(path_tmp, _os.O_EXCL | _os.O_CREAT | _os.O_WRONLY) try: - fd = _os.open(path_tmp, _os.O_EXCL | _os.O_CREAT | _os.O_WRONLY) with _io.FileIO(fd, 'wb') as file: file.write(data) _os.rename(path_tmp, path) diff --git a/Python/import.c b/Python/import.c --- a/Python/import.c +++ b/Python/import.c @@ -1183,42 +1183,6 @@ return co; } -/* Helper to open a bytecode file for writing in exclusive mode */ - -#ifndef MS_WINDOWS -static FILE * -open_exclusive(char *filename, mode_t mode) -{ -#if defined(O_EXCL)&&defined(O_CREAT)&&defined(O_WRONLY)&&defined(O_TRUNC) - /* Use O_EXCL to avoid a race condition when another process tries to - write the same file. When that happens, our open() call fails, - which is just fine (since it's only a cache). - XXX If the file exists and is writable but the directory is not - writable, the file will never be written. Oh well. - */ - int fd; - (void) unlink(filename); - fd = open(filename, O_EXCL|O_CREAT|O_WRONLY|O_TRUNC -#ifdef O_BINARY - |O_BINARY /* necessary for Windows */ -#endif -#ifdef __VMS - , mode, "ctxt=bin", "shr=nil" -#else - , mode -#endif - ); - if (fd < 0) - return NULL; - return fdopen(fd, "wb"); -#else - /* Best we can do -- on Windows this can't happen anyway */ - return fopen(filename, "wb"); -#endif -} -#endif - - /* Write a compiled module to a file, placing the time of last modification of its source into the header. Errors are ignored, if a write error occurs an attempt is made to @@ -1234,15 +1198,13 @@ #ifdef MS_WINDOWS /* since Windows uses different permissions */ mode_t mode = srcstat->st_mode & ~S_IEXEC; #else - mode_t mode = srcstat->st_mode & ~S_IXUSR & ~S_IXGRP & ~S_IXOTH; mode_t dirmode = (srcstat->st_mode | S_IXUSR | S_IXGRP | S_IXOTH | S_IWUSR | S_IWGRP | S_IWOTH); PyObject *dirbytes; #endif -#ifdef MS_WINDOWS int fd; -#else +#ifndef MS_WINDOWS PyObject *cpathbytes, *cpathbytes_tmp; Py_ssize_t cpathbytes_len; #endif @@ -1313,7 +1275,7 @@ return; } cpathbytes_len = PyBytes_GET_SIZE(cpathbytes); - cpathbytes_tmp = PyBytes_FromStringAndSize(NULL, cpathbytes_len + 4); + cpathbytes_tmp = PyBytes_FromStringAndSize(NULL, cpathbytes_len + 6); if (cpathbytes_tmp == NULL) { Py_DECREF(cpathbytes); PyErr_Clear(); @@ -1321,9 +1283,13 @@ } memcpy(PyBytes_AS_STRING(cpathbytes_tmp), PyBytes_AS_STRING(cpathbytes), cpathbytes_len); - memcpy(PyBytes_AS_STRING(cpathbytes_tmp) + cpathbytes_len, ".tmp", 4); - - fp = open_exclusive(PyBytes_AS_STRING(cpathbytes_tmp), mode); + memcpy(PyBytes_AS_STRING(cpathbytes_tmp) + cpathbytes_len, "XXXXXX", 6); + + fd = mkstemp(PyBytes_AS_STRING(cpathbytes_tmp)); + if (0 <= fd) + fp = fdopen(fd, "wb"); + else + fp = NULL; #endif if (fp == NULL) { if (Py_VerboseFlag) -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Backout redundant changeset 1de4d92cd6a4
by antoine.pitrou Oct. 31, 2011

Oct. 31, 2011

http://hg.python.org/cpython/rev/848210b179ab changeset: 73263:848210b179ab branch: 2.7 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Mon Oct 31 20:23:00 2011 +0100 summary: Backout redundant changeset 1de4d92cd6a4 files: Lib/test/test_ssl.py | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -981,8 +981,6 @@ @skip_if_broken_ubuntu_ssl def test_protocol_sslv2(self): """Connecting to an SSLv2 server with various client options""" - if not hasattr(ssl, 'PROTOCOL_SSLv2'): - raise unittest.SkipTest('No SSLv2 available') if test_support.verbose: sys.stdout.write("\n") if not hasattr(ssl, 'PROTOCOL_SSLv2'): -- Repository URL: http://hg.python.org/cpython

1 0

cpython (2.7): Port to 2.7 - issue 10817 - Fix urlretrieve function to raise
by senthil.kumaran Oct. 31, 2011

Oct. 31, 2011

http://hg.python.org/cpython/rev/1ef30d6429db changeset: 73262:1ef30d6429db branch: 2.7 parent: 73260:7f4aaa9c4588 user: Senthil Kumaran <senthil(a)uthcode.com> date: Tue Nov 01 02:44:45 2011 +0800 summary: Port to 2.7 - issue 10817 - Fix urlretrieve function to raise ContentTooShortError even when reporthook is None. Patch by Jyrki Pulliainen. files: Lib/test/test_urllib.py | 91 ++++++++++++++++++++++------ Lib/urllib.py | 4 +- Misc/NEWS | 3 + 3 files changed, 76 insertions(+), 22 deletions(-) diff --git a/Lib/test/test_urllib.py b/Lib/test/test_urllib.py --- a/Lib/test/test_urllib.py +++ b/Lib/test/test_urllib.py @@ -17,6 +17,36 @@ hex_repr = "0%s" % hex_repr return "%" + hex_repr + +class FakeHTTPMixin(object): + def fakehttp(self, fakedata): + class FakeSocket(StringIO.StringIO): + + def sendall(self, str): + pass + def makefile(self, *args, **kwds): + return self + + def read(self, amt=None): + if self.closed: + return "" + return StringIO.StringIO.read(self, amt) + + def readline(self, length=None): + if self.closed: + return "" + return StringIO.StringIO.readline(self, length) + + class FakeHTTPConnection(httplib.HTTPConnection): + def connect(self): + self.sock = FakeSocket(fakedata) + assert httplib.HTTP._connection_class == httplib.HTTPConnection + httplib.HTTP._connection_class = FakeHTTPConnection + + def unfakehttp(self): + httplib.HTTP._connection_class = httplib.HTTPConnection + + class urlopen_FileTests(unittest.TestCase): """Test urlopen() opening a temporary file. @@ -119,28 +149,9 @@ self.assertTrue(urllib.proxy_bypass_environment('anotherdomain.com')) -class urlopen_HttpTests(unittest.TestCase): +class urlopen_HttpTests(unittest.TestCase, FakeHTTPMixin): """Test urlopen() opening a fake http connection.""" - def fakehttp(self, fakedata): - class FakeSocket(StringIO.StringIO): - def sendall(self, str): pass - def makefile(self, mode, name): return self - def read(self, amt=None): - if self.closed: return '' - return StringIO.StringIO.read(self, amt) - def readline(self, length=None): - if self.closed: return '' - return StringIO.StringIO.readline(self, length) - class FakeHTTPConnection(httplib.HTTPConnection): - def connect(self): - self.sock = FakeSocket(fakedata) - assert httplib.HTTP._connection_class == httplib.HTTPConnection - httplib.HTTP._connection_class = FakeHTTPConnection - - def unfakehttp(self): - httplib.HTTP._connection_class = httplib.HTTPConnection - def test_read(self): self.fakehttp('Hello!') try: @@ -330,6 +341,45 @@ self.assertEqual(report[0][1], 8192) self.assertEqual(report[0][2], 8193) + +class urlretrieve_HttpTests(unittest.TestCase, FakeHTTPMixin): + """Test urllib.urlretrieve() using fake http connections""" + + def test_short_content_raises_ContentTooShortError(self): + self.fakehttp('''HTTP/1.1 200 OK +Date: Wed, 02 Jan 2008 03:03:54 GMT +Server: Apache/1.3.33 (Debian GNU/Linux) mod_ssl/2.8.22 OpenSSL/0.9.7e +Connection: close +Content-Length: 100 +Content-Type: text/html; charset=iso-8859-1 + +FF +''') + + def _reporthook(par1, par2, par3): + pass + + try: + self.assertRaises(urllib.ContentTooShortError, urllib.urlretrieve, + 'http://example.com', reporthook=_reporthook) + finally: + self.unfakehttp() + + def test_short_content_raises_ContentTooShortError_without_reporthook(self): + self.fakehttp('''HTTP/1.1 200 OK +Date: Wed, 02 Jan 2008 03:03:54 GMT +Server: Apache/1.3.33 (Debian GNU/Linux) mod_ssl/2.8.22 OpenSSL/0.9.7e +Connection: close +Content-Length: 100 +Content-Type: text/html; charset=iso-8859-1 + +FF +''') + try: + self.assertRaises(urllib.ContentTooShortError, urllib.urlretrieve, 'http://example.com/') + finally: + self.unfakehttp() + class QuotingTests(unittest.TestCase): """Tests for urllib.quote() and urllib.quote_plus() @@ -774,6 +824,7 @@ urlopen_FileTests, urlopen_HttpTests, urlretrieve_FileTests, + urlretrieve_HttpTests, ProxyTests, QuotingTests, UnquotingTests, diff --git a/Lib/urllib.py b/Lib/urllib.py --- a/Lib/urllib.py +++ b/Lib/urllib.py @@ -257,9 +257,9 @@ size = -1 read = 0 blocknum = 0 + if "content-length" in headers: + size = int(headers["Content-Length"]) if reporthook: - if "content-length" in headers: - size = int(headers["Content-Length"]) reporthook(blocknum, bs, size) while 1: block = fp.read(bs) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -74,6 +74,9 @@ Library ------- +- Issue 10817: Fix urlretrieve function to raise ContentTooShortError even + when reporthook is None. Patch by Jyrki Pulliainen. + - Issue 13296: Fix IDLE to clear compile __future__ flags on shell restart. (Patch by Roger Serwy) -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #12797: Added custom opener parameter to builtin open() and FileIO.open().
by ross.lagerwall Oct. 31, 2011

Oct. 31, 2011

http://hg.python.org/cpython/rev/0d64d9ac2b78 changeset: 73261:0d64d9ac2b78 parent: 73257:e3e5b6f03f79 user: Ross Lagerwall <rosslagerwall(a)gmail.com> date: Mon Oct 31 20:34:46 2011 +0200 summary: Issue #12797: Added custom opener parameter to builtin open() and FileIO.open(). files: Doc/library/functions.rst | 11 +++++- Doc/library/io.rst | 11 +++++- Lib/_pyio.py | 10 ++++- Lib/test/test_io.py | 9 ++++ Misc/NEWS | 3 + Modules/_io/_iomodule.c | 18 ++++++--- Modules/_io/fileio.c | 49 ++++++++++++++++++++------ 7 files changed, 89 insertions(+), 22 deletions(-) diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -776,7 +776,7 @@ :meth:`__index__` method that returns an integer. -.. function:: open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True) +.. function:: open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None) Open *file* and return a corresponding stream. If the file cannot be opened, an :exc:`OSError` is raised. @@ -883,6 +883,15 @@ closed. If a filename is given *closefd* has no effect and must be ``True`` (the default). + A custom opener can be used by passing a callable as *opener*. The underlying + file descriptor for the file object is then obtained by calling *opener* with + (*file*, *flags*). *opener* must return an open file descriptor (passing + :mod:`os.open` as *opener* results in functionality similar to passing + ``None``). + + .. versionchanged:: 3.3 + The *opener* parameter was added. + The type of file object returned by the :func:`open` function depends on the mode. When :func:`open` is used to open a file in a text mode (``'w'``, ``'r'``, ``'wt'``, ``'rt'``, etc.), it returns a subclass of diff --git a/Doc/library/io.rst b/Doc/library/io.rst --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -458,7 +458,7 @@ Raw File I/O ^^^^^^^^^^^^ -.. class:: FileIO(name, mode='r', closefd=True) +.. class:: FileIO(name, mode='r', closefd=True, opener=None) :class:`FileIO` represents an OS-level file containing bytes data. It implements the :class:`RawIOBase` interface (and therefore the @@ -479,6 +479,15 @@ The :meth:`read` (when called with a positive argument), :meth:`readinto` and :meth:`write` methods on this class will only make one system call. + A custom opener can be used by passing a callable as *opener*. The underlying + file descriptor for the file object is then obtained by calling *opener* with + (*name*, *flags*). *opener* must return an open file descriptor (passing + :mod:`os.open` as *opener* results in functionality similar to passing + ``None``). + + .. versionchanged:: 3.3 + The *opener* parameter was added. + In addition to the attributes and methods from :class:`IOBase` and :class:`RawIOBase`, :class:`FileIO` provides the following data attributes and methods: diff --git a/Lib/_pyio.py b/Lib/_pyio.py --- a/Lib/_pyio.py +++ b/Lib/_pyio.py @@ -27,7 +27,7 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None, - newline=None, closefd=True): + newline=None, closefd=True, opener=None): r"""Open file and return a stream. Raise IOError upon failure. @@ -122,6 +122,12 @@ be kept open when the file is closed. This does not work when a file name is given and must be True in that case. + A custom opener can be used by passing a callable as *opener*. The + underlying file descriptor for the file object is then obtained by calling + *opener* with (*file*, *flags*). *opener* must return an open file + descriptor (passing os.open as *opener* results in functionality similar to + passing None). + open() returns a file object whose type depends on the mode, and through which the standard file operations such as reading and writing are performed. When open() is used to open a file in a text mode ('w', @@ -176,7 +182,7 @@ (writing and "w" or "") + (appending and "a" or "") + (updating and "+" or ""), - closefd) + closefd, opener=opener) line_buffering = False if buffering == 1 or buffering < 0 and raw.isatty(): buffering = -1 diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -621,6 +621,15 @@ for obj in test: self.assertTrue(hasattr(obj, "__dict__")) + def test_opener(self): + with self.open(support.TESTFN, "w") as f: + f.write("egg\n") + fd = os.open(support.TESTFN, os.O_RDONLY) + def opener(path, flags): + return fd + with self.open("non-existent", "r", opener=opener) as f: + self.assertEqual(f.read(), "egg\n") + class CIOTest(IOTest): def test_IOBase_finalize(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #12797: Added custom opener parameter to builtin open() and + FileIO.open(). + - Issue #10519: Avoid unnecessary recursive function calls in setobject.c. diff --git a/Modules/_io/_iomodule.c b/Modules/_io/_iomodule.c --- a/Modules/_io/_iomodule.c +++ b/Modules/_io/_iomodule.c @@ -95,7 +95,7 @@ */ PyDoc_STRVAR(open_doc, "open(file, mode='r', buffering=-1, encoding=None,\n" -" errors=None, newline=None, closefd=True) -> file object\n" +" errors=None, newline=None, closefd=True, opener=None) -> file object\n" "\n" "Open file and return a stream. Raise IOError upon failure.\n" "\n" @@ -190,6 +190,12 @@ "when the file is closed. This does not work when a file name is given\n" "and must be True in that case.\n" "\n" +"A custom opener can be used by passing a callable as *opener*. The\n" +"underlying file descriptor for the file object is then obtained by\n" +"calling *opener* with (*file*, *flags*). *opener* must return an open\n" +"file descriptor (passing os.open as *opener* results in functionality\n" +"similar to passing None).\n" +"\n" "open() returns a file object whose type depends on the mode, and\n" "through which the standard file operations such as reading and writing\n" "are performed. When open() is used to open a file in a text mode ('w',\n" @@ -210,8 +216,8 @@ { char *kwlist[] = {"file", "mode", "buffering", "encoding", "errors", "newline", - "closefd", NULL}; - PyObject *file; + "closefd", "opener", NULL}; + PyObject *file, *opener = Py_None; char *mode = "r"; int buffering = -1, closefd = 1; char *encoding = NULL, *errors = NULL, *newline = NULL; @@ -228,10 +234,10 @@ _Py_IDENTIFIER(isatty); _Py_IDENTIFIER(fileno); - if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|sizzzi:open", kwlist, + if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|sizzziO:open", kwlist, &file, &mode, &buffering, &encoding, &errors, &newline, - &closefd)) { + &closefd, &opener)) { return NULL; } @@ -331,7 +337,7 @@ /* Create the Raw file stream */ raw = PyObject_CallFunction((PyObject *)&PyFileIO_Type, - "Osi", file, rawmode, closefd); + "OsiO", file, rawmode, closefd, opener); if (raw == NULL) return NULL; diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c --- a/Modules/_io/fileio.c +++ b/Modules/_io/fileio.c @@ -212,9 +212,9 @@ fileio_init(PyObject *oself, PyObject *args, PyObject *kwds) { fileio *self = (fileio *) oself; - static char *kwlist[] = {"file", "mode", "closefd", NULL}; + static char *kwlist[] = {"file", "mode", "closefd", "opener", NULL}; const char *name = NULL; - PyObject *nameobj, *stringobj = NULL; + PyObject *nameobj, *stringobj = NULL, *opener = Py_None; char *mode = "r"; char *s; #ifdef MS_WINDOWS @@ -233,8 +233,9 @@ return -1; } - if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|si:fileio", - kwlist, &nameobj, &mode, &closefd)) + if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|siO:fileio", + kwlist, &nameobj, &mode, &closefd, + &opener)) return -1; if (PyFloat_Check(nameobj)) { @@ -363,15 +364,35 @@ goto error; } - Py_BEGIN_ALLOW_THREADS errno = 0; + if (opener == Py_None) { + Py_BEGIN_ALLOW_THREADS #ifdef MS_WINDOWS - if (widename != NULL) - self->fd = _wopen(widename, flags, 0666); - else + if (widename != NULL) + self->fd = _wopen(widename, flags, 0666); + else #endif - self->fd = open(name, flags, 0666); - Py_END_ALLOW_THREADS + self->fd = open(name, flags, 0666); + Py_END_ALLOW_THREADS + } else { + PyObject *fdobj = PyObject_CallFunction( + opener, "Oi", nameobj, flags); + if (fdobj == NULL) + goto error; + if (!PyLong_Check(fdobj)) { + Py_DECREF(fdobj); + PyErr_SetString(PyExc_TypeError, + "expected integer from opener"); + goto error; + } + + self->fd = PyLong_AsLong(fdobj); + Py_DECREF(fdobj); + if (self->fd == -1) { + goto error; + } + } + if (self->fd < 0) { #ifdef MS_WINDOWS if (widename != NULL) @@ -1017,13 +1038,17 @@ PyDoc_STRVAR(fileio_doc, -"file(name: str[, mode: str]) -> file IO object\n" +"file(name: str[, mode: str][, opener: None]) -> file IO object\n" "\n" "Open a file. The mode can be 'r', 'w' or 'a' for reading (default),\n" "writing or appending. The file will be created if it doesn't exist\n" "when opened for writing or appending; it will be truncated when\n" "opened for writing. Add a '+' to the mode to allow simultaneous\n" -"reading and writing."); +"reading and writing. A custom opener can be used by passing a\n" +"callable as *opener*. The underlying file descriptor for the file\n" +"object is then obtained by calling opener with (*name*, *flags*).\n" +"*opener* must return an open file descriptor (passing os.open as\n" +"*opener* results in functionality similar to passing None)."); PyDoc_STRVAR(read_doc, "read(size: int) -> bytes. read at most size bytes, returned as bytes.\n" -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 2.7 -> 2.7): merge heads
by sandro.tosi Oct. 31, 2011

Oct. 31, 2011

http://hg.python.org/cpython/rev/7f4aaa9c4588 changeset: 73260:7f4aaa9c4588 branch: 2.7 parent: 73258:1de4d92cd6a4 parent: 73259:3a126a459b6f user: Sandro Tosi <sandro.tosi(a)gmail.com> date: Mon Oct 31 19:23:25 2011 +0100 summary: merge heads files: Doc/reference/expressions.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -347,7 +347,7 @@ quotes: .. productionlist:: - string_conversion: "'" `expression_list` "'" + string_conversion: "`" `expression_list` "`" A string conversion evaluates the contained expression list and converts the resulting object into a string according to rules specific to its type. -- Repository URL: http://hg.python.org/cpython

1 0