Mailman 3 March 2013 - Python-Dev

IDLE in the stdlib
by Kurt B. Kaiser 20 Mar '13

20 Mar '13

I apologize for that formatting mess - the Barracuda rejected my original email for some reason (squirrelmail directly from shore.net) and I had to resend it from rejects - it got re-wrapped on transmission. -- KBK

1 0

Recent changes to TextIOWrapper and its tests
by Jeff Allen 20 Mar '13

20 Mar '13

I'm pulling recent changes in the io module across to Jython. I am looking for help understanding the changes in http://hg.python.org/cpython/rev/19a33ef3821d That change set is about what should happen if the underlying buffer does not return bytes when read, but instead, for example, unicode characters. The test test_read_nonbytes() constructs a pathological text stream reader t where the usual BytesIO or BufferedReader is replaced with a StringIO. It then checks that r.read(1) and t.readlines() raise a TypeError, that is, it tests that TextIOWrapper checks the type of what it reads from the buffer. The puzzle is that it requires t.read() to succeed. When I insert a check for bytes type in all the places it seems necessary in my code, I pass the first two conditions, but since t.read() also raises TypeError, the overall test fails. Is reading the stream with read() intended to succeed? Why is this desired? Jeff Allen

2 3

PyCon Sprints - Thank you
by Daniel Wozniak 20 Mar '13

20 Mar '13

David and Senthil, I won't make it to the sprints today because my ride wants to go into San Francisco to do touristy things. I'll be flying back to Arizona this evening. I still have a fair amount of code that has not been submitted to the issue tracker. I will sprint from my house tomorrow and will submit whatever else I have ready tomorrow night. In the future, I'd also like to continue contributing by continuing to improve test coverage since that is helping me get familiar with both the code base, style, and general processes of contributing. Thank you both for all your help and patience. You have been very welcoming. I am sure Python will continue to grow and attract new contributors so long as there are mentors such as yourselves there to support n00bs like me. Thanks again, Daniel Wozniak

3 2

Re: [Python-Dev] cpython: Closes issue 17467. Add readline and readlines support to
by Antoine Pitrou 20 Mar '13

20 Mar '13

On Wed, 20 Mar 2013 01:22:58 +0100 (CET) michael.foord <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/684b75600fa9 > changeset: 82811:684b75600fa9 > user: Michael Foord <michael(a)voidspace.org.uk> > date: Tue Mar 19 17:22:51 2013 -0700 > summary: > Closes issue 17467. Add readline and readlines support to unittest.mock.mock_open Wasn't it possible to re-use an existing implementation (such as TextIOBase or StringIO) rather than re-write your own? (it's not even obvious your implementation is correct, BTW. How about universal newlines?) Regards Antoine.

3 7

Re: [Python-Dev] [Python-checkins] peps: update 2.7.4 release dates
by Andrew Svetlov 20 Mar '13

20 Mar '13

Are you sure about 2.7.4 2012-04-06? I mean 2012 year. On Tue, Mar 19, 2013 at 9:15 PM, benjamin.peterson <python-checkins(a)python.org> wrote: > http://hg.python.org/peps/rev/ce17779c395c > changeset: 4810:ce17779c395c > user: Benjamin Peterson <benjamin(a)python.org> > date: Tue Mar 19 23:15:23 2013 -0500 > summary: > update 2.7.4 release dates > > files: > pep-0373.txt | 4 ++-- > 1 files changed, 2 insertions(+), 2 deletions(-) > > > diff --git a/pep-0373.txt b/pep-0373.txt > --- a/pep-0373.txt > +++ b/pep-0373.txt > @@ -56,8 +56,8 @@ > > Planned future release dates: > > -- 2.7.4rc1 2013-02-02 > -- 2.7.4 2012-02-16 > +- 2.7.4rc1 2013-03-23 > +- 2.7.4 2012-04-06 > > Dates of previous maintenance releases: > > > -- > Repository URL: http://hg.python.org/peps > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins > -- Thanks, Andrew Svetlov

2 1

Re: [Python-Dev] [Python-checkins] cpython: #15927: Fix cvs.reader parsing of escaped \r\n with quoting off.
by Kristján Valur Jónsson 20 Mar '13

20 Mar '13

The compiler complains about this line: if (c == '\n' | c=='\r') { Perhaps you wanted a Boolean operator? -----Original Message----- From: Python-checkins [mailto:python-checkins-bounces+kristjan=ccpgames.com@python.org] On Behalf Of r.david.murray Sent: 19. mars 2013 19:42 To: python-checkins(a)python.org Subject: [Python-checkins] cpython: #15927: Fix cvs.reader parsing of escaped \r\n with quoting off. http://hg.python.org/cpython/rev/940748853712 changeset: 82815:940748853712 parent: 82811:684b75600fa9 user: R David Murray <rdmurray(a)bitdance.com> date: Tue Mar 19 22:41:47 2013 -0400 summary: #15927: Fix cvs.reader parsing of escaped \r\n with quoting off. This fix means that such values are correctly roundtripped, since cvs.writer already does the correct escaping. Patch by Michael Johnson. files: Lib/test/test_csv.py | 9 +++++++++ Misc/ACKS | 1 + Misc/NEWS | 3 +++ Modules/_csv.c | 13 ++++++++++++- 4 files changed, 25 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_csv.py b/Lib/test/test_csv.py --- a/Lib/test/test_csv.py +++ b/Lib/test/test_csv.py @@ -308,6 +308,15 @@ for i, row in enumerate(csv.reader(fileobj)): self.assertEqual(row, rows[i]) + def test_roundtrip_escaped_unquoted_newlines(self): + with TemporaryFile("w+", newline='') as fileobj: + writer = csv.writer(fileobj,quoting=csv.QUOTE_NONE,escapechar="\\") + rows = [['a\nb','b'],['c','x\r\nd']] + writer.writerows(rows) + fileobj.seek(0) + for i, row in enumerate(csv.reader(fileobj,quoting=csv.QUOTE_NONE,escapechar="\\")): + self.assertEqual(row,rows[i]) + class TestDialectRegistry(unittest.TestCase): def test_registry_badargs(self): self.assertRaises(TypeError, csv.list_dialects, None) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -591,6 +591,7 @@ Fredrik Johansson Gregory K. Johnson Kent Johnson +Michael Johnson Simon Johnston Matt Joiner Thomas Jollans diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -289,6 +289,9 @@ Library ------- +- Issue #15927: CVS now correctly parses escaped newlines and carriage + when parsing with quoting turned off. + - Issue #17467: add readline and readlines support to mock_open in unittest.mock. diff --git a/Modules/_csv.c b/Modules/_csv.c --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -51,7 +51,7 @@ typedef enum { START_RECORD, START_FIELD, ESCAPED_CHAR, IN_FIELD, IN_QUOTED_FIELD, ESCAPE_IN_QUOTED_FIELD, QUOTE_IN_QUOTED_FIELD, - EAT_CRNL + EAT_CRNL,AFTER_ESCAPED_CRNL } ParserState; typedef enum { @@ -644,6 +644,12 @@ break; case ESCAPED_CHAR: + if (c == '\n' | c=='\r') { + if (parse_add_char(self, c) < 0) + return -1; + self->state = AFTER_ESCAPED_CRNL; + break; + } if (c == '\0') c = '\n'; if (parse_add_char(self, c) < 0) @@ -651,6 +657,11 @@ self->state = IN_FIELD; break; + case AFTER_ESCAPED_CRNL: + if (c == '\0') + break; + /*fallthru*/ + case IN_FIELD: /* in unquoted field */ if (c == '\n' || c == '\r' || c == '\0') { -- Repository URL: http://hg.python.org/cpython

2 1

Re: [Python-Dev] [Python-checkins] cpython: Issue #17385: Fix quadratic behavior in threading.Condition
by Ezio Melotti 20 Mar '13

20 Mar '13

Hi, On Mon, Mar 11, 2013 at 2:58 AM, raymond.hettinger <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/0f86b51f8f8b > changeset: 82592:0f86b51f8f8b > user: Raymond Hettinger <python(a)rcn.com> > date: Sun Mar 10 17:57:28 2013 -0700 > summary: > Issue #17385: Fix quadratic behavior in threading.Condition > > files: > Lib/threading.py | 10 ++++++++-- > Misc/NEWS | 3 +++ > 2 files changed, 11 insertions(+), 2 deletions(-) > > > diff --git a/Lib/threading.py b/Lib/threading.py > --- a/Lib/threading.py > +++ b/Lib/threading.py > @@ -10,6 +10,12 @@ > from time import time as _time > from traceback import format_exc as _format_exc > from _weakrefset import WeakSet > +try: > + from _itertools import islice as _slice > + from _collections import deque as _deque > +except ImportError: > + from itertools import islice as _islice > + from collections import deque as _deque > Shouldn't the one in the 'try' be _islice too? Best Regards, Ezio Melotti

1 1

Early results from Argument Clinic automation discussion
by Larry Hastings 20 Mar '13

20 Mar '13

Mark Shannon, Dmitry Jemerov (from PyCharm) and I sat down to talk about rearchitecting the Argument Clinic prototype to make it easily to interact with. We came up with the following. The DSL will now produce an intermediate representation. The output will consume this intermediate representation. The two defined outputs from the IR so far: 1. The inplace generated C code (what it has right now), and 2. Argument Clinic DSL code itself. The intermediate representation will be subclasses of inspect.Signature and inspect.Parameter that add the extra bits of information needed by Argument Clinic, as follows: class Function(inspect.Signature): name = 'function name' module = 'module name if any class = 'class name if any, can't actually call this class' docstring = 'function docstring' c_id = 'stub id to use for generated c functions' return_converter = SeeBelow() class Parameter(inspect.Parameter): docstring = 'per-parameter docstring' group = <group membership integer or None> converter = ConverterFunction() Parameter.group is an integer, specifying which "option group" the parameter is in. This must be an integer for positional-only arguments, and None for other argument types. 0 indicates "required positional-only parameter". Left-optional groups get negative numbers, decreasing as they get further from the required parameters; right-optional get positive numbers, increasing as they get further from the required parameters. (Groups cannot nest, so a parameter cannot be in more than one group.) Function.return_converter was suggested by Mark. This is the inverse of the per-parameter converter function: it defines the return type of the impl function, and the conversion process to turn it into the PyObject * that gets returned to Python. And, like the converter functions, it will define the actual return annotation of the function (if any). Mark wants to write something that parses C code implementing builtins and produces the IR; he can then use that to write Argument Clinic DSL. This will make the conversion process go much more quickly. //arry/

1 0

Re: [Python-Dev] [Python-checkins] cpython (2.7): Issue #9090 : Error code 10035 calling socket.recv() on a socket with a timeout
by Guido van Rossum 19 Mar '13

19 Mar '13

On Tue, Mar 19, 2013 at 11:08 AM, kristjan.jonsson < python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/8ec39bfd1f01 > changeset: 82764:8ec39bfd1f01 > branch: 2.7 > parent: 82740:b10ec5083a53 > user: Kristján Valur Jónsson <sweskman(a)gmail.com> > date: Tue Mar 19 10:58:59 2013 -0700 > summary: > Issue #9090 : Error code 10035 calling socket.recv() on a socket with a > timeout > (WSAEWOULDBLOCK - A non-blocking socket operation could not be completed > immediately) > [...] > +- Issue #9090: When a socket with a timeout fails with EWOULDBLOCK or > EAGAIN, > + retry the select() loop instead of bailing out. This is because > select() > + can incorrectly report a socket as ready for reading (for example, if it > + received some data with an invalid checksum). > Might I recommend treating EINTR the same way? It has the same issue of popping up, rarely, when you least expect it, and messing with your code. -- --Guido van Rossum (python.org/~guido)

1 0

Re: [Python-Dev] [Python-checkins] cpython (2.7): Issue #9090 : Error code 10035 calling socket.recv() on a socket with a timeout
by Kristján Valur Jónsson 19 Mar '13

19 Mar '13

Yes, it is a symbol problem on unix. Working on it. -----Original Message----- From: Python-checkins [mailto:python-checkins-bounces+kristjan=ccpgames.com@python.org] On Behalf Of Senthil Kumaran Sent: 19. mars 2013 12:28 To: sweskman(a)gmail.com Cc: python-checkins(a)python.org Subject: Re: [Python-checkins] cpython (2.7): Issue #9090 : Error code 10035 calling socket.recv() on a socket with a timeout Looks like RHEL 2.7 buildbots are unhappy with this change. -- Senthil On Tue, Mar 19, 2013 at 11:08 AM, kristjan.jonsson <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/8ec39bfd1f01 > changeset: 82764:8ec39bfd1f01 > branch: 2.7 > parent: 82740:b10ec5083a53 > user: Kristján Valur Jónsson <sweskman(a)gmail.com> > date: Tue Mar 19 10:58:59 2013 -0700 > summary: > Issue #9090 : Error code 10035 calling socket.recv() on a socket > with a timeout (WSAEWOULDBLOCK - A non-blocking socket operation > could not be completed > immediately) > > files: > Misc/NEWS | 5 + > Modules/socketmodule.c | 104 ++++++++++++++++++++++++---- > Modules/timemodule.c | 7 + > 3 files changed, 101 insertions(+), 15 deletions(-) > > > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -214,6 +214,11 @@ > Library > ------- > > +- Issue #9090: When a socket with a timeout fails with EWOULDBLOCK or > +EAGAIN, > + retry the select() loop instead of bailing out. This is because > +select() > + can incorrectly report a socket as ready for reading (for example, > +if it > + received some data with an invalid checksum). > + > - Issue #1285086: Get rid of the refcounting hack and speed up urllib.unquote(). > > - Issue #17368: Fix an off-by-one error in the Python JSON decoder > that caused diff --git a/Modules/socketmodule.c > b/Modules/socketmodule.c > --- a/Modules/socketmodule.c > +++ b/Modules/socketmodule.c > @@ -473,6 +473,17 @@ > return NULL; > } > > +#ifdef MS_WINDOWS > +#ifndef WSAEAGAIN > +#define WSAEAGAIN WSAEWOULDBLOCK > +#endif > +#define CHECK_ERRNO(expected) \ > + (WSAGetLastError() == WSA ## expected) #else #define > +CHECK_ERRNO(expected) \ > + (errno == expected) > +#endif > + > /* Convenience function to raise an error according to errno > and return a NULL pointer from a function. */ > > @@ -661,7 +672,7 @@ > after they've reacquired the interpreter lock. > Returns 1 on timeout, -1 on error, 0 otherwise. */ static int > -internal_select(PySocketSockObject *s, int writing) > +internal_select_ex(PySocketSockObject *s, int writing, double > +interval) > { > int n; > > @@ -673,6 +684,10 @@ > if (s->sock_fd < 0) > return 0; > > + /* Handling this condition here simplifies the select loops */ > + if (interval < 0.0) > + return 1; > + > /* Prefer poll, if available, since you can poll() any fd > * which can't be done with select(). */ #ifdef HAVE_POLL @@ > -684,7 +699,7 @@ > pollfd.events = writing ? POLLOUT : POLLIN; > > /* s->sock_timeout is in seconds, timeout in ms */ > - timeout = (int)(s->sock_timeout * 1000 + 0.5); > + timeout = (int)(interval * 1000 + 0.5); > n = poll(&pollfd, 1, timeout); > } > #else > @@ -692,8 +707,8 @@ > /* Construct the arguments to select */ > fd_set fds; > struct timeval tv; > - tv.tv_sec = (int)s->sock_timeout; > - tv.tv_usec = (int)((s->sock_timeout - tv.tv_sec) * 1e6); > + tv.tv_sec = (int)interval; > + tv.tv_usec = (int)((interval - tv.tv_sec) * 1e6); > FD_ZERO(&fds); > FD_SET(s->sock_fd, &fds); > > @@ -712,6 +727,49 @@ > return 0; > } > > +static int > +internal_select(PySocketSockObject *s, int writing) { > + return internal_select_ex(s, writing, s->sock_timeout); } > + > +/* > + Two macros for automatic retry of select() in case of false positives > + (for example, select() could indicate a socket is ready for reading > + but the data then discarded by the OS because of a wrong checksum). > + Here is an example of use: > + > + BEGIN_SELECT_LOOP(s) > + Py_BEGIN_ALLOW_THREADS > + timeout = internal_select_ex(s, 0, interval); > + if (!timeout) > + outlen = recv(s->sock_fd, cbuf, len, flags); > + Py_END_ALLOW_THREADS > + if (timeout == 1) { > + PyErr_SetString(socket_timeout, "timed out"); > + return -1; > + } > + END_SELECT_LOOP(s) > +*/ > +PyAPI_FUNC(double) _PyTime_floattime(void); /* defined in timemodule.c */ > +#define BEGIN_SELECT_LOOP(s) \ > + { \ > + double deadline, interval = s->sock_timeout; \ > + int has_timeout = s->sock_timeout > 0.0; \ > + if (has_timeout) { \ > + deadline = _PyTime_floattime() + s->sock_timeout; \ > + } \ > + while (1) { \ > + errno = 0; \ > + > +#define END_SELECT_LOOP(s) \ > + if (!has_timeout || \ > + (!CHECK_ERRNO(EWOULDBLOCK) && !CHECK_ERRNO(EAGAIN))) \ > + break; \ > + interval = deadline - _PyTime_floattime(); \ > + } \ > + } \ > + > /* Initialize a new socket object. */ > > static double defaulttimeout = -1.0; /* Default timeout for new sockets */ > @@ -1656,8 +1714,9 @@ > if (!IS_SELECTABLE(s)) > return select_error(); > > + BEGIN_SELECT_LOOP(s) > Py_BEGIN_ALLOW_THREADS > - timeout = internal_select(s, 0); > + timeout = internal_select_ex(s, 0, interval); > if (!timeout) > newfd = accept(s->sock_fd, SAS2SA(&addrbuf), &addrlen); > Py_END_ALLOW_THREADS > @@ -1666,6 +1725,7 @@ > PyErr_SetString(socket_timeout, "timed out"); > return NULL; > } > + END_SELECT_LOOP(s) > > #ifdef MS_WINDOWS > if (newfd == INVALID_SOCKET) > @@ -2355,8 +2415,9 @@ > } > > #ifndef __VMS > + BEGIN_SELECT_LOOP(s) > Py_BEGIN_ALLOW_THREADS > - timeout = internal_select(s, 0); > + timeout = internal_select_ex(s, 0, interval); > if (!timeout) > outlen = recv(s->sock_fd, cbuf, len, flags); > Py_END_ALLOW_THREADS > @@ -2365,6 +2426,7 @@ > PyErr_SetString(socket_timeout, "timed out"); > return -1; > } > + END_SELECT_LOOP(s) > if (outlen < 0) { > /* Note: the call to errorhandler() ALWAYS indirectly returned > NULL, so ignore its return value */ > @@ -2386,8 +2448,9 @@ > segment = remaining; > } > > + BEGIN_SELECT_LOOP(s) > Py_BEGIN_ALLOW_THREADS > - timeout = internal_select(s, 0); > + timeout = internal_select_ex(s, 0, interval); > if (!timeout) > nread = recv(s->sock_fd, read_buf, segment, flags); > Py_END_ALLOW_THREADS > @@ -2396,6 +2459,8 @@ > PyErr_SetString(socket_timeout, "timed out"); > return -1; > } > + END_SELECT_LOOP(s) > + > if (nread < 0) { > s->errorhandler(); > return -1; > @@ -2559,9 +2624,10 @@ > return -1; > } > > + BEGIN_SELECT_LOOP(s) > Py_BEGIN_ALLOW_THREADS > memset(&addrbuf, 0, addrlen); > - timeout = internal_select(s, 0); > + timeout = internal_select_ex(s, 0, interval); > if (!timeout) { > #ifndef MS_WINDOWS > #if defined(PYOS_OS2) && !defined(PYCC_GCC) > @@ -2582,6 +2648,7 @@ > PyErr_SetString(socket_timeout, "timed out"); > return -1; > } > + END_SELECT_LOOP(s) > if (n < 0) { > s->errorhandler(); > return -1; > @@ -2719,8 +2786,9 @@ > buf = pbuf.buf; > len = pbuf.len; > > + BEGIN_SELECT_LOOP(s) > Py_BEGIN_ALLOW_THREADS > - timeout = internal_select(s, 1); > + timeout = internal_select_ex(s, 1, interval); > if (!timeout) > #ifdef __VMS > n = sendsegmented(s->sock_fd, buf, len, flags); > @@ -2728,13 +2796,14 @@ > n = send(s->sock_fd, buf, len, flags); > #endif > Py_END_ALLOW_THREADS > - > - PyBuffer_Release(&pbuf); > - > if (timeout == 1) { > + PyBuffer_Release(&pbuf); > PyErr_SetString(socket_timeout, "timed out"); > return NULL; > } > + END_SELECT_LOOP(s) > + > + PyBuffer_Release(&pbuf); > if (n < 0) > return s->errorhandler(); > return PyInt_FromLong((long)n); > @@ -2768,8 +2837,9 @@ > } > > do { > + BEGIN_SELECT_LOOP(s) > Py_BEGIN_ALLOW_THREADS > - timeout = internal_select(s, 1); > + timeout = internal_select_ex(s, 1, interval); > n = -1; > if (!timeout) { > #ifdef __VMS > @@ -2784,6 +2854,7 @@ > PyErr_SetString(socket_timeout, "timed out"); > return NULL; > } > + END_SELECT_LOOP(s) > /* PyErr_CheckSignals() might change errno */ > saved_errno = errno; > /* We must run our signal handlers before looping again. > @@ -2863,17 +2934,20 @@ > return NULL; > } > > + BEGIN_SELECT_LOOP(s) > Py_BEGIN_ALLOW_THREADS > - timeout = internal_select(s, 1); > + timeout = internal_select_ex(s, 1, interval); > if (!timeout) > n = sendto(s->sock_fd, buf, len, flags, SAS2SA(&addrbuf), addrlen); > Py_END_ALLOW_THREADS > > - PyBuffer_Release(&pbuf); > if (timeout == 1) { > + PyBuffer_Release(&pbuf); > PyErr_SetString(socket_timeout, "timed out"); > return NULL; > } > + END_SELECT_LOOP(s) > + PyBuffer_Release(&pbuf); > if (n < 0) > return s->errorhandler(); > return PyInt_FromLong((long)n); > diff --git a/Modules/timemodule.c b/Modules/timemodule.c > --- a/Modules/timemodule.c > +++ b/Modules/timemodule.c > @@ -1055,3 +1055,10 @@ > > return 0; > } > + > +/* export floattime to socketmodule.c */ > +PyAPI_FUNC(double) > +_PyTime_floattime(void) > +{ > + return floattime(); > +} > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins > _______________________________________________ Python-checkins mailing list Python-checkins(a)python.org http://mail.python.org/mailman/listinfo/python-checkins

1 1