Mailman 3 April 2013 - Python-checkins

cpython (3.3): Issue #17357: Use more stern wording for
by brett.cannon April 28, 2013

April 28, 2013

http://hg.python.org/cpython/rev/75e32a0bfd74 changeset: 83525:75e32a0bfd74 branch: 3.3 parent: 83517:4b4ed1e11fd0 user: Brett Cannon <brett(a)python.org> date: Sat Apr 27 23:20:32 2013 -0400 summary: Issue #17357: Use more stern wording for importlib.invalidate_caches(). files: Doc/library/importlib.rst | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -103,9 +103,9 @@ Invalidate the internal caches of finders stored at :data:`sys.meta_path`. If a finder implements ``invalidate_caches()`` then it - will be called to perform the invalidation. This function may be needed if - some modules are installed while your program is running and you expect the - program to notice the changes. + will be called to perform the invalidation. This function should be called + if any modules are created/installed while your program is running to + guarantee all finders will notice the new module's existence. .. versionadded:: 3.3 @@ -182,7 +182,7 @@ .. versionadded:: 3.3 - .. method:: find_loader(fullname): + .. method:: find_loader(fullname) An abstract method for finding a :term:`loader` for the specified module. Returns a 2-tuple of ``(loader, portion)`` where ``portion`` @@ -194,7 +194,7 @@ the empty list then no loader or location for a namespace package were found (i.e. failure to find anything for the module). - .. method:: find_module(fullname): + .. method:: find_module(fullname) A concrete implementation of :meth:`Finder.find_module` which is equivalent to ``self.find_loader(fullname)[0]``. -- Repository URL: http://hg.python.org/cpython

2 1

cpython (merge 3.3 -> default): merge for issue #17357
by brett.cannon April 28, 2013

April 28, 2013

http://hg.python.org/cpython/rev/5fac0ac46f54 changeset: 83526:5fac0ac46f54 parent: 83524:d232cff25bbd parent: 83525:75e32a0bfd74 user: Brett Cannon <brett(a)python.org> date: Sat Apr 27 23:20:55 2013 -0400 summary: merge for issue #17357 files: Doc/library/importlib.rst | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -109,9 +109,9 @@ Invalidate the internal caches of finders stored at :data:`sys.meta_path`. If a finder implements ``invalidate_caches()`` then it - will be called to perform the invalidation. This function may be needed if - some modules are installed while your program is running and you expect the - program to notice the changes. + will be called to perform the invalidation. This function should be called + if any modules are created/installed while your program is running to + guarantee all finders will notice the new module's existence. .. versionadded:: 3.3 @@ -197,7 +197,7 @@ .. versionadded:: 3.3 - .. method:: find_loader(fullname): + .. method:: find_loader(fullname) An abstract method for finding a :term:`loader` for the specified module. Returns a 2-tuple of ``(loader, portion)`` where ``portion`` @@ -212,7 +212,7 @@ .. versionchanged:: 3.4 Returns ``(None, [])`` instead of raising :exc:`NotImplementedError`. - .. method:: find_module(fullname): + .. method:: find_module(fullname) A concrete implementation of :meth:`Finder.find_module` which is equivalent to ``self.find_loader(fullname)[0]``. -- Repository URL: http://hg.python.org/cpython

1 0

peps: Add some comments and acknowledgements
by antoine.pitrou April 27, 2013

April 27, 2013

http://hg.python.org/peps/rev/d9be356b12a8 changeset: 4865:d9be356b12a8 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Sat Apr 27 13:24:45 2013 +0200 summary: Add some comments and acknowledgements files: pep-3154.txt | 50 ++++++++++++++++++++++++++++++++------- 1 files changed, 41 insertions(+), 9 deletions(-) diff --git a/pep-3154.txt b/pep-3154.txt --- a/pep-3154.txt +++ b/pep-3154.txt @@ -7,8 +7,9 @@ Type: Standards Track Content-Type: text/x-rst Created: 2011-08-11 -Python-Version: 3.3 +Python-Version: 3.4 Post-History: + http://mail.python.org/pipermail/python-dev/2011-August/112821.html Resolution: TBD @@ -36,10 +37,10 @@ object). The opportunity was not taken at the time to improve the protocol in other ways. -This PEP is an attempt to foster a number of small incremental -improvements in a future new protocol version. The PEP process is -used in order to gather as many improvements as possible, because the -introduction of a new protocol version should be a rare occurrence. +This PEP is an attempt to foster a number of incremental improvements +in a new pickle protocol version. The PEP process is used in order to +gather as many improvements as possible, because the introduction of a +new pickle protocol should be a rare occurrence. Proposed changes @@ -74,11 +75,15 @@ pickles, and the unpickler refuses them. Also, there is no "last frame" marker. The last frame is simply the one which ends with a STOP opcode. +A well-written C implementation doesn't need additional memory copies +for the framing layer, preserving general (un)pickling efficiency. + .. note:: - How the pickler partitions the pickle stream into frames is an - implementation detail. "Closing" a frame as soon as it reaches - ~64 KiB should be ok for both performance and pickle size overhead. + How the pickler decides to partition the pickle stream into frames is an + implementation detail. For example, "closing" a frame as soon as it + reaches ~64 KiB is a reasonable choice for both performance and pickle + size overhead. Binary encoding for all opcodes ------------------------------- @@ -138,10 +143,31 @@ would make many pickles smaller. +Alternative ideas +================= + +Prefetching +----------- + +Serhiy Storchaka suggested to replace framing with a special PREFETCH +opcode (with a 2- or 4-bytes argument) to declare known pickle chunks +explicitly. Large data may be pickled outside such chunks. A naïve +unpickler should be able to skip the PREFETCH opcode and still decode +pickles properly, but good error handling would require checking that +the PREFETCH length falls on an opcode boundary. + + Acknowledgments =============== -(...) +In alphabetic order: + +* Alexandre Vassalotti, for starting the second PEP 3154 implementation [6]_ + +* Serhiy Storchaka, for discussing the framing proposal [6]_ + +* Stefan Mihaila, for starting the first PEP 3154 implementation as a + Google Summer of Code project mentored by Alexandre Vassalotti [7]_. References @@ -162,6 +188,12 @@ .. [5] Lib/multiprocessing/forking.py: http://hg.python.org/cpython/file/baea9f5f973c/Lib/multiprocessing/forking.… +.. [6] Implement PEP 3154, by Alexandre Vassalotti + http://bugs.python.org/issue17810 + +.. [7] Implement PEP 3154, by Stefan Mihaila + http://bugs.python.org/issue15642 + Copyright ========= -- Repository URL: http://hg.python.org/peps

1 0

Daily reference leaks (d232cff25bbd): sum=0
by solipsis＠pitrou.net April 27, 2013

April 27, 2013

results for d232cff25bbd on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogR0f1tJ', '-x']

1 0

peps: Update docs for Handle, add Timer.
by guido.van.rossum April 27, 2013

April 27, 2013

http://hg.python.org/peps/rev/9b67bb8c84ec changeset: 4864:9b67bb8c84ec user: Guido van Rossum <guido(a)python.org> date: Fri Apr 26 20:33:04 2013 -0700 summary: Update docs for Handle, add Timer. files: pep-3156.txt | 29 ++++++++++++++++++++--------- 1 files changed, 20 insertions(+), 9 deletions(-) diff --git a/pep-3156.txt b/pep-3156.txt --- a/pep-3156.txt +++ b/pep-3156.txt @@ -770,8 +770,13 @@ a ``Handle``, although the user never needs to instantiate instances of this class. There is one public method: -- ``cancel()``. Attempt to cancel the callback. - TBD: Exact specification. +- ``cancel()``. Cancel the callback. This just sets the + ``cancelled`` attribute. The event loop must check this. + +- ``run()``. Call the callback with the specified arguments. If this + raises an exception, it is logged and the function returns normally. + This always returns None. Note that this ignores the ``cancelled`` + attribute (i.e., a cancelled ``Handle`` can still be called). Read-only public attributes: @@ -783,15 +788,21 @@ Note that some callbacks (e.g. those registered with ``call_later()``) are meant to be called only once. Others (e.g. those registered with -``add_reader()``) are meant to be called multiple times. +``call_repeatedly()`` or ``add_reader()``) are meant to be called +multiple times. -TBD: An API to call the callback (encapsulating the exception handling -necessary)? Should it record how many times it has been called? -Maybe this API should just be ``__call__()``? (But it should suppress -exceptions.) +The Timer Subclass +'''''''''''''''''' -TBD: Public attribute recording the realtime value when the callback -is scheduled? (Since this is needed anyway for storing it in a heap.) +``Timer`` is a subclass of ``Handle`` returned by ``call_later()`` and +``call_repeatedly()``. + +It has one additional read-only public attribute: + +- ``when``, the real time when the callback is next scheduled to be + called. + +``Timer`` objects are ordered by their ``when`` field. Futures ------- -- Repository URL: http://hg.python.org/peps

1 0

cpython: Issue #17804: New function ``struct.iter_unpack`` allows for streaming struct
by antoine.pitrou April 26, 2013

April 26, 2013

http://hg.python.org/cpython/rev/d232cff25bbd changeset: 83524:d232cff25bbd user: Antoine Pitrou <solipsis(a)pitrou.net> date: Sat Apr 27 00:20:04 2013 +0200 summary: Issue #17804: New function ``struct.iter_unpack`` allows for streaming struct unpacking. files: Doc/library/struct.rst | 20 +++ Lib/struct.py | 1 + Lib/test/test_struct.py | 74 ++++++++++++- Misc/NEWS | 3 + Modules/_struct.c | 165 ++++++++++++++++++++++++++++ 5 files changed, 262 insertions(+), 1 deletions(-) diff --git a/Doc/library/struct.rst b/Doc/library/struct.rst --- a/Doc/library/struct.rst +++ b/Doc/library/struct.rst @@ -66,6 +66,19 @@ format (``len(buffer[offset:])`` must be at least ``calcsize(fmt)``). +.. function:: iter_unpack(fmt, buffer) + + Iteratively unpack from the buffer *buffer* according to the format + string *fmt*. This function returns an iterator which will read + equally-sized chunks from the buffer until all its contents have been + consumed. The buffer's size in bytes must be a multiple of the amount + of data required by the format, as reflected by :func:`calcsize`. + + Each iteration yields a tuple as specified by the format string. + + .. versionadded:: 3.4 + + .. function:: calcsize(fmt) Return the size of the struct (and hence of the bytes object produced by @@ -388,6 +401,13 @@ (``len(buffer[offset:])`` must be at least :attr:`self.size`). + .. method:: iter_unpack(buffer) + + Identical to the :func:`iter_unpack` function, using the compiled format. + (``len(buffer)`` must be a multiple of :attr:`self.size`). + + .. versionadded:: 3.4 + .. attribute:: format The format string used to construct this Struct object. diff --git a/Lib/struct.py b/Lib/struct.py --- a/Lib/struct.py +++ b/Lib/struct.py @@ -1,6 +1,7 @@ __all__ = [ # Functions 'calcsize', 'pack', 'pack_into', 'unpack', 'unpack_from', + 'iter_unpack', # Classes 'Struct', diff --git a/Lib/test/test_struct.py b/Lib/test/test_struct.py --- a/Lib/test/test_struct.py +++ b/Lib/test/test_struct.py @@ -1,4 +1,6 @@ +from collections import abc import array +import operator import unittest import struct import sys @@ -593,8 +595,78 @@ self.check_sizeof('0s', 1) self.check_sizeof('0c', 0) + +class UnpackIteratorTest(unittest.TestCase): + """ + Tests for iterative unpacking (struct.Struct.iter_unpack). + """ + + def test_construct(self): + def _check_iterator(it): + self.assertIsInstance(it, abc.Iterator) + self.assertIsInstance(it, abc.Iterable) + s = struct.Struct('>ibcp') + it = s.iter_unpack(b"") + _check_iterator(it) + it = s.iter_unpack(b"1234567") + _check_iterator(it) + # Wrong bytes length + with self.assertRaises(struct.error): + s.iter_unpack(b"123456") + with self.assertRaises(struct.error): + s.iter_unpack(b"12345678") + # Zero-length struct + s = struct.Struct('>') + with self.assertRaises(struct.error): + s.iter_unpack(b"") + with self.assertRaises(struct.error): + s.iter_unpack(b"12") + + def test_iterate(self): + s = struct.Struct('>IB') + b = bytes(range(1, 16)) + it = s.iter_unpack(b) + self.assertEqual(next(it), (0x01020304, 5)) + self.assertEqual(next(it), (0x06070809, 10)) + self.assertEqual(next(it), (0x0b0c0d0e, 15)) + self.assertRaises(StopIteration, next, it) + self.assertRaises(StopIteration, next, it) + + def test_arbitrary_buffer(self): + s = struct.Struct('>IB') + b = bytes(range(1, 11)) + it = s.iter_unpack(memoryview(b)) + self.assertEqual(next(it), (0x01020304, 5)) + self.assertEqual(next(it), (0x06070809, 10)) + self.assertRaises(StopIteration, next, it) + self.assertRaises(StopIteration, next, it) + + def test_length_hint(self): + lh = operator.length_hint + s = struct.Struct('>IB') + b = bytes(range(1, 16)) + it = s.iter_unpack(b) + self.assertEqual(lh(it), 3) + next(it) + self.assertEqual(lh(it), 2) + next(it) + self.assertEqual(lh(it), 1) + next(it) + self.assertEqual(lh(it), 0) + self.assertRaises(StopIteration, next, it) + self.assertEqual(lh(it), 0) + + def test_module_func(self): + # Sanity check for the global struct.iter_unpack() + it = struct.iter_unpack('>IB', bytes(range(1, 11))) + self.assertEqual(next(it), (0x01020304, 5)) + self.assertEqual(next(it), (0x06070809, 10)) + self.assertRaises(StopIteration, next, it) + self.assertRaises(StopIteration, next, it) + + def test_main(): - support.run_unittest(StructTest) + support.run_unittest(__name__) if __name__ == '__main__': test_main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -49,6 +49,9 @@ Library ------- +- Issue #17804: New function ``struct.iter_unpack`` allows for streaming + struct unpacking. + - Issue #17830: When keyword.py is used to update a keyword file, it now preserves the line endings of the original file. diff --git a/Modules/_struct.c b/Modules/_struct.c --- a/Modules/_struct.c +++ b/Modules/_struct.c @@ -1247,6 +1247,9 @@ return size; } +/* + * Struct object implementation. + */ /* calculate the size of a format string */ @@ -1556,6 +1559,142 @@ } +/* Unpack iterator type */ + +typedef struct { + PyObject_HEAD + PyStructObject *so; + Py_buffer buf; + Py_ssize_t index; +} unpackiterobject; + +static void +unpackiter_dealloc(unpackiterobject *self) +{ + Py_XDECREF(self->so); + PyBuffer_Release(&self->buf); + PyObject_GC_Del(self); +} + +static int +unpackiter_traverse(unpackiterobject *self, visitproc visit, void *arg) +{ + Py_VISIT(self->so); + Py_VISIT(self->buf.obj); + return 0; +} + +static PyObject * +unpackiter_len(unpackiterobject *self) +{ + Py_ssize_t len; + if (self->so == NULL) + len = 0; + else + len = (self->buf.len - self->index) / self->so->s_size; + return PyLong_FromSsize_t(len); +} + +static PyMethodDef unpackiter_methods[] = { + {"__length_hint__", (PyCFunction) unpackiter_len, METH_NOARGS, NULL}, + {NULL, NULL} /* sentinel */ +}; + +static PyObject * +unpackiter_iternext(unpackiterobject *self) +{ + PyObject *result; + if (self->so == NULL) + return NULL; + if (self->index >= self->buf.len) { + /* Iterator exhausted */ + Py_CLEAR(self->so); + PyBuffer_Release(&self->buf); + return NULL; + } + assert(self->index + self->so->s_size <= self->buf.len); + result = s_unpack_internal(self->so, + (char*) self->buf.buf + self->index); + self->index += self->so->s_size; + return result; +} + +PyTypeObject unpackiter_type = { + PyVarObject_HEAD_INIT(&PyType_Type, 0) + "unpack_iterator", /* tp_name */ + sizeof(unpackiterobject), /* tp_basicsize */ + 0, /* tp_itemsize */ + (destructor)unpackiter_dealloc, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_reserved */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + 0, /* tp_as_sequence */ + 0, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + PyObject_GenericGetAttr, /* tp_getattro */ + 0, /* tp_setattro */ + 0, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ + 0, /* tp_doc */ + (traverseproc)unpackiter_traverse, /* tp_traverse */ + 0, /* tp_clear */ + 0, /* tp_richcompare */ + 0, /* tp_weaklistoffset */ + PyObject_SelfIter, /* tp_iter */ + (iternextfunc)unpackiter_iternext, /* tp_iternext */ + unpackiter_methods /* tp_methods */ +}; + +PyDoc_STRVAR(s_iter_unpack__doc__, +"S.iter_unpack(buffer) -> iterator(v1, v2, ...)\n\ +\n\ +Return an iterator yielding tuples unpacked from the given bytes\n\ +source, like a repeated invocation of unpack_from(). Requires\n\ +that the bytes length be a multiple of the struct size."); + +static PyObject * +s_iter_unpack(PyObject *_so, PyObject *input) +{ + PyStructObject *so = (PyStructObject *) _so; + unpackiterobject *self; + + assert(PyStruct_Check(_so)); + assert(so->s_codes != NULL); + + if (so->s_size == 0) { + PyErr_Format(StructError, + "cannot iteratively unpack with a struct of length 0"); + return NULL; + } + + self = (unpackiterobject *) PyType_GenericAlloc(&unpackiter_type, 0); + if (self == NULL) + return NULL; + + if (PyObject_GetBuffer(input, &self->buf, PyBUF_SIMPLE) < 0) { + Py_DECREF(self); + return NULL; + } + if (self->buf.len % so->s_size != 0) { + PyErr_Format(StructError, + "iterative unpacking requires a bytes length " + "multiple of %zd", + so->s_size); + Py_DECREF(self); + return NULL; + } + Py_INCREF(so); + self->so = so; + self->index = 0; + return (PyObject *) self; +} + + /* * Guts of the pack function. * @@ -1776,6 +1915,7 @@ /* List of functions */ static struct PyMethodDef s_methods[] = { + {"iter_unpack", s_iter_unpack, METH_O, s_iter_unpack__doc__}, {"pack", s_pack, METH_VARARGS, s_pack__doc__}, {"pack_into", s_pack_into, METH_VARARGS, s_pack_into__doc__}, {"unpack", s_unpack, METH_O, s_unpack__doc__}, @@ -2025,9 +2165,34 @@ return result; } +PyDoc_STRVAR(iter_unpack_doc, +"iter_unpack(fmt, buffer) -> iterator(v1, v2, ...)\n\ +\n\ +Return an iterator yielding tuples unpacked from the given bytes\n\ +source according to the format string, like a repeated invocation of\n\ +unpack_from(). Requires that the bytes length be a multiple of the\n\ +format struct size."); + +static PyObject * +iter_unpack(PyObject *self, PyObject *args) +{ + PyObject *s_object, *fmt, *input, *result; + + if (!PyArg_ParseTuple(args, "OO:iter_unpack", &fmt, &input)) + return NULL; + + s_object = cache_struct(fmt); + if (s_object == NULL) + return NULL; + result = s_iter_unpack(s_object, input); + Py_DECREF(s_object); + return result; +} + static struct PyMethodDef module_functions[] = { {"_clearcache", (PyCFunction)clearcache, METH_NOARGS, clearcache_doc}, {"calcsize", calcsize, METH_O, calcsize_doc}, + {"iter_unpack", iter_unpack, METH_VARARGS, iter_unpack_doc}, {"pack", pack, METH_VARARGS, pack_doc}, {"pack_into", pack_into, METH_VARARGS, pack_into_doc}, {"unpack", unpack, METH_VARARGS, unpack_doc}, -- Repository URL: http://hg.python.org/cpython

1 0

peps: We could overlap or cross boundaries, but let's straddle them instead.
by antoine.pitrou April 26, 2013

April 26, 2013

http://hg.python.org/peps/rev/96338fd72442 changeset: 4863:96338fd72442 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Fri Apr 26 23:18:25 2013 +0200 summary: We could overlap or cross boundaries, but let's straddle them instead. (dedicated to Larry) files: pep-3154.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-3154.txt b/pep-3154.txt --- a/pep-3154.txt +++ b/pep-3154.txt @@ -70,7 +70,7 @@ etc. To keep the implementation simple, it is forbidden for a pickle opcode -to overlap frame boundaries. The pickler takes care not to produce such +to straddle frame boundaries. The pickler takes care not to produce such pickles, and the unpickler refuses them. Also, there is no "last frame" marker. The last frame is simply the one which ends with a STOP opcode. -- Repository URL: http://hg.python.org/peps

1 0

peps: Major update of the event loop spec.
by guido.van.rossum April 26, 2013

April 26, 2013

http://hg.python.org/peps/rev/9c4e5f027fcc changeset: 4862:9c4e5f027fcc user: Guido van Rossum <guido(a)python.org> date: Fri Apr 26 14:16:59 2013 -0700 summary: Major update of the event loop spec. files: pep-3156.txt | 870 ++++++++++++++++++++++++++------------ 1 files changed, 600 insertions(+), 270 deletions(-) diff --git a/pep-3156.txt b/pep-3156.txt --- a/pep-3156.txt +++ b/pep-3156.txt @@ -92,21 +92,25 @@ difference between transports and protocols is in order. At the highest level, the transport is concerned with *how* bytes are transmitted, while the protocol determines *which* bytes to transmit -(and when). +(and to some extent when). -A transport represents a pair of streams (one in each direction) that -each transmit a sequence of bytes. The most common transport is -probably the TCP connection. Another common transport is SSL. But -there are some other things that can be viewed as transports, for -example an SSH session or a pair of UNIX pipes. Typically there -aren't many different transport implementations, and most of them -come with the event loop implementation. +The most common type of transport is a bidirectional stream transport. +It represents a pair of streams (one in each direction) that each +transmit a sequence of bytes. The most common example of a +bidirectional stream transport is probably a TCP connection. Another +common example is an SSL connection. But there are some other things +that can be viewed this way, for example an SSH session or a pair of +UNIX pipes. Typically there aren't many different transport +implementations, and most of them come with the event loop +implementation. Note that transports don't need to use sockets, not +even if they use TCP -- sockets are a platform-specific implementation +detail. -A transport has two "sides": one side talks to the network (or the -subprocess, or whatever low-level interface it wraps), and the other -side talks to the protocol. The former uses whatever API is necessary -to implement the transport; but the interface between transport and -protocol is standardized by this PEP. +A bidirectional stream transport has two "sides": one side talks to +the network (or another process, or whatever low-level interface it +wraps), and the other side talks to the protocol. The former uses +whatever API is necessary to implement the transport; but the +interface between transport and protocol is standardized by this PEP. A protocol represents some kind of "application-level" protocol such as HTTP or SMTP. Its primary interface is with the transport. While @@ -115,16 +119,16 @@ have libraries of useful 3rd party protocol implementations that can be downloaded and installed from pypi.python.org. -There is also a somewhat more general notion of transport and -protocol, where the transport wraps some other communication -abstraction. Example include an interface for sending and receiving -datagrams, or a subprocess manager. The separation of concerns is the -same as for stream transports and protocols, but the specific -interface between transport and protocol can be different in each -case. +There general notion of transport and protocol includes other +interfaces, where the transport wraps some other communication +abstraction. Examples include interfaces for sending and receiving +datagrams (e.g. UDP), or a subprocess manager. The separation of +concerns is the same as for bidirectional stream transports and +protocols, but the specific interface between transport and protocol +is different in each case. -Details of the interfaces between transports and protocols are given -later. +Details of the interfaces defined by the various standard types of +transports and protocols are given later. Specification @@ -133,9 +137,11 @@ Dependencies ------------ -Python 3.3 is required. No new language or standard library features -beyond Python 3.3 are required. No third-party modules or packages -are required. +Python 3.3 is required for many of the proposed features. The +reference implementation (Tulip) requires no new language or standard +library features beyond Python 3.3, no third-party modules or +packages, and no C code, except for the proactor-based event loop on +Windows. Module Namespace ---------------- @@ -155,57 +161,169 @@ toplevel package name. Classes and functions given without a module name are assumed to be accessed via the toplevel package. -Event Loop Policy: Getting and Setting the Event Loop ------------------------------------------------------ +Event Loop Policy: Getting and Setting the Current Event Loop +------------------------------------------------------------- -To get the current event loop, use ``get_event_loop()``. This returns -an instance of the ``EventLoop`` class defined below or an equivalent -object. It is possible that ``get_event_loop()`` returns a different -object depending on the current thread, or depending on some other -notion of context. +Event loop management is controlled by an event loop policy, which is +a global (per-process) state. There is a default policy, and an API +to change the policy. The policy defines the notion of context; the +default policy's notion of context is defined as the current thread. -To set the current event loop, use ``set_event_loop(event_loop)``, -where ``event_loop`` is an instance of the ``EventLoop`` class or -equivalent. This uses the same notion of context as -``get_event_loop()``. - -For the benefit of unit tests and other special cases there's a third -policy function: ``new_event_loop()``, which creates and returns a new -EventLoop instance according to the policy's default rules. To make -this the current event loop, you must call ``set_event_loop()``. - -To change the way the above three functions work -(including their notion of context), call -``set_event_loop_policy(policy)``, where ``policy`` is an event loop -policy object. The policy object can be any object that has methods -``get_event_loop()``, ``set_event_loop(event_loop)`` -and ``new_event_loop()`` behaving like -the functions described above. The default event loop policy is an -instance of the class ``DefaultEventLoopPolicy``. The current event loop -policy object can be retrieved by calling ``get_event_loop_policy()``. +Certain platforms or programming frameworks may change the default +policy to something more suitable to the expectations of the users of +that platform or framework. Such platforms or frameworks must +document their policy and at what point during their initialization +sequence the policy is set. in order to avoid undefined behavior when +multiple active frameworks want to override the default policy. An event loop policy may but does not have to enforce that there is only one event loop in existence. The default event loop policy does not enforce this, but it does enforce that there is only one event -loop per thread. +loop per thread (as far as ``get_event_loop()`` is concerned). -Event Loop Interface --------------------- +To get the current event loop, use ``get_event_loop()``. This returns +an event loop object implementing the interface specified below, or +``None`` in case no current event loop has been set and the current +policy does not specify how to create one for the current context. It +is expected that ``get_event_loop()`` returns a different object +depending on the context, and the default policy will only create a +default event loop in the main thread; in other threads an event loop +must be explicitly set (but other policies may behave differently). +Event loop creation is lazy; i.e. the first call to +``get_event_loop()`` creates an event loop instance if necessary and +specified by the current policy. + +To set the current event loop, use ``set_event_loop(event_loop)``, +where ``event_loop`` is an event loop object. It is allowed to set +the current event loop to ``None`` (although under the default policy, +if the main thread's current event loop is set to ``None``, and +``get_event_loop()`` is called subsequently, it will create a new +event loop instance. + +For the benefit of unit tests and other special cases there's a third +policy function: ``new_event_loop()``, which creates and returns a new +event loop object according to the policy's default rules. To make +this the current event loop, you must call ``set_event_loop()``. + +To change the event loop policy, call +``set_event_loop_policy(policy)``, where ``policy`` is an event loop +policy object or ``None``. The policy object must be an object that +has methods ``get_event_loop()``, ``set_event_loop(loop)`` and +``new_event_loop()``, all behaving like the functions described above. +Passing a policy value of ``None`` restores the default event loop +policy (overriding the alternate default set by the platform or +framework). The default event loop policy is an instance of the class +``DefaultEventLoopPolicy``. The current event loop policy object can +be retrieved by calling ``get_event_loop_policy()``. (TBD: Require +inheriting from ``AbstractEventLoopPolicy``?) + +Notes for the Event Loop Interface +---------------------------------- A note about times: as usual in Python, all timeouts, intervals and delays are measured in seconds, and may be ints or floats. The accuracy and precision of the clock are up to the implementation; the default implementation uses ``time.monotonic()``. -A note about callbacks and Handles: any function that takes a -callback and a variable number of arguments for it can also be given a -Handle object instead of the callback. Then no arguments should be -given, and the Handle should represent an immediate callback (as -returned from ``call_soon()``), not a delayed callback (as returned -from ``call_later()``). If the Handle is already cancelled, the -call is a no-op. +A note about callbacks and Handles: any function in the API described +below that takes a callback and a variable number of arguments for it +can also be given a Handle object instead of the callback. Then no +arguments should be given, and the Handle should represent an +immediate callback (as returned from ``call_soon()``), not a delayed +callback (as returned from ``call_later()``). If the Handle has +been cancelled, the call is a no-op. -A conforming event loop object has the following methods: +Event Loop Classes +------------------ + +There is no actual class named ``EventLoop``. There is an +``AbstractEventLoop`` class which defines all the methods without +implementations, and serves primarily as documentation. The following +concrete classes are defined: + +- ``SelectorEventLoop`` is a concrete implementation of the full API + based on the ``selectors`` module. (This module is part of Tulip, + but not specified by this PEP. It is separately proposed for + inclusion in the standard library.) The constructor takes one + optional argument, a ``selectors.Selector`` object. By default an + instance of ``selectors.DefaultSelector`` is created and used. + +- ``ProactorEventLoop`` is a concrete implementation of the API except + for the I/O event handling and signal handling methods. It is only + defined on Windows (or on other platforms which support a similar + API for "overlapped I/O"). The constructor takes one optional + argument, a ``Proactor`` object. By default an instance of + ``IocpProactor`` is created and used. (The ``IocpProactor`` class + is not specified by this PEP. Its inclusion in the standard library + is not currently under consideration; it is just an implementation + detail of the ``ProactorEventLoop`` class. + +Event Loop Methods Overview +--------------------------- + +The methods of a conforming event loop are grouped into several +categories. A brief overview of the categories. The first set of +categories must be supported by all conforming event loop +implementations. (However, in some cases a partially-conforming +implementation may choose not to implement the internet/socket +methods, and still conform to the other methods.) + +- Resource management: ``close()``. + +- Starting and stopping: ``run_forever()``, ``run_until_complete()``, + ``stop()``, ``is_running()``. + +- Basic callbacks: ``call_soon()``, ``call_later()``, + ``call_repeatedly()``. + +- Thread interaction: ``call_soon_threadsafe()``, + ``wrap_future()``, ``run_in_executor()``, + ``set_default_executor()``. + +- Internet name lookups: ``getaddrinfo()``, ``getnameinfo()``. + +- Internet connections: ``create_connection()``, ``start_serving()``, + ``stop_serving()``, ``create_datagram_endpoint()``. + +- Wrapped socket methods: ``sock_recv()``, ``sock_sendall()``, + ``sock_connect()``, ``sock_accept()``. + +The second set of categories *may* be supported by conforming event +loop implementations. If not supported, they will raise +``NotImplementedError``. (In the current state of Tulip, +``SelectorEventLoop`` on UNIX systems supports all of these; +``SelectorEventLoop`` on Windows supports the I/O event handling +category; ``ProactorEventLoop`` on Windows supports None. The +intention is to add support for pipes and subprocesses on Windows as +well, using the ``subprocess`` module in the standard library.) + +- I/O callbacks: ``add_reader()``, ``remove_reader()``, + ``add_writer()``, ``remove_writer()``. + +- Pipes and subprocesses: ``connect_read_pipe()``, + ``connect_write_pipe()``, ``spawn_subprocess()``. + +- Signal callbacks: ``add_signal_handler()``, + ``remove_signal_handler()``. + +Required Event Loop Methods +--------------------------- + +Resource Management +''''''''''''''''''' + +- ``close()``. Closes the event loop, releasing any resources it may + hold, such as the file descriptor used by ``epoll()`` or + ``kqueue()``. This should not be called while the event loop is + running. After it has been called the event loop may not be used + again. It may be called multiple times; subsequent calls are + no-ops. + +Starting and Stopping +''''''''''''''''''''' + +An (unclosed) event loop can be in one of two states: running or +stopped. These methods deal with starting and stopping an event loop: - ``run_forever()``. Runs the event loop until ``stop()`` is called. This cannot be called when the event loop is already running. (This @@ -224,20 +342,27 @@ loop is already running. - ``stop()``. Stops the event loop as soon as it is convenient. It - is fine to restart the loop with ``run()`` or ``run_until_complete()`` - subsequently; no scheduled callbacks will be lost if this happens. - Note: ``stop()`` returns normally and the current callback is - allowed to continue. How soon after this point the event loop stops - is up to the implementation, but the intention is to stop short of - polling for I/O, and not to run any callbacks scheduled in the - future; the major freedom an implementation has is how much of the - "ready queue" (callbacks already scheduled with ``call_soon()``) it - processes before stopping. + is fine to restart the loop with ``run_forever()`` or + ``run_until_complete()`` subsequently; no scheduled callbacks will + be lost if this is done. Note: ``stop()`` returns normally and the + current callback is allowed to continue. How soon after this point + the event loop stops is up to the implementation, but the intention + is to stop short of polling for I/O, and not to run any callbacks + scheduled in the future; the major freedom an implementation has is + how much of the "ready queue" (callbacks already scheduled with + ``call_soon()``) it processes before stopping. -- ``close()``. Closes the event loop, releasing any resources it may - hold, such as the file descriptor used by ``epoll()`` or - ``kqueue()``. This should not be called while the event loop is - running. It may be called multiple times. +- ``is_running()``. Returns ``True`` if the event loop is currently + running, ``False`` if it is stopped. (TBD: Do we need another + inquiry method to tell whether the loop is in the process of + stopping?) + +Basic Callbacks +''''''''''''''' + +- ``call_soon(callback, *args)``. This schedules a callback to be + called as soon as possible. It guarantees that callbacks are called + in the order in which they were scheduled. - ``call_later(delay, callback, *args)``. Arrange for ``callback(*args)`` to be called approximately ``delay`` seconds in @@ -256,19 +381,314 @@ callback happens later than scheduled, subsequent callbacks will be delayed for (at least) the same amount. The ``interval`` must be > 0. -- ``call_soon(callback, *args)``. This schedules a callback to be - called as soon as possible. It guarantees that callbacks are called - in the order in which they were scheduled. +Thread interaction +'''''''''''''''''' - ``call_soon_threadsafe(callback, *args)``. Like ``call_soon(callback, *args)``, but when called from another thread while the event loop is blocked waiting for I/O, unblocks the event loop. This is the *only* method that is safe to call from another - thread. (To schedule a callback for a - later time in a threadsafe manner, you can use - ``ev.call_soon_threadsafe(ev.call_later, when, callback, *args)``.) - This is not safe to call from a signal handler (since it may use - locks). + thread. (To schedule a callback for a later time in a threadsafe + manner, you can use ``loop.call_soon_threadsafe(loop.call_later, + when, callback, *args)``.) This is not safe to call from a signal + handler (since it may use locks). + +- ``wrap_future(future)``. This takes a PEP 3148 Future (i.e., an + instance of ``concurrent.futures.Future``) and returns a Future + compatible with the event loop (i.e., a ``tulip.Future`` instance). + +- ``run_in_executor(executor, callback, *args)``. Arrange to call + ``callback(*args)`` in an executor (see PEP 3148). Returns a Future + whose result on success is the return value of that call. This is + equivalent to ``wrap_future(executor.submit(callback, *args))``. If + ``executor`` is ``None``, the default executor set by + ``set_default_executor()`` is used. If no default executor has been + set yet, a ``ThreadPoolExecutor`` with 5 threads is created and set + as the default executor. + +- ``set_default_executor(executor)``. Set the default executor used + by ``run_in_executor()``. The argument must be a PEP 3148 + ``Executor`` instance or ``None``, in order to reset the default + executor. + +Internet name lookups +''''''''''''''''''''' + +These methods are useful if you want to connect or bind a socket to an +address without the risk of blocking for the name lookup. They are +usually called implicitly by ``create_connection()``, +``start_serving()`` or ``create_datagram_endpoint()``. + +- ``getaddrinfo(host, port, family=0, type=0, proto=0, flags=0)``. + Similar to the ``socket.getaddrinfo()`` function but returns a + Future. The Future's result on success will be a list of the same + format as returned by ``socket.getaddrinfo()``, i.e. a list of + ``(address_family, socket_type, socket_protocol, canonical_name, + address)`` where ``address`` is a 2-tuple ``(ipv4_address, port)`` + for IPv4 addresses and a 4-tuple ``(ipv4_address, port, flow_info, + scope_id)`` for IPv6 addresses. If the ``family`` argument is zero + or unspecified, the list returned may contain a mixture of IPv4 and + IPv6 addresses; otherwise the addresses returned are constrained by + the ``family`` value (similar for ``proto`` and ``flags``). The + default implementation calls ``socket.getaddrinfo()`` using + ``run_in_executor()``, but other implementations may choose to + implement their own DNS lookup. The optional arguments *must* be + specified as keyword arguments. + +- ``getnameinfo(sockaddr, flags=0)``. Similar to + ``socket.getnameinfo()`` but returns a Future. The Future's result + on success will be a tuple ``(host, port)``. Same implementation + remarks as for ``getaddrinfo()``. + +Internet connections +'''''''''''''''''''' + +These are the high-level interfaces for managing internet connections. +Their use is recommended over the corresponding lower-level interfaces +because they abstract away the differences between selector-based +and proactor-based event loops. + +Note that the client and server side of stream connections use the +same transport and protocol interface. However, datagram endpoints +use a different transport and protocol interface. + +- ``create_connection(protocol_factory, host, port, **kwargs)``. + Creates a stream connection to a given internet host and port. This + is a task that is typically called from the client side of the + connection. It creates an implementation-dependent (bidirectional + stream) Transport to represent the connection, then calls + ``protocol_factory()`` to instantiate (or retrieve) the user's + Protocol implementation, and finally ties the two together. (See + below for the definitions of Transport and Protocol.) The user's + Protocol implementation is created or retrieved by calling + ``protocol_factory()`` without arguments(*). The coroutine's result + on success is the ``(transport, protocol)`` pair; if a failure + prevents the creation of a successful connection, an appropriate + exception will be raised. Note that when the coroutine completes, + the protocol's ``connection_made()`` method has not yet been called; + that will happen when the connection handshake is complete. + + (*) There is no requirement that ``protocol_factory`` is a class. + If your protocol class needs to have specific arguments passed to + its constructor, you can use ``lambda`` or ``functools.partial()``. + You can also pass a trivial ``lambda`` that returns a previously + constructed Protocol instance. + + Optional keyword arguments: + + - ``ssl``: Pass ``True`` to create an SSL transport (by default a + plain TCP transport is created). Or pass an ``ssl.SSLContext`` + object to override the default SSL context object to be used. + + - ``family``, ``proto``, ``flags``: Address family, protocol and + flags to be passed through to ``getaddrinfo()``. These all + default to ``0``, which means "not specified". (The socket type + is always ``SOCK_STREAM``.) If any of these values are not + specified, the ``getaddrinfo()`` method will choose appropriate + values. Note: ``proto`` has nothing to do with the high-level + Protocol concept or the ``protocol_factory`` argument. + + - ``sock``: An optional socket to be used instead of using the + ``host``, ``port``, ``family``, ``proto``, and ``flags`` + arguments. If this is given, host and port must be omitted; + otherwise, host and port are required. + +- ``start_serving(protocol_factory, host, port, **kwds)``. Enters a + serving loop that accepts connections. This is a Task that + completes once the serving loop is set up to serve. The return + value is a list of one or more sockets in listening mode. (Multiple + sockets may be returned if the specified address allows both IPv4 + and IPv6 connections.) You can use ``stop_serving()`` to stop the + serving loop. Each time a connection is accepted, + ``protocol_factory`` is called without arguments(*) to create a + Protocol, a (bidirectional stream) Transport is created to represent + the network side of the connection, and the two are tied together by + calling ``protocol.connection_made(transport)``. + + (*) See footnote above for ``create_connection()``. However, since + ``protocol_factory()`` is called once for each new incoming + connection, it should return a new Protocol object each time it is + called. + + Optional keyword arguments: + + - ``ssl``: Pass ``True`` to create an SSL transport (by default a + plain TCP transport is created). Or pass an ``ssl.SSLContext`` + object to override the default SSL context object to be used. + + - ``backlog``: Backlog value to be passed to the ``listen()`` call. + Defaults to ``100``. + + - ``reuse_address``: Whether to set the ``SO_REUSEADDR`` option on + the socket. The default is ``True`` on UNIX, ``False`` on + Windows. + + - ``family``, ``flags``: Address family and flags to be passed + through to ``getaddrinfo()``. The family defaults to + ``AF_UNSPEC``; the flags default to ``AI_PASSIVE``. (The socket + type is always ``SOCK_STREAM``; the socket protocol always set to + ``0``, to let ``getaddrinfo()`` choose.) + + - ``sock``: An optional socket to be used instead of using the + ``host``, ``port``, ``family``, and ``flags`` arguments. If this + is given, host and port must be omitted; otherwise, host and port + are required. The return value will be the one-element list + ``[sock]``. + +- ``stop_serving(sock)``. The argument should be a socket from the + list returned by ``start_serving()``. The serving loop associated + with that socket will be stopped. Connections that have already + been accepted will not be affected. + +- ``create_datagram_endpoint(protocol_factory, local_addr, + remote_addr, **kwds)``. Creates an endpoint for sending and + receiving datagrams (typically UDP packets). Because of the nature + of datagram traffic, there are no separate calls to set up client + and server side, since usually a single endpoint acts as both client + and server. This is a coroutine that returns a ``(transport, + protocol)`` pair on success, or raises an exception on failure. If + the coroutine returns successfully, the transport will call + callbacks on the protocol whenever a datagram is received or the + socket is closed; it is up to the protocol to call methods on the + protocol to send datagrams. Note that the transport and protocol + interfaces used here are different than those for stream + connections. + + Arguments: + + - ``protocol_factory``: A class or factory function that will be + called, without arguments, to construct the protocol object to be + returned. The interface between datagram transport and protocol + is described below. + + - ``local_addr``: An optional tuple indicating the address to which + the socket will be bound. If given this must be a ``(host, + port)`` pair. It will be passed to ``getaddrinfo()`` to be + resolved and the result will be passed to the ``bind()`` method of + the socket created. If ``getaddrinfo()`` returns more than one + address, they will be tried in turn. If omitted, no ``bind()`` + call will be made. + + - ``remote_addr``: An optional tuple indicating the address to which + the socket will be "connected". (Since there is no such thing as + a datagram connection, this just specifies a default value for the + destination address of outgoing datagrams.) If given this must be + a ``(host, port)`` pair. It will be passed to ``getaddrinfo()`` + to be resolved and the result will be passed to ``sock_connect()`` + together with the socket created. If ``getaddrinfo()`` returns + more than one address, they will be tried in turn. If omitted, + no ``sock_connect()`` will be made. + + - ``family``, ``proto``, ``flags``: Address family, protocol and + flags to be passed through to ``getaddrinfo()``. These all + default to ``0``, which means "not specified". (The socket type + is always ``SOCK_DGRAM``.) If any of these values are not + specified, the ``getaddrinfo()`` method will choose appropriate + values. + + Note that if both ``local_addr`` and ``remote_addr`` are present, + all combinations of local and remote addresses with matching address + family will be tried. + +Wrapped Socket Methods +'''''''''''''''''''''' + +The following methods for doing async I/O on sockets are not for +general use. They are primarily meant for transport implementations +working with IOCP through the ``ProactorEventLoop`` class. However, +they are easily implementable for other event loop types, so there is +no reason not to require them. The socket argument has to be a +non-blocking socket. + +- ``sock_recv(sock, n)``. Receive up to ``n`` bytes from socket + ``sock``. Returns a Future whose result on success will be a + bytes object. + +- ``sock_sendall(sock, data)``. Send bytes ``data`` to socket + ``sock``. Returns a Future whose result on success will be + ``None``. Note: the name uses ``sendall`` instead of ``send``, to + reflect that the semantics and signature of this method echo those + of the standard library socket method ``sendall()`` rather than + ``send()``. (TBD: but maybe it would be better to emulate + ``send()`` after all? That would be better for datagram sockets.) + +- ``sock_connect(sock, address)``. Connect to the given address. + Returns a Future whose result on success will be ``None``. + +- ``sock_accept(sock)``. Accept a connection from a socket. The + socket must be in listening mode and bound to an address. Returns a + Future whose result on success will be a tuple ``(conn, peer)`` + where ``conn`` is a connected non-blocking socket and ``peer`` is + the peer address. + +Optional Event Loop Methods +--------------------------- + +I/O Callbacks +''''''''''''' + +These methods are primarily meant for transport implementations +working with a selector. They are implemented by +``SelectorEventLoop`` but not by ``ProactorEventLoop``. Custom event +loop implementations may or may not implement them. + +The ``fd`` arguments below may be integer file descriptors, or +"file-like" objects with a ``fileno()`` method that wrap integer file +descriptors. Not all file-like objects or file descriptors are +acceptable. Sockets (and socket file descriptors) are always +accepted. On Windows no other types are supported. On UNIX, pipes +and possibly tty devices are also supported, but disk files are not. +Exactly which special file types are supported may vary by platform +and per selector implementation. (Experimentally, there is at least +one kind of pseudo-tty on OSX that is supported by ``select`` and +``poll`` but not by ``kqueue``: it is used by Emacs shell windows.) + +- ``add_reader(fd, callback, *args)``. Arrange for + ``callback(*args)`` to be called whenever file descriptor ``fd`` is + deemed ready for reading. Returns a ``Handle`` object which can be + used to cancel the callback. Note that, unlike ``call_later()``, + the callback may be called many times. Calling ``add_reader()`` + again for the same file descriptor implies a call to + ``remove_reader()`` for the same file descriptor. Note that, + although cancelling the ``Handle`` returned is nearly equivalent to + calling ``remove_reader()``, it is strongly preferred to call + ``remove_reader()``. + +- ``add_writer(fd, callback, *args)``. Like ``add_reader()``, + but registers the callback for writing instead of for reading. + +- ``remove_reader(fd)``. Cancels the current read callback for file + descriptor ``fd``, if one is set. If no callback is currently set + for the file descriptor, this is a no-op and returns ``False``. + Otherwise, it removes the callback arrangement, cancels the + corresponding ``Handle``, and returns ``True``. + +- ``remove_writer(fd)``. This is to ``add_writer()`` as + ``remove_reader()`` is to ``add_reader()``. + +Pipes and Subprocesses +'''''''''''''''''''''' + +- ``connect_read_pipe(protocol_factory, pipe)``: Create a + unidrectional stream connection from a file-like object wrapping the + read end of a UNIX pipe. The protocol/transport interface is the + read half of the bidirectional stream interface. + +- ``connect_write_pipe(protocol_factory, pipe)``: Create a + unidrectional stream connection from a file-like object wrapping the + write end of a UNIX pipe. The protocol/transport interface is the + write half of the bidirectional stream interface. + +- TBD: A way to run a subprocess with stdin, stdout and stderr + connected to pipe transports. (This is being designed but not yet + ready.) + +TBD: offer the same interface on Windows for e.g. named pipes. (This +should be possible given that the standard library ``subprocess`` +module is supported on Windows.) + +Signal callbacks +'''''''''''''''' - ``add_signal_handler(sig, callback, *args). Whenever signal ``sig`` is received, arrange for ``callback(*args)`` to be called. Returns @@ -284,7 +704,7 @@ signale (e.g. ``SIGKILL``), ``RuntimeError`` if this particular event loop instance cannot handle signals (since signals are global per process, only an event loop associated with the main thread can - handle signals). (TBD: Rename to ``set_signal_handler()``?) + handle signals). - ``remove_signal_handler(sig)``. Removes the handler for signal ``sig``, if one is set. Raises the same exceptions as @@ -293,176 +713,8 @@ ``True`` if a handler was removed successfully, ``False`` if no handler was set. -Some methods in the standard conforming interface return Futures: - -- ``wrap_future(future)``. This takes a PEP 3148 Future (i.e., an - instance of ``concurrent.futures.Future``) and returns a Future - compatible with the event loop (i.e., a ``tulip.Future`` instance). - -- ``run_in_executor(executor, callback, *args)``. Arrange to call - ``callback(*args)`` in an executor (see PEP 3148). Returns a Future - whose result on success is the return value that call. This is - equivalent to ``wrap_future(executor.submit(callback, *args))``. If - ``executor`` is ``None``, a default ``ThreadPoolExecutor`` with 5 - threads is used. - -- ``set_default_executor(executor)``. Set the default executor used - by ``run_in_executor()``. - -- ``getaddrinfo(host, port, family=0, type=0, proto=0, flags=0)``. - Similar to the ``socket.getaddrinfo()`` function but returns a - Future. The Future's result on success will be a list of the same - format as returned by ``socket.getaddrinfo()``. The default - implementation calls ``socket.getaddrinfo()`` using - ``run_in_executor()``, but other implementations may choose to - implement their own DNS lookup. The optional arguments *must* be - specified as keyword arguments. - -- ``getnameinfo(sockaddr, flags=0)``. Similar to - ``socket.getnameinfo()`` but returns a Future. The Future's result - on success will be a tuple ``(host, port)``. Same implementation - remarks as for ``getaddrinfo()``. - -- ``create_connection(protocol_factory, host, port, **kwargs)``. - Creates a stream connection to a given host and port. This creates - an implementation-dependent Transport to represent the connection, - then calls ``protocol_factory()`` to instantiate (or retrieve) the - user's Protocol implementation, and finally ties the two together. - (See below for the definitions of Transport and Protocol.) The - user's Protocol implementation is created or retrieved by calling - ``protocol_factory()`` without arguments(*). The return value is a - Future whose result on success is the ``(transport, protocol)`` - pair; if a failure prevents the creation of a successful connection, - the Future will have an appropriate exception set. Note that when - the Future completes, the protocol's ``connection_made()`` method - has not yet been called; that will happen when the connection - handshake is complete. - - (*) There is no requirement that ``protocol_factory`` is a class. - If your protocol class needs to have specific arguments passed to - its constructor, you can use ``lambda`` or ``functools.partial()``. - You can also pass a trivial ``lambda`` that returns a previously - constructed Protocol instance. - - Optional keyword arguments: - - - ``family``, ``proto``, ``flags``: Address familty, - protcol, and miscellaneous flags to be passed through - to ``getaddrinfo()``. These all default to ``0``. - (The socket type is always ``SOCK_STREAM``.) - - - ``ssl``: Pass ``True`` to create an SSL transport (by default a - plain TCP is created). Or pass an ``ssl.SSLContext`` object to - override the default SSL context object to be used. - -- ``start_serving(protocol_factory, host, port, **kwds)``. Enters a - loop that accepts connections. Returns a Future that completes once - the loop is set up to serve; its return value is None. Each time a - connection is accepted, ``protocol_factory`` is called without - arguments(*) to create a Protocol, a Transport is created to represent - the network side of the connection, and the two are tied together by - calling ``protocol.connection_made(transport)``. - - (*) See footnote above for ``create_connection()``. However, since - ``protocol_factory()`` is called once for each new incoming - connection, it is recommended that it return a new Protocol object - each time it is called. - - Optional keyword arguments: - - - ``family``, ``proto``, ``flags``: Address familty, - protcol, and miscellaneous flags to be passed through - to ``getaddrinfo()``. These all default to ``0``. - (The socket type is always ``SOCK_STREAM``.) - - TBD: Support SSL? I don't even know how to do that synchronously, - and I suppose it needs a certificate. - - TBD: Maybe make the Future's result an object that can be used to - control the serving loop, e.g. to stop serving, abort all active - connections, and (if supported) adjust the backlog or other - parameters? It could also have an API to inquire about active - connections. Alternatively, return a Future (subclass?) that only - completes if the loop stops serving due to an error, or if it cannot - be started? Cancelling it might stop the loop. - -TBD: Some platforms may not be interested in implementing all of -these, e.g. start_serving() may be of no interest to mobile apps. -(Although, there's a Minecraft server on my iPad...) - -The following methods for registering callbacks for file descriptors -are optional. If they are not implemented, accessing the method -(without calling it) returns AttributeError. The default -implementation provides them but the user normally doesn't use these -directly -- they are used by the transport implementations -exclusively. Also, on Windows these may be present or not depending -on whether a select-based or IOCP-based event loop is used. These -take integer file descriptors only, not objects with a fileno() -method. The file descriptor should represent something pollable -- -i.e. no disk files. - -- ``add_reader(fd, callback, *args)``. Arrange for - ``callback(*args)`` to be called whenever file descriptor ``fd`` is - ready for reading. Returns a ``Handle`` object which can be - used to cancel the callback. Note that, unlike ``call_later()``, - the callback may be called many times. Calling ``add_reader()`` - again for the same file descriptor implicitly cancels the previous - callback for that file descriptor. Note: cancelling the handle - may be delayed until the handle would be called. If you plan to - close ``fd``, you should use ``remove_reader(fd)`` instead. - (TBD: Change this to raise an exception if a handle - is already set.) (TBD: Rename to ``set_reader()``?) - -- ``add_writer(fd, callback, *args)``. Like ``add_reader()``, - but registers the callback for writing instead of for reading. - (TBD: Rename to ``set_reader()``?) - -- ``remove_reader(fd)``. Cancels the current read callback for file - descriptor ``fd``, if one is set. A no-op if no callback is - currently set for the file descriptor. (The reason for providing - this alternate interface is that it is often more convenient to - remember the file descriptor than to remember the ``Handle`` - object.) Returns ``True`` if a callback was removed, ``False`` - if not. - -- ``remove_writer(fd)``. This is to ``add_writer()`` as - ``remove_reader()`` is to ``add_reader()``. - -TBD: What about multiple callbacks per fd? The current semantics is -that ``add_reader()/add_writer()`` replace a previously registered -callback. Change this to raise an exception if a callback is already -registered. - -The following methods for doing async I/O on sockets are optional. -They are alternative to the previous set of optional methods, intended -for transport implementations on Windows using IOCP (if the event loop -supports it). The socket argument has to be a non-blocking socket. - -- ``sock_recv(sock, n)``. Receive up to ``n`` bytes from socket - ``sock``. Returns a Future whose result on success will be a - bytes object on success. - -- ``sock_sendall(sock, data)``. Send bytes ``data`` to the socket - ``sock``. Returns a Future whose result on success will be - ``None``. (TBD: Is it better to emulate ``sendall()`` or ``send()`` - semantics? I think ``sendall()`` -- but perhaps it should still - be *named* ``send()``?) - -- ``sock_connect(sock, address)``. Connect to the given address. - Returns a Future whose result on success will be ``None``. - -- ``sock_accept(sock)``. Accept a connection from a socket. The - socket must be in listening mode and bound to an address. Returns a - Future whose result on success will be a tuple ``(conn, peer)`` - where ``conn`` is a connected non-blocking socket and ``peer`` is - the peer address. (TBD: People tell me that this style of API is - too slow for high-volume servers. So there's also - ``start_serving()`` above. Then do we still need this?) - -TBD: Optional methods are not so good. Perhaps these should be -required? It may still depend on the platform which set is more -efficient. Another possibility: document these as "for transports -only" and the rest as "for anyone". +Note: If these methods are statically known to be unsupported, they +may return ``NotImplementedError`` instead of ``RuntimeError``. Callback Sequencing ------------------- @@ -470,8 +722,8 @@ When two callbacks are scheduled for the same time, they are run in the order in which they are registered. For example:: - ev.call_soon(foo) - ev.call_soon(bar) + loop.call_soon(foo) + loop.call_soon(bar) guarantees that ``foo()`` is called before ``bar()``. @@ -612,7 +864,7 @@ ``exception()`` is called on a Future that is cancelled. - ``TimeoutError``. An alias for ``concurrent.futures.TimeoutError``. - May be raised by ``EventLoop.run_until_complete()``. + May be raised by ``run_until_complete()``. A Future is associated with the default event loop when it is created. (TBD: Optionally pass in an alternative event loop instance?) @@ -645,11 +897,9 @@ Transports ---------- -A transport is an abstraction on top of a socket or something similar -(for example, a UNIX pipe or an SSL connection). Transports are -strongly influenced by Twisted and PEP 3153. Users rarely implement -or instantiate transports -- rather, event loops offer utility methods -to set up transports. +Transports and protocols are strongly influenced by Twisted and PEP +3153. Users rarely implement or instantiate transports -- rather, +event loops offer utility methods to set up transports. Transports work in conjunction with protocols. Protocols are typically written without knowing or caring about the exact type of @@ -659,6 +909,29 @@ plain socket transport can be used with many different protocols besides HTTP (e.g. SMTP, IMAP, POP, FTP, IRC, SPDY). +The most common type of transport is a bidirectional stream transport. +There are also unidirectional stream transports (used for pipes) and +datagram transports (used by the ``create_datagram_endpoint()`` +method). + +Methods For All Transports +'''''''''''''''''''''''''' + +- ``get_extra_info(name, default=None)``. This is a catch-all method + that returns implementation-specific information about a transport. + The first argument is the name of the extra field to be retrieved. + The optional second argument is a default value to be returned. + Consult the implementation documentation to find out the supported + extra field names. For an unsupported name, the default is always + returned. + +Bidirectional Stream Transports +''''''''''''''''''''''''''''''' + +A bidrectional stream transport is an abstraction on top of a socket +or something similar (for example, a pair of UNIX pipes or an SSL +connection). + Most connections have an asymmetric nature: the client and server usually have very different roles and behaviors. Hence, the interface between transport and protocol is also asymmetric. From the @@ -669,7 +942,12 @@ other data source), the transport calls the protocol's ``data_received()`` method. -Transports have the following public methods: +Nevertheless, the interface between transport and protocol used by +bidirectional streams is the same for clients as it is for servers, +since the connection between a client and a server is essentially a +pair of streams, one in each direction. + +Bidirectional stream transports have the following public methods: - ``write(data)``. Write some bytes. The argument must be a bytes object. Returns ``None``. The transport is free to buffer the @@ -741,9 +1019,58 @@ protocol doesn't support flow control. (Perhaps different names to avoid confusion between protocols and transports?) +Unidirectional Stream Transports +'''''''''''''''''''''''''''''''' + +A writing stream transport supports the ``write()``, ``writelines()``, +``write_eof()``, ``can_write_eof()``, close() and ``abort()`` methods +described for bidrectional stream transports. + +A reading stream transport supports the ``pause()``, ``resume()`` and +``close()`` methods described for bidrectional stream transports. + +A writing stream transport calls only ``connection_made()`` and +``connection_lost()`` on its associated protocol. + +A reading stream transport can call all protocol methods specified in +the Protocols section below (i.e., the previous two plus +``data_received()`` and ``eof_received()``). + +Datagram Transports +''''''''''''''''''' + +Datagram transports have these methods: + +- ``sendto(data, addr=None)``. Sends a datagram (a bytes object). + The optional second argument is the destination address. If + omitted, ``remote_addr`` must have been specified in the + ``create_datagram_endpoint()`` call that created this transport. If + present, and ``remote_addr`` was specified, they must match. + +- ``abort()``. Immediately close the transport. Buffered data will + be discarded. + +- ``close()``. Close the transport. Buffered data will be + transmitted asynchronously. + +Datagram transports call the following methods on the associated +protocol object: ``connection_made()``, ``connection_lost()``, +``connection_refused()``, and ``datagram_received()``. ("Connection" +in these method names is a slight misnomer, but the concepts still +exist: ``connection_made()`` means the transport representing the +endpoint has been created, and ``connection_lost()`` means the +transport is closed. The ``connection_refused()`` method is called +before ``connection_lost()`` when ``remote_addr`` was given and an +explicit negative acknowledgement was received (this is a UDP +feature). (TBD: Do we need the latter? It seems easy enough to +implement this in the protocol if it needs to make the distinction.) + Protocols --------- +TBD Describe different kinds of protocols (bidrectional stream, +unidirectional stream, datagram). + Protocols are always used in conjunction with transports. While a few common protocols are provided (e.g. decent though not necessarily excellent HTTP client and server implementations), most protocols will @@ -796,9 +1123,10 @@ Most interfaces taking a callback also take positional arguments. For instance, to arrange for ``foo("abc", 42)`` to be called soon, you -call ``ev.call_soon(foo, "abc", 42)``. To schedule the call -``foo()``, use ``ev.call_soon(foo)``. This convention greatly reduces -the number of small lambdas required in typical callback programming. +call ``loop.call_soon(foo, "abc", 42)``. To schedule the call +``foo()``, use ``loop.call_soon(foo)``. This convention greatly +reduces the number of small lambdas required in typical callback +programming. This convention specifically does *not* support keyword arguments. Keyword arguments are used to pass optional extra information about @@ -808,7 +1136,7 @@ keyword argument, you can use a lambda or ``functools.partial``. For example:: - ev.call_soon(functools.partial(foo, "abc", repeat=42)) + loop.call_soon(functools.partial(foo, "abc", repeat=42)) Choosing an Event Loop Implementation ------------------------------------- @@ -1032,7 +1360,7 @@ - We may need APIs to control various timeouts. E.g. we may want to limit the time spent in DNS resolution, connecting, ssl handshake, idle connection, close/shutdown, even per session. Possibly it's - sufficient to add ``timeout`` keyword parameters to some methods, + sufficient to add ``timeout`` keyword arguments to some methods, and other timeouts can probably be implemented by clever use of ``call_later()`` and ``Task.cancel()``. But it's possible that some operations need default timeouts, and we may want to change the @@ -1048,8 +1376,10 @@ References ========== -- PEP 380 describes the semantics of ``yield from``. TBD: Greg - Ewing's tutorial. +- PEP 380 describes the semantics of ``yield from``. + +- Greg Ewing's ``yield from`` tutorials: + http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yield_from.ht… - PEP 3148 describes ``concurrent.futures.Future``. -- Repository URL: http://hg.python.org/peps

1 0

peps: Presentation nits
by antoine.pitrou April 26, 2013

April 26, 2013

http://hg.python.org/peps/rev/e014049192dc changeset: 4861:e014049192dc user: Antoine Pitrou <solipsis(a)pitrou.net> date: Fri Apr 26 23:13:23 2013 +0200 summary: Presentation nits files: pep-3154.txt | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/pep-3154.txt b/pep-3154.txt --- a/pep-3154.txt +++ b/pep-3154.txt @@ -59,11 +59,11 @@ +------+------+ | 0x80 | 0x04 | protocol header (2 bytes) +------+------+-----------+ - | AA BB CC DD EE FF GG HH | frame size (8 bytes, little-endian) + | MM MM MM MM MM MM MM MM | frame size (8 bytes, little-endian) +------+------------------+ - | .... | first frame contents (N bytes) + | .... | first frame contents (M bytes) +------+------+-----------+ - | AA BB CC DD EE FF GG HH | frame size (8 bytes, little-endian) + | NN NN NN NN NN NN NN NN | frame size (8 bytes, little-endian) +------+------------------+ | .... | second frame contents (N bytes) +------+ -- Repository URL: http://hg.python.org/peps

1 0

peps: Protocol 4 has number four in its header
by antoine.pitrou April 26, 2013

April 26, 2013

http://hg.python.org/peps/rev/0e06e5907171 changeset: 4860:0e06e5907171 user: Antoine Pitrou <solipsis(a)pitrou.net> date: Fri Apr 26 23:10:52 2013 +0200 summary: Protocol 4 has number four in its header files: pep-3154.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-3154.txt b/pep-3154.txt --- a/pep-3154.txt +++ b/pep-3154.txt @@ -57,7 +57,7 @@ of a pickle is thus the following:: +------+------+ - | 0x80 | 0x03 | protocol header (2 bytes) + | 0x80 | 0x04 | protocol header (2 bytes) +------+------+-----------+ | AA BB CC DD EE FF GG HH | frame size (8 bytes, little-endian) +------+------------------+ -- Repository URL: http://hg.python.org/peps

1 0