Mailman 3 October 2013 - Python-checkins

cpython: Close #19442: warn_explicit() does nothing when called late during Python
by victor.stinner 31 Oct '13

31 Oct '13

http://hg.python.org/cpython/rev/13a05ed33cf7 changeset: 86816:13a05ed33cf7 user: Victor Stinner <victor.stinner(a)gmail.com> date: Fri Nov 01 00:55:30 2013 +0100 summary: Close #19442: warn_explicit() does nothing when called late during Python shutdown After more tests, I now think that it is the safest option. files: Python/_warnings.c | 18 +++++++++--------- 1 files changed, 9 insertions(+), 9 deletions(-) diff --git a/Python/_warnings.c b/Python/_warnings.c --- a/Python/_warnings.c +++ b/Python/_warnings.c @@ -333,6 +333,13 @@ PyObject *action; int rc; + /* module can be None if a warning is emitted late during Python shutdown. + In this case, the Python warnings module was probably unloaded, filters + are no more available to choose as action. It is safer to ignore the + warning and do nothing. */ + if (module == Py_None) + Py_RETURN_NONE; + if (registry && !PyDict_Check(registry) && (registry != Py_None)) { PyErr_SetString(PyExc_TypeError, "'registry' must be a dict"); return NULL; @@ -635,15 +642,8 @@ if (!setup_context(stack_level, &filename, &lineno, &module, &registry)) return NULL; - if (module != Py_None) { - res = warn_explicit(category, message, filename, lineno, module, registry, - NULL); - } - else { - /* FIXME: emitting warnings at exit does crash Python */ - res = Py_None; - Py_INCREF(res); - } + res = warn_explicit(category, message, filename, lineno, module, registry, + NULL); Py_DECREF(filename); Py_DECREF(registry); Py_DECREF(module); -- Repository URL: http://hg.python.org/cpython

1 0

peps: Huge update to various parts of the spec. Still not done.
by guido.van.rossum 31 Oct '13

31 Oct '13

http://hg.python.org/peps/rev/9e960cac9e94 changeset: 5240:9e960cac9e94 user: Guido van Rossum <guido(a)dropbox.com> date: Thu Oct 31 15:48:48 2013 -0700 summary: Huge update to various parts of the spec. Still not done. files: pep-3156.txt | 630 +++++++++++++++++++++++--------------- 1 files changed, 372 insertions(+), 258 deletions(-) diff --git a/pep-3156.txt b/pep-3156.txt --- a/pep-3156.txt +++ b/pep-3156.txt @@ -39,7 +39,7 @@ force acceptance of the PEP. The expectation is that the package will keep provisional status in Python 3.4 and progress to final status in Python 3.5. Development continues to occur primarily in the Tulip -repo. +repo, with changes occasionally merged into the CPython repo. Dependencies ------------ @@ -47,7 +47,7 @@ Python 3.3 is required for many of the proposed features. The reference implementation (Tulip) requires no new language or standard library features beyond Python 3.3, no third-party modules or -packages, and no C code, except for the proactor-based event loop on +packages, and no C code, except for the (optional) IOCP support on Windows. Module Namespace @@ -99,10 +99,10 @@ implementation. In order to support both directions of adaptation, two separate APIs -are defined: +are specified: -- getting and setting the current event loop object -- the interface of a conforming event loop and its minimum guarantees +- An interface for managing the current event loop +- The interface of a conforming event loop An event loop implementation may provide additional methods and guarantees, as long as these are called out in the documentation as @@ -110,17 +110,19 @@ methods unimplemented if they cannot be implemented in the given environment; however, such deviations from the standard API should be considered only as a last resort, and only if the platform or -environment forces the issue. (An example could be a platform where -there is a system event loop that cannot be started or stopped.) +environment forces the issue. (An example would be a platform where +there is a system event loop that cannot be started or stopped; see +"Embedded Event Loops" below.) The event loop API does not depend on ``yield from``. Rather, it uses a combination of callbacks, additional interfaces (transports and protocols), and Futures. The latter are similar to those defined in PEP 3148, but have a different implementation and are not tied to -threads. In particular, they have no wait() method; the user is -expected to use callbacks. +threads. In particular, the ``result()`` method raises an exception +instead of blocking when a result is not yet ready; the user is +expected to use callbacks (or ``yield from``) to wait for the result. -All event loop methods documented as returning a coroutine are allowed +All event loop methods specified as returning a coroutine are allowed to return either a Future or a coroutine, at the implementation's choice (the standard implementation always returns coroutines). All event loop methods documented as accepting coroutine arguments *must* @@ -180,10 +182,11 @@ that can be viewed this way, for example an SSH session or a pair of UNIX pipes. Typically there aren't many different transport implementations, and most of them come with the event loop -implementation. (But there is no requirement that all transports must -be created by calling an event loop method -- a third party module may -well implement a new transport and provide a constructor or factory -function for it that simply takes an event loop as an argument.) +implementation. However, there is no requirement that all transports +must be created by calling an event loop method: a third party module +may well implement a new transport and provide a constructor or +factory function for it that simply takes an event loop as an argument +or calls ``get_event_loop()``. Note that transports don't need to use sockets, not even if they use TCP -- sockets are a platform-specific implementation detail. @@ -233,6 +236,7 @@ document their policy and at what point during their initialization sequence the policy is set, in order to avoid undefined behavior when multiple active frameworks want to override the default policy. +(See also "Embedded Event Loops" below.) To get the event loop for current context, use ``get_event_loop()``. This returns an event loop object implementing the interface specified @@ -280,6 +284,28 @@ ``DefaultEventLoopPolicy``. The current event loop policy object can be retrieved by calling ``get_event_loop_policy()``. +Specifying Times +---------------- + +As usual in Python, all timeouts, intervals and delays are measured in +seconds, and may be ints or floats. However, absolute times are not +specified as POSIX timestamps. The accuracy, precision and epoch of +the clock are up to the implementation. + +The default implementation uses ``time.monotonic()``. Books could be +written about the implications of this choice. Better read the docs +for the stdandard library ``time`` module. + +Embedded Event Loops +-------------------- + +On some platforms an event loop is provided by the system. Such a +loop may already be running when the user code starts, and there may +be no way to stop or close it without exiting from the program. In +this case, the methods for starting, stopping and closing the event +loop may not be implementable, and ``is_running()`` may always return +``True``. + Event Loop Classes ------------------ @@ -300,89 +326,59 @@ API for "overlapped I/O"). The constructor takes one optional argument, a ``Proactor`` object. By default an instance of ``IocpProactor`` is created and used. (The ``IocpProactor`` class - is not specified by this PEP. Its inclusion in the standard library - is not currently under consideration; it is just an implementation + is not specified by this PEP; it is just an implementation detail of the ``ProactorEventLoop`` class.) Event Loop Methods Overview --------------------------- The methods of a conforming event loop are grouped into several -categories. A brief overview of the categories. The first set of -categories must be supported by all conforming event loop -implementations. (However, in some cases a partially-conforming -implementation may choose not to implement the internet/socket -methods, and still conform to the other methods. Also, it is possible -that a partially-conforming implementation has no way to create, -close, start, or stop a loop.) +categories. The first set of categories must be supported by all +conforming event loop implementations, with the exception that +embedded event loops may not implement the methods for starting, +stopping and closing. (However, a partially-conforming event loop is +still better than nothing. :-) -- Miscellaneous: ``close()``, ``time()``. +- Starting, stopping and closing: ``run_forever()``, + ``run_until_complete()``, ``stop()``, ``is_running()``, ``close()``. -- Starting and stopping: ``run_forever()``, ``run_until_complete()``, - ``stop()``, ``is_running()``. - -- Basic callbacks: ``call_soon()``, ``call_later()``, ``call_at()``. +- Basic and timed callbacks: ``call_soon()``, ``call_later()``, + ``call_at()``, ``time()``. - Thread interaction: ``call_soon_threadsafe()``, ``run_in_executor()``, ``set_default_executor()``. - Internet name lookups: ``getaddrinfo()``, ``getnameinfo()``. -- Internet connections: ``create_connection()``, ``start_serving()``, - ``stop_serving()``, ``create_datagram_endpoint()``. +- Internet connections: ``create_connection()``, ``create_server()``, + ``create_datagram_endpoint()``. - Wrapped socket methods: ``sock_recv()``, ``sock_sendall()``, ``sock_connect()``, ``sock_accept()``. The second set of categories *may* be supported by conforming event loop implementations. If not supported, they will raise -``NotImplementedError``. (In the current state of Tulip, +``NotImplementedError``. (In the default implementation, ``SelectorEventLoop`` on UNIX systems supports all of these; ``SelectorEventLoop`` on Windows supports the I/O event handling -category; ``ProactorEventLoop`` on Windows supports None. The -intention is to add support for pipes and subprocesses on Windows as -well, using the ``subprocess`` module in the standard library.) +category; ``ProactorEventLoop`` on Windows supports the pipes and +subprocess category.) - I/O callbacks: ``add_reader()``, ``remove_reader()``, ``add_writer()``, ``remove_writer()``. - Pipes and subprocesses: ``connect_read_pipe()``, - ``connect_write_pipe()``, ``spawn_subprocess()``. + ``connect_write_pipe()``, ``subprocess_shell()``, + ``subprocess_exec()``. - Signal callbacks: ``add_signal_handler()``, ``remove_signal_handler()``. -Specifying Times ----------------- +Event Loop Methods +------------------ -As usual in Python, all timeouts, intervals and delays are measured in -seconds, and may be ints or floats. The accuracy and precision of the -clock are up to the implementation; the default implementation uses -``time.monotonic()``. Books could be written about the implications -of this choice. Better read the docs for the stdandard library -``time`` module. - -Required Event Loop Methods ---------------------------- - -Miscellaneous -''''''''''''' - -- ``close()``. Closes the event loop, releasing any resources it may - hold, such as the file descriptor used by ``epoll()`` or - ``kqueue()``. This should not be called while the event loop is - running. After it has been called the event loop may not be used - again. It may be called multiple times; subsequent calls are - no-ops. - -- ``time()``. Returns the current time according to the event loop's - clock. This may be ``time.time()`` or ``time.monotonic()`` or some - other system-specific clock, but it must return a float expressing - the time in units of approximately one second since some epoch. - (No clock is perfect -- see PEP 418.) - -Starting and Stopping -''''''''''''''''''''' +Starting, Stopping and Closing +'''''''''''''''''''''''''''''' An (unclosed) event loop can be in one of two states: running or stopped. These methods deal with starting and stopping an event loop: @@ -412,9 +408,14 @@ ``call_soon()``) it processes before stopping. - ``is_running()``. Returns ``True`` if the event loop is currently - running, ``False`` if it is stopped. (TBD: Do we need another - inquiry method to tell whether the loop is in the process of - stopping?) + running, ``False`` if it is stopped. + +- ``close()``. Closes the event loop, releasing any resources it may + hold, such as the file descriptor used by ``epoll()`` or + ``kqueue()``, and the default executor. This should not be called + while the event loop is running. After it has been called the event + loop should not be used again. It may be called multiple times; + subsequent calls are no-ops. Basic Callbacks ''''''''''''''' @@ -426,24 +427,30 @@ shared state isn't changed by another callback. - ``call_soon(callback, *args)``. This schedules a callback to be - called as soon as possible. Returns a Handle representing the - callback, whose ``cancel()`` method can be used to cancel the - callback. It guarantees that callbacks are called in the order in - which they were scheduled. + called as soon as possible. Returns a ``Handle`` (see below) + representing the callback, whose ``cancel()`` method can be used to + cancel the callback. It guarantees that callbacks are called in the + order in which they were scheduled. - ``call_later(delay, callback, *args)``. Arrange for ``callback(*args)`` to be called approximately ``delay`` seconds in - the future, once, unless cancelled. Returns a Handle representing + the future, once, unless cancelled. Returns a ``Handle`` representing the callback, whose ``cancel()`` method can be used to cancel the callback. Callbacks scheduled in the past or at exactly the same time will be called in an undefined order. - ``call_at(when, callback, *args)``. This is like ``call_later()``, but the time is expressed as an absolute time. Returns a similar - Handle. There is a simple equivalency: ``loop.call_later(delay, + ``Handle``. There is a simple equivalency: ``loop.call_later(delay, callback, *args)`` is the same as ``loop.call_at(loop.time() + delay, callback, *args)``. +- ``time()``. Returns the current time according to the event loop's + clock. This may be ``time.time()`` or ``time.monotonic()`` or some + other system-specific clock, but it must return a float expressing + the time in units of approximately one second since some epoch. + (No clock is perfect -- see PEP 418.) + Note: A previous version of this PEP defined a method named ``call_repeatedly()``, which promised to call a callback at regular intervals. This has been withdrawn because the design of such a @@ -464,22 +471,25 @@ - ``call_soon_threadsafe(callback, *args)``. Like ``call_soon(callback, *args)``, but when called from another thread while the event loop is blocked waiting for I/O, unblocks the event - loop. This is the *only* method that is safe to call from another - thread. (To schedule a callback for a later time in a threadsafe - manner, you can use ``loop.call_soon_threadsafe(loop.call_later, - when, callback, *args)``.) Note: this is not safe to call from a - signal handler (since it may use locks). In fact, no API is - signal-safe; if you want to handle signals, use - ``add_signal_handler()`` described below. + loop. Returns a ``Handle``. This is the *only* method that is safe + to call from another thread. (To schedule a callback for a later + time in a threadsafe manner, you can use + ``loop.call_soon_threadsafe(loop.call_later, when, callback, + *args)``.) Note: this is not safe to call from a signal handler + (since it may use locks). In fact, no API is signal-safe; if you + want to handle signals, use ``add_signal_handler()`` described + below. - ``run_in_executor(executor, callback, *args)``. Arrange to call - ``callback(*args)`` in an executor (see PEP 3148). Returns a Future - whose result on success is the return value of that call. This is - equivalent to ``wrap_future(executor.submit(callback, *args))``. If - ``executor`` is ``None``, the default executor set by - ``set_default_executor()`` is used. If no default executor has been - set yet, a ``ThreadPoolExecutor`` with 5 threads is created and set - as the default executor. + ``callback(*args)`` in an executor (see PEP 3148). Returns an + ``asyncio.Future`` instance whose result on success is the return + value of that call. This is equivalent to + ``wrap_future(executor.submit(callback, *args))``. If ``executor`` + is ``None``, the default executor set by ``set_default_executor()`` + is used. If no default executor has been set yet, a + ``ThreadPoolExecutor`` with a default number of threads is created + and set as the default executor. (The default implementation uses + 5 threads in this case.) - ``set_default_executor(executor)``. Set the default executor used by ``run_in_executor()``. The argument must be a PEP 3148 @@ -495,7 +505,7 @@ These methods are useful if you want to connect or bind a socket to an address without the risk of blocking for the name lookup. They are usually called implicitly by ``create_connection()``, -``start_serving()`` or ``create_datagram_endpoint()``. +``create_server()`` or ``create_datagram_endpoint()``. - ``getaddrinfo(host, port, family=0, type=0, proto=0, flags=0)``. Similar to the ``socket.getaddrinfo()`` function but returns a @@ -545,8 +555,8 @@ - ``create_connection(protocol_factory, host, port, <options>)``. Creates a stream connection to a given internet host and port. This is a task that is typically called from the client side of the - connection. It creates an implementation-dependent (bidirectional - stream) Transport to represent the connection, then calls + connection. It creates an implementation-dependent bidirectional + stream Transport to represent the connection, then calls ``protocol_factory()`` to instantiate (or retrieve) the user's Protocol implementation, and finally ties the two together. (See below for the definitions of Transport and Protocol.) The user's @@ -560,7 +570,7 @@ (*) There is no requirement that ``protocol_factory`` is a class. If your protocol class needs to have specific arguments passed to - its constructor, you can use ``lambda`` or ``functools.partial()``. + its constructor, you can use ``lambda``. You can also pass a trivial ``lambda`` that returns a previously constructed Protocol instance. @@ -568,14 +578,13 @@ - ``ssl``: Pass ``True`` to create an SSL/TLS transport (by default a plain TCP transport is created). Or pass an ``ssl.SSLContext`` - object (or something else that implements the same interface) to - override the default SSL context object to be used. If a default - context is created it is up to the implementation to configure - reasonable defaults. The reference implementation currently uses - ``PROTOCOL_SSLv23`` and sets the ``OP_NO_SSLv2`` option, calls - ``set_default_verify_paths()`` and sets verify_mode to - ``CERT_REQUIRED``. In addition, whenever the context (default or - otherwise) specifies a verify_mode of ``CERT_REQUIRED`` or + object to override the default SSL context object to be used. If + a default context is created it is up to the implementation to + configure reasonable defaults. The reference implementation + currently uses ``PROTOCOL_SSLv23`` and sets the ``OP_NO_SSLv2`` + option, calls ``set_default_verify_paths()`` and sets verify_mode + to ``CERT_REQUIRED``. In addition, whenever the context (default + or otherwise) specifies a verify_mode of ``CERT_REQUIRED`` or ``CERT_OPTIONAL``, if a hostname is given, immediately after a successful handshake ``ssl.match_hostname(peercert, hostname)`` is called, and if this raises an exception the conection is closed. @@ -592,9 +601,9 @@ Protocol concept or the ``protocol_factory`` argument. - ``sock``: An optional socket to be used instead of using the - ``host``, ``port``, ``family``, ``proto``, and ``flags`` - arguments. If this is given, host and port must be omitted; - otherwise, host and port are required. + ``host``, ``port``, ``family``, ``proto`` and ``flags`` + arguments. If this is given, ``host`` and ``port`` must be + explicitly set to ``None``. - ``local_addr``: If given, a ``(host, port)`` tuple used to bind the socket to locally. This is rarely needed but on multi-homed @@ -602,21 +611,31 @@ specific address. This is how you would do that. The host and port are looked up using ``getaddrinfo()``. + - ``server_hostname``: This is only relevant when using SSL/TLS; it + should not be used when ``ssl`` is not set. When ``ssl`` is set, + this sets or overrides the hostname that will be verified. By + default the value of the ``host`` argument is used. If ``host`` + is empty, there is no default and you must pass a value for + ``server_hostname``. To disable hostname verification (which is a + serious security risk) you must pass an empty string here and pass + an ``ssl.SSLContext`` object whose ``verify_mode`` is set to + ``ssl.CERT_NONE`` as the ``ssl`` argument. + - ``create_server(protocol_factory, host, port, <options>)``. Enters a serving loop that accepts connections. This is a coroutine that completes once the serving loop is set up to serve. The return value is a ``Server`` object which can be used - to stop the serving loop in a controlled fashion by calling its - ``close()`` method. + to stop the serving loop in a controlled fashion (see below). + Multiple sockets may be bound if the specified address allows + both IPv4 and IPv6 connections. - Multiple sockets may be bound if the specified address allows - both IPv4 and IPv6 connections. Each time a connection is accepted, - ``protocol_factory`` is called without arguments(*) to create a - Protocol, a (bidirectional stream) Transport is created to represent + Each time a connection is accepted, + ``protocol_factory`` is called without arguments(**) to create a + Protocol, a bidirectional stream Transport is created to represent the network side of the connection, and the two are tied together by calling ``protocol.connection_made(transport)``. - (*) See footnote above for ``create_connection()``. However, since + (**) See previous footnote for ``create_connection()``. However, since ``protocol_factory()`` is called once for each new incoming connection, it should return a new Protocol object each time it is called. @@ -625,12 +644,13 @@ - ``ssl``: Pass an ``ssl.SSLContext`` object (or an object with the same interface) to override the default SSL context object to be - used. (Unlike ``create_connection()``, passing ``True`` does not - make sense -- the ``SSLContext`` object is required to specify the - certificate and key.) + used. (Unlike for ``create_connection()``, passing ``True`` does + not make sense here -- the ``SSLContext`` object is needed to + specify the certificate and key.) - ``backlog``: Backlog value to be passed to the ``listen()`` call. - Defaults to ``100``. + The default is implementation-dependent; in the default + implementation the default value is ``100``. - ``reuse_address``: Whether to set the ``SO_REUSEADDR`` option on the socket. The default is ``True`` on UNIX, ``False`` on @@ -643,21 +663,11 @@ ``0``, to let ``getaddrinfo()`` choose.) - ``sock``: An optional socket to be used instead of using the - ``host``, ``port``, ``family``, and ``flags`` arguments. If this - is given, host and port must be omitted; otherwise, host and port - are required. The return value will be the one-element list - ``[sock]``. - -- ``stop_serving(sock)``. The argument should be a socket from the - list returned by ``start_serving()``. The serving loop associated - with that socket will be stopped. Connections that have already - been accepted will not be affected. (TBD: What if - ``start_serving()`` doesn't use sockets? Then it should probably - return a list of opaque objects that can be passed to - ``stop_serving()``.) + ``host``, ``port``, ``family`` and ``flags`` arguments. If this + is given, ``host`` and ``port`` must be explicitly set to ``None``. - ``create_datagram_endpoint(protocol_factory, local_addr=None, - remote_addr=None, **kwds)``. Creates an endpoint for sending and + remote_addr=None, <options>)``. Creates an endpoint for sending and receiving datagrams (typically UDP packets). Because of the nature of datagram traffic, there are no separate calls to set up client and server side, since usually a single endpoint acts as both client @@ -666,16 +676,19 @@ the coroutine returns successfully, the transport will call callbacks on the protocol whenever a datagram is received or the socket is closed; it is up to the protocol to call methods on the - protocol to send datagrams. Note that the transport and protocol - interfaces used here are different than those for stream - connections. + protocol to send datagrams. The transport returned is a + ``DatagramTransport``. The protocol returned is a + ``DatagramProtocol``. These are described later. - Arguments: + Mandatory positional argument: - ``protocol_factory``: A class or factory function that will be - called, without arguments, to construct the protocol object to be - returned. The interface between datagram transport and protocol - is described below. + called exactly once, without arguments, to construct the protocol + object to be returned. The interface between datagram transport + and protocol is described below. + + Optional arguments that may be specified positionally or as keyword + arguments: - ``local_addr``: An optional tuple indicating the address to which the socket will be bound. If given this must be a ``(host, @@ -693,7 +706,9 @@ to be resolved and the result will be passed to ``sock_connect()`` together with the socket created. If ``getaddrinfo()`` returns more than one address, they will be tried in turn. If omitted, - no ``sock_connect()`` will be made. + no ``sock_connect()`` call will be made. + + The <options> are all specified using optional keyword arguments: - ``family``, ``proto``, ``flags``: Address family, protocol and flags to be passed through to ``getaddrinfo()``. These all @@ -737,9 +752,6 @@ where ``conn`` is a connected non-blocking socket and ``peer`` is the peer address. -Optional Event Loop Methods ---------------------------- - I/O Callbacks ''''''''''''' @@ -756,7 +768,7 @@ and possibly tty devices are also supported, but disk files are not. Exactly which special file types are supported may vary by platform and per selector implementation. (Experimentally, there is at least -one kind of pseudo-tty on OSX that is supported by ``select`` and +one kind of pseudo-tty on OS X that is supported by ``select`` and ``poll`` but not by ``kqueue``: it is used by Emacs shell windows.) - ``add_reader(fd, callback, *args)``. Arrange for @@ -779,33 +791,90 @@ Pipes and Subprocesses '''''''''''''''''''''' -TBD: Should these really take stream objects? The stream objects are -not useful for reading or writing because they would cause blocking -I/O. This section of the API is clearly not yet ready for review. +These methods are supported by ``SelectorEventLoop`` on UNIX and +``ProactorEventLoop`` on Windows. + +The transports and protocols used with pipes and subprocesses differ +from those used with regular stream connections. These are described +later. + +Each of the methods below has a ``protocol_factory`` argument, similar +to ``create_connection()``; this will be called exactly once, without +arguments, to construct the protocol object to be returned. + +Each method is a coroutine that returns a ``(transport, protocol)`` +pair on success, or raises an exception on failure. - ``connect_read_pipe(protocol_factory, pipe)``: Create a unidrectional stream connection from a file-like object wrapping the - read end of a UNIX pipe. The protocol/transport interface is the - read half of the bidirectional stream interface. + read end of a UNIX pipe, which must be in non-blocking mode. The + transport returned is a ``ReadTransport``. - ``connect_write_pipe(protocol_factory, pipe)``: Create a unidrectional stream connection from a file-like object wrapping the - write end of a UNIX pipe. The protocol/transport interface is the - write half of the bidirectional stream interface. + write end of a UNIX pipe, which must be in non-blocking mode. The + transport returned is a ``WriteTransport``; it does not have any + read-related methods. The protocol returned is a ``BaseProtocol``. -- TBD: A way to run a subprocess with stdin, stdout and stderr - connected to pipe transports. (This is implemented now but not yet - documented.) (TBD: Document that the subprocess's - connection_closed() won't be called until the process has exited - *and* all pipes are closed, and why -- it's a race condition.) +- ``subprocess_shell(protocol_factory, cmd, <options>)``: Create a + subprocess from ``cmd``, which is a string using the platform's + "shell" syntax. This is similar to the standard library + ``subprocess.Popen()`` class called with ``shell=True``. The + remaining arguments and return value are described below. -TBD: offer the same interface on Windows for e.g. named pipes. (This -should be possible given that the standard library ``subprocess`` -module is supported on Windows.) +- ``subprocess_exec(protocol_factory, *args, <options>)``: Create a + subprocess from one or more string arguments, where the first string + specifies the program to execute, and the remaining strings specify + the program's arguments. (Thus, together the string arguments form + the ``sys.argv`` value of the program, assuming it is a Python + script.) This is similar to the standard library + ``subprocess.Popen()`` class called with ``shell=False`` and the + list of strings passed as the first argument; however, where + ``Popen()`` takes a single argument which is list of strings, + ``subprocess_exec()`` takes multiple string arguments. The + remaining arguments and return value are described below. + +Apart from the way the program to execute is specified, the two +``subprocess_*()`` methods behave the same. The transport returned is +a ``SubprocessTransport`` which has a different interface than the +common bidirectional stream transport. The protocol returned is a +``SubprocessProtocol`` which also has a custom interface. + +The <options> are all specified using optional keyword arguments: + +- ``stdin``: Either a file-like object representing the pipe to be + connected to the subprocess's standard input stream using + ``create_write_pipe()``, or the constant ``subprocess.PIPE`` (the + default). By default a new pipe will be created and connected. + +- ``stdout``: Either a file-like object representing the pipe to be + connected to the subprocess's standard output stream using + ``create_read_pipe()``, or the constant ``subprocess.PIPE`` (the + default). By default a new pipe will be created and connected. + +- ``stderr``: Either a file-like object representing the pipe to be + connected to the subprocess's standard error stream using + ``create_read_pipe()``, or one of the constants ``subprocess.PIPE`` + (the default) or ``subprocess.STDOUT``. By default a new pipe will + be created and connected. When ``subprocess.STDOUT`` is specified, + the subprocess's standard error stream will be connected to the same + pipe as the stdandard output stream. + +- ``bufsize``: The buffer size to be used when creating a pipe; this + is passed to ``subprocess.Popen()``. In the default implementation + this defaults to zero, and on Windows it must be zero; these + defaults deviate from ``subprocess.Popen()``. + +- ``executable``, ``preexec_fn``, ``close_fds``, ``cwd``, ``env``, + ``startupinfo``, ``creationflags``, ``restore_signals``, + ``start_new_session``, ``pass_fds``: These optional arguments are + passed to ``subprocess.Popen()`` without interpretation. Signal callbacks '''''''''''''''' +These methods are only supported on UNIX. + - ``add_signal_handler(sig, callback, *args)``. Whenever signal ``sig`` is received, arrange for ``callback(*args)`` to be called. Specifying another callback for the same signal replaces the @@ -826,36 +895,17 @@ handler was set. Note: If these methods are statically known to be unsupported, they -may return ``NotImplementedError`` instead of ``RuntimeError``. +may raise ``NotImplementedError`` instead of ``RuntimeError``. -Callback Sequencing -------------------- +Mutual Exclusion of Callbacks +----------------------------- -When two callbacks are scheduled for the same time, they are run -in the order in which they are registered. For example:: - - loop.call_soon(foo) - loop.call_soon(bar) - -guarantees that ``foo()`` is called before ``bar()``. - -If ``call_soon()`` is used, this guarantee is true even if the system -clock were to run backwards. This is also the case for -``call_later(0, callback, *args)``. However, if ``call_later()`` is -used with a nonzero delay, all bets are off if the system -clock were to runs backwards. (A good event loop implementation -should use ``time.monotonic()`` to avoid problems when the clock runs -backward. See PEP 418.) - -Context -------- - -All event loops have a notion of context. For the default event loop -implementation, the context is a thread. An event loop implementation -should run all callbacks in the same context. An event loop -implementation should run only one callback at a time, so callbacks -can assume automatic mutual exclusion with other callbacks scheduled -in the same event loop. +An event loop should enforce mutual exclusion of callbacks, i.e. it +should never start a callback while a previously callback is still +running. This should apply across all types of callbacks, regardless +of whether they are scheduled using ``call_soon()``, ``call_later()``, +``call_at()``, ``call_soon_threadsafe()``, ``add_reader()``, +``add_writer()``, or ``add_signal_handler()``. Exceptions ---------- @@ -867,11 +917,11 @@ be passed through by Futures, and they will be logged and ignored when they occur in a callback. -However, exceptions deriving only from ``BaseException`` are never -caught, and will usually cause the program to terminate with a -traceback. (Examples of this category include ``KeyboardInterrupt`` -and ``SystemExit``; it is usually unwise to treat these the same as -most other exceptions.) +However, exceptions deriving only from ``BaseException`` are typically +not caught, and will usually cause the program to terminate with a +traceback. In some cases they are caught and re-raised. (Examples of +this category include ``KeyboardInterrupt`` and ``SystemExit``; it is +usually unwise to treat these the same as most other exceptions.) Handles ------- @@ -880,19 +930,35 @@ (``call_soon()``, ``call_later()``, ``call_at()`` and ``call_soon_threadsafe()``) all return an object representing the registration that can be used to cancel the callback. This object is -called a Handle (although its class name is not necessarily -``Handle``). Handles are opaque and have only one public method: +called a ``Handle``. Handles are opaque and have only one public +method: -- ``cancel()``. Cancel the callback. +- ``cancel()``: Cancel the callback. +Note that ``add_reader()``, ``add_writer()`` and +``add_signal_handler()`` do not return Handles. + +Servers +------- + +The ``create_server()`` method returns a ``Server`` instance, which +wraps the sockets (or other network objects) used to accept requests. +This class has two public methods: + +- ``close()``: Close the service. This stops accepting new requests + but does not cancel requests that have already been accepted and are + currently being handled. + +- ``wait_closed()``: A coroutine that blocks until the service is + closed and all accepted requests have been handled. Futures ------- -The ``tulip.Future`` class here is intentionally similar to the +The ``asyncio.Future`` class here is intentionally similar to the ``concurrent.futures.Future`` class specified by PEP 3148, but there are slight differences. Whenever this PEP talks about Futures or -futures this should be understood to refer to ``tulip.Future`` unless +futures this should be understood to refer to ``asyncio.Future`` unless ``concurrent.futures.Future`` is explicitly mentioned. The supported public API is as follows, indicating the differences with PEP 3148: @@ -968,12 +1034,12 @@ A Future is associated with the default event loop when it is created. (TBD: Optionally pass in an alternative event loop instance?) -A ``tulip.Future`` object is not acceptable to the ``wait()`` and +A ``asyncio.Future`` object is not acceptable to the ``wait()`` and ``as_completed()`` functions in the ``concurrent.futures`` package. -However, there are similar APIs ``tulip.wait()`` and -``tulip.as_completed()``, described below. +However, there are similar APIs ``asyncio.wait()`` and +``asyncio.as_completed()``, described below. -A ``tulip.Future`` object is acceptable to a ``yield from`` expression +A ``asyncio.Future`` object is acceptable to a ``yield from`` expression when used in a coroutine. This is implemented through the ``__iter__()`` interface on the Future. See the section "Coroutines and the Scheduler" below. @@ -981,9 +1047,9 @@ When a Future is garbage-collected, if it has an associated exception but neither ``result()`` nor ``exception()`` nor ``__iter__()`` has ever been called (or the latter hasn't raised the exception yet -- -details TBD), the exception should be logged. TBD: At what level? +details TBD), the exception is logged. -In the future (pun intended) we may unify ``tulip.Future`` and +In the future (pun intended) we may unify ``asyncio.Future`` and ``concurrent.futures.Future``, e.g. by adding an ``__iter__()`` method to the latter that works with ``yield from``. To prevent accidentally blocking the event loop by calling e.g. ``result()`` on a Future @@ -995,16 +1061,16 @@ There are some public functions related to Futures: -- ``async(arg)``. This takes an argument that is either a coroutine - object or a Future (i.e., anything you can use with ``yield from``) - and returns a Future. If the argument is a Future, it is returned - unchanged; if it is a coroutine object, it wraps it in a Task - (remember that ``Task`` is a subclass of ``Future``). +- ``asyncio.async(arg)``. This takes an argument that is either a + coroutine object or a Future (i.e., anything you can use with + ``yield from``) and returns a Future. If the argument is a Future, + it is returned unchanged; if it is a coroutine object, it wraps it + in a Task (remember that ``Task`` is a subclass of ``Future``). -- ``wrap_future(future)``. This takes a PEP 3148 Future (i.e., an - instance of ``concurrent.futures.Future``) and returns a Future - compatible with the event loop (i.e., a ``tulip.Future`` instance). - +- ``asyncio.wrap_future(future)``. This takes a PEP 3148 Future + (i.e., an instance of ``concurrent.futures.Future``) and returns a + Future compatible with the event loop (i.e., a ``asyncio.Future`` + instance). Transports ---------- @@ -1172,7 +1238,7 @@ Datagram transports call the following methods on the associated protocol object: ``connection_made()``, ``connection_lost()``, -``connection_refused()``, and ``datagram_received()``. ("Connection" +``connection_refused()`` and ``datagram_received()``. ("Connection" in these method names is a slight misnomer, but the concepts still exist: ``connection_made()`` means the transport representing the endpoint has been created, and ``connection_lost()`` means the @@ -1182,6 +1248,11 @@ feature). (TBD: Do we need the latter? It seems easy enough to implement this in the protocol if it needs to make the distinction.) +Subprocess Transports +''''''''''''''''''''' + +TBD. + Protocols --------- @@ -1275,6 +1346,11 @@ 3. ``connection_refused()`` -- at most once 4. ``connection_lost()`` -- exactly once +Subprocess Protocol +''''''''''''''''''' + +TBD. + Callback Style -------------- @@ -1290,10 +1366,9 @@ the callback. This allows graceful evolution of the API without having to worry about whether a keyword might be significant to a callee somewhere. If you have a callback that *must* be called with a -keyword argument, you can use a lambda or ``functools.partial``. For -example:: +keyword argument, you can use a lambda. For example:: - loop.call_soon(functools.partial(foo, "abc", repeat=42)) + loop.call_soon(lambda: foo('abc', repeat=42)) Coroutines and the Scheduler @@ -1310,7 +1385,7 @@ A coroutine is a generator that follows certain conventions. For documentation purposes, all coroutines should be decorated with -``(a)tulip.coroutine``, but this cannot be strictly enforced. +``(a)asyncio.coroutine``, but this cannot be strictly enforced. Coroutines use the ``yield from`` syntax introduced in PEP 380, instead of the original ``yield`` syntax. @@ -1319,7 +1394,7 @@ different (though related) concepts: - The function that defines a coroutine (a function definition - decorated with ``tulip.coroutine``). If disambiguation is needed + decorated with ``asyncio.coroutine``). If disambiguation is needed we will call this a *coroutine function*. - The object obtained by calling a coroutine function. This object @@ -1362,7 +1437,7 @@ ``wait()`` and ``as_completed()`` APIs in the ``concurrent.futures`` package are provided: -- ``tulip.wait(fs, timeout=None, return_when=ALL_COMPLETED)``. This +- ``asyncio.wait(fs, timeout=None, return_when=ALL_COMPLETED)``. This is a coroutine that waits for the Futures or coroutines given by ``fs`` to complete. Coroutine arguments will be wrapped in Tasks (see below). This returns a Future whose result on success is a @@ -1389,7 +1464,7 @@ Futures from the condition is surprising, but PEP 3148 does it this way.) -- ``tulip.as_completed(fs, timeout=None)``. Returns an iterator whose +- ``asyncio.as_completed(fs, timeout=None)``. Returns an iterator whose values are Futures or coroutines; waiting for successive values waits until the next Future or coroutine from the set ``fs`` completes, and returns its result (or raises its exception). The @@ -1406,13 +1481,13 @@ your ``for`` loop may not make progress (since you are not allowing other tasks to run). -- ``tulip.wait_for(f, timeout)``. This is a convenience to wait for a +- ``asyncio.wait_for(f, timeout)``. This is a convenience to wait for a single coroutine or Future with a timeout. It is a simple wrapper - around ``tulip.wait()`` with a single item in the first argument, + around ``asyncio.wait()`` with a single item in the first argument, returning the result or raising the exception if it is completed within the timeout, raising ``TimeoutError`` otherwise. -- ``tulip.gather(f1, f2, ...)``. Returns a Future which waits until +- ``asyncio.gather(f1, f2, ...)``. Returns a Future which waits until all arguments (Futures or coroutines) are done and return a list of their corresponding results. If one or more of the arguments is cancelled or raises an exception, the returned Future is cancelled @@ -1420,12 +1495,12 @@ argument), and the remaining arguments are left running in the background. Cancelling the returned Future does not affect the arguments. Note that coroutine arguments are converted to Futures - using ``tulip.async()``. + using ``asyncio.async()``. Sleeping -------- -The coroutine ``sleep(delay)`` returns after a given time delay. +The coroutine ``asyncio.sleep(delay)`` returns after a given time delay. (TBD: Should the optional second argument, ``result``, be part of the spec?) @@ -1440,22 +1515,22 @@ becomes the task's result, if it raises an exception, that becomes the task's exception. -Cancelling a task that's not done yet prevents its coroutine from -completing. In this case a ``CancelledError`` exception is thrown -into the coroutine, which it may catch to propagate cancellation to -other Futures. If the exception is not caught, the generator will be -properly finalized anyway, as described in PEP 342. +Cancelling a task that's not done yet throws an +``asyncio.CancelledError`` exception into the coroutine. If the +coroutine doesn't catch this (or if it re-raises it) the task will be +marked as cancelled; but if the coroutine somehow catches and ignores +the exception it may continue to execute. Tasks are also useful for interoperating between coroutines and callback-based frameworks like Twisted. After converting a coroutine into a Task, callbacks can be added to the Task. To convert a coroutine into a task, call the coroutine function and -pass the resulting coroutine object to the ``tulip.Task()`` -constructor. You may also use ``tulip.async()`` for this purpose. +pass the resulting coroutine object to the ``asyncio.Task()`` +constructor. You may also use ``asyncio.async()`` for this purpose. You may ask, why not automatically convert all coroutines to Tasks? -The ``(a)tulip.coroutine`` decorator could do this. However, this would +The ``(a)asyncio.coroutine`` decorator could do this. However, this would slow things down considerably in the case where one coroutine calls another (and so on), as switching to a "bare" coroutine has much less overhead than switching to a Task. @@ -1487,23 +1562,49 @@ off the coroutine when ``connection_made()`` is called. +TO DO +===== + +- Document pause/resume reading/writing. + +- Document subprocess/pipe protocols/transports. + +- Document locks and queues. + +- Describe task cancellation details (and update future cancellation + spec to promise less). + +- Explain that eof_received() shouldn't return True with SSL/TLS. + +- Document SIGCHILD handling API (once it lands). + +- Compare all APIs with the source code to be sure there aren't any + undocumented or unimplemented features. + + +Wish List +========= + +- Support a "start TLS" operation to upgrade a TCP socket to SSL/TLS. + +- UNIX domain sockets. + +- A per-loop error handling callback. + + Open Issues =========== -- Make ``loop.stop()`` optional (and hence - ``loop.run_until_complete()`` too). +- Why do ``create_connection()`` and ``create_datagram_endpoint()`` + have a ``proto`` argument but not ``create_server()``? And why are + the family, flag, proto arguments for ``getaddrinfo()`` sometimes + zero and sometimes named constants (whose value is also zero)? + +- Do we need another inquiry method to tell whether the loop is in the + process of stopping?) - A fuller public API for Handle? What's the use case? -- Should we require all event loops to implement ``sock_recv()`` and - friends? Is the use case strong enough? (Maybe some transports - usable with both ``SelectorEventLoop`` and ``ProactorEventLoop`` use - them? That'd be a good enough use case for me.) - -- Should we require callers of ``create_connection()`` to create and - pass an ``SSLContext`` explicitly, instead of allowing ssl=True? - (For ``start_serving()`` we already require an ``SSLContext``.) - - A debugging API? E.g. something that logs a lot of stuff, or logs unusual conditions (like queues filling up faster than they drain) or even callbacks taking too much time... @@ -1511,16 +1612,11 @@ - Do we need introspection APIs? E.g. asking for the read callback given a file descriptor. Or when the next scheduled call is. Or the list of file descriptors registered with callbacks. Right now - these would all require using Tulip internals. + these all require using internals. -- Locks and queues? The Tulip implementation contains implementations - of most types of locks and queues modeled after the standard library - ``threading`` and ``queue`` modules. These should be incorporated - into the PEP. - -- Probably need more socket I/O methods, e.g. ``sock_sendto()`` and - ``sock_recvfrom()``, and perhaps others like ``pipe_read()``. Or - users can write their own (it's not rocket science). +- Do we need more socket I/O methods, e.g. ``sock_sendto()`` and + ``sock_recvfrom()``, and perhaps others like ``pipe_read()``? + I guess users can write their own (it's not rocket science). - We may need APIs to control various timeouts. E.g. we may want to limit the time spent in DNS resolution, connecting, ssl/tls handshake, @@ -1564,16 +1660,34 @@ =============== Apart from PEP 3153, influences include PEP 380 and Greg Ewing's -tutorial for ``yield from``, Twisted, Tornado, ZeroMQ, pyftpdlib, tulip -(the author's attempts at synthesis of all these), wattle (Steve -Dower's counter-proposal), numerous discussions on python-ideas from -September through December 2012, a Skype session with Steve Dower and -Dino Viehland, email exchanges with Ben Darnell, an audience with -Niels Provos (original author of libevent), and two in-person meetings -with several Twisted developers, including Glyph, Brian Warner, David -Reid, and Duncan McGreggor. Also, the author's previous work on async -support in the NDB library for Google App Engine was an important -influence. +tutorial for ``yield from``, Twisted, Tornado, ZeroMQ, pyftpdlib, and +wattle (Steve Dower's counter-proposal). My previous work on +asynchronous support in the NDB library for Google App Engine provided +an important starting point. + +I am grateful for the numerous discussions on python-ideas from +September through December 2012, and many more on python-tulip since +then; a Skype session with Steve Dower and Dino Viehland; email +exchanges with and a visit by Ben Darnell; an audience with Niels +Provos (original author of libevent); and in-person meetings (as well +as frequent email exchanges) with several Twisted developers, +including Glyph, Brian Warner, David Reid, and Duncan McGreggor. + +Contributors to the implementation include +Eli Bendersky, +Saúl Ibarra Corretgé, +Geert Jansen, +A. Jesse Jiryu Davis, +Nikolay Kim, +Charles-François Natali, +Richard Oudkerk, +Antoine Pitrou, +Giampaolo Rodolá, +Andrew Svetlov, +and many others who submitted bugs and/or fixes. + +I thank Antoine Pitrou for his feedback in his role of official PEP +BDFL. Copyright -- Repository URL: http://hg.python.org/peps

1 0

cpython (merge 3.3 -> default): Closes #19349: Merged fix from 3.3.
by vinay.sajip 31 Oct '13

31 Oct '13

http://hg.python.org/cpython/rev/e97d9926da83 changeset: 86815:e97d9926da83 parent: 86813:a3c68d919cd1 parent: 86814:731bdec35fdd user: Vinay Sajip <vinay_sajip(a)yahoo.co.uk> date: Thu Oct 31 18:44:48 2013 +0000 summary: Closes #19349: Merged fix from 3.3. files: Lib/venv/__init__.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/venv/__init__.py b/Lib/venv/__init__.py --- a/Lib/venv/__init__.py +++ b/Lib/venv/__init__.py @@ -345,7 +345,7 @@ elif not hasattr(sys, 'base_prefix'): compatible = False if not compatible: - raise ValueError('This script is only for use with Python 3.3') + raise ValueError('This script is only for use with Python >= 3.3') else: import argparse -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.3): Issue #19349: Corrected error message.
by vinay.sajip 31 Oct '13

31 Oct '13

http://hg.python.org/cpython/rev/731bdec35fdd changeset: 86814:731bdec35fdd branch: 3.3 parent: 86811:54bf7b5ec3b6 user: Vinay Sajip <vinay_sajip(a)yahoo.co.uk> date: Thu Oct 31 18:44:04 2013 +0000 summary: Issue #19349: Corrected error message. files: Lib/venv/__init__.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/venv/__init__.py b/Lib/venv/__init__.py --- a/Lib/venv/__init__.py +++ b/Lib/venv/__init__.py @@ -336,7 +336,7 @@ elif not hasattr(sys, 'base_prefix'): compatible = False if not compatible: - raise ValueError('This script is only for use with Python 3.3') + raise ValueError('This script is only for use with Python >= 3.3') else: import argparse -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Close resources owned by subclass before calling super().close().
by guido.van.rossum 31 Oct '13

31 Oct '13

http://hg.python.org/cpython/rev/a3c68d919cd1 changeset: 86813:a3c68d919cd1 user: Guido van Rossum <guido(a)dropbox.com> date: Thu Oct 31 11:01:40 2013 -0700 summary: Close resources owned by subclass before calling super().close(). files: Lib/selectors.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/selectors.py b/Lib/selectors.py --- a/Lib/selectors.py +++ b/Lib/selectors.py @@ -351,8 +351,8 @@ return ready def close(self): + self._epoll.close() super().close() - self._epoll.close() if hasattr(select, 'kqueue'): @@ -414,8 +414,8 @@ return ready def close(self): + self._kqueue.close() super().close() - self._kqueue.close() # Choose the best implementation: roughly, epoll|kqueue > poll > select. -- Repository URL: http://hg.python.org/cpython

1 0

cpython (merge 3.3 -> default): Null merge
by tim.golden 31 Oct '13

31 Oct '13

http://hg.python.org/cpython/rev/6a2d27f5d48c changeset: 86812:6a2d27f5d48c parent: 86810:995173ed248a parent: 86811:54bf7b5ec3b6 user: Tim Golden <mail(a)timgolden.me.uk> date: Thu Oct 31 17:40:16 2013 +0000 summary: Null merge files: -- Repository URL: http://hg.python.org/cpython

1 0

cpython (3.3): Issue #19418 Fix some warnings on Win64
by tim.golden 31 Oct '13

31 Oct '13

http://hg.python.org/cpython/rev/54bf7b5ec3b6 changeset: 86811:54bf7b5ec3b6 branch: 3.3 parent: 86798:e6941225e014 user: Tim Golden <mail(a)timgolden.me.uk> date: Thu Oct 31 17:38:24 2013 +0000 summary: Issue #19418 Fix some warnings on Win64 files: Modules/audioop.c | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Modules/audioop.c b/Modules/audioop.c --- a/Modules/audioop.c +++ b/Modules/audioop.c @@ -27,7 +27,8 @@ #endif static const int maxvals[] = {0, 0x7F, 0x7FFF, 0x7FFFFF, 0x7FFFFFFF}; -static const int minvals[] = {0, -0x80, -0x8000, -0x800000, -0x80000000}; +/* -1 trick is needed on Windows to support -0x80000000 without a warning */ +static const int minvals[] = {0, -0x80, -0x8000, -0x800000, -0x7FFFFFFF-1}; static const unsigned int masks[] = {0, 0xFF, 0xFFFF, 0xFFFFFF, 0xFFFFFFFF}; static int @@ -385,7 +386,9 @@ signed char *cp; Py_ssize_t len, i; int size, val = 0; - int min = 0x7fffffff, max = -0x80000000; + /* -1 trick below is needed on Windows to support -0x80000000 without + a warning */ + int min = 0x7fffffff, max = -0x7FFFFFFF-1; if (!PyArg_ParseTuple(args, "s#i:minmax", &cp, &len, &size)) return NULL; -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #19437: Cleanup r_ref() of the marshal module
by victor.stinner 31 Oct '13

31 Oct '13

http://hg.python.org/cpython/rev/995173ed248a changeset: 86810:995173ed248a user: Victor Stinner <victor.stinner(a)gmail.com> date: Thu Oct 31 17:09:01 2013 +0100 summary: Issue #19437: Cleanup r_ref() of the marshal module files: Python/marshal.c | 11 ++++++----- 1 files changed, 6 insertions(+), 5 deletions(-) diff --git a/Python/marshal.c b/Python/marshal.c --- a/Python/marshal.c +++ b/Python/marshal.c @@ -827,11 +827,12 @@ static PyObject * r_ref(PyObject *o, int flag, RFILE *p) { - if (o != NULL && flag) { /* currently only FLAG_REF is defined */ - if (PyList_Append(p->refs, o) < 0) { - Py_DECREF(o); /* release the new object */ - return NULL; - } + assert(flag & FLAG_REF); + if (o == NULL) + return NULL; + if (PyList_Append(p->refs, o) < 0) { + Py_DECREF(o); /* release the new object */ + return NULL; } return o; } -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #19437: Fix r_object() of marshal module, handle PyDict_SetItem() failure
by victor.stinner 31 Oct '13

31 Oct '13

http://hg.python.org/cpython/rev/fc54e2848c98 changeset: 86809:fc54e2848c98 user: Victor Stinner <victor.stinner(a)gmail.com> date: Thu Oct 31 17:14:52 2013 +0100 summary: Issue #19437: Fix r_object() of marshal module, handle PyDict_SetItem() failure for TYPE_DICT and stop immedialty on first r_object() failure files: Python/marshal.c | 15 ++++++++++++--- 1 files changed, 12 insertions(+), 3 deletions(-) diff --git a/Python/marshal.c b/Python/marshal.c --- a/Python/marshal.c +++ b/Python/marshal.c @@ -1117,6 +1117,7 @@ R_REF(v); if (v == NULL) break; + for (i = 0; i < n; i++) { v2 = r_object(p); if ( v2 == NULL ) { @@ -1170,10 +1171,17 @@ if (key == NULL) break; val = r_object(p); - if (val != NULL) - PyDict_SetItem(v, key, val); + if (val == NULL) { + Py_DECREF(key); + break; + } + if (PyDict_SetItem(v, key, val) < 0) { + Py_DECREF(key); + Py_DECREF(val); + break; + } Py_DECREF(key); - Py_XDECREF(val); + Py_DECREF(val); } if (PyErr_Occurred()) { Py_DECREF(v); @@ -1204,6 +1212,7 @@ } if (v == NULL) break; + for (i = 0; i < n; i++) { v2 = r_object(p); if ( v2 == NULL ) { -- Repository URL: http://hg.python.org/cpython

1 0

cpython: Issue #19437: Fix r_object() of marshal module, handle r_byte() failure for
by victor.stinner 31 Oct '13

31 Oct '13

http://hg.python.org/cpython/rev/9e836a945ea9 changeset: 86808:9e836a945ea9 user: Victor Stinner <victor.stinner(a)gmail.com> date: Thu Oct 31 17:07:08 2013 +0100 summary: Issue #19437: Fix r_object() of marshal module, handle r_byte() failure for TYPE_SMALL_TUPLE files: Python/marshal.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Python/marshal.c b/Python/marshal.c --- a/Python/marshal.c +++ b/Python/marshal.c @@ -1101,6 +1101,8 @@ case TYPE_SMALL_TUPLE: n = (unsigned char) r_byte(p); + if (PyErr_Occurred()) + break; goto _read_tuple; case TYPE_TUPLE: n = r_long(p); -- Repository URL: http://hg.python.org/cpython

1 0