[Python-Dev] Socket timeout and completion based sockets

Guido van Rossum guido at python.org
Wed Nov 28 18:04:26 CET 2012

On Wed, Nov 28, 2012 at 4:13 AM, Kristján Valur Jónsson
<kristjan at ccpgames.com> wrote:
> I'm sorry, I thought it was something that people did more often, to create different implementations of of the "socket" api, for which cPython provided a mere reference implementation.  I know of at least three different alternative implementations, so I thought that the question were clear enough:  Is the timeout mechanism "supposed" to be re-startable for an api that aims to conform to the "socket" module, or is that a mere coincidence falling out from the select/bsd based reference implementation in cPython?  The docs don't say either way.

We're going to have to decide here, since nobody has thought about
this enough apparently. I see two possible answers: we can make it
implementation-dependent, or we can require conforming implementations
to implement properly restartable semantics (if they support timeouts
at all). A third option would be to require these semantics *if the
timeout option is supported* but leave it up to the implementation to
support it at all (ditto for nonblocking, i.e. timeout=0).

> (For c-level timeout mechanisms implemented for various c implementations of the bsd socket api,  it is not uncommon to see it stated that after a socket operation times out, the socket is in an undefined state and should be discarded, e.g. here:  http://msdn.microsoft.com/en-us/library/windows/desktop/ms740476(v=vs.85).aspx " If a send or receive operation times out on a socket, the socket state is indeterminate, and should not be used; TCP sockets in this state have a potential for data loss, since the operation could be canceled at the same moment the operation was to be completed.")

Is this relevant to CPython though? Its socket implementation uses
select() to implement timeouts, even on Windows, AFAIK.

> Anyway, as for concrete requirements:  The issue I have always seen with various asynchronous libraries is their lack of composability.  Everyone writes their own application loop and event queue.  Merely having a standard spec and reference implementation of an application main loop object, and main event queue object, in the spirit of WSGI, would possibly remedy this.  You could then hopefully assemble various different libraries in the same application, including greenlet(*) based ones.

Hm. I agree with the first part of this -- and indeed I am planning to
make it so that tulip's event loop can easily be replaced by another
one. I'm less sure about the yield-from-based scheduler, that's the
kind of thing for which it doesn't really make sense to have multiple
implementations. If greenlets can work with the standard event loop
interface, good for them. (Either by providing a conforming
implementation that also supports greenlets, or by just using the
standard implementation.)

> (*) Greenlets or stackless can be just another way of hiding asynchronous operations from the programmer.  My favourite one, in fact.  The main trick here is unwinding and replaying of calling contexts, the specific implementation by stack-slicking is mere technical detail, since it can be achieved in other ways (see soft-switching in stackless python)

Yes. While none of these belong in the stdlib, I certainly want to
support their viability as a 3rd party alternative to yield-from.

Note however that even Christian Tismer has expressed doubts about
hiding async blocks completely -- for the same reasons I'm not keen on
them myself: when it's not obvious whether a particular call can cause
a context switch (e.g. due to something it uses indirectly doing some
kind of I/O that requires a context switch to avoid blocking), you're
back in the world of threads and explicit locks and all the nightmares
that come with that. Using yield or yield-from to mark switch points
means you will always be aware of the possibility that a call switches
(unfortunately there are other costs).


> Cheers,
> K
>> -----Original Message-----
>> From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf
>> Of Guido van Rossum
>> Sent: 27. nóvember 2012 15:54
> with stackless python.
>> It would have been nice if you had given more context and stated your
>> objective upfront instead of asking what appeared to be an obscure question
>> about a technical detail
>> Finally, I am not at all interested in greenlets
>> ...
>> very much unspecified at this point. NOW WOULD BE A GOOD TIME TO
>> (*) Greenlets are a fine mechanism for some application areas, but ultimately
>> not fit for the standard library, and they have some significant downsides.

--Guido van Rossum (python.org/~guido)

More information about the Python-Dev mailing list