[Python-Dev] Socket timeout and completion based sockets

[Guido van Rossum]

On Tue, Nov 27, 2012 at 1:23 AM, Kristján Valur Jónsson
<kristjan at ccpgames.com> wrote:
> Yes, well, as a matter of fact, I do have an IOCP based socket implementation with stackless python.

It would have been nice if you had given more context and stated your
objective upfront instead of asking what appeared to be an obscure
question about a technical detail. I have been distracted by other
stuff temporarily, but I do plan to continue working on tulip more,
and I want to assure you that EVERYTHING IS STILL OPEN. I.e. don't
take the current tulip code base as a draft PEP -- it is just my way
of learning about the issues. Also note that Richard Oudkerk has
developed an alternative, which you can find in the "proactor" branch
of the tulip repository.

> From the programmer's perspective, operations appear blocking while IOCP is used to switch tasklets in the background.
> But my socket timeout implementation does not guarantee that the socket is left in a valid state for retrying a recv() operation.
> I was under the (perhaps mistaken) impression that the current async work _could_ result in a standardized way to create such
> alternative socket implementations, ones that might do their magic using greenlets, tasklets, or generators.  But if that were
> the case, such loosely defined features of the socket API would need clearer definitions.

If you have a specific requirement, please state it explicitly, rather
than just hoping tulip will be your solution. FWIW, it is likely that
tulip will hide the actual socket object completely inside a
higher-level abstraction, so that the actual semantics of sockets
(which are extremely platform-dependent) cannot affect the
specification of the API. There will be platform-specific ways to
reach inside, but they will be of limited use -- it's possible that on
Windows (at least when using IOCP) there won't be a socket object at
all.

Finally, I am not at all interested in greenlets(*). Tulip will
support two API surfaces: a low-level one, suitable for interfacing
with e.g. Twisted or Tornado, that uses callbacks to signal ready-ness
or completion. And a high-level one, useful for e.g. writing protocol
handling libraries and applications, that will somehow use yield
and/or yield-from. The details of each layer are very much unspecified
at this point. NOW WOULD BE A GOOD TIME TO WRITE DOWN YOUR
REQUIREMENTS.

(*) Greenlets are a fine mechanism for some application areas, but
ultimately not fit for the standard library, and they have some
significant downsides.

--Guido

> K
>> -----Original Message-----
>> From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf
>> Of Guido van Rossum
>> Sent: 26. nóvember 2012 15:59
>> To: Kristján Valur Jónsson
>> Cc: Python-Dev (python-dev at python.org)
>> Subject: Re: [Python-Dev] Socket timeout and completion based sockets
>>
>> If you're talking about the standard socket module, I'm not aware that it uses
>> IOCP on Windows. Are you asking this just in the abstract, or do you know of
>> a Python implementation that uses IOCP to implement the standard socket
>> type?
>>
>> As to the design of the async I/O library (which I am still working on!), I
>> cannot guarantee anything, and the issue will probably be moot
>> -- the API won't have the same kind of timeout as the current socket object
>> (it will have other ways to set deadlines though).
>
>



-- 
--Guido van Rossum (python.org/~guido)