On Tue, Jan 29, 2013 at 12:08 PM, Saúl Ibarra Corretgé <saghul@gmail.com> wrote:
[snip] [snip*2] I'm no windows expert either :-) AFAIS, IOCP provides a completion-based interface, but many people/libraries are used to level-triggered readiness notifications. It's apparently not easy to have unix style file descriptor polling in Windows, but that AFD Poll stuff (fairy dust to me, to be honest) does the trick. It only works for sockets, but I guess that's ok.
Yeah, so do the other polling things on Windows. (Well, mostly sockets. There are some other things supported like named pipes.) I guess in order to support this we'd need some kind of abstraction away from socket objects and file descriptors, at least for event loop methods like sock_recv() and add_reader(). But those are mostly meant for transports to build upon, so I think that would be fine.
- The transport abstraction seems quite tight to socket objects.
I'm confused to hear you say this, since the APIs for transports and protocols are one of the few places of PEP 3156 where sockets are *not* explicitly mentioned. (Though they are used in the implementations, but I am envisioning alternate implementations that don't use sockets.)
Indeed I meant the implementation. For example right now start_serving returns a Python socket object maybe some sort of ServerHandler class could hide that and provide some some convenience methods such as getsockname. If the eventloop implementation uses Python sockets it could just call the function in the underlying sockets, but some other implementations may have other means so gather that information.
Ah, yes, the start_serving() API. It is far from ready. :-(
pyuv provides a TCP and UDP handles, which provide a completion-style API and use a better approach than Poll handles.
So it implements TCP and UDP without socket objects? I actually like this, because it validates my decision to keep socket objects out of the transport/protocol APIs. (Note that PEP 3156 and Tulip currently don't support UDP; it will require a somewhat different API between transports and protocols.)
Yes, the TCP and UDP handles from pyuv are wrappers to their corresponding types in libuv. They exist because JS doesn't have sockets so the had to create them for nodejs. The API, however, is completion style, here is a simple example on how data is read from a TCP handle:
def on_data_received(handle, data, error): if error == pyuv.error.UV_EOF: # Remove closed the connection handle.close() return print(data)
tcp_handle.start_read(on_data_received)
This model actually fits pretty well in tulip's transport/protocol mechanism.
Yeah, I see. If we squint and read "handle" instead of "socket" we could even make it so that loop.sock_recv() takes one of these -- it would return a Future and your callback would set the Future's result, or its exception if an error was set.
They should give better performance since EINTR in handled internally and there are less roundtrips between Python-land and C-land.
Why would EINTR handling be important? That should occur almost never. Or did you mean EAGAIN?
Actually, both. If the process receives signal epoll_wait would be interrupted, and libuv takes care of rearming the file descriptor, which happens in C without the GIL. Same goes for EAGAIN, basically libuv tries to read 64k chunks when start_read is called, and it automatically retires on EAGAIN. I don't have number to back this up (yet) but conceptually sounds pretty plausible.
Hm. Anything that uses signals for its normal operation sounds highly suspect to me. But it probably doesn't matter either way.
Was it ever considered to provide some sort of abstraction so that transports can be used on top of something other than regular sockets? For example I see no way to get the remote party from the transport, without checking the underlying socket.
This we are considering in another thread -- there are in fact two proposals on the table, one to add transport methods get_name() and get_peer(), which should return (host, port) pairs if possible, or None if the transport is not talking to an IP connection (or there are too many layers in between to dig out that information). The other proposal is a more generic API to get info out of the transport, e.g. get_extra_info("name") and get_extra_info("peer"), which can be more easily extended (without changing the PEP) to support other things, e.g. certificate info if the transport implements SSL.
The second model seems more flexible indeed. I guess the SSL transport could be tricky, because while currently Tulip uses the ssl module I have no TLS handle on pyuv so I'd have to build one on top of a TCP handle with pyOpenSSL (I have a prototype here [1]), so object types / APIs wouldn't match, unless Tulip provides some wrappers for SSL related objects such as certificates...
Hm, I thought certificates were just blobs of data? We should probably come up with a standard way to represent these that isn't tied to the stdlib's ssl module. But I don't think this should be part of PEP 3156 -- it's too big already.
Saúl Ibarra Corretgé http://saghul.net/blog | http://about.me/saghul
-- --Guido van Rossum (python.org/~guido)