(Trimming stuff that doesn't need a reply -- this doesn't mean I
agree, just that I don't see a need for more discussion.)
On Sun, Jan 6, 2013 at 7:45 AM, Markus
Exactly - signals are a mess, threading and signals make things worse - I'm no expert here, but I just have had experienced problems with signal handling and threads, basically the same problems you describe. Creating the threads after installing signal handlers (in the main thread) works, and signals get delivered to the main thread, installing the signal handlers (in the main thread) after creating the threads - and the signals ended up in *some thread*. Additionally it depended on if you'd install your signal handler with signal() or sigaction() and flags when creating threads.
So I suppose you're okay with the signal handling API I proposed? I'll add it to the PEP then, with a note that it may raise an exception if not supported.
I'd expect the EventLoop never to create threads on his own behalf, it's just wrong.
Here's the way it works. You can call run_in_executor(executor, function, *args) where executor is an executor (a fancy thread pool) that you create. You have full control. However you can pass executor=None and then the event loop will create its own, default executor -- or it will use a default executor that you have created and given to it previously. It needs the default executor so that it can implement getaddrinfo() by calling the stdlib socket.getaddrinfo() in a thread -- and getaddrinfo() is essential for creating transports. The user can take full control over the executor though -- you could set the default to something that always raises an exception.
If you can't provide some functionality without threads, don't provide the functionality.
I don't see this as an absolute requirement. The threads are an implementation detail (other event loop implementations could implement getaddrinfo() differently, taking directly to DNS using tasklets or callbacks), and you can control its use of threads.
Besides, getaddrinfo() is a bad choice, as it relies on distribution specific flags. For example ip6 link local scope exists on every current platform, but - when resolving an link local scope address -not domain- with getaddrinfo, getaddrinfo will fail if no global routed ipv6 address is available on debian/ubuntu.
Nevertheless it is the only thing available in the stdlib. If you want to improve it, that's fine, but just use the issue tracker.
As Transport are part of the PEP - some more:
EventLoop * create_transport(protocol_factory, host, port, **kwargs) kwargs requires "local" - local address as tuple like ('fe80::14ad:1680:54e1:6a91%eth0',0) - so you can bind when using ipv6 link local scope. or ('192.168.2.1',5060) - bind local port for udp
Not sure I understand. What socket.connect() (or other API) call parameters does this correspond to? What can't expressed through the host and port parameters?
In case you have multiple interfaces, and multiple gateways, you need to assign the connection to an address - so the kernel knows which interface to use for the connection - else he'd default to "the first" interface. In IPv6 link-local scope you can have multiple addresses in the same subnet fe80:: - IIRC if you want to connect somewhere, you have to either set the scope_id of the remote, or bind the "source" address before - I don't know how to set the scope_id in python, it's in sockaddr_in6.
In terms of socket. it is a bind before a connect.
s = socket.socket(AF_INET6,SOCK_DGRAM,0) s.bind(('fe80::1',0)) s.connect(('fe80::2',4712))
same for ipv4 in case you are multi homed and rely on source based routing.
Ok, this seems a useful option to add to create_transport(). Your example shows SOCK_DGRAM -- is it also relevant for SOCK_STREAM?
Handler: Requiring 2 handlers for every active connection r/w is highly ineffective.
How so? What is the concern?
Of course you can fold the fdsets, but in case you need a seperate handler for write, you re-create it for every write - see below.
That would seem to depend on the write rate.
Additionally, I can .stop() the handler without having to know the fd, .stop() the handler, change the events the handler is looking for, restart the handler with .start(). In your proposal, I'd create a new handler every time I want to sent something, poll for readability - discard the handler when I'm done, create a new one for the next sent.
The questions are, does it make any difference in efficiency (when using Python -- the performance of the C API is hardly relevant here), and how often does this pattern occur.
Every time you send - you poll for write-ability, you get the callback, you write, you got nothing left, you stop polling for write-ability.
That's not quite how it's implemented. The code first tries to send without polling. Since the socket is non-blocking, if this succeeds, great -- only if it returns a partial send or EAGAIN we register a callback. If the protocol keeps the buffer filled the callback doesn't have to be recreated each time. If the protocol doesn't keep the buffer full, we must unregister the callback to prevent select/poll/etc. from calling it over and over again, there's nothing you can do about that.
* reconnect() - I'd love to be able to reconnect a transport
But what does that mean in general? It depends on the protocol (e.g. FTP, HTTP, IRC, SMTP) how much state must be restored/renegotiated upon a reconnect, and how much data may have to be re-sent. This seems a higher-level feature that transports and protocols will have to implement themselves.
I don't need the EventLoop to sync my state upon reconnect - just have the Transport providing the ability. Protocols are free to use this, but do not have to.
Aha, I get it. You want to be able to call transport.reconnect() from connection_lost() and it should respond by eventually calling protocol.connection_made(transport) again. Of course, this only applies to clients -- for a server to reconnect to a client makes no sense (it would be up to the client). That seems simple enough to implement, but Glyph recommended strongly against this, because reusing the protocol object often means that some private state of the protocol may not be properly reinitialized. It would also be difficult to decide where errors from the reconnect attempt should go -- reconnect() itself must return immediately (since connection_lost() cannot wait for I/O, it can only schedule async I/O events). But at a higher level in your app it would be easy to set this up: you just call eventloop.create_transport(lambda: protocol, ...) where protocol is a protocol instance you've created earlier.
Twisted suggested something here which I haven't implemented yet but which seems reasonable -- using a series of short timeouts try connecting to the various addresses and keep the first one that connects successfully. If multiple addresses connect after the first timeout, too bad, just close the redundant sockets, little harm is done (though the timeouts should be tuned that this is relatively rare, because a server may waste significant resources on such redundant connects).
Fast, yes - reasonable? - no. How would you feel if web browsers behaved like this?
I have no idea -- who says they aren't doing this? Browsers do tons of stuff that I am not aware of.
domain name has to be resolved, addresses ordered according to rfc X which says prefer ipv6 etc., try connecting linear.
Sure. It was just an idea. I'll see what Twisted actually does.
For almost all the timeouts I mentioned - the protocol needs to take care - so the protocol has to exist before the connection is established in case of outbound connections.
I'm not sure I follow. Can you sketch out some code to help me here? ISTM that e.g. the DNS, connect and handshake timeouts can be implemented by the machinery that tries to set up the connection behind the scenes, and the user's protocol won't know anything of these shenanigans. The code that calls create_transport() (actually it'll probably be renamed create_client()) will just get a Future that either indicates success (and then the protocol and transport are successfully hooked up) or an error (and then no protocol was created -- whether or not a transport was created is an implementation detail).
From my understanding the Future does not provide any information which connection to which host using which protocol and credentials failed?
That's not up to the Future -- it just passes an exception object along. We could make this info available as attributes on the exception object, if there is a need.
I'd create the Procotol when trying to create a connection, so the Protocol is informed when the Transport fails and can take action - retry, whatever.
I had this in an earlier version, but Glyph convinced me that this is the wrong design -- and it doesn't work for servers anyway, you must have a protocol factory there.
* data_received(data) - if it was possible to return the number of bytes consumed by the protocol, and have the Transport buffer the rest for the next io in call, one would avoid having to do this in every Protocol on it's own - learned from experience.
Twisted has a whole slew of protocol implementation subclasses that implement various strategies like line-buffering (including a really complex version where you can turn the line buffering on and off) and "netstrings". I am trying to limit the PEP's size by not including these, but I fully expect that in practice a set of useful protocol implementations will be created that handles common cases. I'm not convinced that putting this in the transport/protocol interface will make user code less buggy: it seems easy for the user code to miscount the bytes or not return a count at all in a rarely taken code branch.
Please don't drop this.
You never know how much data you'll receive, you never know how much data you need for a message, so the Protocol needs a buffer.
That all depends on what the protocol is trying to do. (The ECHO protocol certainly doesn't need a buffer. :-)
Having this io in buffer in the Transports allows every Protocol to benefit, they try to read a message from the data passed to data_received(), if the data received is not sufficient to create a full message, they need to buffer it and wait for more data.
Having it in a Protocol base class also allows every protocol that wants it to benefit, without complicating the transport. I can also see problems where the transport needs to keep calling data_received() until either all data is consumed or it returns 0 (no data consumed). It just doesn't seem right to make the transport responsible for this logic, since it doesn't know enough about the needs of the protocol.
So having the Protocol.data_received return the number of bytes the Protocol could process, the Transport can do the job, saving it for every Protocol. Still - a protocol can have it's own buffering strategy, i.e. in case of a incremental XML parser which does it's own buffering, and always return len(data), so the Transport does not buffer anything.
Right, data_received() is closely related to the concept of a "feed parser" which is used in a few places in the stdlib (http://docs.python.org/3/search.html?q=feed&check_keywords=yes&area=default) and even has a 3rd party implementation (http://pypi.python.org/pypi/feedparser/), and there the parser (i.e. the protocol equivalent) is always responsible for buffering data it cannot immediately process.
Next, what happens if a dns can not be resolved, ssl handshake (in case ssl is transport) or connecting fails - in my opinion it's an error the protocol is supposed to take care of + error_dns + error_ssl + error_connecting
The future returned by create_transport() (aka create_client()) will raise the exception.
When do I get this exception - the EventLoop.run() raises?
No, the eventloop doesn't normally raise, just whichever task is waiting for that future using 'yield from' will get the exception. Or you can use eventloop.run_until_complete(<future>) and then that call will raise. -- --Guido van Rossum (python.org/~guido)