On Fri, Jan 4, 2013 at 6:53 PM, Markus wrote:

On Fri, Jan 4, 2013 at 11:33 PM, Guido van Rossum wrote:

...

On Wed, Dec 26, 2012 at 2:38 PM, Markus wrote:

...

First shoot should be getting a well established event loop into python.

Perhaps. What is your definition of an event loop?

I ask the loop to notify me via callback if something I care about happens.

Heh. That's rather too general -- it depends on "something I care about" which could be impossible to guess. :-)

Usually that's fds and read/writeability.

Ok, although on some platforms it can't be a fd (UNIX-style small integer) but some other abstraction, e.g. a socket *object* in Jython or a "handle" on Windows (but I am already starting to repeat myself :-).

I create a data structure which has the fd, the event I care about, the callback and userdata, pass it to the loop, and the loop will take care.

Next, timers, same story, I create a data structure which has the time I care about, the callback and userdata, pass it to the loop, and the loop will take care.

The "create data structure" part is a specific choice of interface style, not necessarily the best for Python. Most event loop implementations I've seen for Python (pyev excluded) just have various methods that express everything through the argument list, not with a separate data structure.

Signals - sometimes having signals in the event loop is handy too. Same story.

Agreed, I've added this to the open issues section in the PEP. Do you have a suggestion for a minimal interface for signal handling? I could imagine the following: - add_signal_handler(sig, callback, *args). Whenever signal 'sig' is received, arrange for callback(*args) to be called. Returns a Handler which can be used to cancel the signal callback. Specifying another callback for the same signal replaces the previous handler (only one handler can be active per signal). - remove_signal_handler(sig). Removes the handler for signal 'sig', if one is set. Is anything else needed? Note that Python only receives signals in the main thread, and the effect may be undefined if the event loop is not running in the main thread, or if more than one event loop sets a handler for the same signal. It also can't work for signals directed to a specific thread (I think POSIX defines a few of these, but I don't know of any support for these in Python.)

...

But sockets are not native on Windows, and I am making some effort with PEP 3156 to efficiently support higher-level abstractions without tying them to sockets. (The plan is to support IOCP on Windows. The previous version of Tulip already had a branch that did support that, as a demonstration of the power of this abstraction.)

Supporting IOCP on windows is absolutely required, as WSAPoll is broken and won't be fixed. http://social.msdn.microsoft.com/Forums/hu/wsk/thread/18769abd-fca0-4d3c-988...

Wow. Now I'm even more glad that we're planning to support IOCP.

...

Only if the C code also uses libev, of course. But C programs may use other event mechanisms -- e.g. AFAIK there are alternatives to libev (during the early stages of Tulip development I chatted a bit with one of the original authors of libevent, Niels Provos, and I believe there's also something called libuv), and GUI frameworks (e.g. X, Qt, Gtk, Wx) tend to have their own event loop.

libuv is a wrapper around libev -adding IOCP- which adds some other things besides an event loop and is developed for/used in node.js.

Ah, that's helpful. I did not realize this after briefly skimming the libuv page. (And the github logs suggest that it may no longer be the case: https://github.com/joyent/libuv/commit/1282d64868b9c560c074b9c9630391f3b18ef...

...

PEP 3156 is designed to let alternative *implementations* of the same *interface* be selected at run time. Hopefully it is possible to provide a conforming implementation using libev -- then your goal (smooth interoperability with C code using libev) is obtained.

Smooth interoperability is not a major goal here - it's great if you get it for free. I'm just looking forward an event loop in the stdlib I want to use.

Heh, so stop objecting. :-)

...

(It would also be harder to implement initially as a 3rd party framework. At the lowest level, no changes to Python itself are needed -- it already supports non-blocking sockets, for example. But adding optional callbacks to existing low-level APIs would require changes throughout the stdlib.)

As a result - making the stdlib async io aware - the complete stdlib. Would be great.

No matter what API style is chosen, making the entire stdlib async aware will be tough. No matter what you do, the async support will have to be "pulled through" every abstraction layer -- e.g. making sockets async-aware doesn't automatically make socketserver or urllib2 async-aware(*). With the strong requirements for backwards compatibility, in many cases it may be easier to define a new API that is suitable for async use instead of trying to augment existing APIs. (*) Unless you use microthreads, like gevent, but this has its own set of problems -- I don't want to get into that here, since we seem to at least agree on the need for an event loop with callbacks.

...

I am not so concerned about naming (it seems inevitable that everyone uses somewhat different terminology anyway, and it is probably better not to reuse terms when the meaning is different), but I do like to look at guarantees (or the absence thereof!) and best practices for dealing with the differences between platforms.

Handler - the best example for not re-using terms.

??? (Can't tell if you're sarcastic or agreeing here.)

...

You haven't convinced me about this.

Fine, if you include transports, I'll pick on the transports as well ;)

??? (Similar.)

...

However, you can help me by comparing the event loop part of PEP 3156 (ignoring anything that returns or takes a Future) to libev and pointing out things (either specific APIs or certain guarantees or requirements) that would be hard to implement using libev, as well as useful features in libev that you think every event loop should have.

Note: In libev only the "default event loop" can have timers.

Interesting. This seems an odd constraint.

EventLoop

* run() - ev_run(struct ev_loop) * stop() - ev_break(EV_UNLOOP_ALL) * run_forever() - registering an idle watcher will keep the loop alive * run_once(timeout=None) - registering an timer, have the timer stop() the loop * call_later(delay, callback, *args) - ev_timer * call_repeatedly(interval, callback, **args) - ev_timer (periodic) * call_soon(callback, *args) - Equivalent to call_later(0, callback, *args). - call_soon_threadsafe(callback, *args) - it would be better to have

the event loops taking care of signals too, else waking up an ev_async in the loop which checks a async queue which contains the required information to register the call_soon callback would be possible

Not sure I understand. PEP 3156/Tulip uses a self-pipe to prevent race conditions when call_soon_threadsafe() is called from a signal handler or other thread(*) -- but I don't know if that is relevant or not. (*) http://code.google.com/p/tulip/source/browse/tulip/unix_events.py#448 and http://code.google.com/p/tulip/source/browse/tulip/unix_events.py#576

- getaddrinfo(host, port, family=0, type=0, proto=0, flags=0) - libev does not do dns - getnameinfo(sockaddr, flags=0) - libev does not do dns

Note that these exist at least in part so that an event loop implementation may *choose* to implement its own DNS handling (IIUC Twisted has this), whereas the default behavior is just to run socket.getaddrinfo() -- but in a separate thread because it blocks. (This is a useful test case for run_in_executor() too.)

- create_transport(protocol_factory, host, port, **kwargs) - libev does not do transports - start_serving(protocol_factory, host, port, **kwds) - libev does not do transports * add_reader(fd, callback, *args) - create a ev_io watcher with EV_READ * add_writer(fd, callback, *args) - create ev_io watcher with EV_WRITE * remove_reader(fd) - in libev you have to name the watcher you want to stop, you can not remove watchers/handlers by fd, workaround is maintaining a dict with fd:Handler in the EventLoop

Ok, this does not sound like a show-stopper for a conforming PEP 3156 implementation on top of libev then, right? Just a minor inconvenience. I'm sure everyone has *some* impedance mismatches to deal with.

* remove_writer(fd) - same * add_connector(fd, callback, *args) - poll for writeability, getsockopt, done

TBH, I'm not 100% convinced of the need for add_connector(), but Richard Oudkerk claims that it is needed for Windows. (OTOH if WSAPoll() is too broken to bother, maybe we don't need it. It's a bit of a nuisance because code that uses add_writer() instead works just fine on UNIX but would be subtly broken on Windows, leading to disappointments when porting apps to Windows. I'd rather have things break on all platforms, or on none...)

* remove_connector(fd) - same as with all other remove-by-fd methods

As Transport are part of the PEP - some more:

EventLoop * create_transport(protocol_factory, host, port, **kwargs) kwargs requires "local" - local address as tuple like ('fe80::14ad:1680:54e1:6a91%eth0',0) - so you can bind when using ipv6 link local scope. or ('192.168.2.1',5060) - bind local port for udp

Not sure I understand. What socket.connect() (or other API) call parameters does this correspond to? What can't expressed through the host and port parameters?

* start_serving(protocol_factory, host, port, **kwds) what is the behaviour for SOCK_DGRAM - does this multiplex sessions based on src host/port / dst host/port - I'd love it.

TBH I haven't thought much about datagram transports. It's been years since I used UDP. I guess the API may have to distinguish between connected and unconnected UDP. I think the transport/protocol API will be different than for SOCK_STREAM: for every received datagram, the transport will call protocol.datagram_received(data, address), (the address will be a dummy for connected use) and to send a datagram, the protocol must call tranport.write_datagram(data, [address]), which returns immediately. Flow control (if supported) should work the same as for streams: if the transport finds its buffers exceed a certain limit it will tell the protocol to back off by calling protocol.pause().

Handler: Requiring 2 handlers for every active connection r/w is highly ineffective.

How so? What is the concern? The actions of the read and write handler are typically completely different, so the first thing the handler would have to do is to decide whether to call the read or the write code. Also, depending on flow control, only one of the two may be active. If you are after minimizing the number of records passed to [e]poll or kqueue, you can always collapse the handlers at that level and distinguish between read/write based on the mask and recover the appropriate user-level handler from the readers/writers array (and this is what Tulip's epoll pollster class does). PS. Also check out this issue, where an implementation of *just* Tulip's pollster class for the stdlib is being designed: http://bugs.python.org/issue16853; also check out the code reviews here; http://bugs.python.org/review/16853/

I'd prefer to be able to create a Handler from a loop. Handler = EventLoop.create_handler(socket, callback, events) and have the callback called with the returned events, so I can multiplex read/write op in the callback.

Hm. See above.

Additionally, I can .stop() the handler without having to know the fd, .stop() the handler, change the events the handler is looking for, restart the handler with .start(). In your proposal, I'd create a new handler every time I want to sent something, poll for readability - discard the handler when I'm done, create a new one for the next sent.

The questions are, does it make any difference in efficiency (when using Python -- the performance of the C API is hardly relevant here), and how often does this pattern occur.

Timers: Not in the PEP - re-arming a timer lets say I want to do something if nothing happens for 5 seconds. I create a timer call_later(5.,cb), if something happens, I need to cancel the timer and create a new one. If there was a Timer: Timer.stop() Timer.set(5) Timer.start()

Actually it's one less call using the PEP's proposed API: timer.cancel() timer = loop.call_later(5, callback) Which of the two idioms is faster? Who knows? libev's pattern is probably faster in C, but that has little to bear on the cost in Python. My guess is that the amount of work is about the same -- the real cost is that you have to make some changes the heap used to keep track of all timers in the order in which they will trigger, and those changes are the same regardless of how you style the API.

Transports: I think SSL should be a Protocol not a transport - implemented using BIO pairs. If you can chain protocols, like Transport / ProtocolA / ProtocolB you can have TCP / SSL / HTTP as https or TCP / SSL / SOCKS / HTTP as https via ssl enabled socks proxy without having to much problems. Another example, shaping a connection TCP / RATELIMIT / HTTP.

Interesting idea. This may be up to the implementation -- not every implementation may have BIO wrappers available (AFAIK the stdlib doesn't), so the stackability may not be easy to implement everywhere. In any case, when you stack things like this, the stack doesn't look like transport<-->protocol<-->protocol<-->protocol; rather, it's A<-->B<-->C<-->D where each object has a "left" and a "right" API. Each arrow connects the "transport (right) half" of the object on its left (e.g. A) to the "protocol (left) half" of the object on the arrow's right (e.g. B). So maybe we can visualise this as T1 <--> P2:T2 <--> P3:T3 <--> P4.

Having SSL as a Protocol allows closing the SSL connection without closing the TCP connection, re-using the TCP connection, re-using a SSL session cookie during reconnect of the SSL Protocol.

That seems a pretty esoteric use case (though given your background in honeypots maybe common for you :-). It also seems hard to get both sides acting correctly when you do this (but I'm certainly no SSL expert -- I just want it supported because half the web is inaccessible these days if you don't speak SSL, regardless of whether you do any actual verification). All in all I think that stackable transports/protocols are mostly something that is enabled by the interfaces defined here (the PEP takes care not to specify any base classes from which you must inherit -- you must just implement certain methods, and the rest is duck typing) but otherwise does not concern the PEP much. The only concern I have, really, is that the PEP currently hints that both protocols and transports might have pause() and resume() methods for flow control, where the protocol calls transport.pause() if protocol.data_received() is called too frequently, and the transport calls protocol.pause() if transport.write() has buffered more data than sensible. But for an object that is both a protocol and a transport, this would make it impossible to distinguish between pause() calls by its left and right neighbors. So maybe the names must differ. Given the tendency of transport method names to be shorter (e.g. write()) vs. the longer protocol method names (data_received(), connection_lost() etc.), perhaps it should be transport.pause() and protocol.pause_writing() (and similar for resume()).

* reconnect() - I'd love to be able to reconnect a transport

But what does that mean in general? It depends on the protocol (e.g. FTP, HTTP, IRC, SMTP) how much state must be restored/renegotiated upon a reconnect, and how much data may have to be re-sent. This seems a higher-level feature that transports and protocols will have to implement themselves.

* timers - Transports need timers

I think you mean timeouts?

* dns-resolve-timeout - dns can be slow * connecting-timeout - connecting can take too much time, more than we want to wait * idle-timeout ( no action on the connection for a while ) - call protocol.timeout_idle() * sustain-timeout ( max session time ) - close() transport * ssl-handshake-timeout ( in case ssl is a Transport ) - close transport * close-timeout (shutdown is async) - close transport hard * reconnect-timeout - (wait some seconds before reconnecting) - reconnect connection

This is an interesting point. I think some of these really do need APIs in the PEP, others may be implemented using existing machinery (e.g. call_later() to schedule a callback that calls cancel() on a task). I've added a bullet on this to Open Issue.

Now, in case we connect to a host by name, and have multiple addresses resolved, and the first connection can not be established, there is no way to 'reconnect()' - as the protocol does not yet exist.

Twisted suggested something here which I haven't implemented yet but which seems reasonable -- using a series of short timeouts try connecting to the various addresses and keep the first one that connects successfully. If multiple addresses connect after the first timeout, too bad, just close the redundant sockets, little harm is done (though the timeouts should be tuned that this is relatively rare, because a server may waste significant resources on such redundant connects).

For almost all the timeouts I mentioned - the protocol needs to take care - so the protocol has to exist before the connection is established in case of outbound connections.

I'm not sure I follow. Can you sketch out some code to help me here? ISTM that e.g. the DNS, connect and handshake timeouts can be implemented by the machinery that tries to set up the connection behind the scenes, and the user's protocol won't know anything of these shenanigans. The code that calls create_transport() (actually it'll probably be renamed create_client()) will just get a Future that either indicates success (and then the protocol and transport are successfully hooked up) or an error (and then no protocol was created -- whether or not a transport was created is an implementation detail).

In case aconnection is lost and reconnecting is required - .reconnect() is handy, so the protocol can request reconnecting.

I'd need more details of how you would like to specify this.

As this does not work with the current Protocols callbacks I propose Protocols.connection_established() therefore.

How does this differ from connection_made()? (I'm trying to follow Twisted's guidance here, they seem to have the longest experience doing these kinds of things. When I talked to Glyph IIRC he was skeptical about reconnecting in general.)

Protocols I'd outline protocol_factory can be a instance of a class, which can set specific parameters for 'things' class p: def __init__(self, a=1,b=2,c=3): self.a = a self.b = b self.c = c def __call__(self): return p(a=self.a, b=self.b, c=self.c) def ... all protocol methods ...: pass

EventLoop.start_serving(p(a=5,b=7), ...) EventLoop.start_serving(p(a=9,b=4), ...)

Same Protocol, different parameters for it.

No such helper method (or class) is needed. You can use a lambda or functools.partial for the same effect. I'll add a note to the PEP to remind people of this.

+ connection_established() + timeout_dns() + timeout_idle() + timeout_connecting()

Signatures please?

* data_received(data) - if it was possible to return the number of bytes consumed by the protocol, and have the Transport buffer the rest for the next io in call, one would avoid having to do this in every Protocol on it's own - learned from experience.

Twisted has a whole slew of protocol implementation subclasses that implement various strategies like line-buffering (including a really complex version where you can turn the line buffering on and off) and "netstrings". I am trying to limit the PEP's size by not including these, but I fully expect that in practice a set of useful protocol implementations will be created that handles common cases. I'm not convinced that putting this in the transport/protocol interface will make user code less buggy: it seems easy for the user code to miscount the bytes or not return a count at all in a rarely taken code branch.

* eof_received()/connection_lost(exc) - a connection can be closed clean recv()=0, unclean recv()=-1, errno, SIGPIPE when writing and in case of SSL even more, it is required to distinguish.

Well, this is why eof_received() exists -- to indicate a clean close. We should never receive SIGPIPE (Python disables this signal, so you always get the errno instead). According to Glyph, SSL doesn't support sending eof, so you have to use Content-length or a chunked encoding. What other conditions do you expect from SSL that wouldn't be distinguished by the exception instance passed to connection_lost()?

+ nextlayer_is_empty() - called if the Transport (or underlying Protocol in case of chaining) write buffer is empty - Imagine an http server sending a 1GB file, you do not want to sent 1GB at once - as you do not have that much memory, but get a callback if the transport done sending the chunk you've queued, so you can send the next chunk of data.

That's what the pause()/resume() flow control protocol is for. You read the file (presumably it's a file) in e.g. 16K blocks and call write() for each block; if the transport can't keep up and exceeds its buffer space, it calls protocol.pause() (or perhaps protocol.pause_writing(), see discussion above).

Next, what happens if a dns can not be resolved, ssl handshake (in case ssl is transport) or connecting fails - in my opinion it's an error the protocol is supposed to take care of + error_dns + error_ssl + error_connecting

The future returned by create_transport() (aka create_client()) will raise the exception.

I'm not that much into futures - so I may have got some things wrong.

No problem. You may want to read PEP 3148, it explains Futures and much of that explanation remains valid; just in PEP 3156 to wait for a future you must use "yield from <future>". -- --Guido van Rossum (python.org/~guido)