Mailman 3 Add a generic async IO poller/reactor to select module - Python-ideas

Add a generic async IO poller/reactor to select module

Giampaolo Rodolà

24 May 2012 24 May '12

7:02 a.m.

Including an established async IO framework such as Twisted, gevent or Tornado in the Python stdlib has always been a controversial subject. PEP-3153 (http://www.python.org/dev/peps/pep-3153/) tried to face this problem in the most agnostic way as possible, and it's a good starting point IMO. Nevertheless, it's still vague about what the actual API should look like and AFAIK it remained stagnant so far. There's one thing in the whole async stack which is basically the same for all implementations though: the poller/reactor. Could it make sense to add something similar to select module? Differently from PEP-3153, providing such a layer on top of select(), poll() & co. is easier and could possibly be an incentive to avoid such code duplication. I'm coming up with this because I recently did something similar in pyftpdlib as an hack on top of asyncore to add support for epoll() and kqueue(), using the excellent Tornado's io loop as source of inspiration: http://code.google.com/p/pyftpdlib/issues/detail?id=203 http://code.google.com/p/pyftpdlib/source/browse/trunk/pyftpdlib/lib/ioloop.... The way I imagine it:

...

...
...
import select dir(select) [..., 'EpollPoller', 'PollPoller', 'SelectPoller', 'KqueuePoller'] poller = select.EpollPoller() poller.register(fd, handler, poller.READ | poller.WRITE) poller.socket_map {2 : } poller.modify(fd, poller.READ) poller.poll() # will call handler.handle_read_event() if/when it's the case ^C KeyboardInterrupt poller.remove(fd) poller.close()

The handler is supposed to provide 3 methods: - handle_read_event - handle_write_event - handle_error_event Users willing to support multiple event loops such as wx, gtk etc can do:

...

...
...
while 1: ... poller.poll(timeout=0.1, blocking=False) ... otherpoller.poll()

Basically, this would be the whole API. Thoughts? --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/

Show replies by thread

Giampaolo Rodolà

24 May 24 May

7:13 a.m.

2012/5/24 Giampaolo Rodolà :

...

The handler is supposed to provide 3 methods: - handle_read_event - handle_write_event - handle_error_event

Further note: this is the approach I used in pyftpdlib. An even more abstracted approach would be having poller.poll() return a dict of {fd: events, fd, events, ...}, similarly to what Tornado currently does. This way we wouldn't be forcing the user to provide a handler class with the 3 methods described above. --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/

Ronald Oussoren

12:17 p.m.

New subject: Add a generic async IO poller/reactor to select module

On 24 May, 2012, at 3:32, Giampaolo Rodolà wrote:

...

...
...
...
The handler is supposed to provide 3 methods: - handle_read_event - handle_write_event - handle_error_event

Users willing to support multiple event loops such as wx, gtk etc can do:

...
...
...
while 1: ... poller.poll(timeout=0.1, blocking=False) ... otherpoller.poll()

Basically, this would be the whole API.

Isn't this a limited version of asyncore? (poller.poll == asyncore.loop, the handler is a subset of asyncore.dispatcher). Ronald

Giampaolo Rodolà

5:20 p.m.

2012/5/24 Ronald Oussoren :

...

On 24 May, 2012, at 3:32, Giampaolo Rodolà wrote:

...
...
...
...
The handler is supposed to provide 3 methods: - handle_read_event - handle_write_event - handle_error_event

Users willing to support multiple event loops such as wx, gtk etc can do:

...
...
...
while 1: ... poller.poll(timeout=0.1, blocking=False) ... otherpoller.poll()

Basically, this would be the whole API.

Isn't this a limited version of asyncore? (poller.poll == asyncore.loop, the handler is a subset of asyncore.dispatcher).

Ronald

poller.poll serves the same purpose of asyncore.loop, yes, but this is supposed to be independent from asyncore. --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/

Antoine Pitrou

5:33 p.m.

On Thu, 24 May 2012 13:50:27 +0200 Giampaolo Rodolà wrote:

...

2012/5/24 Ronald Oussoren :

...
On 24 May, 2012, at 3:32, Giampaolo Rodolà wrote:

...
...
...
...
The handler is supposed to provide 3 methods: - handle_read_event - handle_write_event - handle_error_event

Users willing to support multiple event loops such as wx, gtk etc can do:

...
...
...
while 1: ... poller.poll(timeout=0.1, blocking=False) ... otherpoller.poll()

Basically, this would be the whole API.

Isn't this a limited version of asyncore? (poller.poll == asyncore.loop, the handler is a subset of asyncore.dispatcher).

Ronald

poller.poll serves the same purpose of asyncore.loop, yes, but this is supposed to be independent from asyncore.

I agree with Ronald that it looks like a less-braindead version of asyncore. I don't think the select module is the right place. Also, I don't know why you would specify poller.READ or poller.WRITE explicitly. Usually you are interested in all events, no? Regards Antoine.

Nick Coghlan

6:07 p.m.

On Thu, May 24, 2012 at 9:50 PM, Giampaolo Rodolà wrote:

...

poller.poll serves the same purpose of asyncore.loop, yes, but this is supposed to be independent from asyncore.

I'd actually like to see something like this pitched as a "concurrent.eventloop" PEP. PEP 3153 really wasn't what I was expecting after the discussions at the PyCon US 2011 language summit - I was expecting "here's a common event loop all the async frameworks can hook into", but instead we got something a *lot* more ambitious taht tried to merge the entire IO stack for the async frameworks, rather than just provide a standard way for their event loops to cooperate. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Giampaolo Rodolà

6:15 p.m.

2012/5/24 Antoine Pitrou :

...

On Thu, 24 May 2012 13:50:27 +0200 Giampaolo Rodolà wrote:

...
2012/5/24 Ronald Oussoren :

...
On 24 May, 2012, at 3:32, Giampaolo Rodolà wrote:

...
...
...
>

The handler is supposed to provide 3 methods: - handle_read_event - handle_write_event - handle_error_event

Users willing to support multiple event loops such as wx, gtk etc can do:

...
...
> while 1: ... poller.poll(timeout=0.1, blocking=False) ... otherpoller.poll()

Basically, this would be the whole API.

Isn't this a limited version of asyncore? (poller.poll == asyncore.loop, the handler is a subset of asyncore.dispatcher).

Ronald

poller.poll serves the same purpose of asyncore.loop, yes, but this is supposed to be independent from asyncore.

I agree with Ronald that it looks like a less-braindead version of asyncore. I don't think the select module is the right place.

Yeah, probably. Usually when I post here I'm the first one not being sure whether what I propose is a good idea or not. =) Anyway, it must be clear that what I have in mind is not related to asyncore per-se. The proposal is to add a *generic* poller/reactor to select module as an abstraction layer on top of select(), poll(), epoll() and kqueue(), that's all.

...

Also, I don't know why you would specify poller.READ or poller.WRITE explicitly. Usually you are interested in all events, no?

Nope, that's what asyncore does and that's why it is significantly slower compared to more modern and clever async loops (independenly from the lack of epoll() / kqueue() support in asyncore). You should only be interested in reading for accepting sockets (servers) or when you want to receive data. You should only be interested in writing for connecting sockets (clients) or when you want to send data. Being interested in both when, say, you only intend to receive data is a considerable waste of time, especially when there are many concurrent connections. The performance degradation if you wildly look for both read and write events is *huge*, see benchmarks referring to old vs. new select() implementation here (~8.5x slowdown with 200 concurrent clients): http://code.google.com/p/pyftpdlib/issues/detail?id=203#c6 --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/

Nick Coghlan

6:21 p.m.

On Thu, May 24, 2012 at 10:37 PM, Nick Coghlan wrote:

...

On Thu, May 24, 2012 at 9:50 PM, Giampaolo Rodolà wrote:

...
poller.poll serves the same purpose of asyncore.loop, yes, but this is supposed to be independent from asyncore.

I'd actually like to see something like this pitched as a "concurrent.eventloop" PEP. PEP 3153 really wasn't what I was expecting after the discussions at the PyCon US 2011 language summit - I was expecting "here's a common event loop all the async frameworks can hook into", but instead we got something a *lot* more ambitious taht tried to merge the entire IO stack for the async frameworks, rather than just provide a standard way for their event loops to cooperate.

See the final section of my notes here: http://www.boredomandlaziness.org/2011/03/python-language-summit-rough-notes... Turns out the idea of a PEP 3153 level API *was* raised at the summit, but I'd still like to see a competing PEP that targets the reactor level API directly. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Ronald Oussoren

6:22 p.m.

New subject: Add a generic async IO poller/reactor to select module

On 24 May, 2012, at 14:03, Antoine Pitrou wrote:

...

On Thu, 24 May 2012 13:50:27 +0200 Giampaolo Rodolà wrote:

...
2012/5/24 Ronald Oussoren :

...
On 24 May, 2012, at 3:32, Giampaolo Rodolà wrote:

...
...
...
>

The handler is supposed to provide 3 methods: - handle_read_event - handle_write_event - handle_error_event

Users willing to support multiple event loops such as wx, gtk etc can do:

...
...
> while 1: ... poller.poll(timeout=0.1, blocking=False) ... otherpoller.poll()

Basically, this would be the whole API.

Isn't this a limited version of asyncore? (poller.poll == asyncore.loop, the handler is a subset of asyncore.dispatcher).

Ronald

poller.poll serves the same purpose of asyncore.loop, yes, but this is supposed to be independent from asyncore.

I agree with Ronald that it looks like a less-braindead version of asyncore. I don't think the select module is the right place.

What worries me most is that it might only look like a beter version of asyncore. I'd much rather see something based on the event-handling core of Twisted because that code base is used in production and is hence more likely to be correct w.r.t. odd real-world conditions. IIRC doing this was discussed at the language summit in 2011, but as Nick mentions that doesn't seem to be the focus of PEP 3153. I am by the way not using Twisted myself, I'm at this time still using homebrew select loops and asyncore.

...

Also, I don't know why you would specify poller.READ or poller.WRITE explicitly. Usually you are interested in all events, no?

You're not always interested in write events, those are only interesting when you have data that must be written to a socket. Ronald

Giampaolo Rodolà

6:36 p.m.

2012/5/24 Ronald Oussoren :

...

On 24 May, 2012, at 14:03, Antoine Pitrou wrote:

...
On Thu, 24 May 2012 13:50:27 +0200 Giampaolo Rodolà wrote:

...
2012/5/24 Ronald Oussoren :

...
On 24 May, 2012, at 3:32, Giampaolo Rodolà wrote:

...
...
>>

The handler is supposed to provide 3 methods: - handle_read_event - handle_write_event - handle_error_event

Users willing to support multiple event loops such as wx, gtk etc can do:

...
>> while 1: ... poller.poll(timeout=0.1, blocking=False) ... otherpoller.poll()

Basically, this would be the whole API.

Isn't this a limited version of asyncore? (poller.poll == asyncore.loop, the handler is a subset of asyncore.dispatcher).

Ronald

poller.poll serves the same purpose of asyncore.loop, yes, but this is supposed to be independent from asyncore.

I agree with Ronald that it looks like a less-braindead version of asyncore. I don't think the select module is the right place.

What worries me most is that it might only look like a beter version of asyncore.

Please, forget about asyncore: this has nothing to do with it per-se as it's just a reactor - it doesn't aim to provide any connection handling. Given the poor asyncore API I doubt it would be even integrable with it without breaking backward compatibility. --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/

Antoine Pitrou

6:36 p.m.

On Thu, 24 May 2012 22:37:03 +1000 Nick Coghlan wrote:

...

On Thu, May 24, 2012 at 9:50 PM, Giampaolo Rodolà wrote:

...
poller.poll serves the same purpose of asyncore.loop, yes, but this is supposed to be independent from asyncore.

I'd actually like to see something like this pitched as a "concurrent.eventloop" PEP.

Sounds like a good idea to me. By the way, it should also have some support for delayed calls to be actually useful (something that asyncore *still* doesn't have, AFAIK). Regards Antoine.

Giampaolo Rodolà

25 May 25 May

12:10 a.m.

2012/5/24 Nick Coghlan :

...

On Thu, May 24, 2012 at 10:37 PM, Nick Coghlan wrote:

...
On Thu, May 24, 2012 at 9:50 PM, Giampaolo Rodolà wrote:

...
poller.poll serves the same purpose of asyncore.loop, yes, but this is supposed to be independent from asyncore.

I'd actually like to see something like this pitched as a "concurrent.eventloop" PEP. PEP 3153 really wasn't what I was expecting after the discussions at the PyCon US 2011 language summit - I was expecting "here's a common event loop all the async frameworks can hook into", but instead we got something a *lot* more ambitious taht tried to merge the entire IO stack for the async frameworks, rather than just provide a standard way for their event loops to cooperate.

See the final section of my notes here: http://www.boredomandlaziness.org/2011/03/python-language-summit-rough-notes...

Turns out the idea of a PEP 3153 level API *was* raised at the summit, but I'd still like to see a competing PEP that targets the reactor level API directly.

Cheers, Nick.

It's not clear to me what such a PEP should address in particular, anyway here's a bunch of semi-random ideas. === Idea #1 === 4 classes (SelectPoller, PollPoller, EpollPoller, KqueuePoller) within concurrent.eventloop namespace all sharing the same API: - register(fd, events, callback) # callback gets called with events as arg - modify(fd, events) - unregister(fd) - call_later(timeout, callback, errback=None) - call_every(timeout, callback, errback=None) - poll(timeout=1.0, blocking=True) - close() call_later() and call_every() can return an object having cancel() and reset() methods. The user willing to register a new handler will do:

...

...
...
poller.register(sock.fileno(), poller.READ | poller.WRITE, callback)

...then, in the callback: def callback(events): if events & poller.ERROR and not events & poller.READ: disconnect() else: if events & poller.READ: read() if events & poller.WRITE: write() pros: highly customizable cons: too low level, requires manual handling === Idea #2 === same as #1 except: - register(fd, events) - poll(timeout=1.0) # desn't block, return {fd:events, fd:events, ...} === Idea #3 === same as #1 except: - register(fd, events, handler) - poll(timeout=1.0, blocking=True) ...poll() will call handler.handle_X_event() depending on the current event (READ, WRITE or ERROR). An internal map such as {fd:handler, fd:handler} will be maintaned internally. - pros: easier to use - cons: more rigid, requires a "contract" with the handler --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/

Terry Reedy

1:34 a.m.

On 5/24/2012 2:40 PM, Giampaolo Rodolà wrote:

...

It's not clear to me what such a PEP should address in particular, anyway here's a bunch of semi-random ideas.

I have been reading for perhaps a decade how bad asyncore is. So I hope you stick with trying to thrash out something different, even if the discussion gets tedious or contentions.

...

=== Idea #1 ===

4 classes (SelectPoller, PollPoller, EpollPoller, KqueuePoller) within concurrent.eventloop namespace all sharing the same API:

For new classes, the first question is what concept (and data/function grouping) they and their instances represent. As a naive event loop user, I might think in terms of event sources (or sets of sources) and corresponding handlers. For events generated by 'file' polling, the particular method would seem like a secondary issue. Your proposed classes are named after methods and you give no initialization api. This suggests to me that you mean for all files being polled by the same method to be grouped together. If so, there would only need 0 or 1 instance of each 'class', in while case, they could just as well be modules. In other words, I am unsure what concept these classes would represent. I am perhaps thinking at too high a level.

...

- register(fd, events, callback) # callback gets called with events as arg - modify(fd, events) - unregister(fd) - call_later(timeout, callback, errback=None) - call_every(timeout, callback, errback=None) - poll(timeout=1.0, blocking=True) - close()

call_later() and call_every() can return an object having cancel() and reset() methods.

-- Terry Jan Reedy

Cameron Simpson

4:07 a.m.

New subject: Add a generic async IO poller/reactor to select module

On 24May2012 14:03, Antoine Pitrou wrote: | Also, I don't know why you would specify poller.READ or poller.WRITE | explicitly. Usually you are interested in all events, no? Personally, I would want specificity. If I only care about write (eg I'm only sending), I would only specify poller.WRITE and have my handler only know and care about that. Possibly it would be good to be able to raise an exception for events I hadn't handled, but I'd be half inclined to have my handler do that, were it wanted (yes, there is some tension in this sentence). Unless I'm missing something here. Just my 2c, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ I just didn't give up, not riding it out wasn't an option. You don't crash, until you do. The longer you ride it out the more likely you are to ride it out. Throwing it away, saves nothing. - J. Pridmore

Ronald Oussoren

12:09 p.m.

New subject: Add a generic async IO poller/reactor to select module

On 24 May, 2012, at 20:40, Giampaolo Rodolà wrote:

...

2012/5/24 Nick Coghlan :

...
On Thu, May 24, 2012 at 10:37 PM, Nick Coghlan wrote:

...
On Thu, May 24, 2012 at 9:50 PM, Giampaolo Rodolà wrote:

...
poller.poll serves the same purpose of asyncore.loop, yes, but this is supposed to be independent from asyncore.

I'd actually like to see something like this pitched as a "concurrent.eventloop" PEP. PEP 3153 really wasn't what I was expecting after the discussions at the PyCon US 2011 language summit - I was expecting "here's a common event loop all the async frameworks can hook into", but instead we got something a *lot* more ambitious taht tried to merge the entire IO stack for the async frameworks, rather than just provide a standard way for their event loops to cooperate.

See the final section of my notes here: http://www.boredomandlaziness.org/2011/03/python-language-summit-rough-notes...

Turns out the idea of a PEP 3153 level API *was* raised at the summit, but I'd still like to see a competing PEP that targets the reactor level API directly.

Cheers, Nick.

It's not clear to me what such a PEP should address in particular, anyway here's a bunch of semi-random ideas.

All of these are probably too low level to be the only API because they don't encapsulate error handling. A slightly higher level API would have a callback with received data and a buffered API for sending data. That way the networking library can deal with lowlevel socket API errors and translate them to usefull abtract errors. It would also handle some errors like and EGAIN error itself. Also: how would you use SSL with these APIs? The API would probably end up with functionality simular to Twisted's reactor and transport APIs (and possibly endpoints but I don't know how stable that API is). Ronald

Calvin Spealman

3:02 p.m.

On Wed, May 23, 2012 at 9:32 PM, Giampaolo Rodolà wrote:

...

Including an established async IO framework such as Twisted, gevent or Tornado in the Python stdlib has always been a controversial subject. PEP-3153 (http://www.python.org/dev/peps/pep-3153/) tried to face this problem in the most agnostic way as possible, and it's a good starting point IMO. Nevertheless, it's still vague about what the actual API should look like and AFAIK it remained stagnant so far.

There's one thing in the whole async stack which is basically the same for all implementations though: the poller/reactor. Could it make sense to add something similar to select module? Differently from PEP-3153, providing such a layer on top of select(), poll() & co. is easier and could possibly be an incentive to avoid such code duplication.

I'm coming up with this because I recently did something similar in pyftpdlib as an hack on top of asyncore to add support for epoll() and kqueue(), using the excellent Tornado's io loop as source of inspiration: http://code.google.com/p/pyftpdlib/issues/detail?id=203 http://code.google.com/p/pyftpdlib/source/browse/trunk/pyftpdlib/lib/ioloop....

The way I imagine it:

...
...
...
import select dir(select) [..., 'EpollPoller', 'PollPoller', 'SelectPoller', 'KqueuePoller'] poller = select.EpollPoller() poller.register(fd, handler, poller.READ | poller.WRITE) poller.socket_map {2 : } poller.modify(fd, poller.READ) poller.poll() # will call handler.handle_read_event() if/when it's the case ^C KeyboardInterrupt poller.remove(fd) poller.close()

The handler is supposed to provide 3 methods: - handle_read_event - handle_write_event - handle_error_event

Users willing to support multiple event loops such as wx, gtk etc can do:

...
...
...
while 1: ... poller.poll(timeout=0.1, blocking=False) ... otherpoller.poll()

Basically, this would be the whole API.

Thoughts?

Frankly, I don't think this deserves a PEP at all, or even to consider one *yet*. Building a new API and a new library from scratch seems a frail comparison to testing a library in the real world, it having real uses, and then being incorporated into the stdlib. The problem here, of course, is that all the real-world solutions (ie, Twisted) include far more than the reactor.

...

--- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/ _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

Devin Jeanpierre

4:03 p.m.

On Fri, May 25, 2012 at 5:32 AM, Calvin Spealman wrote:

...

Frankly, I don't think this deserves a PEP at all, or even to consider one *yet*.

Building a new API and a new library from scratch seems a frail comparison to testing a library in the real world, it having real uses, and then being incorporated into the stdlib. The problem here, of course, is that all the real-world solutions (ie, Twisted) include far more than the reactor.

To be fair, PEP-3153 was built based largely on experience from the Twisted project and input from Twisted developers, who know what they are talking about and how to build a useful system. The entire transport/protocol separation is lifted directly out of it. -- Devin

Calvin Spealman

5:24 p.m.

On Fri, May 25, 2012 at 6:33 AM, Devin Jeanpierre wrote:

...

On Fri, May 25, 2012 at 5:32 AM, Calvin Spealman wrote:

...
Frankly, I don't think this deserves a PEP at all, or even to consider one *yet*.

Building a new API and a new library from scratch seems a frail comparison to testing a library in the real world, it having real uses, and then being incorporated into the stdlib. The problem here, of course, is that all the real-world solutions (ie, Twisted) include far more than the reactor.

To be fair, PEP-3153 was built based largely on experience from the Twisted project and input from Twisted developers, who know what they are talking about and how to build a useful system. The entire transport/protocol separation is lifted directly out of it.

-- Devin

My comments were in response to this post, not PEP-3153 -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

Nick Coghlan

7:23 p.m.

On May 25, 2012 7:33 PM, "Calvin Spealman" wrote:

...

On Wed, May 23, 2012 at 9:32 PM, Giampaolo Rodolà wrote:

...
Users willing to support multiple event loops such as wx, gtk etc can do:

...
...
...
while 1: ... poller.poll(timeout=0.1, blocking=False) ... otherpoller.poll()

Basically, this would be the whole API.

Thoughts?

Frankly, I don't think this deserves a PEP at all, or even to consider one *yet*.

Building a new API and a new library from scratch seems a frail comparison to testing a library in the real world, it having real uses, and then being incorporated into the stdlib. The problem here, of course, is that all the real-world solutions (ie, Twisted) include far more than the reactor.

No, the specific call at the PyCon US 2011 language summit was for a PEP that proposed a *new* event loop for the standard library that: 1. Provides simple event loop functionality in the standard library, as an improved alternative to asyncore for small apps that don't require the full power of a framework like Twisted (think things like little IRC bots, TCP echo servers, or testing of async components) 2. Provides a clean migration path to a production grade reactor like Twisted's 3. Makes it easier for multiple event loop based frameworks (e.g. tkinter, wxPython, PySide, Twisted) to all cooperate within the same process What we're after is something for the stdlib that is to event loops/reactors as wsgiref is to production grade WSGI servers like mod_wsgi and nginx. asyncore isn't it, because the migration path isn't clean. PEP 3153 currently spends a lot of time talking about transports and protocols, but doesn't answer those 3 core questions: 1. How do I write a simple IRC bot or TCP echo server? 2. How do I migrate my simple app to a production grade reactor like Twisted's? 3. How do I run two different concurrent.eventloop compatible reactors in the same process? As far as I can tell, PEP 3153 wants to handle all that by merging the I/O stacks of all the frameworks first, which strikes me as being *way* too ambitious for a first step. If we can't even figure out a common abstraction for the reactor level (ala WSGI), how are we ever going to agree on a standard async I/O abstraction? Cheers, Nick.

Nick Coghlan

7:24 p.m.

On Fri, May 25, 2012 at 11:53 PM, Nick Coghlan wrote:

...

as wsgiref is to production grade WSGI servers like mod_wsgi and nginx.

s/nginx/gunicorn/ Confusing-my-software-stack-levels'ly, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Calvin Spealman

27 May 27 May

2:35 a.m.

On Fri, May 25, 2012 at 9:53 AM, Nick Coghlan wrote:

...

On May 25, 2012 7:33 PM, "Calvin Spealman" wrote:

...
On Wed, May 23, 2012 at 9:32 PM, Giampaolo Rodolà wrote:

...
Users willing to support multiple event loops such as wx, gtk etc can do:

...
...
...
while 1: ... poller.poll(timeout=0.1, blocking=False) ... otherpoller.poll()

Basically, this would be the whole API.

Thoughts?

Frankly, I don't think this deserves a PEP at all, or even to consider one *yet*.

Building a new API and a new library from scratch seems a frail comparison to testing a library in the real world, it having real uses, and then being incorporated into the stdlib. The problem here, of course, is that all the real-world solutions (ie, Twisted) include far more than the reactor.

No, the specific call at the PyCon US 2011 language summit was for a PEP that proposed a *new* event loop for the standard library that: 1. Provides simple event loop functionality in the standard library, as an improved alternative to asyncore for small apps that don't require the full power of a framework like Twisted (think things like little IRC bots, TCP echo servers, or testing of async components) 2. Provides a clean migration path to a production grade reactor like Twisted's 3. Makes it easier for multiple event loop based frameworks (e.g. tkinter, wxPython, PySide, Twisted) to all cooperate within the same process

What we're after is something for the stdlib that is to event loops/reactors as wsgiref is to production grade WSGI servers like mod_wsgi and nginx. asyncore isn't it, because the migration path isn't clean.

PEP 3153 currently spends a lot of time talking about transports and protocols, but doesn't answer those 3 core questions:

1. How do I write a simple IRC bot or TCP echo server? 2. How do I migrate my simple app to a production grade reactor like Twisted's? 3. How do I run two different concurrent.eventloop compatible reactors in the same process?

As far as I can tell, PEP 3153 wants to handle all that by merging the I/O stacks of all the frameworks first, which strikes me as being *way* too ambitious for a first step. If we can't even figure out a common abstraction for the reactor level (ala WSGI), how are we ever going to agree on a standard async I/O abstraction?

Obviously, for a man with many opinions I miss out on too many conversations and too many potential actions. I should make steps to correct this in the future. Thanks for clearing this up.

...

Cheers, Nick.

-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

4351

Age (days ago)

4353

Last active (days ago)

List overview

Download

20 comments

8 participants

participants (8)

Antoine Pitrou
Calvin Spealman
Cameron Simpson
Devin Jeanpierre
Giampaolo Rodolà
Nick Coghlan
Ronald Oussoren
Terry Reedy

Add a generic async IO poller/reactor to select module

Cameron Simpson

tags

participants (8)