[Twisted-Python] Event queue in the Python std lib
Hi, I know some of you guys have already thinking about this but ... A couple of weeks ago we had a bit of a chat (flame? ;-)) in #twisted about the idea of moving parts of Twisted, specifically the event loop, into the std lib. So, at Europython I had a quick chat with GvR about it too. GvR would really, really like to get a simple, flexible event loop into the std lib and thinks that Twisted is likely the best starting point since Twisted probably integrates with more event driven toolkits than anything else. Obviously, having the event loop part in the std lib has a number of benefits but it should also put pressure on all event driven toolkits to conform to the common API. Everything would then work together and the world would be a happier, shinier place ;-). GvR suggested that someone should start a PEP to talk around, which seems like a sensible thing to do, although I don't know if anyone with enough knowledge of the reactor and working with other event loops has time right now. Cheers, Matt -- __ / \__ Matt Goodall, Pollenation Internet Ltd \__/ \ w: http://www.pollenation.net __/ \__/ e: matt@pollenation.net / \__/ \ t: +44 (0)113 2252500 \__/ \__/ / \ Any views expressed are my own and do not necessarily \__/ reflect the views of my employer.
On Fri, 2005-07-01 at 10:56 +0100, Matt Goodall wrote:
Obviously, having the event loop part in the std lib has a number of benefits but it should also put pressure on all event driven toolkits to conform to the common API. Everything would then work together and the world would be a happier, shinier place ;-).
Which event driven toolkits are you talking about, besides Twisted?
GvR suggested that someone should start a PEP to talk around, which seems like a sensible thing to do, although I don't know if anyone with enough knowledge of the reactor and working with other event loops has time right now.
On the whole I don't think now, if ever, is the time to add Twisted or a
Twisted-based event loop to the Python standard library. There are a
number of issues:
1. Release cycles: Python releases on its own schedule, which may not
match up to the needs of bug fixes or feature additions to Twisted.
One solution is a separate Twisted package you can install, overriding
the stdlib's. Since Twisted, even split up, is quite large, this would
probably be necessary for most non-trivial functionality anyway, which
leads to:
2. Installing separate package issues. As PyXML has shown, having a
separate package which overrides or extends a built-in stdlib API works
terribly. http://www.amk.ca/conceit/xmlplus.html covers some of the
issues, but there's also -
3. Limited usefulness of minimal APIs: lets say we add Twisted to the
stdlib. Because it's large, we don't want to add *everything*. We can't
add SSL probably cause that depends on pyOpenSSL which is LGPLed,
probably don't want the pygtk reactor because that occasionally needs
updates for new pygtk releases, etc.. So Jane User decides to build a
network application, plays a bit with the TCP client code, and now says
"OK great, now to add a GUI and TLS support" - and suddenly she needs to
install the full Twisted framework to do that.
I would argue that Python needs less in the stdlib, not more; specific
external software packages will probably grow into de-facto standard
packages, but there's no reason for them to be in stdlib. PyXML is
usually installed separately because people need the extended features
the full package has, and the same would happen to Twisted. The only
real benefit to inclusion in the stdlib is ease of distribution, and a
better way to deal with that is to put more work into things like Eggs
(http://peak.telecommunity.com/DevCenter/PythonEggs).
--
Itamar Shtull-Trauring
Itamar Shtull-Trauring wrote:
On Fri, 2005-07-01 at 10:56 +0100, Matt Goodall wrote:
GvR suggested that someone should start a PEP to talk around, which seems like a sensible thing to do, although I don't know if anyone with enough knowledge of the reactor and working with other event loops has time right now.
Well, it's a subject that comes up periodically, and I have some new thoughts, so let me play devil's advocate here:
On the whole I don't think now, if ever, is the time to add Twisted or a Twisted-based event loop to the Python standard library. There are a number of issues:
1. Release cycles: Python releases on its own schedule, which may not match up to the needs of bug fixes or feature additions to Twisted.
This will always be a concern, but Python manages to get releases out every so often, and the stdlib *does* change sometimes. We've been pretty good about preserving compatibility for a release or two ourselves. I think it is a reasonable concern but not a reason to stop this entirely.
One solution is a separate Twisted package you can install, overriding the stdlib's. Since Twisted, even split up, is quite large, this would probably be necessary for most non-trivial functionality anyway, which leads to:
IMHO the point here is the trivial functionality. As far as I can tell from the few examples of event-driven applications that I use which I didn't write :-), there is a progression that many event-driven Python programs go through these days: - Implement something with asyncore - realize asyncore's deficiencies, start enhancing it - discover Twisted, re-implement networking code (from scratch) If the event loop in Python provided similar interfaces to the one in Twisted, the amount of work involved in stage 3 could be greatly reduced.
2. Installing separate package issues. As PyXML has shown, having a separate package which overrides or extends a built-in stdlib API works terribly. http://www.amk.ca/conceit/xmlplus.html covers some of the issues, but there's also -
So let's not pretend that they're the same thing. 'from eventloop import listenTCP' -> 'from twisted.internet import reactor'. They can just share a set of common interfaces. The big problem with xmlplus was that it changed the behavior of a built-in package, so let's not ever do that again. amk's article specifically recommends this solution.
3. Limited usefulness of minimal APIs: lets say we add Twisted to the stdlib. Because it's large, we don't want to add *everything*. We can't add SSL probably cause that depends on pyOpenSSL which is LGPLed, probably don't want the pygtk reactor because that occasionally needs updates for new pygtk releases, etc.. So Jane User decides to build a network application, plays a bit with the TCP client code, and now says "OK great, now to add a GUI and TLS support" - and suddenly she needs to install the full Twisted framework to do that.
Right. But, her code uses Twisted-like interfaces so she doesn't have to re-write it all. Given that a lot of network applications are trivial one-offs, this step will not always happen.
I would argue that Python needs less in the stdlib, not more; specific external software packages will probably grow into de-facto standard packages, but there's no reason for them to be in stdlib. PyXML is usually installed separately because people need the extended features the full package has, and the same would happen to Twisted. The only real benefit to inclusion in the stdlib is ease of distribution, and a better way to deal with that is to put more work into things like Eggs (http://peak.telecommunity.com/DevCenter/PythonEggs).
I don't think the issue is "less" or "more" but "better". The Python standard library is of a highly non-uniform quality. Whether or not Twisted gets added, I think asyncore should be removed, or at least strongly deprecated.
On Jul 1, 2005, at 8:24 AM, Glyph Lefkowitz wrote:
Itamar Shtull-Trauring wrote:
On Fri, 2005-07-01 at 10:56 +0100, Matt Goodall wrote:
One solution is a separate Twisted package you can install, overriding the stdlib's. Since Twisted, even split up, is quite large, this would probably be necessary for most non-trivial functionality anyway, which leads to:
IMHO the point here is the trivial functionality. As far as I can tell from the few examples of event-driven applications that I use which I didn't write :-), there is a progression that many event- driven Python programs go through these days:
- Implement something with asyncore - realize asyncore's deficiencies, start enhancing it - discover Twisted, re-implement networking code (from scratch)
If the event loop in Python provided similar interfaces to the one in Twisted, the amount of work involved in stage 3 could be greatly reduced.
A fringe benefit of having something reactor-like in the Python stdlib is that it gives Twisted more credibility (amongst those that don't already know that we're right).
3. Limited usefulness of minimal APIs: lets say we add Twisted to the stdlib. Because it's large, we don't want to add *everything*. We can't add SSL probably cause that depends on pyOpenSSL which is LGPLed, probably don't want the pygtk reactor because that occasionally needs updates for new pygtk releases, etc.. So Jane User decides to build a network application, plays a bit with the TCP client code, and now says "OK great, now to add a GUI and TLS support" - and suddenly she needs to install the full Twisted framework to do that.
Right. But, her code uses Twisted-like interfaces so she doesn't have to re-write it all. Given that a lot of network applications are trivial one-offs, this step will not always happen.
If something like threadedselectreactor made it into stdlib, then GUI integration (and integration with otherwise blocking code) would be easy enough. Personally, since this is the 2.5+ timeframe we're talking about here, I would rather see something PEP-342 based on the outside, as that would require less brain enhancement for most people to use properly. Deferreds are just as important as anything else.. I mean hell, I have implemented the Deferred API (with cancellation) in both JavaScript and ActionScript 2.0 for various purposes because it's the Right Way to represent that sort of thing given the language constraints. (the JavaScript stuff, which should make a lot of Pythonistas very happy, will probably make it into open source in 2 weeks or so.. the AS2 stuff I have no immediate plans of releasing). -bob
On Fri, 2005-07-01 at 14:24 -0400, Glyph Lefkowitz wrote:
- Implement something with asyncore - realize asyncore's deficiencies, start enhancing it - discover Twisted, re-implement networking code (from scratch)
If the event loop in Python provided similar interfaces to the one in Twisted, the amount of work involved in stage 3 could be greatly reduced.
Alternatively, asyncore could just be deprecated, which would be a lot easier to do, and people would start from the correct solution :)
2. Installing separate package issues. As PyXML has shown, having a separate package which overrides or extends a built-in stdlib API works terribly. http://www.amk.ca/conceit/xmlplus.html covers some of the issues, but there's also -
So let's not pretend that they're the same thing. 'from eventloop import listenTCP' -> 'from twisted.internet import reactor'. They can just share a set of common interfaces. The big problem with xmlplus was that it changed the behavior of a built-in package, so let's not ever do that again. amk's article specifically recommends this solution.
Hm. That's a lot better than moving parts of Twisted, yes. I still think it's silly.
I don't think the issue is "less" or "more" but "better". The Python standard library is of a highly non-uniform quality. Whether or not Twisted gets added, I think asyncore should be removed, or at least strongly deprecated.
Adding yet another event loop would just continue the tradition, I'm sure; it took us quite a while to get to the quality we have now just in terms of platform support. I suppose you could butcher our code by ripping features out... but you'd just get a crippled piece of software. The motivation is "so then people can move to something less suck"? Why not just have them start from something good to begin with?
Hi,
IMHO the point here is the trivial functionality. As far as I can tell from the few examples of event-driven applications that I use which I didn't write :-), there is a progression that many event-driven Python programs go through these days:
- Implement something with asyncore - realize asyncore's deficiencies, start enhancing it - discover Twisted, re-implement networking code (from scratch)
If the event loop in Python provided similar interfaces to the one in Twisted, the amount of work involved in stage 3 could be greatly reduced.
As a humble lurker on this list, I just wanna say that this argument rings a bell. I had to rewrite a P2P program which had chosen to implement its own event loop instead of using Twisted... ;) It would be nice if more people knew about Twisted. Regards Antoine.
On Jul 1, 2005, at 9:39 AM, Antoine Pitrou wrote:
IMHO the point here is the trivial functionality. As far as I can tell from the few examples of event-driven applications that I use which I didn't write :-), there is a progression that many event-driven Python programs go through these days:
- Implement something with asyncore - realize asyncore's deficiencies, start enhancing it - discover Twisted, re-implement networking code (from scratch)
If the event loop in Python provided similar interfaces to the one in Twisted, the amount of work involved in stage 3 could be greatly reduced.
As a humble lurker on this list, I just wanna say that this argument rings a bell. I had to rewrite a P2P program which had chosen to implement its own event loop instead of using Twisted... ;)
I did this once too, s/P2P program/XML Socket Server/ It turned out that doing one in Twisted was so easy, that I never bothered to package another one again. Everything else I had in there was more or less YAGNI because Twisted already did it, or made implementing the features so trivial as it wasn't worth including it. These days I just whip up app-specific LineReceivers with a '\0' delimiter that parse/generate with ElementTree.. takes like 5 minutes. -bob
One more argument: Inclusion in the stdlib requires more stringent backwards compatibility in APIs. Now, on the whole this is a good thing, but it can have serious long term consequences. Asyncore is the perfect example; Twisted's event loop API started out being no better than asyncore's, whereas now it is far superior (can work much better with stateful OS APIs like epoll(), can support asynchronous rather than just non-blocking, supports scheduling, much better TCP client support, etc..) The reason is years and years of working on improving it, whereas asyncore was basically stuck where it was because it had to be backwards compatible, and because adding major changes into the stdlib is harder. So, for example, a number of people independently added scheduling to the asyncore API, but it never made it into the stdlib. asyncore is now almost identical to the way it was 5 years ago.
On Jul 1, 2005, at 5:56 AM, Matt Goodall wrote:
Hi,
I know some of you guys have already thinking about this but ...
A couple of weeks ago we had a bit of a chat (flame? ;-)) in #twisted about the idea of moving parts of Twisted, specifically the event loop, into the std lib. So, at Europython I had a quick chat with GvR about it too.
GvR would really, really like to get a simple, flexible event loop into the std lib and thinks that Twisted is likely the best starting point since Twisted probably integrates with more event driven toolkits than anything else.
Obviously, having the event loop part in the std lib has a number of benefits but it should also put pressure on all event driven toolkits to conform to the common API. Everything would then work together and the world would be a happier, shinier place ;-).
GvR suggested that someone should start a PEP to talk around, which seems like a sensible thing to do, although I don't know if anyone with enough knowledge of the reactor and working with other event loops has time right now.
I think adding a subset of Twisted to the python stdlib would be a disservice to both the stdlib users and Twisted users, especially now that Python is on the cusp of having an easy way to distribute, find, and install 3rd party packages automatically. Saying that Twisted should be a "starting point" bothers me especially, as that seems to indicate that the two will diverge over time. That is worrisome. Who is going to maintain the stdlib fork? I would suggest instead that python should distribute a selected set of "recommended extras" as part of the installer, which would basically be complete 3rd party packages, included just for convenience. Twisted core would make a good package to include there, as would many others. James
James Y Knight wrote:
I would suggest instead that python should distribute a selected set of "recommended extras" as part of the installer, which would basically be complete 3rd party packages, included just for convenience. Twisted core would make a good package to include there, as would many others.
I agree with James's position: Twisted should be one of a number of Python "recommended extras" (I also agree with Glyph and Itamar that asyncore should be deprecated). IMO, that approach accomplishes everything that could be hoped for by incorporating a reactor-like event loop in the std library: (1) makes Twisted more visible to anyone looking for a "simple, flexible event loop", and (equally important) add the context of the deferred api and some well-tested goodies with asynchronous api's (like adbapi) and (2) does not create any gratuitous wrinkles (like a potential stdlib fork) for the current well-oiled Twisted development machine's minions to have to deal with. ;) Bob's point about "credibility" is well-taken, but I think having Twisted designated as an official Python "recommended extra" would accomplish that as well. Cheers, Steve
On Jul 1, 2005, at 10:01 AM, Stephen Waterbury wrote:
James Y Knight wrote:
I would suggest instead that python should distribute a selected set of "recommended extras" as part of the installer, which would basically be complete 3rd party packages, included just for convenience. Twisted core would make a good package to include there, as would many others.
I agree with James's position: Twisted should be one of a number of Python "recommended extras" (I also agree with Glyph and Itamar that asyncore should be deprecated).
IMO, that approach accomplishes everything that could be hoped for by incorporating a reactor-like event loop in the std library: (1) makes Twisted more visible to anyone looking for a "simple, flexible event loop", and (equally important) add the context of the deferred api and some well-tested goodies with asynchronous api's (like adbapi) and (2) does not create any gratuitous wrinkles (like a potential stdlib fork) for the current well-oiled Twisted development machine's minions to have to deal with. ;)
Bob's point about "credibility" is well-taken, but I think having Twisted designated as an official Python "recommended extra" would accomplish that as well.
I just wanted to point out that I agree with James' position too, but was speaking towards the "what if ideas from Twisted were to be included", rather than "what should actually happen". Python's standard library should dissolve into just the bits required to support simple scripts and EasyInstall/eggs. -bob
Bob Ippolito wrote:
I just wanted to point out that I agree with James' position too, but was speaking towards the "what if ideas from Twisted were to be included", rather than "what should actually happen".
Python's standard library should dissolve into just the bits required to support simple scripts and EasyInstall/eggs.
Rat on! And the library docs will hopefully evolve into a nice distributed documentation system, too ... :) Steve
I am not yet a twisted programmer, so my contribution to this thread has more to do with English interpretation than with twisted. A principle problem with event mainloops is that each toolkit wants to provide one, and the program can only have one running at a time (barring multithreading fanciness). When I read the original post referring to the discussion with GvR, I thought, cool, a resolution to the different-mainloops-for-different-tool kits problem: a mainloop that can integrate various tool kits. It seems that all other respondents have interpreted it as a call to include twisted, or parts of it _as is_, in the stdlib. I did not see in the original post a call for twisted _networking_ inclusion at all. When I read about twisted on other lists (the pygame list comes to mind), the examples almost always use the poll() reactor method, because it allows you to use twisted from your own mainloop; you can then integrate other tool kits into that home-spun mainloop by whatever provisions they allow for doing so. We shouldn't have to be that creative. There should be one std dist mainloop that can accommodate, via callbacks perhaps, whatever (multiple) services need a mainloop. So a tkinter-twisted mainloop and a gtk-twisted mainloop would not be needed as separate components. I am too naive to write such a library, nor even to write a PEP, so I won't be running with this, but is there something impossible about my interpretation? David
On Jul 1, 2005, at 3:54 PM, David wrote:
I am not yet a twisted programmer, so my contribution to this thread has more to do with English interpretation than with twisted.
A principle problem with event mainloops is that each toolkit wants to provide one, and the program can only have one running at a time (barring multithreading fanciness). When I read the original post referring to the discussion with GvR, I thought, cool, a resolution to the different-mainloops-for-different-tool kits problem: a mainloop that can integrate various tool kits.
It seems that all other respondents have interpreted it as a call to include twisted, or parts of it _as is_, in the stdlib. I did not see in the original post a call for twisted _networking_ inclusion at all.
When I read about twisted on other lists (the pygame list comes to mind), the examples almost always use the poll() reactor method, because it allows you to use twisted from your own mainloop; you can then integrate other tool kits into that home-spun mainloop by whatever provisions they allow for doing so.
We shouldn't have to be that creative. There should be one std dist mainloop that can accommodate, via callbacks perhaps, whatever (multiple) services need a mainloop. So a tkinter-twisted mainloop and a gtk-twisted mainloop would not be needed as separate components.
I am too naive to write such a library, nor even to write a PEP, so I won't be running with this, but is there something impossible about my interpretation?
That's a nice idea, but it can't possibly work. If you want a general way to interoperate with different toolkits, you basically have to use multithreaded fanciness, because not many of them have timers or socket notifiers built in worth their salt. Starting a single thread to sit around and wait for select() or equivalent is pretty sane, and makes the interface a lot more flexible and generic since you only need one facility in the toolkit -- namely a thread- safe way to call back into the main thread. Fortunately, there is one in Twisted 2.1, threadedselectreactor, which DOES integrate with various runloops (pygame and wxPython included) without polling. However, it's not the end of the road. threadedselectreactor should be refactored into a general interface (interleave) such that other backends (epoll, kqueue, windows junk, etc.) can be used rather than select. -bob
On Saturday 02 July 2005 04:55, James Y Knight wrote:
I would suggest instead that python should distribute a selected set of "recommended extras" as part of the installer, which would basically be complete 3rd party packages, included just for convenience. Twisted core would make a good package to include there, as would many others.
I don't think this should be bundled into the base installer.
a: The installer's already up around 10M - adding Twisted
makes it that much larger
b: Shipping it as part of the Official Installer suggests
that Twisted will follow then standard Python release
process, which means you have to worry about deprecations
and the like
c: There will be confusion for users with bugs - do they
submit an entry to the python bug tracker?
But I _would_ like to see asyncore more aggressively deprecated...
Anthony
--
Anthony Baxter
participants (9)
-
Anthony Baxter -
Antoine Pitrou -
Bob Ippolito -
David -
Glyph Lefkowitz -
Itamar Shtull-Trauring -
James Y Knight -
Matt Goodall -
Stephen Waterbury