[Twisted-Python] Other ways to integrate foreign event loops?
![](https://secure.gravatar.com/avatar/826694d326649b5e681474e8e2116dc1.jpg?s=120&d=mm&r=g)
Is there a way to do the opposite of task.coiterate? I'm using pyglet for my UI, which was working great until I upgraded from pyglet 1.0 to 1.1. Now for some reason, the pyglet performance tanked. Since I'm only using twisted to send AMP messages very rarely (on the order of one every few seconds), it seems like I could try to work around the problem by giving pyglet back control of the main loop and occasionally giving twisted some time to breath (occasionally like every 50ms or something). Can that be done? Here's what I'm doing _now_ that technically works, but pyglet 1.1 is really choking on:
By the way, the preferred method for the runloop on pyglet 1.1 is to use a builtin one like so:
So if twisted can stand giving up it's run loop, I could periodically call it from that "on_draw()" method. ~ Nathan
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On 31 Mar, 10:45 pm, nathan.stocks@gmail.com wrote:
Twisted's job is to run the main loop; it's best just to let it do that. If Pyglet slowed down, it would be best to examine what's gone wrong with Pyglet and simply submit a fix for that. We spoke to Richard Jones about Pyglet/Twisted integration at Chris Armstrong's excellent gaming+twisted openspace at PyCon, and I believe that some future version of Pyglet will include Twisted integration. Of course, this is just a plan right now, not code, so perhaps your time would be better spent contributing this feature to Pyglet, rather than turning Twisted inside-out :).
![](https://secure.gravatar.com/avatar/d6304567ada7ac5e8c6f4e5902270831.jpg?s=120&d=mm&r=g)
On Mon, Mar 31, 2008 at 10:17 PM, <glyph@divmod.com> wrote:
Yeah, I've experienced some performance issues with pyglet's app.runtime (maybe Linux-specific or something). Given the time, I'll submit a meaningful bug report concerning this. Regarding integration with Twisted, though, I think the best thing to do (for now) is to just ignore pyglet.app.runtime and write a pump function, roughly: def pump(): dt = clock.tick(True) # polling tick w.clear() w.dispatch_events() ... w.flip() if w.has_exit and reactor.running: reactor.stop() Be sure to call clock.tick(True) - since clock.tick will otherwise block. The next thing to do is customize clock where appropriate - that is, to used Twisted's scheduling functions. Coincidentally, I did this about 45 minutes ago for a project I'm working on: http://hg.enterthefoo.com/roboto/file/db931f29b98e/roboto/clock.py Beware - the above is half-baked and written under the influence of Hobgoblin Dark English Ale. Finally, you need schedule the pump function. To do it as fast as possible: from twisted.internet import task gameLoop = task.LoopingCall(pump) gameLoop.start(0., True) reactor.run() Or with a set framerate: ... gameLoop.start(1/60., True) Cheers, -- \\\\\/\"/\\\\\\\\\\\ \\\\/ // //\/\\\\\\\ \\\/ \\// /\ \/\\\\ \\/ /\/ / /\/ /\ \\\ \/ / /\/ /\ /\\\ \\ / /\\\ /\\\ \\\\\/\ \/\\\\\/\\\\\/\\\\\\ d.p.s
![](https://secure.gravatar.com/avatar/826694d326649b5e681474e8e2116dc1.jpg?s=120&d=mm&r=g)
On Mon, Mar 31, 2008 at 10:18 PM, Drew Smathers <drew.smathers@gmail.com> wrote:
Thanks for the great advice. I implemented it all and it runs, but unfortunately there was no noticeable change (for better or worse) to pyglet's performance. I'm currently working on setting up a stripped-down twisted+pyglet example that reproduces the problem. I'll post it when I get it up and running. ~ Nathan
![](https://secure.gravatar.com/avatar/a61e243764490913906c773c9acb0d3c.jpg?s=120&d=mm&r=g)
Nathan <nathan.stocks@gmail.com> writes:
While in general I'm always in favor of letting Twisted own the main loop, I've also had quite good success in the past not doing so, even without multiple threads, primarily in cases where networking was a secondary function (e.g., GUI applications) and less time crucial than the primary UI interaction. In our case, with wxPython based applications, things worked great just having wxPython handle a timer that triggered reactor iterations. It involved: * During initialization, start reactor with reactor.startRunning() * At some periodic frequency, run reactor.iterate(0) * During termination, set up a reactor.callLater for reactor.stop, and then call reactor.mainLoop(). This was the cleanest way we found to permit all reactor finalization (events, final I/O, etc...). Note that the first point violates the IReactorCore interface (startRunning() should only be accessed through run()). I don't consider running mainLoop() at the end a violation since it's the only time it is called. Yes, the above means that Twisted's loop might incur higher latency (we ran the timer at about 150ms) than it might be if in control, but it worked very well for us, including during times when the win32/wxPython reactors weren't particularly stable, and _threadedselect wasn't yet an option. And even if in control, once Twisted calls out to pump GUI messages, large latency is still theoretically possible until it regains control. So while I wouldn't usually suggest making Twisted's main loop subservient to some other loop (and trying to identify the issue in Pyglet would be a good thing), it's also something I'm not averse to doing when needed. It might be worth experimenting with using Pyglet's main loop, and pyglet.clock.schedule or pyglet.clock.schedule_interval to iterate Twisted's reactor. -- David
![](https://secure.gravatar.com/avatar/826694d326649b5e681474e8e2116dc1.jpg?s=120&d=mm&r=g)
On Thu, Apr 3, 2008 at 1:35 PM, David Bolen <db3l.net@gmail.com> wrote:
That's exactly what I was looking for, thanks! I'll save this as the technique to try if all else fails and the time comes that I've got to deploy. Hopefully we'll just be able to figure out what's going on and fix it before then. ~ Nathan
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On 08:32 pm, nathan.stocks@gmail.com wrote:
Definitely for an "if all else fails" scenario, this should work well enough :). But definitely reserve it for that. For PyGame, there aren't too many other considerations which will practically cause problems, since you're spinning at full-throttle all the time and you are unlikely to want to integrate with too much other code. (And if everything's written to just expect a normal, running reactor anyway, you can easily fix it later.) For a regular GUI app though, waking up every 150ms or so can be more serious than the apparently negligible CPU usage it causes. Try, for example, running your program under http://en.wikipedia.org/wiki/PowerTOP to see if this reactor integration mechanism is running down your laptop users' battery :). (This message written from a laptop whose battery is no longer being devoured by Firefox's too-frequent wakeups...)
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Thu, 03 Apr 2008 15:35:36 -0400, David Bolen <db3l.net@gmail.com> wrote:
Let me emphasize this. The above relies on implementation details and may be broken by any subsequent release (assuming it works now, which I won't comment on). Jean-Paul
![](https://secure.gravatar.com/avatar/313c3d40f833c31895423ce0621cc489.jpg?s=120&d=mm&r=g)
The Qt reactor, http://code.tarbox.org/qtreactor, inverts the "ownership" of the callback loop as well. However, Qt provides well behaved callbacks for timers as well as socket activity so the reactor behaves properly. The qt reactor design allows for the conventional reactor.run() at the end of the script as well as a call to reactor.runReturn() which starts up the reactor, nails up the timers and socket IO but returns to the calling application. Hence, you can startup twisted sometime after your gui app is running. Obviously, you need to "behave" properly and make sure that any user code doesn't hang onto the thread but this is required for proper gui handling code anyway... Twisted's current wx reactor uses another thread due to difficulties experienced with an earlier version of wx not behaving well. There are complexities WRT integrating those threads which lie at the heart of the testing failures for the current wx reactor. It might be easy to fix... However, I've been thinking about taking the qtreactor approach and (re) trying to fix the wx reactor by first evaluating whether wx is "better behaved". If it isn't, perhaps its straightforward to fix wx (???) Thing is, once you get the logic down... and (I think) its all correct in the qtreactor, it should be straightforward to use the wx calls to get servicing for timing and socket IO. -glenn On Thu, 2008-04-03 at 17:35 -0500, Jean-Paul Calderone wrote:
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On 31 Mar, 10:45 pm, nathan.stocks@gmail.com wrote:
Twisted's job is to run the main loop; it's best just to let it do that. If Pyglet slowed down, it would be best to examine what's gone wrong with Pyglet and simply submit a fix for that. We spoke to Richard Jones about Pyglet/Twisted integration at Chris Armstrong's excellent gaming+twisted openspace at PyCon, and I believe that some future version of Pyglet will include Twisted integration. Of course, this is just a plan right now, not code, so perhaps your time would be better spent contributing this feature to Pyglet, rather than turning Twisted inside-out :).
![](https://secure.gravatar.com/avatar/d6304567ada7ac5e8c6f4e5902270831.jpg?s=120&d=mm&r=g)
On Mon, Mar 31, 2008 at 10:17 PM, <glyph@divmod.com> wrote:
Yeah, I've experienced some performance issues with pyglet's app.runtime (maybe Linux-specific or something). Given the time, I'll submit a meaningful bug report concerning this. Regarding integration with Twisted, though, I think the best thing to do (for now) is to just ignore pyglet.app.runtime and write a pump function, roughly: def pump(): dt = clock.tick(True) # polling tick w.clear() w.dispatch_events() ... w.flip() if w.has_exit and reactor.running: reactor.stop() Be sure to call clock.tick(True) - since clock.tick will otherwise block. The next thing to do is customize clock where appropriate - that is, to used Twisted's scheduling functions. Coincidentally, I did this about 45 minutes ago for a project I'm working on: http://hg.enterthefoo.com/roboto/file/db931f29b98e/roboto/clock.py Beware - the above is half-baked and written under the influence of Hobgoblin Dark English Ale. Finally, you need schedule the pump function. To do it as fast as possible: from twisted.internet import task gameLoop = task.LoopingCall(pump) gameLoop.start(0., True) reactor.run() Or with a set framerate: ... gameLoop.start(1/60., True) Cheers, -- \\\\\/\"/\\\\\\\\\\\ \\\\/ // //\/\\\\\\\ \\\/ \\// /\ \/\\\\ \\/ /\/ / /\/ /\ \\\ \/ / /\/ /\ /\\\ \\ / /\\\ /\\\ \\\\\/\ \/\\\\\/\\\\\/\\\\\\ d.p.s
![](https://secure.gravatar.com/avatar/826694d326649b5e681474e8e2116dc1.jpg?s=120&d=mm&r=g)
On Mon, Mar 31, 2008 at 10:18 PM, Drew Smathers <drew.smathers@gmail.com> wrote:
Thanks for the great advice. I implemented it all and it runs, but unfortunately there was no noticeable change (for better or worse) to pyglet's performance. I'm currently working on setting up a stripped-down twisted+pyglet example that reproduces the problem. I'll post it when I get it up and running. ~ Nathan
![](https://secure.gravatar.com/avatar/a61e243764490913906c773c9acb0d3c.jpg?s=120&d=mm&r=g)
Nathan <nathan.stocks@gmail.com> writes:
While in general I'm always in favor of letting Twisted own the main loop, I've also had quite good success in the past not doing so, even without multiple threads, primarily in cases where networking was a secondary function (e.g., GUI applications) and less time crucial than the primary UI interaction. In our case, with wxPython based applications, things worked great just having wxPython handle a timer that triggered reactor iterations. It involved: * During initialization, start reactor with reactor.startRunning() * At some periodic frequency, run reactor.iterate(0) * During termination, set up a reactor.callLater for reactor.stop, and then call reactor.mainLoop(). This was the cleanest way we found to permit all reactor finalization (events, final I/O, etc...). Note that the first point violates the IReactorCore interface (startRunning() should only be accessed through run()). I don't consider running mainLoop() at the end a violation since it's the only time it is called. Yes, the above means that Twisted's loop might incur higher latency (we ran the timer at about 150ms) than it might be if in control, but it worked very well for us, including during times when the win32/wxPython reactors weren't particularly stable, and _threadedselect wasn't yet an option. And even if in control, once Twisted calls out to pump GUI messages, large latency is still theoretically possible until it regains control. So while I wouldn't usually suggest making Twisted's main loop subservient to some other loop (and trying to identify the issue in Pyglet would be a good thing), it's also something I'm not averse to doing when needed. It might be worth experimenting with using Pyglet's main loop, and pyglet.clock.schedule or pyglet.clock.schedule_interval to iterate Twisted's reactor. -- David
![](https://secure.gravatar.com/avatar/826694d326649b5e681474e8e2116dc1.jpg?s=120&d=mm&r=g)
On Thu, Apr 3, 2008 at 1:35 PM, David Bolen <db3l.net@gmail.com> wrote:
That's exactly what I was looking for, thanks! I'll save this as the technique to try if all else fails and the time comes that I've got to deploy. Hopefully we'll just be able to figure out what's going on and fix it before then. ~ Nathan
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On 08:32 pm, nathan.stocks@gmail.com wrote:
Definitely for an "if all else fails" scenario, this should work well enough :). But definitely reserve it for that. For PyGame, there aren't too many other considerations which will practically cause problems, since you're spinning at full-throttle all the time and you are unlikely to want to integrate with too much other code. (And if everything's written to just expect a normal, running reactor anyway, you can easily fix it later.) For a regular GUI app though, waking up every 150ms or so can be more serious than the apparently negligible CPU usage it causes. Try, for example, running your program under http://en.wikipedia.org/wiki/PowerTOP to see if this reactor integration mechanism is running down your laptop users' battery :). (This message written from a laptop whose battery is no longer being devoured by Firefox's too-frequent wakeups...)
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Thu, 03 Apr 2008 15:35:36 -0400, David Bolen <db3l.net@gmail.com> wrote:
Let me emphasize this. The above relies on implementation details and may be broken by any subsequent release (assuming it works now, which I won't comment on). Jean-Paul
![](https://secure.gravatar.com/avatar/313c3d40f833c31895423ce0621cc489.jpg?s=120&d=mm&r=g)
The Qt reactor, http://code.tarbox.org/qtreactor, inverts the "ownership" of the callback loop as well. However, Qt provides well behaved callbacks for timers as well as socket activity so the reactor behaves properly. The qt reactor design allows for the conventional reactor.run() at the end of the script as well as a call to reactor.runReturn() which starts up the reactor, nails up the timers and socket IO but returns to the calling application. Hence, you can startup twisted sometime after your gui app is running. Obviously, you need to "behave" properly and make sure that any user code doesn't hang onto the thread but this is required for proper gui handling code anyway... Twisted's current wx reactor uses another thread due to difficulties experienced with an earlier version of wx not behaving well. There are complexities WRT integrating those threads which lie at the heart of the testing failures for the current wx reactor. It might be easy to fix... However, I've been thinking about taking the qtreactor approach and (re) trying to fix the wx reactor by first evaluating whether wx is "better behaved". If it isn't, perhaps its straightforward to fix wx (???) Thing is, once you get the logic down... and (I think) its all correct in the qtreactor, it should be straightforward to use the wx calls to get servicing for timing and socket IO. -glenn On Thu, 2008-04-03 at 17:35 -0500, Jean-Paul Calderone wrote:
participants (6)
-
David Bolen
-
Drew Smathers
-
Glenn H Tarbox, PhD
-
glyph@divmod.com
-
Jean-Paul Calderone
-
Nathan