[Twisted-Python] Twisted and python-ogre
Hello, I am currently part of a MMORPG project in python, using Twisted and python-ogre. I found some posts talking about the cohabitation between the two. However I couldn't find a definitive answer. What I am seeking for is more a general method than technical details. What I would like to do is to put properly Twisted and python-ogre together. To do that I have already tried the following ways : - First run python-ogre in the main loop. At a time, this loop launches the twisted reactor in another thread with Thread(target = sthg_which_runs_reactor.run). The problem I am experiencing is described here [1]. The sendLine method is executed immediately, but the data is received by the other side only 20/30 seconds after. - So I tried not to run Twisted in another thread, by using the way described here [2]. Twisted is given the main loop and the rendering is done frame by frame with the coiterate. It works sometimes, the renderOneFrame in the coiterration is not really working. But it seems more logical to run ptyhon-ogre in the main loop and twisted on the side (in another thread for example, as soon as the client has decided to connect to the server). Have you any ideas how I could do this properly (with threads, without threads, with the coiterrate ...)? If Twisted was originally made for MMORPGs, why the reactor wants so much to be in the main loop and to manage itself the other threads (with callInThread for example)? Cheers [1] http://archives.free.net.ph/message/20080102.113837.4c4513eb.en.html [2] http://www.ogre3d.org/phpBB2addons/viewtopic.php?p=35187&sid=ce193664e1d3d7c4af509e6f4e2718c6 -- Thomas Boucher
On Thu, 14 Feb 2008 16:32:30 +0100, Thomas Boucher <thomas.boucher@student.ecp.fr> wrote:
Hello,
I am currently part of a MMORPG project in python, using Twisted and python-ogre. I found some posts talking about the cohabitation between the two. However I couldn't find a definitive answer. What I am seeking for is more a general method than technical details.
What I would like to do is to put properly Twisted and python-ogre together. To do that I have already tried the following ways :
- First run python-ogre in the main loop. At a time, this loop launches the twisted reactor in another thread with Thread(target = sthg_which_runs_reactor.run). The problem I am experiencing is described here [1]. The sendLine method is executed immediately, but the data is received by the other side only 20/30 seconds after.
Chris Armstrong's response to the post you refer to is accurate. Twisted APIs are not threadsafe. In order to call them from a thread other than the one in which the reactor is running, it is absolutely required that you use reactor.callFromThread. This is quite simple. Instead of: proto.sendLine("line") Simple do: reactor.callFromThread(proto.sendLine, "line") You will have to do this for every call you make to a Twisted API (except for reactor.callFromThread, of course ;). Since this is quite inconvenient, inefficient, and error prone, it's best if you keep the interaction between different threads to the barest minumum you can manage.
- So I tried not to run Twisted in another thread, by using the way described here [2]. Twisted is given the main loop and the rendering is done frame by frame with the coiterate. It works sometimes, the renderOneFrame in the coiterration is not really working.
What do you mean "not really working"?
But it seems more logical to run ptyhon-ogre in the main loop and twisted on the side (in another thread for example, as soon as the client has decided to connect to the server). Have you any ideas how I could do this properly (with threads, without threads, with the coiterrate ...)?
The best solution would be to really integrate the two loops. I don't know the details of OGRE though, so I can't tell you specifically how to do this. It is tightly dependent on the details of the loops being integrated.
If Twisted was originally made for MMORPGs, why the reactor wants so much to be in the main loop and to manage itself the other threads (with callInThread for example)?
Because most other event loops are very bad at managing events. :) If you want reasonable network performance, you need a good network event loop to be in control. Jean-Paul
on 14.02.2008 17:01 Jean-Paul Calderone said the following:
On Thu, 14 Feb 2008 16:32:30 +0100, Thomas Boucher <thomas.boucher@student.ecp.fr> wrote: <snip>
What I would like to do is to put properly Twisted and python-ogre together. To do that I have already tried the following ways :
<snip>
But it seems more logical to run ptyhon-ogre in the main loop and twisted on the side (in another thread for example, as soon as the client has decided to connect to the server). Have you any ideas how I could do this properly (with threads, without threads, with the coiterrate ...)?
The best solution would be to really integrate the two loops. I don't know the details of OGRE though, so I can't tell you specifically how to do this. It is tightly dependent on the details of the loops being integrated.
I use python-ogre plus twisted: no threads, twisted has control and a LoopingCall triggers a basic Ogre iteration step (similar to the coiteration idea the OP referenced). Using a LoopingCall provides automatic frame-rate limitation, the coiteration code gives render-as-often-as-possible. Twisted's main loop is by far more complicated than Ogre's default loop, the latter simply runs pumpmessages-render-callframelisteners as often as possible and is easier to integrate into twisted's. cheers, stefan
Jean-Paul Calderone a écrit :
Chris Armstrong's response to the post you refer to is accurate. Twisted APIs are not threadsafe. In order to call them from a thread other than the one in which the reactor is running, it is absolutely required that you use reactor.callFromThread. This is quite simple. Instead of:
proto.sendLine("line")
Simple do:
reactor.callFromThread(proto.sendLine, "line")
You will have to do this for every call you make to a Twisted API (except for reactor.callFromThread, of course ;).
I have just tried this method, and it perfectly works. The twisted reactor is run in another thread and the calls to Twisted are managed by the callFromThread method. The main advantage I see to it (excepting the fact that it works) is that it does not require changes in the way the graphic rendering is done : you do not care about how python-ogre runs to code the twisted part.
The best solution would be to really integrate the two loops. I don't know the details of OGRE though, so I can't tell you specifically how to do this. It is tightly dependent on the details of the loops being integrated.
What I can't see for the moment is how much better it would be to do like the other ways delivered in the discussion thread (by Werner, Bernie an Stefan). These ways consist in running Twisted first and in letting it manage python-ogre (by doing the rendering frame by frame with coiteration, or loopingcalls). Is this better than doing separate threads ?
Because most other event loops are very bad at managing events. :) If you want reasonable network performance, you need a good network event loop to be in control.
Don't you think using basic sockets (easy to code in Python) to make the network connection from a client to a server is more efficient (considering the time spent to do it or the complexity of the code) and more easy to keep in a project ? I suppose the connexion is better managed with Twisted with a lot of control (and moreover there are huge libs) but to what extend ? Anyway, thanks for all the suggestions already given by everybody. Cheers -- Thomas Boucher
On Fri, 15 Feb 2008 15:01:21 +0100, Thomas Boucher <thomas.boucher@student.ecp.fr> wrote:
Jean-Paul Calderone a écrit :
Chris Armstrong's response to the post you refer to is accurate. Twisted APIs are not threadsafe. In order to call them from a thread other than the one in which the reactor is running, it is absolutely required that you use reactor.callFromThread. This is quite simple. Instead of:
proto.sendLine("line")
Simple do:
reactor.callFromThread(proto.sendLine, "line")
You will have to do this for every call you make to a Twisted API (except for reactor.callFromThread, of course ;).
I have just tried this method, and it perfectly works. The twisted reactor is run in another thread and the calls to Twisted are managed by the callFromThread method. The main advantage I see to it (excepting the fact that it works) is that it does not require changes in the way the graphic rendering is done : you do not care about how python- ogre runs to code the twisted part.
The best solution would be to really integrate the two loops. I don't know the details of OGRE though, so I can't tell you specifically how to do this. It is tightly dependent on the details of the loops being integrated.
What I can't see for the moment is how much better it would be to do like the other ways delivered in the discussion thread (by Werner, Bernie an Stefan). These ways consist in running Twisted first and in letting it manage python-ogre (by doing the rendering frame by frame with coiteration, or loopingcalls). Is this better than doing separate threads ?
There are arguments about efficiency. Using threads has overhead which is not present when not using threads. These arguments are probably not very convincing in this case, since you would only have two threads and their overhead will be dwarfed by your rendering costs. You should be aware that reactor.callFromThread _does_ have some non-trivial overhead. Using it for each sendLine call _could_ have a noticable detrimental impact on your app's performance. There are arguments about simplicity. It's harder to understand what a multithreaded program is doing. It's harder to debug a multithreaded program. It's very, very hard to unit test a multithreaded program. This is probably the compelling argument to try to avoid threads.
Because most other event loops are very bad at managing events. :) If you want reasonable network performance, you need a good network event loop to be in control.
Don't you think using basic sockets (easy to code in Python) to make the network connection from a client to a server is more efficient (considering the time spent to do it or the complexity of the code) and more easy to keep in a project ?
That might be nice. However, almost without exception, the network code I read which does not use Twisted is buggy in the most trivial ways. It is rare for it to take more than a cursory glance to notice critical bugs (the sort which will result in data being lost). Of course, this doesn't mean that all network code written without Twisted is fatally flawed, but it does give me a reasonable basis to assume that it will have issues. :) Twisted's APIs aren't just easier to use correctly than the BSD socket APIs, they're also consistent across platforms. Are you sure you know the BSD APIs well enough to write some that is correct on Linux and on Windows? (Let alone on OS X or FreeBSD) Even if you are, would you rather spend your time writing the error handling logic for each of these platforms or adding features to your game? Twisted may seems to be harder than the socket module. It probably does have a higher up-front cost. However, in the long run, I think that you'll save yourself a lot of time using it rather than using sockets directly. You also get the advantages which the structure Twisted attempts to impose on your application confers. For example, protocol/transport separation means that you can unit test your protocol implementation very easily. And that's assuming you even need to implement your own protocol, rather than using one of the protocols Twisted already includes an implementation of. You also leave open the possibility of easily taking advantage of some other Twisted features which don't seem useful now but which you may benefit from later in development. Hope this helps, Jean-Paul
Jean-Paul Calderone wrote:
On Fri, 15 Feb 2008 15:01:21 +0100, Thomas Boucher <thomas.boucher@student.ecp.fr> wrote:
Jean-Paul Calderone a écrit :
Chris Armstrong's response to the post you refer to is accurate. Twisted APIs are not threadsafe. In order to call them from a thread other than the one in which the reactor is running, it is absolutely required that you use reactor.callFromThread. This is quite simple. Instead of:
proto.sendLine("line")
Simple do:
reactor.callFromThread(proto.sendLine, "line")
You will have to do this for every call you make to a Twisted API (except for reactor.callFromThread, of course ;).
I have just tried this method, and it perfectly works. The twisted reactor is run in another thread and the calls to Twisted are managed by the callFromThread method. The main advantage I see to it (excepting the fact that it works) is that it does not require changes in the way the graphic rendering is done : you do not care about how python- ogre runs to code the twisted part.
The best solution would be to really integrate the two loops. I don't know the details of OGRE though, so I can't tell you specifically how to do this. It is tightly dependent on the details of the loops being integrated.
What I can't see for the moment is how much better it would be to do like the other ways delivered in the discussion thread (by Werner, Bernie an Stefan). These ways consist in running Twisted first and in letting it manage python-ogre (by doing the rendering frame by frame with coiteration, or loopingcalls). Is this better than doing separate threads ?
There are arguments about efficiency. Using threads has overhead which is not present when not using threads. These arguments are probably not very convincing in this case, since you would only have two threads and their overhead will be dwarfed by your rendering costs. You should be aware that reactor.callFromThread _does_ have some non-trivial overhead. Using it for each sendLine call _could_ have a noticable detrimental impact on your app's performance.
There are arguments about simplicity. It's harder to understand what a multithreaded program is doing. It's harder to debug a multithreaded program. It's very, very hard to unit test a multithreaded program. This is probably the compelling argument to try to avoid threads.
Because most other event loops are very bad at managing events. :) If you want reasonable network performance, you need a good network event loop to be in control.
Don't you think using basic sockets (easy to code in Python) to make the network connection from a client to a server is more efficient (considering the time spent to do it or the complexity of the code) and more easy to keep in a project ?
That might be nice. However, almost without exception, the network code I read which does not use Twisted is buggy in the most trivial ways. It is rare for it to take more than a cursory glance to notice critical bugs (the sort which will result in data being lost). Of course, this doesn't mean that all network code written without Twisted is fatally flawed, but it does give me a reasonable basis to assume that it will have issues. :)
Twisted's APIs aren't just easier to use correctly than the BSD socket APIs, they're also consistent across platforms. Are you sure you know the BSD APIs well enough to write some that is correct on Linux and on Windows? (Let alone on OS X or FreeBSD) Even if you are, would you rather spend your time writing the error handling logic for each of these platforms or adding features to your game?
Twisted may seems to be harder than the socket module. It probably does have a higher up-front cost. However, in the long run, I think that you'll save yourself a lot of time using it rather than using sockets directly.
You also get the advantages which the structure Twisted attempts to impose on your application confers. For example, protocol/transport separation means that you can unit test your protocol implementation very easily. And that's assuming you even need to implement your own protocol, rather than using one of the protocols Twisted already includes an implementation of.
You also leave open the possibility of easily taking advantage of some other Twisted features which don't seem useful now but which you may benefit from later in development.
Hope this helps,
It certainly lists a goodly number of reasons why people should strongly consider using Twisted, and there are a goodly number one doesn't come across every day. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/
Hi Thomas Looks like the twisted event loop should be in control and Ogre should be called like hb = task.LoopingCall(ogre.heartBeat) #system pump hb.start(0.05) with ogre.heartbeat operating along the line of Ogre::Root::getSingletonPtr()->renderOneFrame(); //C++ code I don't know if this particular C++ call is exposed in PyOgre, but we have Ogre running in a networked multiplayer wxWidgets app written in C++ with exactly that setup, although the limit is - only one Ogre window per app when using this approach. I would be really interested in how you solved your problem! In the meantime HTH, Werner Thomas Boucher wrote:
Hello,
I am currently part of a MMORPG project in python, using Twisted and python-ogre. I found some posts talking about the cohabitation between the two. However I couldn't find a definitive answer. What I am seeking for is more a general method than technical details.
What I would like to do is to put properly Twisted and python-ogre together. To do that I have already tried the following ways :
- First run python-ogre in the main loop. At a time, this loop launches the twisted reactor in another thread with Thread(target = sthg_which_runs_reactor.run). The problem I am experiencing is described here [1]. The sendLine method is executed immediately, but the data is received by the other side only 20/30 seconds after.
- So I tried not to run Twisted in another thread, by using the way described here [2]. Twisted is given the main loop and the rendering is done frame by frame with the coiterate. It works sometimes, the renderOneFrame in the coiterration is not really working.
But it seems more logical to run ptyhon-ogre in the main loop and twisted on the side (in another thread for example, as soon as the client has decided to connect to the server). Have you any ideas how I could do this properly (with threads, without threads, with the coiterrate ...)?
If Twisted was originally made for MMORPGs, why the reactor wants so much to be in the main loop and to manage itself the other threads (with callInThread for example)?
Cheers
[1] http://archives.free.net.ph/message/20080102.113837.4c4513eb.en.html [2] http://www.ogre3d.org/phpBB2addons/viewtopic.php?p=35187&sid=ce193664e1d3d7c4af509e6f4e2718c6
On Thu, Feb 14, 2008 at 10:32 AM, Thomas Boucher < thomas.boucher@student.ecp.fr> wrote:
I am currently part of a MMORPG project in python, using Twisted and python-ogre.
Hey, me too! I wonder if anyone else out there is doing this -- it seems like a natural combination of technologies. - First run python-ogre in the main loop. At a time, this loop launches
the twisted reactor in another thread
Everyone I've spoken to has recommended not using threads with Twisted.
- So I tried not to run Twisted in another thread, by using the way described here [2]. Twisted is given the main loop and the rendering is done frame by frame with the coiterate. It works sometimes, the renderOneFrame in the coiterration is not really working.
I hadn't seen that discussion thread before, and I'm not that familiar with coiteration. What I'm currently doing is this: def renderFrame(): ogre.WindowEventUtilities.messagePump() root.renderOneFrame() renderSystem._initRenderTargets() root.clearEventTimes() renderTask = task.LoopingCall(renderFrame) renderTask.start(0.01, False) reactor.run() I do my keyboard and mouse capture in a frame listener which I register with Ogre. It works, but I've only done a little bit of network activity (e.g. an IRC client). If any Twisted experts would care to comment on the above code, I'd appreciate it. I'm also wondering what would be the advantages of using a coiterator...? -- Bernie Roehl Mail: broehl@gmail.com
On Thu, Feb 14, 2008 at 1:30 PM, Bernie Roehl <broehl@gmail.com> wrote:
I'm also wondering what would be the advantages of using a coiterator...?
As an experiment, I tried switching to using a coiterator. I got a 30% increase in framerate! My code now looks like this: class RendererIterator: # used if usingCoiterator is True def __iter__(self): return self def next(self): ogre.WindowEventUtilities.messagePump() root.renderOneFrame() from twisted.internet.task import coiterate renderSystem._initRenderTargets() root.clearEventTimes() coiterate(RendererIterator()) reactor.run() Seems to work okay, but haven't tested it extensively. Any Twisted-gurus have any comments/suggestions/cautions? -- Bernie Roehl Mail: broehl@gmail.com
On Fri, 15 Feb 2008 10:31:14 -0500, Bernie Roehl <broehl@gmail.com> wrote:
On Thu, Feb 14, 2008 at 1:30 PM, Bernie Roehl <broehl@gmail.com> wrote:
I'm also wondering what would be the advantages of using a coiterator...?
As an experiment, I tried switching to using a coiterator. I got a 30% increase in framerate!
Hmmm. :)
My code now looks like this:
class RendererIterator: # used if usingCoiterator is True def __iter__(self): return self def next(self): ogre.WindowEventUtilities.messagePump() root.renderOneFrame()
from twisted.internet.task import coiterate
renderSystem._initRenderTargets() root.clearEventTimes() coiterate(RendererIterator()) reactor.run()
Seems to work okay, but haven't tested it extensively.
Any Twisted-gurus have any comments/suggestions/cautions?
There shouldn't really be any performance difference between using coiterate and using LoopingCall. The significant behavioral difference is that coiterate will call your code (RendererIterator.next in this case) as fast as it can (with some constraints). LoopingCall will call your code no faster than you specify. So my guess is that the 30% increase in FPS is because your system is capable of doing that much additional work above what you asked to do when you used LoopingCall. If you make a corresponding decrease in the interval you use with LoopingCall, do you not see any increase in FPS? If not, then you may be getting a bit of extra performance from coiterate from its behavior of sometimes calling your code multiple times without letting the reactor run. It will do this if it notices your code runs very quickly and so decides that it can afford to call it again before giving control back to the reactor to let network events be processed. This may give you a somewhat unevent frame rate, as compared to LoopingCall, but that might be suitable for your application. Jean-Paul
On Fri, Feb 15, 2008 at 10:51 AM, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Fri, 15 Feb 2008 10:31:14 -0500, Bernie Roehl <broehl@gmail.com> wrote:
As an experiment, I tried switching to using a coiterator. I got a 30% increase in framerate!
Hmmm. :)
Hmmm indeed!
There shouldn't really be any performance difference between using coiterate and using LoopingCall. The significant behavioral difference is that coiterate will call your code (RendererIterator.next in this case) as fast as it can (with some constraints). LoopingCall will call your code no faster than you specify.
So my guess is that the 30% increase in FPS is because your system is capable of doing that much additional work above what you asked to do when you used LoopingCall. If you make a corresponding decrease in the interval you use with LoopingCall, do you not see any increase in FPS?
Well, I was using renderTask = task.LoopingCall(renderFrame) renderTask.start(0.01, False) If I change the 0.01 to 0.0, I do indeed get a framerate comparable to using the coiterator methord. I wouldn't have thought that waiting ten milliseconds between calls would make such a substantial difference. -- Bernie Roehl Mail: broehl@gmail.com
* Bernie Roehl <broehl@gmail.com> [2008-02-15 11:09:07 -0500]:
If I change the 0.01 to 0.0, I do indeed get a framerate comparable to using the coiterator methord.
I wouldn't have thought that waiting ten milliseconds between calls would make such a substantial difference.
It limits you to a maximum of 100 frames/sec or so (since there are 100 ten millisecond intervals in a second); what sort of framerates are you seeing? Also, you should probably pick a number greater than 0, since there's not much point in rendering more than a certain number of frames every second -- the exact threshold at which it becomes pointless depends on a variety of application / environment specific factors, though. -- mithrandi, i Ainil en-Balandor, a faer Ambar
On the (relatively low-end) machine I'm running on, I get 33 fps with with 0.01, and 43 fps with 0.0 or with the coiterator. I'm leaning towards the coiterator approach, since I don't have to worry about figuring out what value to pass. On Fri, Feb 15, 2008 at 1:45 PM, Tristan Seligmann < mithrandi@mithrandi.za.net> wrote:
* Bernie Roehl <broehl@gmail.com> [2008-02-15 11:09:07 -0500]:
If I change the 0.01 to 0.0, I do indeed get a framerate comparable to using the coiterator methord.
I wouldn't have thought that waiting ten milliseconds between calls would make such a substantial difference.
It limits you to a maximum of 100 frames/sec or so (since there are 100 ten millisecond intervals in a second); what sort of framerates are you seeing? Also, you should probably pick a number greater than 0, since there's not much point in rendering more than a certain number of frames every second -- the exact threshold at which it becomes pointless depends on a variety of application / environment specific factors, though. -- mithrandi, i Ainil en-Balandor, a faer Ambar
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.8 (GNU/Linux)
iEYEARECAAYFAke13cAACgkQpNuXDQIV94oMSgCghgH4cj3lwyDsxLHn4lrtygkq ic8An1XBicz3/lObeGZkv7OGxVd2Nyqb =HJIx -----END PGP SIGNATURE-----
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Bernie Roehl Mail: broehl@gmail.com
That's consistent with a frame rendering time of roughly 23 ms. Waiting 10 ms after each rendering would indeed lower the rate by about 10 fps. regards Steve Bernie Roehl wrote:
On the (relatively low-end) machine I'm running on, I get 33 fps with with 0.01, and 43 fps with 0.0 or with the coiterator.
I'm leaning towards the coiterator approach, since I don't have to worry about figuring out what value to pass.
On Fri, Feb 15, 2008 at 1:45 PM, Tristan Seligmann <mithrandi@mithrandi.za.net <mailto:mithrandi@mithrandi.za.net>> wrote:
* Bernie Roehl <broehl@gmail.com <mailto:broehl@gmail.com>> [2008-02-15 11:09:07 -0500]:
> If I change the 0.01 to 0.0, I do indeed get a framerate comparable to using > the coiterator methord. > > I wouldn't have thought that waiting ten milliseconds between calls would > make such a substantial difference.
It limits you to a maximum of 100 frames/sec or so (since there are 100 ten millisecond intervals in a second); what sort of framerates are you seeing? Also, you should probably pick a number greater than 0, since there's not much point in rendering more than a certain number of frames every second -- the exact threshold at which it becomes pointless depends on a variety of application / environment specific factors, though. -- mithrandi, i Ainil en-Balandor, a faer Ambar
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.8 (GNU/Linux)
iEYEARECAAYFAke13cAACgkQpNuXDQIV94oMSgCghgH4cj3lwyDsxLHn4lrtygkq ic8An1XBicz3/lObeGZkv7OGxVd2Nyqb =HJIx -----END PGP SIGNATURE-----
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Bernie Roehl Mail: broehl@gmail.com <mailto:broehl@gmail.com>
------------------------------------------------------------------------
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/
* Steve Holden <steve@holdenweb.com> [2008-02-15 21:45:03 -0500]:
That's consistent with a frame rendering time of roughly 23 ms. Waiting 10 ms after each rendering would indeed lower the rate by about 10 fps.
But LoopingCall tries to schedule the next call for N seconds after the start of the previous call, not the end of the previous call, so that doesn't quite explain it. (Unless I'm misunderstanding something...) -- mithrandi, i Ainil en-Balandor, a faer Ambar
participants (7)
-
Bernie Roehl -
Jean-Paul Calderone -
Stefan Rank -
Steve Holden -
Thomas Boucher -
Tristan Seligmann -
Werner Thie