Anthony Baxter wrote:

The problem has been that wx's timer promises "no worse than 1s" resolution. In practice, it's been utterly unable to hit less than 100ms on Windows, and 20ms on Linux is a sometimes-thing. The only real solution that works robustly is to run wx and twisted in separate threads. If there's a new version of wx, you should probably try running wxreactor with a LoopingCall set to 1ms, and see what timer it can hit reliably.

I wonder if anyone has a minimalist demo of wx and twisted running in different threads? (I ask only because I'm under the proverbial gun to GUIify my twisted app.)

Cheers, Darran.

I am new to twisted and also experienced problems using wxreactor. After reading some earlier posts to this message list I wrote a wx.App sub-class that polls the reactor ONLY when the GUI requires data from twisted:

# This is a method of derived wx.App class I'm leaving out init/clean up methods: def twistedCall(self,deferredFunction,timeOut=3.0): """ Call a remote method and return when it is done (or times out) """ class WaitForDeferred(object): def __call__(self,deferredFunction,timeOut=3.0): self.loop=True; self.returnValue=None deferredFunction().addBoth(self.__callback) if timeOut>0: self.timeOut=reactor.callLater( timeOut,self.__callback, Failure("Timeout while connecting to host",TimeoutError) ) else: self.timeOut=None

while self.loop: reactor.runUntilCurrent(); reactor.iterate(0)

if isinstance(self.returnValue,Failure): self.returnValue.trap() return self.returnValue

def __callback(self,arg): if self.timeOut is not None: self.timeOut.cancel() self.timeOut=None; self.returnValue=arg; self.loop=False

try: return WaitForDeferred()(deferredFunction,timeOut) # This is to reconnect to server after the connection is broken (client factory is overridden to # reconnect as well) Is there a better way of doing this???: except pb.DeadReferenceError: self.twistedPerspective=self.twistedCall(lambda: self.twistedFactory.login(self.twistedCredentials) ) return WaitForDeferred()(deferredFunction,timeOut)

So when the application needs to login, request data, etc., it calls wx.GetApp().twistedCall(...) and the wx app is "blocked" until the deferred completes. In addition I have a wx.Timer poll set at one second to do one reactor iteration "just in case" although things seem to run fine without this extra poll.

In preliminary tests I was able to run 10+ client instances each requesting 551k blocks of data at 1 second intervals and everything seemed to run fine (some of the wx polls were missed but twisted never dropped anything). Note that this is a PB based client/server application running (both clients and sever) on a 1.7 Ghz P4 w/1 GB under Linux.

I was pretty impressed with this performance esp. since the server was reading the 551k from disk each time it was requested with a simple "return file(read(...))" statement and I did not even wrap the server code in a deferred like I should of. I did this test to see if the the reactor loop, as implemented, would "starve" and drop data producing timeout errors.

This testing was done before I read this latest post and before any actual production application code has been developed. So am I better off just scrapping this approach and adopting a threaded method? The reason I adopted the above approach was that, in general, the applications I write NEED to wait for data before they can continue. For example, the user enter a userid and password and hits the "login" button. No further (interactive) processing can be performed until the server validates the input and authorizes/rejects the login.

I have over 20 years applications exp but minimal networking/systems type exp so forgive me if my questions (and testing approach) are naive. Also, I apologize for the length of this post. I tried to avoid posting so much code but could not find a better way to explain what I was doing (I will be happy to provide complete examples if anyone is interested).

Shawn Church Information Systems Consultant sl_church@sbcglobal.net http://SChurchComputers.com

Uwe C. Schroeder (uwe@oss4u.com) wrote:

You probably better switch to using separate threads. Although I basically invented the timer driven "eventloop starving" quite a while ago, published it in the python cookbook (which eventually was turned into wxreactor by Itamar), I meanwhile switched my application to multiple threads. Works way better, particularly on windows. The timer-eventloop works fine for smaller applications with little to moderate network usage, the moment you start transferring more data over the network you basically starve both eventloops to death, because neither of the loops gets enough CPU attention to do the job properly.

You might want to check out this example:

http://solipsis.netofpeers.net/wiki/wikka.php?wakka=WxTwistedExample

It's a basic example of multiple threads, but it will give you the idea. I've written a network wrapper layer to make this whole thing more easy to handle, but I can't put this online just yet (contract reasons) - and it's very specialized for my application anyways. Just to give you a hint: once you understood how the above example works, you can write wrapper classes for the remote referenceable interfaces of twisted. This allows you to use the twisted interface the way it's documented without having to fiddle around with the threads.

Hope that gives you some clues

UC

On Wednesday 13 April 2005 19:58, Anthony Baxter wrote:

...

The problem has been that wx's timer promises "no worse than 1s" resolution. In practice, it's been utterly unable to hit less than 100ms on Windows, and 20ms on Linux is a sometimes-thing. The only real solution that works robustly is to run wx and twisted in separate threads. If there's a new version of wx, you should probably try running wxreactor with a LoopingCall set to 1ms, and see what timer it can hit reliably.

Anthony

-- Open Source Solutions 4U, LLC 2570 Fleetwood Drive Phone: +1 650 872 2425 San Bruno, CA 94066 Cell: +1 650 302 2405 United States Fax: +1 650 872 2417

---

Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

On Apr 15, 2005, at 10:27 AM, Glyph Lefkowitz wrote:

Shawn Church wrote:

...

I am new to twisted and also experienced problems using wxreactor. After reading some earlier posts to this message list I wrote a wx.App sub-class that polls the reactor ONLY when the GUI requires data from twisted:

Eep! This is definitely broken.

The simplest way to explain why is: what if you are writing a chat program, and your GUI might need to be notified about network events at any time? Deferreds are just one way that Twisted can trigger events. There are lots of other callbacks which your program might be interested in.

Well, if you know that can't happen, (or that you aren't interested if it does happen) then you're all set. :)

James

It is entirely possible that twisted is NOT what I should be using. What I want to do basically is: 1) Provide a consistant login for users independent of any DB; 2) Provide a security framework based upon number 1 (i.e. user sam can do a,b, and c but user john can only do a); and 3) Provide the abililty for the server to query a DB backend in response to requests from the client, and return the data. In addition, it would be very nice to wrap all of this up in SSL to protect casual "snooping" of data.

I have looked at implementations of Soap, CORBA, and Pyro, none of which have really statisfied me. I thought that by using the PB interface of twisted, along with a SSL connection, I could achieve all of my objectives fairly simply -- once I worked out the details I was able connect, login, and transfer data with very little code.

I fully realize that if I want to write a GUI application that depends upon arbitrary responses from the server (i.e. a chat client to use your example) then my approach will not work. What I want to do is have my clients request data from the server and wait till the data is ready (seems prettty simple :-)) Twisted seems to do what I want (assuming I can interface with wx), but if you can point to another package please let me know.

Glyph Lefkowitz (glyph@divmod.com) wrote:

James Y Knight wrote:

...

Well, if you know that can't happen, (or that you aren't interested if it does happen) then you're all set. :)

True enough, but in that case, don't use Twisted - it's just extra complexity. urllib will do just fine.

---

Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

OK, I will adopt the threaded approach. Despite it's mainloop problems (they can surface even without twisted) I still like wx. I develop on a linux system but support mostly windows clients and wx makes this easy. Thanks to everyone for their valuable responses to my questions.

Glyph Lefkowitz wrote:

Shawn Church wrote:

...

I have looked at implementations of Soap, CORBA, and Pyro, none of which have really statisfied me. I thought that by using the PB interface of twisted, along with a SSL connection, I could achieve all of my objectives fairly simply -- once I worked out the details I was able connect, login, and transfer data with very little code.

All of this, and more, is possible with Twisted.

...

I fully realize that if I want to write a GUI application that depends upon arbitrary responses from the server (i.e. a chat client to use your example) then my approach will not work.

That was an example of where the brokenness would be instantly visible to any user of your program. Internally Twisted makes many assumptions about the reactor generally being running which may cause you lots of little surprises if you use the technique you've proposed.

...

What I want to do is have my clients request data from the server and wait till the data is ready (seems prettty simple :-))

the Twisted way is to invert it so that rather than saying "Wait until Y is done, then do X", you say "Do X when Y is done." Among other things, this approach lets you do more than one thing at a time. For example, you might want to put a "cancel" button in your UI if your application has to run over the internet. Responding to the cancel button at the same time as waiting for a request's response is a second thing, so you need to be worried about concurrency even in simple applications.

...

Twisted seems to do what I want (assuming I can interface with wx),

...

but if you can point to another package please let me know.

I would point you at gtk, Tkinter, or win32api, not an alternative to Twisted :). wx is the problem here: its mainloop is badly broken, on all platforms. Twisted works fine. The easiest fix, if wx is a genuine requirement, is to run the wx main loop in a Twisted thread, using callInThread, so that the Twisted event loop (i.e. the one that isn't horribly broken) is "in charge".

I believe that future releases of wx are fixing this issue, as well, since you're far from the first person to have complained about it. I hope that this happens soon, since despite its many flaws wx is very popular in the Python community.

P.S.: If I seem negative towards WX, it is only because my _only_ use of it is answering this same question over and over again. I'm sure it has some positive points too, but I never discover them, because for my own development projects "non-broken mainloop" is criterion #1 for the GUI framework ;-)

---

Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

On Apr 15, 2005, at 4:29 PM, Uwe C. Schroeder wrote:

On Friday 15 April 2005 12:53, Glyph Lefkowitz wrote:

...

Shawn Church wrote:

...

I have looked at implementations of Soap, CORBA, and Pyro, none of which have really statisfied me. I thought that by using the PB interface of twisted, along with a SSL connection, I could achieve all of my objectives fairly simply -- once I worked out the details I was able connect, login, and transfer data with very little code.

All of this, and more, is possible with Twisted.

...

I fully realize that if I want to write a GUI application that depends upon arbitrary responses from the server (i.e. a chat client to use your example) then my approach will not work.

That was an example of where the brokenness would be instantly visible to any user of your program. Internally Twisted makes many assumptions about the reactor generally being running which may cause you lots of little surprises if you use the technique you've proposed.

...

What I want to do is have my clients request data from the server and wait till the data is ready (seems prettty simple

:-))

the Twisted way is to invert it so that rather than saying "Wait until Y is done, then do X", you say "Do X when Y is done." Among other things, this approach lets you do more than one thing at a time. For example, you might want to put a "cancel" button in your UI if your application has to run over the internet. Responding to the cancel button at the same time as waiting for a request's response is a second thing, so you need to be worried about concurrency even in simple applications.

...

Twisted seems to do what I want (assuming I can interface with wx),

but if you can point to another package please let me know.

I would point you at gtk, Tkinter, or win32api, not an alternative to Twisted :). wx is the problem here: its mainloop is badly broken, on all platforms. Twisted works fine. The easiest fix, if wx is a genuine requirement, is to run the wx main loop in a Twisted thread, using callInThread, so that the Twisted event loop (i.e. the one that isn't horribly broken) is "in charge".

I believe that future releases of wx are fixing this issue, as well, since you're far from the first person to have complained about it. I hope that this happens soon, since despite its many flaws wx is very popular in the Python community.

P.S.: If I seem negative towards WX, it is only because my _only_ use of it is answering this same question over and over again. I'm sure it has some positive points too, but I never discover them, because for my own development projects "non-broken mainloop" is criterion #1 for the GUI framework ;-)

There are certainly people better suited to respond to this, I'll try anyways. The main problem with wx's mainloop is that wx uses the eventloops of the platform dependant gui's. If you run wx on Linux with GTK2 it uses the GTK2 eventloop. On win it uses the windows loop etc. etc. I'm not sure this can be fixed - but as said, there are people who have more in-depth knowledge about the issue. Wx is so popular because it works on all platforms in a (mostly) consistent way. The only alternative I could think of is TK or JAVA SWING (and both look really ugly). If you have to write a gui application that has to run on multiple platforms and needs to look a bit more modern, you have very few choices. Another idea here. Why not write a "wxreactor" that handles the threading already and provides a "twisted consistent" API to the gui thread? That way one could encapsulate any broken mainloop out there. Just a thought.

It would make perfect sense to provide an abstract reactor implementation that did the select/poll/etc. in a separate thread which sends an event to the mainloop to wake up the reactor... it could be used to interoperate with just about anything, pygame, wxPython, pygtk, Cocoa, etc. and would require very little work to "port".

-bob