[Twisted-Python] 3 questions about async, threading and dynamic protocol selection
Q1: How to determine a function is 'blocking action' or not? Any function needs CPU times. Indeed, a computational-intensive function is blocking action. But how small/fast function can be classified as non-blocking? Twist requires all user functions to be non-blocked. If reactor calls a blocking function, what will happen? In my mind, reactor maintains a command queue internally (just like windows message queue). The blocking function only postpones the execution of other queued functions, but it does not break the logic of the program. Is that right? Q2: Today when I go through the twist document, I am confused about the threading problem in reactor. What is the different between 'reactor.callFromThread' and a plain call in reactor loop? def callFromThread(f): self.threadCallQueue.append((f, args, kw)) self.wakeUp() It seems equivalent to reactor.callLater(0, f...). What is the real circumstance for calling callFromThread? Further more, how to determine a function is thread-safe in python? Why the twist doc says: "writing data to a transport from a protocol is not thread-safe."? Q3: In my application, I need a facility to dynamically select a protocol to communicate with the server. Eg: When connected, the server sent a string to client to indicate the version of protocol it used. Then, the client can load the proper protocol. But I don't know how to implement this. A 'Factory' can only create one kind of 'Protocol', and it seems the instances of 'Protocol' cannot share the connection (Transport object) to a server. Could you give me some clues? Thank you. -- ShenLei
Hi u1CtDC, On 10:11 am, littlesweetmelon@gmail.com wrote:
Q1: How to determine a function is 'blocking action' or not? Any function needs CPU times. Indeed, a computational-intensive function is blocking action. But how small/fast function can be classified as non-blocking? Twist requires all user functions to be non-blocked. If reactor calls a blocking function, what will happen? In my mind, reactor maintains a command queue internally (just like windows message queue). The blocking function only postpones the execution of other queued functions, but it does not break the logic of the program. Is that right?
You've basically answered your own question here. A "blocking action" is one where your users will not want to wait for it :).
Q2: Today when I go through the twist document, I am confused about the threading problem in reactor. What is the different between 'reactor.callFromThread' and a plain call in reactor loop? def callFromThread(f): self.threadCallQueue.append((f, args, kw)) self.wakeUp() It seems equivalent to reactor.callLater(0, f...). What is the real circumstance for calling callFromThread?
callLater is not thread-safe. callFromThread is designed to be called, well, from a thread. You use callFromThread from threads *other* than the thread where the reactor is running, in order to wake up the reactor thread and have it do something - usually something that involves calling a non-thread-safe Twisted API.
Further more, how to determine a function is thread-safe in python?
As in any other language, ask the person who wrote it. There is no other way to determine if a function is thread-safe. In any language with dynamic run-time dispatch, determining this without talking to the author of the code in question reduces to the halting problem.
Why the twist doc says: "writing data to a transport from a protocol is not thread-safe."?
Nothing in Twisted is thread-safe (other than callFromThread) but this is a particularly common error and we wanted to stress it.
Q3: In my application, I need a facility to dynamically select a protocol to communicate with the server. Eg: When connected, the server sent a string to client to indicate the version of protocol it used. Then, the client can load the proper protocol. But I don't know how to implement this. A 'Factory' can only create one kind of 'Protocol', and it seems the instances of 'Protocol' cannot share the connection (Transport object) to a server. Could you give me some clues?
The protocol that you are implementing includes a "version" message. Unless you have a "Protocol" object connected to receive the data and decode that message, you can't decide which version to use for subsequent messages. Simply implement a "Protocol" which understands that "version" message and changes its behavior accordingly. For an example of how you might switch to a completely different protocol object, see http://twistedmatrix.com/trac/browser/trunk/twisted/protocols/amp.py#L1524 The techniques involved are quite nuanced, however, and are probably not appropriate for someone just learning about Twisted.
littlesweetmelon@gmail.com wrote:
Q1: How to determine a function is 'blocking action' or not? Any function needs CPU times. Indeed, a computational-intensive function is blocking action. But how small/fast function can be classified as non-blocking?
glyph@divmod.com wrote:
You've basically answered your own question here. A "blocking action" is one where your users will not want to wait for it :).
A value typically used as a reference point, on a human time scale, is a hundredth of a second, that is, 0.01s, that is, 10ms. If that's the maximum processing time of each event, you may expect a rate of at least 100 events per second, with a good overall event flow in the system, depending on the system load.
Twist requires all user functions to be non-blocked. If reactor calls a blocking function, what will happen? In my mind, reactor maintains a command queue internally (just like windows message queue). The blocking function only postpones the execution of other queued functions, but it does not break the logic of the program. Is that right?
Yes, that's right. The processing of subsequent events will be delayed, but the program will still run correctly, *if* its correctness does not depend on the timing of event processing. -- Nicola Larosa - http://www.tekNico.net/ E-mail can be saved by your ISP or by the IT department in your corporation. Gmail, for example, saves everything, even if you delete it. -- Bruce Schneier, November 2006
On 4/4/07, Nicola Larosa <nico@teknico.net> wrote:
littlesweetmelon@gmail.com wrote:
Twist requires all user functions to be non-blocked. If reactor calls a blocking function, what will happen? In my mind, reactor maintains a command queue internally (just like windows message queue). The blocking function only postpones the execution of other queued functions, but it does not break the logic of the program. Is that right?
Yes, that's right. The processing of subsequent events will be delayed, but the program will still run correctly, *if* its correctness does not depend on the timing of event processing.
However, I want to stress that technically Twisted is not maintaining a command queue relevant to this situation. Yes, it does have a queue of scheduled calls, but when you're just talking about running some code that blocks, it's not Twisted which is preventing further things from happening, it is the sequential nature of the code itself. There's no need to queue things up when Python itself, when executing "a(); b()", prevents 'b' from executing before 'a' has returned. -- Christopher Armstrong International Man of Twistery http://radix.twistedmatrix.com/ http://twistedmatrix.com/ http://canonical.com/
participants (4)
-
Christopher Armstrong -
glyph＠divmod.com -
Nicola Larosa -
甜瓜