[Twisted-Python] client connecting to 2 servers (non simultaneously)
Hi, I'm new to twisted and I have started to write a new protocol with a TCP client and server. In my protocol, a client should be able to connect to 2 servers (master/slave node - only the master accepts connection). The client should try to connect to server1. If it fails, try to connect to server2 (after a specific timeout). If that fails, try server1... I came up with a solution (see below). As I'm new to twisted and I haven't seen anything like that in the examples, I'd like to check if that's a proper way to do it. Any comments is welcome. Thanks Benjamin *********************************************** class MyClientFactory(ClientFactory): protocol = MyClientProtocol def __init__(self, host2=None): self.host1 = None self.host2 = host2 def clientConnectionFailed(self, connector, reason): from twisted.internet import reactor if self.host2 is None: # host2 is not defined, reconnect to host1 reactor.callLater(2.0, connector.connect) else: destination = connector.getDestination() if self.host1 is None: # First connection failed, initialize host1, and try host2 self.host1 = destination.host host = self.host2 elif destination.host == self.host1: # Connection to host1 failed, try host2 host = self.host2 else: # Connection to host2 failed, try host1 host = self.host1 reactor.callLater(2.0, reactor.connectTCP, host, destination.port, self) factory = MyClientFactory(server2) from twisted.internet import reactor reactor.connectTCP(server1, 8010, factory) reactor.run() ***********************************************
You could probably generalize and simplify by using a collections.deque of hosts/ports and using the rotate() method before each reactor.connectTCP. Also, no need for multiple reactor imports---one at the top of the code is fine. Note that if the connection is "lost" in a non-clean fashion, you may also want to reconnect. IIUC, "fail" only handles the case that no connection is made (protocol is never created). import collections from twisted.internet import reactor class MyClientFactory(object): protocol = MyClientProtocol def __init__(self, hosts): self.hosts = collections.deque(hosts) reactor.callWhenRunning(reactor.connectTCP, self.hosts[0][0], self.hosts[0][1], self) self.hosts.rotate(1) def clientConnectionFailed(self, connector, reason): reactor.callLater(2.0, connectTCP, self.hosts[0][0], self.hosts[0][1], self) self.hosts.rotate(1) factory = MyClientFactory([('host1', port1), ('host2', port2), ...]) reactor.run() Cheers, Jason On Tue, Jan 11, 2011 at 5:17 AM, <benjamin.bertrand@lfv.se> wrote:
-- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
Using a deque for this is a fantastic idea, though I would have used rotate(-1), as your example looks like it rotate's "backwards" to me. Matter of taste I suppose. It doesn't make much actual difference. Kevin Horn On Tue, Jan 11, 2011 at 7:11 AM, Jason Rennie <jrennie@gmail.com> wrote:
I agree that rotate(-1) is more intuitive. I barely gave it any thought---almost forgot to include an argument for rotate as I initially incorrectly assumed no argument would do something like rotate(-1) :-) Jason On Tue, Jan 11, 2011 at 1:36 PM, Kevin Horn <kevin.horn@gmail.com> wrote:
-- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
Nice, this looks much cleaner. I thought about the connection "lost" case. It's quite similar to "fail" (but client should always try to reconnect to host1 in that case). So easy to implement. Thanks a lot! Benjamin Från: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] För Jason Rennie Skickat: den 11 januari 2011 14:11 Till: Twisted general discussion Ämne: Re: [Twisted-Python] client connecting to 2 servers (nonsimultaneously) You could probably generalize and simplify by using a collections.deque of hosts/ports and using the rotate() method before each reactor.connectTCP. Also, no need for multiple reactor imports---one at the top of the code is fine. Note that if the connection is "lost" in a non-clean fashion, you may also want to reconnect. IIUC, "fail" only handles the case that no connection is made (protocol is never created). import collections from twisted.internet import reactor class MyClientFactory(object): protocol = MyClientProtocol def __init__(self, hosts): self.hosts = collections.deque(hosts) reactor.callWhenRunning(reactor.connectTCP, self.hosts[0][0], self.hosts[0][1], self) self.hosts.rotate(1) def clientConnectionFailed(self, connector, reason): reactor.callLater(2.0, connectTCP, self.hosts[0][0], self.hosts[0][1], self) self.hosts.rotate(1) factory = MyClientFactory([('host1', port1), ('host2', port2), ...]) reactor.run() Cheers, Jason On Tue, Jan 11, 2011 at 5:17 AM, <benjamin.bertrand@lfv.se> wrote: Hi, I'm new to twisted and I have started to write a new protocol with a TCP client and server. In my protocol, a client should be able to connect to 2 servers (master/slave node - only the master accepts connection). The client should try to connect to server1. If it fails, try to connect to server2 (after a specific timeout). If that fails, try server1... I came up with a solution (see below). As I'm new to twisted and I haven't seen anything like that in the examples, I'd like to check if that's a proper way to do it. Any comments is welcome. Thanks Benjamin *********************************************** class MyClientFactory(ClientFactory): protocol = MyClientProtocol def __init__(self, host2=None): self.host1 = None self.host2 = host2 def clientConnectionFailed(self, connector, reason): from twisted.internet import reactor if self.host2 is None: # host2 is not defined, reconnect to host1 reactor.callLater(2.0, connector.connect) else: destination = connector.getDestination() if self.host1 is None: # First connection failed, initialize host1, and try host2 self.host1 = destination.host host = self.host2 elif destination.host == self.host1: # Connection to host1 failed, try host2 host = self.host2 else: # Connection to host2 failed, try host1 host = self.host1 reactor.callLater(2.0, reactor.connectTCP, host, destination.port, self) factory = MyClientFactory(server2) from twisted.internet import reactor reactor.connectTCP(server1, 8010, factory) reactor.run() *********************************************** _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
One small question about the following code: Why did you use reactor.callWhenRunning in the __init__ method? Why not calling directly reactor.connectTCP? Cheers, Benjamin Från: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] För Jason Rennie Skickat: den 11 januari 2011 14:11 Till: Twisted general discussion Ämne: Re: [Twisted-Python] client connecting to 2 servers (nonsimultaneously) You could probably generalize and simplify by using a collections.deque of hosts/ports and using the rotate() method before each reactor.connectTCP. Also, no need for multiple reactor imports---one at the top of the code is fine. Note that if the connection is "lost" in a non-clean fashion, you may also want to reconnect. IIUC, "fail" only handles the case that no connection is made (protocol is never created). import collections from twisted.internet import reactor class MyClientFactory(object): protocol = MyClientProtocol def __init__(self, hosts): self.hosts = collections.deque(hosts) reactor.callWhenRunning(reactor.connectTCP, self.hosts[0][0], self.hosts[0][1], self) self.hosts.rotate(1) def clientConnectionFailed(self, connector, reason): reactor.callLater(2.0, connectTCP, self.hosts[0][0], self.hosts[0][1], self) self.hosts.rotate(1) factory = MyClientFactory([('host1', port1), ('host2', port2), ...]) reactor.run() Cheers, Jason On Tue, Jan 11, 2011 at 5:17 AM, <benjamin.bertrand@lfv.se> wrote: Hi, I'm new to twisted and I have started to write a new protocol with a TCP client and server. In my protocol, a client should be able to connect to 2 servers (master/slave node - only the master accepts connection). The client should try to connect to server1. If it fails, try to connect to server2 (after a specific timeout). If that fails, try server1... I came up with a solution (see below). As I'm new to twisted and I haven't seen anything like that in the examples, I'd like to check if that's a proper way to do it. Any comments is welcome. Thanks Benjamin *********************************************** class MyClientFactory(ClientFactory): protocol = MyClientProtocol def __init__(self, host2=None): self.host1 = None self.host2 = host2 def clientConnectionFailed(self, connector, reason): from twisted.internet import reactor if self.host2 is None: # host2 is not defined, reconnect to host1 reactor.callLater(2.0, connector.connect) else: destination = connector.getDestination() if self.host1 is None: # First connection failed, initialize host1, and try host2 self.host1 = destination.host host = self.host2 elif destination.host == self.host1: # Connection to host1 failed, try host2 host = self.host2 else: # Connection to host2 failed, try host1 host = self.host1 reactor.callLater(2.0, reactor.connectTCP, host, destination.port, self) factory = MyClientFactory(server2) from twisted.internet import reactor reactor.connectTCP(server1, 8010, factory) reactor.run() *********************************************** _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
Habit, mostly. I think it's a good habit, though. The reason is that I wanted to make sure the __init__ code completed before the connectTCP was called. In this case, it doesn't matter, but if I had a number of things I wanted to do in __init__, it might matter. Consider: class MyClientProtocol(Protocol): def connectionMade(self): self.factory.numConnections += 1 class MyClientFactory(Factory): def __init__(self): reactor.connectTCP(host, port, self) self.numConnections = 0 'course, I'm sure you'd put reactor.connectTCP *after* the numConnections initializer, but using callWhenRunning ensures that the order doesn't matter, so it's one less thing you have to worry about :-) Jason On Wed, Jan 12, 2011 at 4:49 AM, <benjamin.bertrand@lfv.se> wrote:
-- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
On 03:15 pm, jrennie@gmail.com wrote:
Although note that if the reactor is already running when you instantiate the factory, then callWhenRunning will immediately call the function you pass to it, as demonstrated by this transcript in which the reactor is already running: >>> def foo(): ... print 'foo running' ... >>> from twisted.internet import reactor >>> reactor.callWhenRunning(foo) foo running >>> Jean-Paul
Benjamin, sorry for providing you with an incorrect explanation. Jean-Paul, thank you for correcting my broken understanding of callWhenRunning. Is there a call which puts a function into the reactor's queue of tasks to be completed? Thanks, Jason On Wed, Jan 12, 2011 at 10:26 AM, <exarkun@twistedmatrix.com> wrote:
-- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
On Wed, Jan 12, 2011 at 10:52 AM, Jason Rennie <jrennie@gmail.com> wrote:
The reactor doesn't have a queue of tasks to be completed. It has sets of various event sources, which it executes in no particular order. Scheduling a timed event with callLater(0,...) might do what you want, though.
On Wed, Jan 12, 2011 at 11:32 AM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
The reactor doesn't have a queue of tasks to be completed. It has sets of various event sources, which it executes in no particular order.
"queue of tasks" was a guess on my part, but I looked through the BaseReactor code and found something like that. 'course, I could easily be reading the code wrong. How would you describe threadCallQueue? When I said "tasks", I meant something akin to "calls". That could be confusing---did you interpret "tasks" differently?
Scheduling a timed event with callLater(0,...) might do what you want, though.
Yes. Thanks! I see that the callFromThread documentation even recommends using callLater for this behavior. (doh!) http://twistedmatrix.com/documents/10.2.0/api/twisted.internet.interfaces.IR...
Interesting that you can substitute callFromThread for callLater(0, ...) in the above code and get the same behavior... Sorry to prolong the tangent, but I'd like to better understand the differences between callWhenRunning, callFromThread and callLater. I think reactor.wakeUp() is the one missing piece for me. How exactly does it work? Reading the BaseReactor code... callFromThread adds the call to a queue, then calls wakeUp. IIUC, wakeUp simply runs self.port.postEvent(0, KEY_WAKEUP, None) Does this cause the reactor thread to jump to the front of the queue of threads to be run by python/linux? If there were no wakeUp in callFromThread, would the call simply be delayed until the thread round-robin got to the reactor thread? Or, without wakeUp, is it possible the callFromThread call would never be made? Cheers, Jason -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
On 06:45 pm, jrennie@gmail.com wrote:
Some reactors have a different implementation of wakeUp than the above, but they all have the same goal. The reactor thread may be doing something which is going to block indefinitely (WaitForMultipleObjects, select(), epoll_wait(), etc). The purpose of the wakeUp call is to cause that blocking call to end. Once it ends, calls in the threadCallQueue can be processed. The reactor is written such that if you are running code in the same thread it is running in (ie, you are "in the reactor thread") then any event source you create (be it network or time or whatever), the reactor will be sure to service it in a timely manner. For network event sources, this means it will include a descriptor in the select() (etc) call. For timing event sources (ie callLater), it means the reactor will set a timeout on the select() (etc) call so that it returns before it is time for that delayed call to execute. However, if you are not in the reactor thread, then really the only thing you're allowed to do to Twisted is use reactor.callFromThread. Since this might happen at any time with respect to what the reactor thread is doing, there's no way to be sure a (let's call it a) thread call event will get handled in a timely fashion. So this is the problem that wakeUp solves. After the thread call event is added to threadCallQueue, wakeUp makes some event source the reactor is monitoring signal readiness (on POSIX, it writes a byte to a pipe the reactor is select()ing (etc) on; on Windows, it posts an event like you pasted above). This ensures that if the reactor is asleep waiting for an event it will wake up soon, and then it will notice there are thread call events to process. Jean-Paul
Great explanation. Thanks! Jason On Wed, Jan 12, 2011 at 4:40 PM, <exarkun@twistedmatrix.com> wrote:
-- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
No need to apologize Jason :-) Got a lot of useful information in this thread! Thanks! Benjamin Från: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] För Jason Rennie Skickat: den 12 januari 2011 16:52 Till: Twisted general discussion Ämne: Re: [Twisted-Python] client connecting to 2 servers (nonsimultaneously) Benjamin, sorry for providing you with an incorrect explanation. Jean-Paul, thank you for correcting my broken understanding of callWhenRunning. Is there a call which puts a function into the reactor's queue of tasks to be completed? Thanks, Jason On Wed, Jan 12, 2011 at 10:26 AM, <exarkun@twistedmatrix.com> wrote: Although note that if the reactor is already running when you instantiate the factory, then callWhenRunning will immediately call the function you pass to it, as demonstrated by this transcript in which the reactor is already running: >>> def foo(): ... print 'foo running' ... >>> from twisted.internet import reactor >>> reactor.callWhenRunning(foo) foo running >>> Jean-Paul -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
You could probably generalize and simplify by using a collections.deque of hosts/ports and using the rotate() method before each reactor.connectTCP. Also, no need for multiple reactor imports---one at the top of the code is fine. Note that if the connection is "lost" in a non-clean fashion, you may also want to reconnect. IIUC, "fail" only handles the case that no connection is made (protocol is never created). import collections from twisted.internet import reactor class MyClientFactory(object): protocol = MyClientProtocol def __init__(self, hosts): self.hosts = collections.deque(hosts) reactor.callWhenRunning(reactor.connectTCP, self.hosts[0][0], self.hosts[0][1], self) self.hosts.rotate(1) def clientConnectionFailed(self, connector, reason): reactor.callLater(2.0, connectTCP, self.hosts[0][0], self.hosts[0][1], self) self.hosts.rotate(1) factory = MyClientFactory([('host1', port1), ('host2', port2), ...]) reactor.run() Cheers, Jason On Tue, Jan 11, 2011 at 5:17 AM, <benjamin.bertrand@lfv.se> wrote:
-- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
Using a deque for this is a fantastic idea, though I would have used rotate(-1), as your example looks like it rotate's "backwards" to me. Matter of taste I suppose. It doesn't make much actual difference. Kevin Horn On Tue, Jan 11, 2011 at 7:11 AM, Jason Rennie <jrennie@gmail.com> wrote:
I agree that rotate(-1) is more intuitive. I barely gave it any thought---almost forgot to include an argument for rotate as I initially incorrectly assumed no argument would do something like rotate(-1) :-) Jason On Tue, Jan 11, 2011 at 1:36 PM, Kevin Horn <kevin.horn@gmail.com> wrote:
-- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
Nice, this looks much cleaner. I thought about the connection "lost" case. It's quite similar to "fail" (but client should always try to reconnect to host1 in that case). So easy to implement. Thanks a lot! Benjamin Från: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] För Jason Rennie Skickat: den 11 januari 2011 14:11 Till: Twisted general discussion Ämne: Re: [Twisted-Python] client connecting to 2 servers (nonsimultaneously) You could probably generalize and simplify by using a collections.deque of hosts/ports and using the rotate() method before each reactor.connectTCP. Also, no need for multiple reactor imports---one at the top of the code is fine. Note that if the connection is "lost" in a non-clean fashion, you may also want to reconnect. IIUC, "fail" only handles the case that no connection is made (protocol is never created). import collections from twisted.internet import reactor class MyClientFactory(object): protocol = MyClientProtocol def __init__(self, hosts): self.hosts = collections.deque(hosts) reactor.callWhenRunning(reactor.connectTCP, self.hosts[0][0], self.hosts[0][1], self) self.hosts.rotate(1) def clientConnectionFailed(self, connector, reason): reactor.callLater(2.0, connectTCP, self.hosts[0][0], self.hosts[0][1], self) self.hosts.rotate(1) factory = MyClientFactory([('host1', port1), ('host2', port2), ...]) reactor.run() Cheers, Jason On Tue, Jan 11, 2011 at 5:17 AM, <benjamin.bertrand@lfv.se> wrote: Hi, I'm new to twisted and I have started to write a new protocol with a TCP client and server. In my protocol, a client should be able to connect to 2 servers (master/slave node - only the master accepts connection). The client should try to connect to server1. If it fails, try to connect to server2 (after a specific timeout). If that fails, try server1... I came up with a solution (see below). As I'm new to twisted and I haven't seen anything like that in the examples, I'd like to check if that's a proper way to do it. Any comments is welcome. Thanks Benjamin *********************************************** class MyClientFactory(ClientFactory): protocol = MyClientProtocol def __init__(self, host2=None): self.host1 = None self.host2 = host2 def clientConnectionFailed(self, connector, reason): from twisted.internet import reactor if self.host2 is None: # host2 is not defined, reconnect to host1 reactor.callLater(2.0, connector.connect) else: destination = connector.getDestination() if self.host1 is None: # First connection failed, initialize host1, and try host2 self.host1 = destination.host host = self.host2 elif destination.host == self.host1: # Connection to host1 failed, try host2 host = self.host2 else: # Connection to host2 failed, try host1 host = self.host1 reactor.callLater(2.0, reactor.connectTCP, host, destination.port, self) factory = MyClientFactory(server2) from twisted.internet import reactor reactor.connectTCP(server1, 8010, factory) reactor.run() *********************************************** _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
One small question about the following code: Why did you use reactor.callWhenRunning in the __init__ method? Why not calling directly reactor.connectTCP? Cheers, Benjamin Från: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] För Jason Rennie Skickat: den 11 januari 2011 14:11 Till: Twisted general discussion Ämne: Re: [Twisted-Python] client connecting to 2 servers (nonsimultaneously) You could probably generalize and simplify by using a collections.deque of hosts/ports and using the rotate() method before each reactor.connectTCP. Also, no need for multiple reactor imports---one at the top of the code is fine. Note that if the connection is "lost" in a non-clean fashion, you may also want to reconnect. IIUC, "fail" only handles the case that no connection is made (protocol is never created). import collections from twisted.internet import reactor class MyClientFactory(object): protocol = MyClientProtocol def __init__(self, hosts): self.hosts = collections.deque(hosts) reactor.callWhenRunning(reactor.connectTCP, self.hosts[0][0], self.hosts[0][1], self) self.hosts.rotate(1) def clientConnectionFailed(self, connector, reason): reactor.callLater(2.0, connectTCP, self.hosts[0][0], self.hosts[0][1], self) self.hosts.rotate(1) factory = MyClientFactory([('host1', port1), ('host2', port2), ...]) reactor.run() Cheers, Jason On Tue, Jan 11, 2011 at 5:17 AM, <benjamin.bertrand@lfv.se> wrote: Hi, I'm new to twisted and I have started to write a new protocol with a TCP client and server. In my protocol, a client should be able to connect to 2 servers (master/slave node - only the master accepts connection). The client should try to connect to server1. If it fails, try to connect to server2 (after a specific timeout). If that fails, try server1... I came up with a solution (see below). As I'm new to twisted and I haven't seen anything like that in the examples, I'd like to check if that's a proper way to do it. Any comments is welcome. Thanks Benjamin *********************************************** class MyClientFactory(ClientFactory): protocol = MyClientProtocol def __init__(self, host2=None): self.host1 = None self.host2 = host2 def clientConnectionFailed(self, connector, reason): from twisted.internet import reactor if self.host2 is None: # host2 is not defined, reconnect to host1 reactor.callLater(2.0, connector.connect) else: destination = connector.getDestination() if self.host1 is None: # First connection failed, initialize host1, and try host2 self.host1 = destination.host host = self.host2 elif destination.host == self.host1: # Connection to host1 failed, try host2 host = self.host2 else: # Connection to host2 failed, try host1 host = self.host1 reactor.callLater(2.0, reactor.connectTCP, host, destination.port, self) factory = MyClientFactory(server2) from twisted.internet import reactor reactor.connectTCP(server1, 8010, factory) reactor.run() *********************************************** _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
Habit, mostly. I think it's a good habit, though. The reason is that I wanted to make sure the __init__ code completed before the connectTCP was called. In this case, it doesn't matter, but if I had a number of things I wanted to do in __init__, it might matter. Consider: class MyClientProtocol(Protocol): def connectionMade(self): self.factory.numConnections += 1 class MyClientFactory(Factory): def __init__(self): reactor.connectTCP(host, port, self) self.numConnections = 0 'course, I'm sure you'd put reactor.connectTCP *after* the numConnections initializer, but using callWhenRunning ensures that the order doesn't matter, so it's one less thing you have to worry about :-) Jason On Wed, Jan 12, 2011 at 4:49 AM, <benjamin.bertrand@lfv.se> wrote:
-- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
On 03:15 pm, jrennie@gmail.com wrote:
Although note that if the reactor is already running when you instantiate the factory, then callWhenRunning will immediately call the function you pass to it, as demonstrated by this transcript in which the reactor is already running: >>> def foo(): ... print 'foo running' ... >>> from twisted.internet import reactor >>> reactor.callWhenRunning(foo) foo running >>> Jean-Paul
Benjamin, sorry for providing you with an incorrect explanation. Jean-Paul, thank you for correcting my broken understanding of callWhenRunning. Is there a call which puts a function into the reactor's queue of tasks to be completed? Thanks, Jason On Wed, Jan 12, 2011 at 10:26 AM, <exarkun@twistedmatrix.com> wrote:
-- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
On Wed, Jan 12, 2011 at 10:52 AM, Jason Rennie <jrennie@gmail.com> wrote:
The reactor doesn't have a queue of tasks to be completed. It has sets of various event sources, which it executes in no particular order. Scheduling a timed event with callLater(0,...) might do what you want, though.
On Wed, Jan 12, 2011 at 11:32 AM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
The reactor doesn't have a queue of tasks to be completed. It has sets of various event sources, which it executes in no particular order.
"queue of tasks" was a guess on my part, but I looked through the BaseReactor code and found something like that. 'course, I could easily be reading the code wrong. How would you describe threadCallQueue? When I said "tasks", I meant something akin to "calls". That could be confusing---did you interpret "tasks" differently?
Scheduling a timed event with callLater(0,...) might do what you want, though.
Yes. Thanks! I see that the callFromThread documentation even recommends using callLater for this behavior. (doh!) http://twistedmatrix.com/documents/10.2.0/api/twisted.internet.interfaces.IR...
Interesting that you can substitute callFromThread for callLater(0, ...) in the above code and get the same behavior... Sorry to prolong the tangent, but I'd like to better understand the differences between callWhenRunning, callFromThread and callLater. I think reactor.wakeUp() is the one missing piece for me. How exactly does it work? Reading the BaseReactor code... callFromThread adds the call to a queue, then calls wakeUp. IIUC, wakeUp simply runs self.port.postEvent(0, KEY_WAKEUP, None) Does this cause the reactor thread to jump to the front of the queue of threads to be run by python/linux? If there were no wakeUp in callFromThread, would the call simply be delayed until the thread round-robin got to the reactor thread? Or, without wakeUp, is it possible the callFromThread call would never be made? Cheers, Jason -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
On 06:45 pm, jrennie@gmail.com wrote:
Some reactors have a different implementation of wakeUp than the above, but they all have the same goal. The reactor thread may be doing something which is going to block indefinitely (WaitForMultipleObjects, select(), epoll_wait(), etc). The purpose of the wakeUp call is to cause that blocking call to end. Once it ends, calls in the threadCallQueue can be processed. The reactor is written such that if you are running code in the same thread it is running in (ie, you are "in the reactor thread") then any event source you create (be it network or time or whatever), the reactor will be sure to service it in a timely manner. For network event sources, this means it will include a descriptor in the select() (etc) call. For timing event sources (ie callLater), it means the reactor will set a timeout on the select() (etc) call so that it returns before it is time for that delayed call to execute. However, if you are not in the reactor thread, then really the only thing you're allowed to do to Twisted is use reactor.callFromThread. Since this might happen at any time with respect to what the reactor thread is doing, there's no way to be sure a (let's call it a) thread call event will get handled in a timely fashion. So this is the problem that wakeUp solves. After the thread call event is added to threadCallQueue, wakeUp makes some event source the reactor is monitoring signal readiness (on POSIX, it writes a byte to a pipe the reactor is select()ing (etc) on; on Windows, it posts an event like you pasted above). This ensures that if the reactor is asleep waiting for an event it will wake up soon, and then it will notice there are thread call events to process. Jean-Paul
Great explanation. Thanks! Jason On Wed, Jan 12, 2011 at 4:40 PM, <exarkun@twistedmatrix.com> wrote:
-- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
No need to apologize Jason :-) Got a lot of useful information in this thread! Thanks! Benjamin Från: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] För Jason Rennie Skickat: den 12 januari 2011 16:52 Till: Twisted general discussion Ämne: Re: [Twisted-Python] client connecting to 2 servers (nonsimultaneously) Benjamin, sorry for providing you with an incorrect explanation. Jean-Paul, thank you for correcting my broken understanding of callWhenRunning. Is there a call which puts a function into the reactor's queue of tasks to be completed? Thanks, Jason On Wed, Jan 12, 2011 at 10:26 AM, <exarkun@twistedmatrix.com> wrote: Although note that if the reactor is already running when you instantiate the factory, then callWhenRunning will immediately call the function you pass to it, as demonstrated by this transcript in which the reactor is already running: >>> def foo(): ... print 'foo running' ... >>> from twisted.internet import reactor >>> reactor.callWhenRunning(foo) foo running >>> Jean-Paul -- Jason Rennie Research Scientist, ITA Software 617-714-2645 http://www.itasoftware.com/
participants (6)
-
benjamin.bertrand＠lfv.se -
exarkun＠twistedmatrix.com -
Glyph Lefkowitz -
Itamar Turner-Trauring -
Jason Rennie -
Kevin Horn