[Twisted-Python] How can a tcp client connect with multi servers?
![](https://secure.gravatar.com/avatar/32c19d529b3b9e4f3ab0d8a56703c0f5.jpg?s=120&d=mm&r=g)
Here is the situation: a tcp control client need connect to multi-agent. Is there a light way to connect with server , instead make a factory to do each connect?
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Wed, Nov 30, 2005 at 03:19:45PM +0800, Xu Ryan wrote:
ClientCreator is a simpler way to make connections. See the "Simple, single-use clients" section of the Writing Clients developer guide: http://twistedmatrix.com/projects/core/documentation/howto/clients.html#auto... -Andrew.
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Thu, Dec 01, 2005 at 10:35:56AM +0800, Xu Ryan wrote:
ClientCreator is simply an easy way to establish a single connection to a server. It has nothing to do with what happens to the connection after that; that's what the Protocol does. I can't really give you any more specific help without more information about your problem. -Andrew.
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
[please don't top-post] On Thu, Dec 01, 2005 at 04:22:03PM +0800, Xu Ryan wrote:
Does Client Creator load protocol's "connectionMade" automatically , same as factory?
ClientCreator is just a shortcut for creating the factory and so forth manually. The event handlers (connectionMade, dataReceived, connectionLost) on the resulting protocol will be invoked as normal. So yes, connectionMade will be called (assuming a connection is made, if the connection fails the Deferred returned from ClientCreator.connectTCP will have an error instead), but this isn't really anything to do with ClientCreator: once the connection is established, ClientCreator is not involved. Experimenting with the example at http://twistedmatrix.com/projects/core/documentation/howto/clients.html#auto... might be a good idea. The source code for ClientCreator is quite simple. Perhaps it would be worth reading and understanding that, so you can see how little code is involved. The definition of ClientCreator is found in twisted/internet/protocol.py: http://svn.twistedmatrix.com/cvs/trunk/twisted/internet/protocol.py?view=auto&rev=14194 -Andrew.
![](https://secure.gravatar.com/avatar/32c19d529b3b9e4f3ab0d8a56703c0f5.jpg?s=120&d=mm&r=g)
I meet another problem. After i create some ClientCreate instance and let the reactor run , it works ok. But how could I send message when reactor is running? Because there are not only one services that twisted provide. And when I need send the messages i have told above , perhaps there are other service running , using the same reactor(becaus it's global), and obviously I can't stop it , re-create ClientCreator, then call reactor.run() to send the message. On 12/1/05, Andrew Bennetts <andrew-twisted@puzzling.org> wrote:
-- Xu Ryans
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Thu, Dec 01, 2005 at 05:48:12PM +0800, Xu Ryan wrote:
You seem to misunderstand how to do concurrent operations in Twisted. You don't need to stop the reactor. If at some point in your code you want to make another connection with ClientCreator, just call ClientCreator's connectTCP method as usual -- without stopping the reactor. The point of the reactor is that it takes care of managing multiple connections and the like at once. So, it's not "stop [the reactor] , re-create ClientCreator, then call reactor.run() to send the message", it's just "re-create ClientCreator". It may even be just "call connectTCP on your existing ClientCreator again." -Andrew.
![](https://secure.gravatar.com/avatar/32c19d529b3b9e4f3ab0d8a56703c0f5.jpg?s=120&d=mm&r=g)
Would you like show me an simple example about this? I wrote one, but it dosn't work. __sender = None class p(Protocol): def connectionMade(self): print "Connecttion make" def sendMessage(self): self.transport.write("some message") def connectionLost(self, reason): print "Lost, reason", reason def sendCmd(addr, port, cmd, task, options = ""): """Send Cmd to """ global __sender __sender.connectTCP(addr, port).addCallback(send,\ cmd, task, options) def send(p,cmd, task, options): print "send" p.sendMessage() return p def finishConnection(p): print "lostConnection" p.transport.loseConnection() def initSendCommand(): global __sender if not __sender: __sender = ClientCreator(reactor, p) if __name__ == "__main__": def testSendMessage(): time.sleep(2) sendCmd("localhost", 8009, "test send") initSendCommand() thread.start_new_thread(testSendMessage, ()) reactor.run() On 12/1/05, Andrew Bennetts <andrew-twisted@puzzling.org> wrote:
-- Xu Ryans
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Fri, Dec 02, 2005 at 11:42:49AM +0800, Xu Ryan wrote:
Would you like show me an simple example about this? I wrote one, but it dosn't work.
Sure. Thanks for taking the time to write a simple demonstration of your problem, it makes it much easier to understand and discuss!
This all looks ok (except for the unused 'finishConnection' function).
This is the problem. There's two issues here. The first is that in general, a thread CANNOT call any Twisted functions aside from reactor.callFromThread. Twisted is not thread safe. See http://twistedmatrix.com/projects/core/documentation/howto/threading.html The second, and more fundamental, is that threads are totally unnecessary for this. You can write that code block as: if __name__ == "__main__": initSendCommand() reactor.callLater(2, sendCmd, "localhost", 8009, "test send") reactor.run() -Andrew.
![](https://secure.gravatar.com/avatar/32c19d529b3b9e4f3ab0d8a56703c0f5.jpg?s=120&d=mm&r=g)
I use the thread to make the sendCmd be called after the reactor run. Because in the real application, I should start the server(reactor) first, and other logic will call the sendCmd, so I can't fix it before reactor.run() (as callLater(foo)). What should I do ? On 12/2/05, Andrew Bennetts <andrew-twisted@puzzling.org> wrote:
-- Xu Ryans
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Fri, Dec 02, 2005 at 01:44:06PM +0800, Xu Ryan wrote:
It's hard to know what you need here. Can you elaborate more on this "other logic"? Note though that the callLater will also "make the sendCmd be called after the reactor run" -- although a clearer phrasing would be "make the sendCmd be called after the reactor starts", so I don't understand why it is inappropriate for you. Also, saying "the server(reactor)" suggests to me that you misunderstand the role of the reactor. The reactor isn't your server; it's the event loop that runs all the networking Twisted does, servers, clients, whatever. You don't want to stop and start it for each task, or have a seperate one for each task, it's a single object that handles the events that occur (like sockets receiving data) and dispatching them to the right object. -Andrew.
![](https://secure.gravatar.com/avatar/32c19d529b3b9e4f3ab0d8a56703c0f5.jpg?s=120&d=mm&r=g)
I have a master machine ,that will control hundreds remote servers(agent). It will send authenticate and command message to remote agent. The master will run months. And I can't sure which command will be sent, and when it will be sent. So I cann't write it in codes as "reactor.callLater" then "reactor.run()". I need sth like a task pool, I can push a command into it in anytime, and the pool will send out the command immediately. Those commands will be send to different agent. The "other logic" is "pushing commands into the pool". It's unsure before running.
-- Xu Ryans
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Fri, Dec 02, 2005 at 02:47:12PM +0800, Xu Ryan wrote:
As you saying, reactor handles the events. Can i register new event after the event loop running?
Absolutely. You can, for instance, call connectTCP as many times as you like while the event loop is running to establish multiple client connections, which may be concurrent. You can call listenTCP at any time too, if during the course of running the code needs to start listening on another TCP port. And so on. Most of the reactor methods can be called while the reactor is running, with a couple of obvious exceptions (e.g. you can't call reactor.run). Many of the other actions you can take in a Twisted program, like writing to or disconnecting a transport, are also at some level instructing the reactor to do something, and again this happens inside the event loop. -Andrew.
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Fri, Dec 02, 2005 at 03:25:48PM +0800, Xu Ryan wrote:
But when reactor.run() is called, the program lock here. and the codes after "reactor.run()" are not been called until reactor is stop.
Right. You generally don't want to have any code after reactor.run(). The idea is to react to events that occur, like receiving messages off the network. For example, if you want to make a new connection while the reactor is running, you do that from within an event handler. There's some examples of this in Twisted, e.g. look at twisted/protocols/portforward.py (http://svn.twistedmatrix.com/cvs/trunk/twisted/protocols/portforward.py?view=auto&rev=12914). An example of using this module would be: from twisted.internet import reactor from twisted.protocols.portforward import ProxyFactory reactor.listenTCP(1234, ProxyFactory('somewhere.com', 5678)) reactor.run() If you run that it will start listening on port 1234, and do nothing else until you connect to that port. As soon as a connection is established, a ProxyServer protocol is made, and its connectionMade handler will call reactor.connectTCP to establish a connection to somewhere.com's port 5678. Here's another example of making new connections in response to events that occur while the program is running. You may find it simpler to understand, although I haven't tested it so there may be bugs, and it has essentially no error handling...: ---- from twisted.protocols import basic from twisted.internet import reactor, protocol class MessageReceiver(basic.LineReceiver): def lineReceived(self, line): # expects lines like "host.somewhere.com 1234 here's a message". host, port, message = line.split(' ', 2) sendMessage(host, port, message) class MessageSender(protocol.Protocol): def __init__(self, message): self.message = message def connectionMade(self): # send the message self.transport.write(message) # close the connection self.transport.loseConnection() def sendMessage(host, port, message): cc = protocol.ClientCreator(reactor, MessageSender, message) cc.connectTCP(host, port) f = protocol.ServerFactory() f.protocol = MessageReceiver reactor.listenTCP(1234, f) reactor.run() ---- You may also want to work through the finger tutorial -- it's a bit long, but it does cover this sort of thing. -Andrew.
![](https://secure.gravatar.com/avatar/32c19d529b3b9e4f3ab0d8a56703c0f5.jpg?s=120&d=mm&r=g)
Thank you for you patient:) The codes below seems have other problems: 1. one port bind one Proxy. And the proxy is also fixed before reactor run. 2. Should use other method make connect with localhost. Perhaps it make my problem complicated. I think i should make a schedule and check the command list, if it's not empty, send the commands in the list. twisted has some module like that, i rememberd.
-- Xu Ryans
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Wed, Nov 30, 2005 at 03:19:45PM +0800, Xu Ryan wrote:
ClientCreator is a simpler way to make connections. See the "Simple, single-use clients" section of the Writing Clients developer guide: http://twistedmatrix.com/projects/core/documentation/howto/clients.html#auto... -Andrew.
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Thu, Dec 01, 2005 at 10:35:56AM +0800, Xu Ryan wrote:
ClientCreator is simply an easy way to establish a single connection to a server. It has nothing to do with what happens to the connection after that; that's what the Protocol does. I can't really give you any more specific help without more information about your problem. -Andrew.
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
[please don't top-post] On Thu, Dec 01, 2005 at 04:22:03PM +0800, Xu Ryan wrote:
Does Client Creator load protocol's "connectionMade" automatically , same as factory?
ClientCreator is just a shortcut for creating the factory and so forth manually. The event handlers (connectionMade, dataReceived, connectionLost) on the resulting protocol will be invoked as normal. So yes, connectionMade will be called (assuming a connection is made, if the connection fails the Deferred returned from ClientCreator.connectTCP will have an error instead), but this isn't really anything to do with ClientCreator: once the connection is established, ClientCreator is not involved. Experimenting with the example at http://twistedmatrix.com/projects/core/documentation/howto/clients.html#auto... might be a good idea. The source code for ClientCreator is quite simple. Perhaps it would be worth reading and understanding that, so you can see how little code is involved. The definition of ClientCreator is found in twisted/internet/protocol.py: http://svn.twistedmatrix.com/cvs/trunk/twisted/internet/protocol.py?view=auto&rev=14194 -Andrew.
![](https://secure.gravatar.com/avatar/32c19d529b3b9e4f3ab0d8a56703c0f5.jpg?s=120&d=mm&r=g)
I meet another problem. After i create some ClientCreate instance and let the reactor run , it works ok. But how could I send message when reactor is running? Because there are not only one services that twisted provide. And when I need send the messages i have told above , perhaps there are other service running , using the same reactor(becaus it's global), and obviously I can't stop it , re-create ClientCreator, then call reactor.run() to send the message. On 12/1/05, Andrew Bennetts <andrew-twisted@puzzling.org> wrote:
-- Xu Ryans
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Thu, Dec 01, 2005 at 05:48:12PM +0800, Xu Ryan wrote:
You seem to misunderstand how to do concurrent operations in Twisted. You don't need to stop the reactor. If at some point in your code you want to make another connection with ClientCreator, just call ClientCreator's connectTCP method as usual -- without stopping the reactor. The point of the reactor is that it takes care of managing multiple connections and the like at once. So, it's not "stop [the reactor] , re-create ClientCreator, then call reactor.run() to send the message", it's just "re-create ClientCreator". It may even be just "call connectTCP on your existing ClientCreator again." -Andrew.
![](https://secure.gravatar.com/avatar/32c19d529b3b9e4f3ab0d8a56703c0f5.jpg?s=120&d=mm&r=g)
Would you like show me an simple example about this? I wrote one, but it dosn't work. __sender = None class p(Protocol): def connectionMade(self): print "Connecttion make" def sendMessage(self): self.transport.write("some message") def connectionLost(self, reason): print "Lost, reason", reason def sendCmd(addr, port, cmd, task, options = ""): """Send Cmd to """ global __sender __sender.connectTCP(addr, port).addCallback(send,\ cmd, task, options) def send(p,cmd, task, options): print "send" p.sendMessage() return p def finishConnection(p): print "lostConnection" p.transport.loseConnection() def initSendCommand(): global __sender if not __sender: __sender = ClientCreator(reactor, p) if __name__ == "__main__": def testSendMessage(): time.sleep(2) sendCmd("localhost", 8009, "test send") initSendCommand() thread.start_new_thread(testSendMessage, ()) reactor.run() On 12/1/05, Andrew Bennetts <andrew-twisted@puzzling.org> wrote:
-- Xu Ryans
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Fri, Dec 02, 2005 at 11:42:49AM +0800, Xu Ryan wrote:
Would you like show me an simple example about this? I wrote one, but it dosn't work.
Sure. Thanks for taking the time to write a simple demonstration of your problem, it makes it much easier to understand and discuss!
This all looks ok (except for the unused 'finishConnection' function).
This is the problem. There's two issues here. The first is that in general, a thread CANNOT call any Twisted functions aside from reactor.callFromThread. Twisted is not thread safe. See http://twistedmatrix.com/projects/core/documentation/howto/threading.html The second, and more fundamental, is that threads are totally unnecessary for this. You can write that code block as: if __name__ == "__main__": initSendCommand() reactor.callLater(2, sendCmd, "localhost", 8009, "test send") reactor.run() -Andrew.
![](https://secure.gravatar.com/avatar/32c19d529b3b9e4f3ab0d8a56703c0f5.jpg?s=120&d=mm&r=g)
I use the thread to make the sendCmd be called after the reactor run. Because in the real application, I should start the server(reactor) first, and other logic will call the sendCmd, so I can't fix it before reactor.run() (as callLater(foo)). What should I do ? On 12/2/05, Andrew Bennetts <andrew-twisted@puzzling.org> wrote:
-- Xu Ryans
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Fri, Dec 02, 2005 at 01:44:06PM +0800, Xu Ryan wrote:
It's hard to know what you need here. Can you elaborate more on this "other logic"? Note though that the callLater will also "make the sendCmd be called after the reactor run" -- although a clearer phrasing would be "make the sendCmd be called after the reactor starts", so I don't understand why it is inappropriate for you. Also, saying "the server(reactor)" suggests to me that you misunderstand the role of the reactor. The reactor isn't your server; it's the event loop that runs all the networking Twisted does, servers, clients, whatever. You don't want to stop and start it for each task, or have a seperate one for each task, it's a single object that handles the events that occur (like sockets receiving data) and dispatching them to the right object. -Andrew.
![](https://secure.gravatar.com/avatar/32c19d529b3b9e4f3ab0d8a56703c0f5.jpg?s=120&d=mm&r=g)
I have a master machine ,that will control hundreds remote servers(agent). It will send authenticate and command message to remote agent. The master will run months. And I can't sure which command will be sent, and when it will be sent. So I cann't write it in codes as "reactor.callLater" then "reactor.run()". I need sth like a task pool, I can push a command into it in anytime, and the pool will send out the command immediately. Those commands will be send to different agent. The "other logic" is "pushing commands into the pool". It's unsure before running.
-- Xu Ryans
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Fri, Dec 02, 2005 at 02:47:12PM +0800, Xu Ryan wrote:
As you saying, reactor handles the events. Can i register new event after the event loop running?
Absolutely. You can, for instance, call connectTCP as many times as you like while the event loop is running to establish multiple client connections, which may be concurrent. You can call listenTCP at any time too, if during the course of running the code needs to start listening on another TCP port. And so on. Most of the reactor methods can be called while the reactor is running, with a couple of obvious exceptions (e.g. you can't call reactor.run). Many of the other actions you can take in a Twisted program, like writing to or disconnecting a transport, are also at some level instructing the reactor to do something, and again this happens inside the event loop. -Andrew.
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Fri, Dec 02, 2005 at 03:25:48PM +0800, Xu Ryan wrote:
But when reactor.run() is called, the program lock here. and the codes after "reactor.run()" are not been called until reactor is stop.
Right. You generally don't want to have any code after reactor.run(). The idea is to react to events that occur, like receiving messages off the network. For example, if you want to make a new connection while the reactor is running, you do that from within an event handler. There's some examples of this in Twisted, e.g. look at twisted/protocols/portforward.py (http://svn.twistedmatrix.com/cvs/trunk/twisted/protocols/portforward.py?view=auto&rev=12914). An example of using this module would be: from twisted.internet import reactor from twisted.protocols.portforward import ProxyFactory reactor.listenTCP(1234, ProxyFactory('somewhere.com', 5678)) reactor.run() If you run that it will start listening on port 1234, and do nothing else until you connect to that port. As soon as a connection is established, a ProxyServer protocol is made, and its connectionMade handler will call reactor.connectTCP to establish a connection to somewhere.com's port 5678. Here's another example of making new connections in response to events that occur while the program is running. You may find it simpler to understand, although I haven't tested it so there may be bugs, and it has essentially no error handling...: ---- from twisted.protocols import basic from twisted.internet import reactor, protocol class MessageReceiver(basic.LineReceiver): def lineReceived(self, line): # expects lines like "host.somewhere.com 1234 here's a message". host, port, message = line.split(' ', 2) sendMessage(host, port, message) class MessageSender(protocol.Protocol): def __init__(self, message): self.message = message def connectionMade(self): # send the message self.transport.write(message) # close the connection self.transport.loseConnection() def sendMessage(host, port, message): cc = protocol.ClientCreator(reactor, MessageSender, message) cc.connectTCP(host, port) f = protocol.ServerFactory() f.protocol = MessageReceiver reactor.listenTCP(1234, f) reactor.run() ---- You may also want to work through the finger tutorial -- it's a bit long, but it does cover this sort of thing. -Andrew.
![](https://secure.gravatar.com/avatar/32c19d529b3b9e4f3ab0d8a56703c0f5.jpg?s=120&d=mm&r=g)
Thank you for you patient:) The codes below seems have other problems: 1. one port bind one Proxy. And the proxy is also fixed before reactor run. 2. Should use other method make connect with localhost. Perhaps it make my problem complicated. I think i should make a schedule and check the command list, if it's not empty, send the commands in the list. twisted has some module like that, i rememberd.
-- Xu Ryans
participants (2)
-
Andrew Bennetts
-
Xu Ryan