[Twisted-Python] writing back to the connection is blocking my code
Hi, I have a server that tries to write back to several clients after it processed some that. I keep a hash with the currently connected clients (protocol instances) and write back to those as soon as the workers finish their job. The thing is I don't know where to call that instance.writeDataToClient() to not block my code. (I have tried in buildProtocol, but of course, it waits that another connection to be made until it writes back). Thanks for All your help.
On 03:03 pm, coder_gus@lavabit.com wrote:
Hi, I have a server that tries to write back to several clients after it processed some that. I keep a hash with the currently connected clients (protocol instances) and write back to those as soon as the workers finish their job. The thing is I don't know where to call that instance.writeDataToClient() to not block my code. (I have tried in buildProtocol, but of course, it waits that another connection to be made until it writes back).
It sounds like you might be having a problem answered by this FAQ question: http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#HowdoImakeTwiste... or perhaps the following one.
Thanks for All your help.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Ok, this link answered a few of my questions (stupid of me that I haven't looked on that before I posted), but my question still stands: how do I send data to a client whenever I want from wherever I want and that data to be send as fast as possible, without the code waiting for the reactor to answer to some events? Sorry for being too pushy but I am not getting the picture right and I really want to get what's going on with this. Thanks. glyph@divmod.com wrote:
On 03:03 pm, coder_gus@lavabit.com wrote:
Hi, I have a server that tries to write back to several clients after it processed some that. I keep a hash with the currently connected clients (protocol instances) and write back to those as soon as the workers finish their job. The thing is I don't know where to call that instance.writeDataToClient() to not block my code. (I have tried in buildProtocol, but of course, it waits that another connection to be made until it writes back).
It sounds like you might be having a problem answered by this FAQ question:
http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#HowdoImakeTwiste...
or perhaps the following one.
Thanks for All your help.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
____________________________________________________________________________________
Click for quotes on adjustable mortgages. http://ads.lavabit.com/fc/REAK6bkMOiFnNX2sGS485h9324PbTgrM9xKyt0ZhxqimEnvQxv...
____________________________________________________________________________________
coder_gus wrote:
Ok, this link answered a few of my questions (stupid of me that I haven't looked on that before I posted), but my question still stands: how do I send data to a client whenever I want from wherever I want and that data to be send as fast as possible,
The code in the faq that glyph pointed you to will do what you want, if I understand your requirements correctly. Heres the link again: http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#HowdoImakeinputo...
without the code waiting for the reactor to answer to some events?
Your transmission *might* have to wait if the operating system transmit buffers are already full, and, if so, the reactor is the mechanism which means your new data can wait without blocking the whole process.
Well, by default the Nagle algorithm is turned off so it should transmit the data asap. I don't think it's OS dependent. Toby Dickenson wrote:
coder_gus wrote:
Ok, this link answered a few of my questions (stupid of me that I haven't looked on that before I posted), but my question still stands: how do I send data to a client whenever I want from wherever I want and that data to be send as fast as possible,
The code in the faq that glyph pointed you to will do what you want, if I understand your requirements correctly. Heres the link again:
http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#HowdoImakeinputo...
without the code waiting for the reactor to answer to some events?
Your transmission *might* have to wait if the operating system transmit buffers are already full, and, if so, the reactor is the mechanism which means your new data can wait without blocking the whole process.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
____________________________________________________________________________________ Save hundreds on an Unsecured Loan - Click here. http://ads.lavabit.com/fc/REAK6bkM9e5ejOJrLUP0P44A0oQrI53GoIDlgN8VVyT8BnEAxl... ____________________________________________________________________________________
coder_gus wrote:
Well, by default the Nagle algorithm is turned off so it should transmit the data asap. I don't think it's OS dependent.
It's only off if you've turned it off. It's on by default AFAIK on most OSes. Anyway, Nagle is irrelevant - if the TCP window is closed i.e. you've sent N bytes and the far end has not acked, and N == window, your TCP stack WILL NOT (MUST NOT) send any more.
Toby Dickenson wrote:
coder_gus wrote:
Ok, this link answered a few of my questions (stupid of me that I haven't looked on that before I posted), but my question still stands: how do I send data to a client whenever I want from wherever I want and that data to be send as fast as possible,
The code in the faq that glyph pointed you to will do what you want, if I understand your requirements correctly. Heres the link again:
http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#HowdoImakeinputo...
without the code waiting for the reactor to answer to some events?
Your transmission *might* have to wait if the operating system transmit buffers are already full, and, if so, the reactor is the mechanism which means your new data can wait without blocking the whole process.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
____________________________________________________________________________________
Save hundreds on an Unsecured Loan - Click here. http://ads.lavabit.com/fc/REAK6bkM9e5ejOJrLUP0P44A0oQrI53GoIDlgN8VVyT8BnEAxl...
____________________________________________________________________________________
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Agreed, I was talking about transport.tcpKeepAlive(). The thing I am asking (and call me stupid, but I am trying to figure it out for days now and no one seems to answer my question or is it me too stupid that can see the answer in front of my eyes) is how do I send data to clients that are connected to the twisted server waiting for data to be processed, without sending to the server any data or causing any event to the server? In the link you guys showed me it's true that the data gets back to the clients, but only when the server receives other data, that is in the dataReceived method. How can I send it back to them (having a reference to each client's protocol class) without receiving data, client connecting etc. ? Phil Mayers wrote:
coder_gus wrote:
Well, by default the Nagle algorithm is turned off so it should transmit the data asap. I don't think it's OS dependent.
It's only off if you've turned it off. It's on by default AFAIK on most OSes.
Anyway, Nagle is irrelevant - if the TCP window is closed i.e. you've sent N bytes and the far end has not acked, and N == window, your TCP stack WILL NOT (MUST NOT) send any more.
Toby Dickenson wrote:
coder_gus wrote:
Ok, this link answered a few of my questions (stupid of me that I haven't looked on that before I posted), but my question still stands: how do I send data to a client whenever I want from wherever I want and that data to be send as fast as possible,
The code in the faq that glyph pointed you to will do what you want, if I understand your requirements correctly. Heres the link again:
http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#HowdoImakeinputo...
without the code waiting for the reactor to answer to some events?
Your transmission *might* have to wait if the operating system transmit buffers are already full, and, if so, the reactor is the mechanism which means your new data can wait without blocking the whole process.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
____________________________________________________________________________________
Save hundreds on an Unsecured Loan - Click here. http://ads.lavabit.com/fc/REAK6bkM9e5ejOJrLUP0P44A0oQrI53GoIDlgN8VVyT8BnEAxl...
____________________________________________________________________________________
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
____________________________________________________________________________________
Handyman Franchises. Click Here. http://ads.lavabit.com/fc/REAK6bkONUs9yhym0lTTSbxUFXmZYqNkZwLDRJhQY0xBalLSyO...
____________________________________________________________________________________
Quoting coder_gus <coder_gus@lavabit.com>:
Agreed, I was talking about transport.tcpKeepAlive(). The thing I am asking (and call me stupid, but I am trying to figure it out for days now and no one seems to answer my question or is it me too stupid that can see the answer in front of my eyes) is how do I send data to clients that are connected to the twisted server waiting for data to be processed, without sending to the server any data or causing any event to the server?
This is where you are confused I think. There is no way to send data to clients if you don't have *any* events on the server. Because how would you now to send data if you don't have something telling you to do so? You should have a time event, or a network event from another client, or a GUI input, or whatever.
In the link you guys showed me it's true that the data gets back to the clients, but only when the server receives other data, that is in the dataReceived method. How can I send it back to them (having a reference to each client's protocol class) without receiving data, client connecting etc. ?
It's not totally clear to understand what you're trying to achieve, so maybe you want to explain more clearly your problem. I suspect you want to use reactor.callLater, to generate a time event. -- Thomas
Hi, thanks for answering and clarifying this - until now you where the only one that told me there no way of sending data if there is no event on the server. So what I am trying to achieve, I have asked a couple of weeks ago too about this matter (actually about the architecure): I want to write an application server that receives data from clients (client app written in delphi), data is in xml format (I want to write it using a bare-bone protocol and not an xml-rpc server). The data the client sends are some requests for some specific data hold in a database. So, the client makes a request, request that is hold in a synchronized queue together with an instance of the client's connection. A coordinator takes the data from that queue and gives it to some xml threads to parse it and the same coordinator take the parsed data from the xml threads and give some workers to do the bussiness work. This workers place the result in a queue from where the coordinator takes care of sending it back to the clients. I will be using sqlalchemy for database communication. I know the architecture is pretty weird from the twisted perspective, but when the dev-manager wants to do it that way, what can you do? So that is the thing I want to achieve and from this started my question; when the workers finish their work, they have the result, place it in a queue and from there I have to send it to the clients asap, without waiting the server for events. Not sure if I made myself clear as I am not a native-English speaker. Thanks for you patience. :) Thomas Hervé wrote:
Quoting coder_gus <coder_gus@lavabit.com>:
Agreed, I was talking about transport.tcpKeepAlive(). The thing I am asking (and call me stupid, but I am trying to figure it out for days now and no one seems to answer my question or is it me too stupid that can see the answer in front of my eyes) is how do I send data to clients that are connected to the twisted server waiting for data to be processed, without sending to the server any data or causing any event to the server?
This is where you are confused I think. There is no way to send data to clients if you don't have *any* events on the server. Because how would you now to send data if you don't have something telling you to do so? You should have a time event, or a network event from another client, or a GUI input, or whatever.
In the link you guys showed me it's true that the data gets back to the clients, but only when the server receives other data, that is in the dataReceived method. How can I send it back to them (having a reference to each client's protocol class) without receiving data, client connecting etc. ?
It's not totally clear to understand what you're trying to achieve, so maybe you want to explain more clearly your problem. I suspect you want to use reactor.callLater, to generate a time event.
Quoting coder_gus <coder_gus@lavabit.com>:
Hi, thanks for answering and clarifying this - until now you where the only one that told me there no way of sending data if there is no event on the server. So what I am trying to achieve, I have asked a couple of weeks ago too about this matter (actually about the architecure): I want to write an application server that receives data from clients (client app written in delphi), data is in xml format (I want to write it using a bare-bone protocol and not an xml-rpc server). The data the client sends are some requests for some specific data hold in a database. So, the client makes a request, request that is hold in a synchronized queue together with an instance of the client's connection. A coordinator takes the data from that queue and gives it to some xml threads to parse it and the same coordinator take the parsed data from the xml threads and give some workers to do the bussiness work. This workers place the result in a queue from where the coordinator takes care of sending it back to the clients. I will be using sqlalchemy for database communication. I know the architecture is pretty weird from the twisted perspective, but when the dev-manager wants to do it that way, what can you do? So that is the thing I want to achieve and from this started my question; when the workers finish their work, they have the result, place it in a queue and from there I have to send it to the clients asap, without waiting the server for events.
This is were you're wrong: pushing the result is an event! So to the event 'I got some result in a thread', you'll fire a connection write. Let me try to write some pseudo code to solve your problem: def blockingTask(content): import sqlalchemy # sqlalchemy stuff query = session.query("bla") .... return result class CommandProtocol(Protocol): def __init__(self): self.buf = '' def dataReceived(self, data): self.buf += data # This is stupid, don't do that while '</end_command>' in self.buf: content, self.buf = self.buf.split('</end_command>', 1) content += '</end_command>' deferToThread(blockingTask, content).addCallback(self.response) def response(self, result): # I got the response event! self.sendLine.write("response %s" % (result,)) See? deferToThread generates an event, which is used the pass the result of the command to a callback. I hope it clarifies some things. -- Thomas
Yup, that made it alot clear and thanks so much for it :) But now another question comes up: what if for a special client I want to inject some data ... let's say that while he is processing some tasks, the backend observes that some data in the database has altered and wants to notify the client that he can't use that data anymore - if I am using deferToThread I don't think sending data back trough that is pretty easy ... or am I wrong? Thanks for all. Thomas Hervé wrote:
Quoting coder_gus <coder_gus@lavabit.com>:
Hi, thanks for answering and clarifying this - until now you where the only one that told me there no way of sending data if there is no event on the server. So what I am trying to achieve, I have asked a couple of weeks ago too about this matter (actually about the architecure): I want to write an application server that receives data from clients (client app written in delphi), data is in xml format (I want to write it using a bare-bone protocol and not an xml-rpc server). The data the client sends are some requests for some specific data hold in a database. So, the client makes a request, request that is hold in a synchronized queue together with an instance of the client's connection. A coordinator takes the data from that queue and gives it to some xml threads to parse it and the same coordinator take the parsed data from the xml threads and give some workers to do the bussiness work. This workers place the result in a queue from where the coordinator takes care of sending it back to the clients. I will be using sqlalchemy for database communication. I know the architecture is pretty weird from the twisted perspective, but when the dev-manager wants to do it that way, what can you do? So that is the thing I want to achieve and from this started my question; when the workers finish their work, they have the result, place it in a queue and from there I have to send it to the clients asap, without waiting the server for events.
This is were you're wrong: pushing the result is an event! So to the event 'I got some result in a thread', you'll fire a connection write. Let me try to write some pseudo code to solve your problem:
def blockingTask(content): import sqlalchemy # sqlalchemy stuff query = session.query("bla") .... return result
class CommandProtocol(Protocol): def __init__(self): self.buf = ''
def dataReceived(self, data): self.buf += data # This is stupid, don't do that while '</end_command>' in self.buf: content, self.buf = self.buf.split('</end_command>', 1) content += '</end_command>' deferToThread(blockingTask, content).addCallback(self.response)
def response(self, result): # I got the response event! self.sendLine.write("response %s" % (result,))
See? deferToThread generates an event, which is used the pass the result of the command to a callback.
I hope it clarifies some things.
Le Friday 21 March 2008 18:11:58 coder_gus, vous avez écrit :
Yup, that made it alot clear and thanks so much for it :)
But now another question comes up: what if for a special client I want to inject some data ... let's say that while he is processing some tasks, the backend observes that some data in the database has altered and wants to notify the client that he can't use that data anymore - if I am using deferToThread I don't think sending data back trough that is pretty easy ... or am I wrong?
Indeed, that's not a problem. You just have to pass something to your task that will allow it to send data back, and remember that Twisted is not threadsafe so that you must use reactor.callFromThread. Let's get back to our example: def blockingTask(content, safeWrite): import sqlalchemy # sqlalchemy stuff query = session.query("bla") data = query.getresult() if data == "unexpected": safeWrite("Not expected!") # But I continue my task .... return result class CommandProtocol(Protocol): def __init__(self): self.buf = '' def safeWrite(self, data): reactor.callFromThread(self.transport.write, data) def dataReceived(self, data): self.buf += data # This is stupid, don't do that while '</end_command>' in self.buf: content, self.buf = self.buf.split('</end_command>', 1) content += '</end_command>' deferToThread(blockingTask, content, self.safeWrite).addCallback(self.response) def response(self, result): # I got the response event! self.sendLine.write("response %s" % (result,)) See? We're passing a method to the task, so that you can send notification whenever you want. In practice, you'll have to check if the connection is still alive, but the main thing to remind is to use reactor.callFromThread. -- Thomas
On Fri, Mar 21, 2008 at 8:50 AM, coder_gus <coder_gus@lavabit.com> wrote:
How can I send it back to them (having a reference to each client's protocol class) without receiving data, client connecting etc. ?
1. Without receiving data This is simple. When a client connects, your protocol can capture the connection event with connectionMade method on the protocol. That's when you'll likely store a reference to the protocol instance on the factory. If you need to write back to the connection at a later time, simply call write on the protocol's transport. For example, a chat server might do something like this: for c in self.chatters: c.transport.write(msg) I'm guessing that your concern is that write is going to block. It's not. 2. Client connecting The client has to connect first - unless you are writing an application where each participant in the system is known up front - and you can establish connections with them (no firewalls). I would suggest rereading these documents carefully - they should really cover your questions: http://twistedmatrix.com/projects/core/documentation/howto/servers.html http://twistedmatrix.com/projects/core/documentation/howto/clients.html Otherwise, you should provide a more detailed explanation of your use case. -- \\\\\/\"/\\\\\\\\\\\ \\\\/ // //\/\\\\\\\ \\\/ \\// /\ \/\\\\ \\/ /\/ / /\/ /\ \\\ \/ / /\/ /\ /\\\ \\ / /\\\ /\\\ \\\\\/\ \/\\\\\/\\\\\/\\\\\\ d.p.s
participants (6)
-
coder_gus -
Drew Smathers -
glyph＠divmod.com -
Phil Mayers -
Thomas Hervé -
Toby Dickenson