[Twisted-Python] Regarding Twisted Matrix
Hi, I have a client server model in twisted, where the server spawns a thread ( basically a test script in python that runs for about 20 mins) I want to track the progress of the thread, and send the progress to the client back So, I write something like this in my server: parent_conn, child_conn = Pipe() thread = Process(target = start_test.main_func, args=(SCRIPT_PATH, TEMP_OUTPUT_PATH, self.output_name, child_conn)) thread.start() response = parent_conn.recv() print response //prints like: initialization done self.transport.write(response) response = parent_conn.recv() print response // configuration done self.transport.write(response) . . . thread.join() But the transport.write calls don't send at the same time. instead they wait for the thread to finish (coz of thread.join) and then append all the response and send it back; like "initialization doneconfiguration done...done" thereby defeating the purpose of creating a thread. How do I give the control to the reactor to write the data back, and still keep the thread running? or is there any other way these kinda of progress can be tracked ? I am kinda stuck with this :(
On Mon, Oct 26, 2009 at 11:44 PM, naman jain <namanvit@gmail.com> wrote:
I'm no expert with threads, but I don't see any deferToThread in there, which is what people on this list tend to mention whenever threading comes up. Perhaps you need to switch to the "twisted way" to create your thread? http://twistedmatrix.com/documents/8.2.0/api/twisted.internet.threads.html ~ Nathan
I tried to implement this: parent_conn, child_conn = Pipe() f = defer.Deferred() f = threads.deferToThread(start_test.main_func, SCRIPT_PATH, TEMP_OUTPUT_PATH, self.output_name, child_conn) response = parent_conn.recv() print response //prints like: initialization done self.transport.write(response) response = parent_conn.recv() print response // configuration done self.transport.write(response) But get the same output ie all status messages concatenated and sent at once.! [Do we need to put some deffered/callback mechanism on parent_conn.recv() ?? as that is the call blocking the thread waiting for reading.] Any ideas ? How is such stuff done in twisted servers?? Naman On Tue, Oct 27, 2009 at 7:03 AM, Nathan <nathan.stocks@gmail.com> wrote:
Hi Naman,
Do you really want to spawn a thread or a process? Since it seems like you're trying to run a python script, you'd want to spawn a process (since it also looks like you want to use a pipe for communication with that process). Twisted has support for spawning processes - see reactor.spawnProcess and ProcessProtocol: http://twistedmatrix.com/projects/core/documentation/howto/process.html#auto... Cheers, Reza -- Reza Lotun mobile: +44 (0)7521 310 763 email: rlotun@gmail.com work: reza@tweetdeck.com twitter: @rlotun
Naman, Another problem you might (possibly) be encountering - is the way your server and/or client protocol is implemented. If you just inherit from "plain" twisted.internet.protocol.Protocol - you are automatically using write () and dataReceived() methods that do not implement buffering (it is left up to developer). To enforce that line (string) oriented communication works correctly - you might want to refactor to use twisted.protocols.basic.LineReceiver instead. Kind regards, Valeriy Pogrebitskiy Email: vpogrebi@iname.com -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of naman jain Sent: Tuesday, October 27, 2009 6:36 AM To: Twisted general discussion Subject: Re: [Twisted-Python] Regarding Twisted Matrix I tried to implement this: parent_conn, child_conn = Pipe() f = defer.Deferred() f = threads.deferToThread(start_test.main_func, SCRIPT_PATH, TEMP_OUTPUT_PATH, self.output_name, child_conn) response = parent_conn.recv() print response //prints like: initialization done self.transport.write(response) response = parent_conn.recv() print response // configuration done self.transport.write(response) But get the same output ie all status messages concatenated and sent at once.! [Do we need to put some deffered/callback mechanism on parent_conn.recv() ?? as that is the call blocking the thread waiting for reading.] Any ideas ? How is such stuff done in twisted servers?? Naman On Tue, Oct 27, 2009 at 7:03 AM, Nathan <nathan.stocks@gmail.com> wrote: On Mon, Oct 26, 2009 at 11:44 PM, naman jain <namanvit@gmail.com> wrote:
I'm no expert with threads, but I don't see any deferToThread in there, which is what people on this list tend to mention whenever threading comes up. Perhaps you need to switch to the "twisted way" to create your thread? http://twistedmatrix.com/documents/8.2.0/api/twisted.internet.threads.html ~ Nathan _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hey guys, Thanks for your responses. Nathan your pointer to the "deferToThread" really helped. I finally did this: ran my external script in a thread using: threads.deferToThread(test.main....) and started a function called status_check in another thread threads.deferToThread(self.check_status) then immidiately returned control to reactor. def check_status: self.testprocess_status = self.parent_conn.recv() now, this call on pipe blocks the thread not the reactor main loop. in the reactor when the value is changed of this self.testprocess_status I ping my client with the new status. "Threads are twisted :)" Thanks Naman On Tue, Oct 27, 2009 at 4:31 PM, Valeriy Pogrebitskiy <vpogrebi@verizon.net>wrote:
Naman, Using threads - is not a good way of testing Twisted code. Instead, you can implement unit test script (using twisted.trial.unittest) - implementing test versions of your Twisted server and/or client class(es). This can be done by extending original class(es) and overwriting appropriate methods (like connectionMade(), connectionLost(), dataReceived(), etc.) - by adding deferreds and executing callbacks when given event occurs. These callbacks would be methods within your unittest script (these deferreds would be instantiated and added to these classes by the test script) - thus allowing you to test client's or server's behavior, or track progress Kind regards, Valeriy Pogrebitskiy Email: vpogrebi@iname.com -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of naman jain Sent: Tuesday, October 27, 2009 1:45 AM To: twisted-python@twistedmatrix.com Subject: [Twisted-Python] Regarding Twisted Matrix Hi, I have a client server model in twisted, where the server spawns a thread ( basically a test script in python that runs for about 20 mins) I want to track the progress of the thread, and send the progress to the client back So, I write something like this in my server: parent_conn, child_conn = Pipe() thread = Process(target = start_test.main_func, args=(SCRIPT_PATH, TEMP_OUTPUT_PATH, self.output_name, child_conn)) thread.start() response = parent_conn.recv() print response //prints like: initialization done self.transport.write(response) response = parent_conn.recv() print response // configuration done self.transport.write(response) . . . thread.join() But the transport.write calls don't send at the same time. instead they wait for the thread to finish (coz of thread.join) and then append all the response and send it back; like "initialization doneconfiguration done...done" thereby defeating the purpose of creating a thread. How do I give the control to the reactor to write the data back, and still keep the thread running? or is there any other way these kinda of progress can be tracked ? I am kinda stuck with this :(
On Mon, Oct 26, 2009 at 11:44 PM, naman jain <namanvit@gmail.com> wrote:
I'm no expert with threads, but I don't see any deferToThread in there, which is what people on this list tend to mention whenever threading comes up. Perhaps you need to switch to the "twisted way" to create your thread? http://twistedmatrix.com/documents/8.2.0/api/twisted.internet.threads.html ~ Nathan
I tried to implement this: parent_conn, child_conn = Pipe() f = defer.Deferred() f = threads.deferToThread(start_test.main_func, SCRIPT_PATH, TEMP_OUTPUT_PATH, self.output_name, child_conn) response = parent_conn.recv() print response //prints like: initialization done self.transport.write(response) response = parent_conn.recv() print response // configuration done self.transport.write(response) But get the same output ie all status messages concatenated and sent at once.! [Do we need to put some deffered/callback mechanism on parent_conn.recv() ?? as that is the call blocking the thread waiting for reading.] Any ideas ? How is such stuff done in twisted servers?? Naman On Tue, Oct 27, 2009 at 7:03 AM, Nathan <nathan.stocks@gmail.com> wrote:
Hi Naman,
Do you really want to spawn a thread or a process? Since it seems like you're trying to run a python script, you'd want to spawn a process (since it also looks like you want to use a pipe for communication with that process). Twisted has support for spawning processes - see reactor.spawnProcess and ProcessProtocol: http://twistedmatrix.com/projects/core/documentation/howto/process.html#auto... Cheers, Reza -- Reza Lotun mobile: +44 (0)7521 310 763 email: rlotun@gmail.com work: reza@tweetdeck.com twitter: @rlotun
Naman, Another problem you might (possibly) be encountering - is the way your server and/or client protocol is implemented. If you just inherit from "plain" twisted.internet.protocol.Protocol - you are automatically using write () and dataReceived() methods that do not implement buffering (it is left up to developer). To enforce that line (string) oriented communication works correctly - you might want to refactor to use twisted.protocols.basic.LineReceiver instead. Kind regards, Valeriy Pogrebitskiy Email: vpogrebi@iname.com -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of naman jain Sent: Tuesday, October 27, 2009 6:36 AM To: Twisted general discussion Subject: Re: [Twisted-Python] Regarding Twisted Matrix I tried to implement this: parent_conn, child_conn = Pipe() f = defer.Deferred() f = threads.deferToThread(start_test.main_func, SCRIPT_PATH, TEMP_OUTPUT_PATH, self.output_name, child_conn) response = parent_conn.recv() print response //prints like: initialization done self.transport.write(response) response = parent_conn.recv() print response // configuration done self.transport.write(response) But get the same output ie all status messages concatenated and sent at once.! [Do we need to put some deffered/callback mechanism on parent_conn.recv() ?? as that is the call blocking the thread waiting for reading.] Any ideas ? How is such stuff done in twisted servers?? Naman On Tue, Oct 27, 2009 at 7:03 AM, Nathan <nathan.stocks@gmail.com> wrote: On Mon, Oct 26, 2009 at 11:44 PM, naman jain <namanvit@gmail.com> wrote:
I'm no expert with threads, but I don't see any deferToThread in there, which is what people on this list tend to mention whenever threading comes up. Perhaps you need to switch to the "twisted way" to create your thread? http://twistedmatrix.com/documents/8.2.0/api/twisted.internet.threads.html ~ Nathan _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hey guys, Thanks for your responses. Nathan your pointer to the "deferToThread" really helped. I finally did this: ran my external script in a thread using: threads.deferToThread(test.main....) and started a function called status_check in another thread threads.deferToThread(self.check_status) then immidiately returned control to reactor. def check_status: self.testprocess_status = self.parent_conn.recv() now, this call on pipe blocks the thread not the reactor main loop. in the reactor when the value is changed of this self.testprocess_status I ping my client with the new status. "Threads are twisted :)" Thanks Naman On Tue, Oct 27, 2009 at 4:31 PM, Valeriy Pogrebitskiy <vpogrebi@verizon.net>wrote:
Naman, Using threads - is not a good way of testing Twisted code. Instead, you can implement unit test script (using twisted.trial.unittest) - implementing test versions of your Twisted server and/or client class(es). This can be done by extending original class(es) and overwriting appropriate methods (like connectionMade(), connectionLost(), dataReceived(), etc.) - by adding deferreds and executing callbacks when given event occurs. These callbacks would be methods within your unittest script (these deferreds would be instantiated and added to these classes by the test script) - thus allowing you to test client's or server's behavior, or track progress Kind regards, Valeriy Pogrebitskiy Email: vpogrebi@iname.com -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of naman jain Sent: Tuesday, October 27, 2009 1:45 AM To: twisted-python@twistedmatrix.com Subject: [Twisted-Python] Regarding Twisted Matrix Hi, I have a client server model in twisted, where the server spawns a thread ( basically a test script in python that runs for about 20 mins) I want to track the progress of the thread, and send the progress to the client back So, I write something like this in my server: parent_conn, child_conn = Pipe() thread = Process(target = start_test.main_func, args=(SCRIPT_PATH, TEMP_OUTPUT_PATH, self.output_name, child_conn)) thread.start() response = parent_conn.recv() print response //prints like: initialization done self.transport.write(response) response = parent_conn.recv() print response // configuration done self.transport.write(response) . . . thread.join() But the transport.write calls don't send at the same time. instead they wait for the thread to finish (coz of thread.join) and then append all the response and send it back; like "initialization doneconfiguration done...done" thereby defeating the purpose of creating a thread. How do I give the control to the reactor to write the data back, and still keep the thread running? or is there any other way these kinda of progress can be tracked ? I am kinda stuck with this :(
participants (4)
-
naman jain -
Nathan -
Reza Lotun -
Valeriy Pogrebitskiy