[Twisted-Python] question about threading
hey folks, i've got a quick question for anyone that can help me out. i'm also in the process of looking this up in the twisted docs/code, but this is some last minute coding for a big demo tomorrow, so i'm hedging my bets by posting here. i've got a relatively simple async twisted server that has a custom protocol. most everything the server needs to do on a receipt of a message is short running, but i have one task that isn't. i need to spawn a thread that executes a couple of binaries (html2ps and ps2pdf) which are generally long-running. after executing those binaries, i need to send a message back to the client that sent the original command to generate the pdf. i tried just creating a basic thread that holds onto a reference to the client object, but it doesn't seem to send the command back to the client until i send another message to the server (any kind of message). i know there's a bunch of reactor methods that deal with threads, but i've never used them before, and i'm not sure which one will fix this issue. here's my thread subclass: class PDFGenerator(threading.Thread): def __init__(self, client): threading.Thread.__init__(self) self.client = client def run(self): pres = event.getActivePresentation() t = datetime.datetime.now() base_name = 'sample-file-name' out_html = file('/tmp/' + base_name + '.html', 'w') # [snip snip snip] # write some html to the file out_html.close() os.system('html2ps /tmp/' + base_name + '.html > /tmp/' + base_name + '.ps') # > /tmp/' + base_name + '_ps.log') os.system('mkdir data/transcripts/' + str(pres.id)) os.system('ps2pdf /tmp/' + base_name + '.ps data/ transcripts/' + str(pres.id) + '/' + base_name + '.pdf > /tmp/' + base_name + '_pdf.log') lock = threading.Lock() lock.acquire() self.client.sendCommand('presentTranscript', ['/ transcripts/' + str(pres.id) + '/' + base_name + '.pdf']) lock.release() the client object holds a reference to the protocol object, and sendCommand basically just executes: print "Sending " + message + " to " + str(self.protocol.source) self.protocol.transport.write(message + "\0") i made sure to set the lock, but i think the problem is in how i'm using threads. the clue i'm seeing is that in the log, i see the following: 2005/10/25 10:40 EDT [SSProtocol,25,127.0.0.1] Got generateTranscript ('') from IPv4Address(TCP, '127.0.0.1', 49179) 2005/10/25 10:40 EDT [-] Sending presentTranscript,/transcripts/ phil_bubblehouse.org_2005-10-25-10-40-13.pdf to IPv4Address(TCP, '127.0.0.1', 49179) you can see that the second log entry doesn't print the protocol object that's generating the message. i know there's something simple i'm not doing; any help would be appreciated. -phil
On Tue, 25 Oct 2005 11:05:21 -0400, Phil Christensen <phil@bubblehouse.org> wrote:
hey folks,
Let me start from the other end of your message:
i need to spawn a thread
danger, danger will robinson. Here is where your trouble started :). Actually you *don't* need to spawn a thread, you need to spawn a process - twisted supports processes - and I can't guarantee that os.system will work properly from within a Twisted application. Handling of SIGCHILD has proven to be a sticky wicket in the past.
i know there's a bunch of reactor methods that deal with threads, but i've never used them before, and i'm not sure which one will fix this issue."
There are really only 2: callInThread and callFromThread. There is a convenience API, twisted.internet.threads.deferToThread, which might be what you wanted, if what you wanted was in fact a thread. However, you want spawnProcess in any event. As I said, os.system may not work at _all_ from within Twisted, depending on your operating system.
here's my thread subclass:
Don't subclass thread. Twisted implements its own threadpool; use callInThread with what used to be your 'run' function.
^ This bit could actually be threaded, if it's slow and blocking. My suggestion: If it's for a demo, just block.
os.system('html2ps /tmp/' + base_name + '.html > /tmp/' + base_name + '.ps') # > /tmp/' + base_name + '_ps.log')
Since reactor.spawnProcess might be a bit tedious for simply running this here, try this: twisted.internet.utils.getProcessOutput('/usr/bin/html2ps', ['html2ps', ...).addCallback(keepGoing)
os.system('mkdir data/transcripts/' + str(pres.id))
^ UGH! Why are you spawning another process here?? os.mkdir, please.
os.system('ps2pdf /tmp/' + base_name + '.ps data/ transcripts/' + str(pres.id) + '/' + base_name + '.pdf > /tmp/' + base_name + '_pdf.log')
^ Another getProcessOutput here. Return the resultant Deferred from within your keepGoing callback...
lock = threading.Lock() lock.acquire()
^ Hooray, now you can forget about this garbage
^ Do this in the final callback of the Deferred that you've just created. I hope this helped.
On Tue, 2005-10-25 at 11:05 -0400, Phil Christensen wrote:
Twisted is *not* thread-safe. You can't call its methods from another thread like that. Also, os.system() will probably not work on Unix in Twisted; instead, you can use Twisted's non-blocking process support to run commands (reactor.spawnProcess, or in your case twisted.internet.utils.getProcessOutput()). Then you won't need to use threads at all. There is a way of doing what you want with threads (see http://twistedmatrix.com/projects/core/documentation/howto/threading.html) but you should not be using threads in this case as you don't need to and it's just a source of bugs if you're not careful.
On Tue, 25 Oct 2005 11:05:21 -0400, Phil Christensen <phil@bubblehouse.org> wrote:
hey folks,
Let me start from the other end of your message:
i need to spawn a thread
danger, danger will robinson. Here is where your trouble started :). Actually you *don't* need to spawn a thread, you need to spawn a process - twisted supports processes - and I can't guarantee that os.system will work properly from within a Twisted application. Handling of SIGCHILD has proven to be a sticky wicket in the past.
i know there's a bunch of reactor methods that deal with threads, but i've never used them before, and i'm not sure which one will fix this issue."
There are really only 2: callInThread and callFromThread. There is a convenience API, twisted.internet.threads.deferToThread, which might be what you wanted, if what you wanted was in fact a thread. However, you want spawnProcess in any event. As I said, os.system may not work at _all_ from within Twisted, depending on your operating system.
here's my thread subclass:
Don't subclass thread. Twisted implements its own threadpool; use callInThread with what used to be your 'run' function.
^ This bit could actually be threaded, if it's slow and blocking. My suggestion: If it's for a demo, just block.
os.system('html2ps /tmp/' + base_name + '.html > /tmp/' + base_name + '.ps') # > /tmp/' + base_name + '_ps.log')
Since reactor.spawnProcess might be a bit tedious for simply running this here, try this: twisted.internet.utils.getProcessOutput('/usr/bin/html2ps', ['html2ps', ...).addCallback(keepGoing)
os.system('mkdir data/transcripts/' + str(pres.id))
^ UGH! Why are you spawning another process here?? os.mkdir, please.
os.system('ps2pdf /tmp/' + base_name + '.ps data/ transcripts/' + str(pres.id) + '/' + base_name + '.pdf > /tmp/' + base_name + '_pdf.log')
^ Another getProcessOutput here. Return the resultant Deferred from within your keepGoing callback...
lock = threading.Lock() lock.acquire()
^ Hooray, now you can forget about this garbage
^ Do this in the final callback of the Deferred that you've just created. I hope this helped.
On Tue, 2005-10-25 at 11:05 -0400, Phil Christensen wrote:
Twisted is *not* thread-safe. You can't call its methods from another thread like that. Also, os.system() will probably not work on Unix in Twisted; instead, you can use Twisted's non-blocking process support to run commands (reactor.spawnProcess, or in your case twisted.internet.utils.getProcessOutput()). Then you won't need to use threads at all. There is a way of doing what you want with threads (see http://twistedmatrix.com/projects/core/documentation/howto/threading.html) but you should not be using threads in this case as you don't need to and it's just a source of bugs if you're not careful.
participants (4)
-
glyph＠divmod.com -
Itamar Shtull-Trauring -
Nicola Larosa -
Phil Christensen