[Twisted-Python] Twistd thread problem
Hi all, I coded an SMTP proxy that receives incoming e-mail requests and pushes the received e-mail onto a Queue were it is processed by a worker thread that finishes by sending the e-mail on to another SMTP server. The server part of the proxy was written in Twisted, but the worker thread uses smtplib to fire off the processed e-mails (I didn't know how to use the twisted smtp client in a thread). When I run the application using twistd -noy <appname>, it works great. When I try to daemonize it (twistd -oy <appname>) the server part works, but the worker thread doesn't. It creates and then I suspect it immediately dies, as no error is returned upon creation. A debug of the queue reveals that the thread is not pulling items off it. A debugging statement placed at the top of the thread never runs, and the twistd log reports nothing wrong. I'm using reactor.callInThread to start the worker function. I tried running twistd --spew -oy <appname> and captured the log it produced. I did a filter for anything with 'thread' in it, this is the result: 2005/05/16 09:15 PDT [-] function ? in /usr/lib/python2.3/site-packages/twisted/python/threadpool.py, line 11 2005/05/16 09:15 PDT [-] function WorkerStop in /usr/lib/python2.3/site-packages/twisted/python/threadpool.py, line 23 2005/05/16 09:15 PDT [-] function init in /usr/lib/python2.3/site-packages/twisted/python/threadable.py, line 219 2005/05/16 09:15 PDT [-] function ThreadPool in /usr/lib/python2.3/site-packages/twisted/python/threadpool.py, line 31 2005/05/16 09:15 PDT [-] function ThreadSafeList in /usr/lib/python2.3/site-packages/twisted/python/threadpool.py, line 199 2005/05/16 09:15 PDT [-] function init in /usr/lib/python2.3/site-packages/twisted/python/threadable.py, line 219 2005/05/16 09:15 PDT [-] method __init__ of twisted.python.threadpool.ThreadPool at -1216569236 2005/05/16 09:15 PDT [-] method start of twisted.python.threadpool.ThreadPool at -1216569236 2005/05/16 09:15 PDT [-] method adjustPoolsize of twisted.python.threadpool.ThreadPool at -1216569236 2005/05/16 09:15 PDT [-] method _startSomeWorkers of twisted.python.threadpool.ThreadPool at -1216569236 2005/05/16 09:15 PDT [-] method callInThread of twisted.python.threadpool.ThreadPool at -1216569236 2005/05/16 09:15 PDT [-] method _startSomeWorkers of twisted.python.threadpool.ThreadPool at -1216569236 2005/05/16 09:15 PDT [-] method startAWorker of twisted.python.threadpool.ThreadPool at -1216569236 2005/05/16 09:15 PDT [-] method __init__ of threading.Thread at -1216568116 2005/05/16 09:15 PDT [-] method __init__ of threading.Thread at -1216568116 2005/05/16 09:15 PDT [-] method _set_daemon of threading.Thread at -1216568116 2005/05/16 09:15 PDT [-] function currentThread in /usr/lib/python2.3/threading.py, line 669 2005/05/16 09:15 PDT [-] method isDaemon of threading._MainThread at -1214935060 2005/05/16 09:15 PDT [-] function Condition in /usr/lib/python2.3/threading.py, line 147 2005/05/16 09:15 PDT [-] method __init__ of threading._Condition at -1216567604 2005/05/16 09:15 PDT [-] method __init__ of threading._Condition at -1216567604 2005/05/16 09:15 PDT [-] method start of threading.Thread at -1216568116 2005/05/16 09:15 PDT [-] method _note of threading.Thread at -1216568116 2005/05/16 09:15 PDT [-] method run of threading.Thread at -1216568116 2005/05/16 09:15 PDT [-] method _worker of twisted.python.threadpool.ThreadPool at -1216569236 2005/05/16 09:15 PDT [-] function currentThread in /usr/lib/python2.3/threading.py, line 669 2005/05/16 09:15 PDT [-] function threadMailer in twisted-copyuser.py, line 201 2005/05/16 09:15 PDT [-] method acquire of twisted.python.threadable._XLock at -1216639284 2005/05/16 09:15 PDT [-] method release of twisted.python.threadable._XLock at -1216639284 2005/05/16 09:15 PDT [-] method acquire of twisted.python.threadable._XLock at -1216639284 2005/05/16 09:15 PDT [-] method release of twisted.python.threadable._XLock at -1216639284 2005/05/16 09:15 PDT [-] method acquire of twisted.python.threadable._XLock at -1216639284 2005/05/16 09:15 PDT [-] method release of twisted.python.threadable._XLock at -1216639284 2005/05/16 09:15 PDT [-] function registerAsIOThread in /usr/lib/python2.3/site-packages/twisted/python/threadable.py, line 262 Why would the worker thread act differently with twistd daemonized? Any suggestions, advice, or hints would be welcome. Thanks, Jan -- Jan Van Uytven Computer Systems & Network Engineer Convergent Media Network wyvern@crm3.com Only in silence the word, only in dark the light, only in dying life: Bright the hawk's flight on the empty sky. - Ursula K. LeGuin, 'A Wizard of Earthsea' *** LEGAL STUFF *** This electronic message transmission, including any attached files, contains information which may be confidential and/or privileged. The information is intended to be for the SOLE use of the individual or entity named above. If you are not the intended recipient, be aware that any review, disclosure, copying, distribution or use of the contents of this information is prohibited. This message is not to be forwarded for any reason without prior consent of the sender. If you have received this electronic transmission in error, please notify the sender immediately by telephone (250) 386-4266 or by a 'reply to sender' only message and destroy all electronic and hard copies of the communication, including attachments, without reading or saving in any manner.
If your making a proxy you shouldn't a queue or worker threads. Just connect to the destination and send your incoming data to it and it's outgoing data back to the client. This can all be done in one thread. Even if you don't do it like this you also don't need a thread to send the data, sending smtp is fairly trivial you can do it by hand, or use the smtp client that comes with twisted (might as well learn sometime). On May 16, 2005, at 7:50 PM, Jan Van Uytven wrote:
All, I am still trying to get a grip of what Twisted is and how to use it. So my opinions might not be completely accurate. But, this is not regarding Twisted anycase. Orbitz, can you please state why do you think it is not a good idea to use a worker thread. If the OP wanted to create a high volume SMTP proxy with each worker thread handling every invocation from the smtp client (smtplib in this case), then don't you think this can scale better than a single thread, wherein the remaining clients are waiting in the wait queue (if something like that exists, am not sure about that) while one smtp data transfer is being done. Also, if the OP chose to use a master-servant model wherein the master thread keeps a tab on each of the worker thread to see the status and (may be) notify the user (via a ui) of the status, the thread model will work ? Please share your thoughts on this. Thank You Gangadhar On 5/18/05, orbitz@ezabel.com <orbitz@ezabel.com> wrote:
On 5/21/05, Gangadhar NPK <npk.gangadhar@gmail.com> wrote:
There's no reason to use a thread because all network I/O is non-blocking in Twisted. There is no "wait queue"; as soon as you write data to the transport, it either gets sent immediately or buffered and execution continues on. There's no performance reason to use a thread and it just complicates things when you can use the non-blocking SMTP code. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash
If your making a proxy you shouldn't a queue or worker threads. Just connect to the destination and send your incoming data to it and it's outgoing data back to the client. This can all be done in one thread. Even if you don't do it like this you also don't need a thread to send the data, sending smtp is fairly trivial you can do it by hand, or use the smtp client that comes with twisted (might as well learn sometime). On May 16, 2005, at 7:50 PM, Jan Van Uytven wrote:
All, I am still trying to get a grip of what Twisted is and how to use it. So my opinions might not be completely accurate. But, this is not regarding Twisted anycase. Orbitz, can you please state why do you think it is not a good idea to use a worker thread. If the OP wanted to create a high volume SMTP proxy with each worker thread handling every invocation from the smtp client (smtplib in this case), then don't you think this can scale better than a single thread, wherein the remaining clients are waiting in the wait queue (if something like that exists, am not sure about that) while one smtp data transfer is being done. Also, if the OP chose to use a master-servant model wherein the master thread keeps a tab on each of the worker thread to see the status and (may be) notify the user (via a ui) of the status, the thread model will work ? Please share your thoughts on this. Thank You Gangadhar On 5/18/05, orbitz@ezabel.com <orbitz@ezabel.com> wrote:
On 5/21/05, Gangadhar NPK <npk.gangadhar@gmail.com> wrote:
There's no reason to use a thread because all network I/O is non-blocking in Twisted. There is no "wait queue"; as soon as you write data to the transport, it either gets sent immediately or buffered and execution continues on. There's no performance reason to use a thread and it just complicates things when you can use the non-blocking SMTP code. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash
participants (5)
-
Christopher Armstrong -
Gangadhar NPK -
Jan Van Uytven -
Jp Calderone -
orbitz＠ezabel.com