[Twisted-Python] locking threads when deferToThread is used
Hi, I'm writing an application which will be periodically testing servers. I will have a global list of these servers(domain names) and need to do few tasks. 1) DNS checks - I will use asynchronous twisted-names for it - in case there is a difference comparing to the list it should update the list(then also in DB) 2) ICMP pings - should be also possible to do it asynchronously 3) Blocking function which will be pinging with SIP requests - here I will use function deferToThread to make it non-blocking. Questions: 1) How do I lock each thread when writing to a global list in twisted? 2) How will I put together all three results mentioned above in the global list - is it by using function callLater ? 3) Could you help me with a rough skeleton of this application, please? Thank you, Lada
On Wed, 08 Aug 2007 09:56:16 +0200, Ladislav Andel <ladaan@iptel.org> wrote:
Hi, I'm writing an application which will be periodically testing servers. I will have a global list of these servers(domain names) and need to do few tasks. 1) DNS checks - I will use asynchronous twisted-names for it - in case there is a difference comparing to the list it should update the list(then also in DB) 2) ICMP pings - should be also possible to do it asynchronously 3) Blocking function which will be pinging with SIP requests - here I will use function deferToThread to make it non-blocking.
Questions: 1) How do I lock each thread when writing to a global list in twisted?
Don't. Don't share state between different threads. If you need to mutate state as a result of code which has run in a thread, do it in the reactor thread based on the result of the Deferred returned by deferToThread. Also, why do you need threads to send SIP messages?
2) How will I put together all three results mentioned above in the global list - is it by using function callLater ?
Don't use globals. You can just put your state variables on an instance of a class. There's nothing unique to Twisted about this. I'm not sure how callLater would be involved. callLater is for scheduling functions to run based on timing events. Jean-Paul
Jean-Paul Calderone wrote:
On Wed, 08 Aug 2007 09:56:16 +0200, Ladislav Andel <ladaan@iptel.org> wrote:
Hi, I'm writing an application which will be periodically testing servers. I will have a global list of these servers(domain names) and need to do few tasks. 1) DNS checks - I will use asynchronous twisted-names for it - in case there is a difference comparing to the list it should update the list(then also in DB) 2) ICMP pings - should be also possible to do it asynchronously 3) Blocking function which will be pinging with SIP requests - here I will use function deferToThread to make it non-blocking.
Questions: 1) How do I lock each thread when writing to a global list in twisted?
Don't. Don't share state between different threads. If you need to mutate state as a result of code which has run in a thread, do it in the reactor thread based on the result of the Deferred returned by deferToThread.
I'm aiming to use results of the Deffered returned by deferToThread. Can you give me an example of combining two Deferred results? This will be done through callbacks but what if the blocking application gives me result in 5 sec and e.g. DNS check in 1 sec.
Also, why do you need threads to send SIP messages? I use third party application where I'm just handling parameters to the application. I know it's possible to do it somehow easily via twisted but I don't have time to explore it now. I'm also quite new to python and also to twisted so that's why I don't want to get involved in something more complex for now.
2) How will I put together all three results mentioned above in the global list - is it by using function callLater ?
Don't use globals. You can just put your state variables on an instance of a class. There's nothing unique to Twisted about this. I'm not sure how callLater would be involved. callLater is for scheduling functions to run based on timing events.
Can you give me a little example? Lada
On Wed, 08 Aug 2007 13:11:09 +0200, Ladislav Andel <ladaan@iptel.org> wrote:
Jean-Paul Calderone wrote:
On Wed, 08 Aug 2007 09:56:16 +0200, Ladislav Andel <ladaan@iptel.org> wrote:
Hi, I'm writing an application which will be periodically testing servers. I will have a global list of these servers(domain names) and need to do few tasks. 1) DNS checks - I will use asynchronous twisted-names for it - in case there is a difference comparing to the list it should update the list(then also in DB) 2) ICMP pings - should be also possible to do it asynchronously 3) Blocking function which will be pinging with SIP requests - here I will use function deferToThread to make it non-blocking.
Questions: 1) How do I lock each thread when writing to a global list in twisted?
Don't. Don't share state between different threads. If you need to mutate state as a result of code which has run in a thread, do it in the reactor thread based on the result of the Deferred returned by deferToThread. I'm aiming to use results of the Deffered returned by deferToThread. Can you give me an example of combining two Deferred results? This will be done through callbacks but what if the blocking application gives me result in 5 sec and e.g. DNS check in 1 sec.
twisted.internet.defer.gatherResults takes a list of Deferreds and returns a Deferred which fires with a list of results. If you want to wait for both results, you can use this. If you only want the first result, whichever that may be, then you just need to write a little class that acknowledges the first Deferred's callback and disregards callbacks from any subsequent Deferreds.
Also, why do you need threads to send SIP messages? I use third party application where I'm just handling parameters to the application. I know it's possible to do it somehow easily via twisted but I don't have time to explore it now. I'm also quite new to python and also to twisted so that's why I don't want to get involved in something more complex for now.
Alright.
2) How will I put together all three results mentioned above in the global list - is it by using function callLater ?
Don't use globals. You can just put your state variables on an instance of a class. There's nothing unique to Twisted about this. I'm not sure how callLater would be involved. callLater is for scheduling functions to run based on timing events.
Can you give me a little example?
I'm not really sure what you have in mind, but maybe an approach along these lines would be useful: class PingTracker: def __init__(self): self.dns = None self.sip = None self.ping = None def check(self, host): checks = [] checks.append(checkDNS().addCallback(self.setDNSResult)) checks.append(checkSIP().addCallback(self.setSIPResult)) checks.append(checkPing().addCallback(self.setPingResult)) return gatherResults(checks) def setDNSResult(self, result): self.dns = result ... Jean-Paul
I've posted this a month or two ago, but thought I'd try again in case someone missed it. Does anyone have a Persistent SSH Class for Twisted that they'd be willing to share? I need it for a GUI application that we really like to build. I've looked at plenty of examples of connect/execute/disconnect scripts, but converting those to a persistent connection is a bit too daunting for my programming abilities. Any help is much appreciated. Paul.
On 8/8/07, Ladislav Andel <ladaan@iptel.org> wrote:
Don't. Don't share state between different threads. If you need to mutate state as a result of code which has run in a thread, do it in the reactor thread based on the result of the Deferred returned by deferToThread.
I'm aiming to use results of the Deffered returned by deferToThread. Can you give me an example of combining two Deferred results? This will be done through callbacks but what if the blocking application gives me result in 5 sec and e.g. DNS check in 1 sec.
check gatherResults from twisted.internet.defer. basicaly it's: d3 = defer.gatherResults(d1, d2) where d1 is call to blocking app using deferToThread and d2 is DNS check and d3 gets called back with results of d1 and d2 (if they both succeeded) cu, MS
Thank you for your help. Well, I'm trying to get into the problem step by step so please receive my apologies about not so professional approach to the problem. I have following code below with a few problems. 1) I need to call getServerResponse of SIP servers for given list domains periodically. I have tried to use callLater method but probably at wrong places. This way it runs only once. In the future it should be at least every 5 minutes. 2) In my code is something wrong. I get the Server response but also following error. Traceback (most recent call last): File "/home/data/iptel_summer_2007/sipcraft1/src/sipcraft/multideferred.py", line 46, in ? reactor.run() File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 220, in run self.mainLoop() File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 228, in mainLoop self.runUntilCurrent() --- <exception caught here> --- File "/usr/lib/python2.4/site-packages/twisted/internet/base.py", line 533, in runUntilCurrent f(*a, **kw) File "/usr/lib/python2.4/site-packages/twisted/internet/defer.py", line 239, in callback self._startRunCallbacks(result) File "/usr/lib/python2.4/site-packages/twisted/internet/defer.py", line 290, in _startRunCallbacks raise AlreadyCalledError twisted.internet.defer.AlreadyCalledError: ----------------------------- code --------------------------------- from twisted.internet import reactor, defer, threads from siptest import siptest_f from twisted.internet.threads import deferToThread test_opts = {'username':'test','transport':'udp','localport':'5085','password':'test'} domain = ['sip.1und1.de', 'sip.babble.net'] class Getter: def gotResults(self, domain, test_opts): if len(domain) != 0: self.d.callback(siptest_f(domain, test_opts)) else: self.d.errback(ValueError("You used an odd number!")) def _print(self, r, domain): return domain, r def getServerResponse(self, domain, test_opts): self.d = threads.deferToThread(self.gotResults, domain, test_opts) self.d.addCallback(self._print, domain) return self.d def printData(d): print d def printError(failure): import sys sys.stderr.write(str(failure)) def test(): for dom in domain: g = Getter() d = g.getServerResponse(dom, test_opts) d.addCallback(printData) d.addErrback(printError) if __name__ == '__main__': test() reactor.run()
class Getter: def gotResults(self, domain, test_opts): if len(domain) != 0: self.d.callback(siptest_f(domain, test_opts)) else: self.d.errback(ValueError("You used an odd number!"))
gotResults is inside a thread, and you're touching the deferred. Don't. Just do this: class MyWeirdTester: def gotResults(...): if domain: return siptest_f(domain, test_opts) raise ValueError("you used an odd number") def getServerResponse(...): d = deferToThread(self.gotResults, ...) d.addCallback(self._print, ...) return d
participants (5)
-
Jean-Paul Calderone -
Ladislav Andel -
Marian Schubert -
Paul_S_Johnson＠mnb.uscourts.gov -
Phil Mayers