[Twisted-Python] dumb question on qt4reactor
calling reactor.callLater within qt4reactor in an attempt to async non webkit work but not seeing expected behavior. Webkit locks while the callLater function executes. Any insight appreciated. Should I move real work into separate reactor and "broker" calls into qt4reactor/webkit. I have xmlrpc and websocket hooks into browser...but looking for guidance as to most direct method of brokering calls between reactors. Any insight appreciated. Having a blast using QT browser as a back-end, see demo site www.athleets.com. See below for amazon S3 upload worker that I expected to async, but in qt4reactor does not appear to work as planned. from pyscewpt import fixed from pyscewpt.s3 import bucket_util import boto from boto.s3.bucket import Bucket from boto.s3.key import Key from twisted.internet import defer, reactor bucket_queue = defer.DeferredQueue() def upload_worker(data): print 'macth upload_worker', data save_match( data['league'], data['twitter'], data['local_file'], data['meta'], data['profile']) return len(bucket_queue.pending) def match_queue(ign=None): print 'match_queue:', len(bucket_queue.pending), len(bucket_queue.waiting) if ign is None or ign >= 2: print 'requeue:', len(bucket_queue.pending) , len(bucket_queue.waiting) d = bucket_queue.get() d.addCallback(upload_worker) d.addBoth(match_queue) return d else: return None def queue_save_match(queue_dic): bucket_queue.put(queue_dic) print 'queue_save_match', len(bucket_queue.pending), len(bucket_queue.waiting), queue_dic['twitter'] # if len(bucket_queue.pending) >= 5 and len(bucket_queue.waiting) == 0: print 'queue drop worker' reactor.callLater(0,match_queue,None) def save_match(league, twitter, systemfile, meta, profile): print 'save_match:', twitter, profile s3_filename = league + '/' + twitter + '.png' bucket_util.save_s3( boto.connect_s3().get_bucket('~bucket'), s3_filename, None, systemfile, 'image/png', 'public-read', meta + [('yes', False), ('no', False), ('maybe', False), ('profile', profile)] )
On 5 Jul, 08:35 pm, kebin70@gmail.com wrote:
`callLater` does not make a blocking function into a non-blocking one. It just calls a function after a specified delay. If the function blocks, then the reactor will block when it gets around to calling it. If you need to call blocking APIs you may need to call them in a separate thread (Twisted includes a threadpool that can help you with this) or a separate process (Twisted has some APIs for launching and interacting with other processes, too). Of course, the ideal solution is often to find a non-blocking equivalent to the blocking API and use that instead (for example, I know that txaws is a project that exists - perhaps it can do the things to S3 that you want to do). Jean-Paul
Another "Oh right" twisted moment. callInThread instead of callLater? This is a case where I've been laying tracks for 2 years without totally understanding the train that was eventually coming down the tracks. I guess I implicitly understood by wanting to move to a separate reactor as a separate thread, and in the past when I've used callLater it's encapsulated non-blocking code. In this sense is callLater ever utilized as a pseudo asynchronous method? On Sat, Jul 5, 2014 at 5:24 PM, <exarkun@twistedmatrix.com> wrote:
It might be, but I can't think of a reason off hand why it ever _should_ be. I can kind of see how it might work, but if your "async" process ever takes longer than your callLater delay it will break, and probably quite horribly. On Wed, Jul 9, 2014 at 12:11 PM, Kevin Mcintyre <kebin70@gmail.com> wrote:
-- -- Kevin Horn
finalizing - maybe I'm not describing my usage correctly. I've used reactor.callLater(0,blah) as a means of starting a unit of work the by-product of which the caller didn't need. this is really a case of laying tracks for 2 years not fully comprehending the train coming down the tracks behind me! I sorta refused to learn/use inlinecallbacks or generators and I've never used callInThread until now. I inserted that into my S3 upload pipeline and it works, simplifying my code but detaches the caller from reactive failure. The mechanism you've described is different ~ a callLater to an estimated point of work completion, maybe a holding pattern if work isn't completed after the delay, and maybe not a bad means of observation (?)...but yes disastrous if the callLater function is wired poorly. Unless I'm misreading. Anyway I'm sure my code is illogical since I've was left to my own devices to solve problems of my own choosing. The by-product you can see here www.pokertalon.com - a single connection html5 app. On Wed, Jul 9, 2014 at 10:51 AM, Kevin Horn <kevin.horn@gmail.com> wrote:
On 5 Jul, 08:35 pm, kebin70@gmail.com wrote:
`callLater` does not make a blocking function into a non-blocking one. It just calls a function after a specified delay. If the function blocks, then the reactor will block when it gets around to calling it. If you need to call blocking APIs you may need to call them in a separate thread (Twisted includes a threadpool that can help you with this) or a separate process (Twisted has some APIs for launching and interacting with other processes, too). Of course, the ideal solution is often to find a non-blocking equivalent to the blocking API and use that instead (for example, I know that txaws is a project that exists - perhaps it can do the things to S3 that you want to do). Jean-Paul
Another "Oh right" twisted moment. callInThread instead of callLater? This is a case where I've been laying tracks for 2 years without totally understanding the train that was eventually coming down the tracks. I guess I implicitly understood by wanting to move to a separate reactor as a separate thread, and in the past when I've used callLater it's encapsulated non-blocking code. In this sense is callLater ever utilized as a pseudo asynchronous method? On Sat, Jul 5, 2014 at 5:24 PM, <exarkun@twistedmatrix.com> wrote:
It might be, but I can't think of a reason off hand why it ever _should_ be. I can kind of see how it might work, but if your "async" process ever takes longer than your callLater delay it will break, and probably quite horribly. On Wed, Jul 9, 2014 at 12:11 PM, Kevin Mcintyre <kebin70@gmail.com> wrote:
-- -- Kevin Horn
finalizing - maybe I'm not describing my usage correctly. I've used reactor.callLater(0,blah) as a means of starting a unit of work the by-product of which the caller didn't need. this is really a case of laying tracks for 2 years not fully comprehending the train coming down the tracks behind me! I sorta refused to learn/use inlinecallbacks or generators and I've never used callInThread until now. I inserted that into my S3 upload pipeline and it works, simplifying my code but detaches the caller from reactive failure. The mechanism you've described is different ~ a callLater to an estimated point of work completion, maybe a holding pattern if work isn't completed after the delay, and maybe not a bad means of observation (?)...but yes disastrous if the callLater function is wired poorly. Unless I'm misreading. Anyway I'm sure my code is illogical since I've was left to my own devices to solve problems of my own choosing. The by-product you can see here www.pokertalon.com - a single connection html5 app. On Wed, Jul 9, 2014 at 10:51 AM, Kevin Horn <kevin.horn@gmail.com> wrote:
participants (3)
-
exarkun＠twistedmatrix.com -
Kevin Horn -
Kevin Mcintyre