Thanks Moshe & Meejah & Gelin for the suggestions and advice. This is super helpful.
I think, I am able to move forward with this.
Let me just summarize this..
My usecase is.. fetch the data.. and then you assemble the data. Fetching data is network bound, assembling data is like CPU bound.
Can you guys confirm if what I am doing makes sense?
def compute_heavy_function(x):
return x*x
@defer.inlinecallbacks
def network_get(x):
content = yield x.content()
defer.returnValue(val)
@defer.inlinecallbacks
def twisted_do_your_magic():
nets, cpus = [], []
for i in range(10):
t = defer.ensureDeferred(network_get(i))
nets.append(t)
d = threads.deferToThread(compute_heavy_function, i)
cpus.append(d)
cpu_res = yield defer.gatherResults(cpus)
network_res = yield defer.gatherResults(nets)
defer.returnValue({'cpu': cpu_res, 'network': network_res})
if __name__ == '__main__':
twisted_do_your_magic()
reactor.callLater(2, reactor.stop)
reactor.run()
I ran it locally.. it seems to be running fine. But just want to make sure that I got the concept on what to deferToThread & what to "ensureDeferred".
From the SO, I got the impression that network based IO benefits from `deferToThread` but from video tutorial.. I got the impression that ensureDefer followed by gatherResults seems to be the right way to go?
Moshe.. One last question..
I was trying to follow the tutorial on video lecture..
But, I wasnt able to make it run on python3.
Say, I have an async function
async def foo():
resp = await treq.get("localhost:1234/foo")
content = await resp.content()
return json.loads(content.decode("utf-8")
async def func():
d1 = defer.ensureDeffered(foo())
d2 = defer.ensureDeffered(foo())
res = await defer.gatherResults([d1, d2])
return res
if __name__ == '__main__'
x = func()
reactor.callLater(2, reactor.stop)
reactor.run()
In this case, I get an error (x = func() in main code block)..
RuntimeWarning: coroutine 'func' was never awaited
How do i fix this.
Again, thanks for all the help, support and advice in getting me started with twisted.