Hi Cheng,
deferToThread *returns* a deferred, and sends the work to a thread pool. The other two functions are useful for manipulating deferreds or other promises, but will not send work to a separate thread, so they cannot be used instead.
The simple example is pretty simple, it's pretty much
@inlineCallbacks
def some_function():
d1 = deferToThread(cpu_heavy_square, 5)
d2 = deferToThread(cpu_heavy_square, 6)
five_squared, six_squared = yield gatherResults([d1, d2])
returnValue(five_squared + six_squared)
However, as explained, this will *defer to threads*, but will *not* use multiple cores, since Python is single-core. This is not entirely accurate: if your heavy functions are using numpy, numpy will release the GIL sometimes. Otherwise, you might want to use ampoule, but there are no easy tutorials for it that I'm aware of. You might want to consider something like dask + deferToThread for using multiple cores. See
https://github.com/dask/dask-tutorial/blob/master/01_dask.delayed.ipynb for a simple example: you just plan your computation, and run the `compute` method in a thread. In this case, the only thing the thread is doing is avoiding blocking the reactor, while dask internally apportions computations.
(Thank you for the kind of words about the tutorial -- I'm one of the presenters.)
Moshe Z.
On Mon, Jun 24, 2019, at 11:34, Chengi Liu wrote:
Thanks for pointing that out. Is there a dead simply example on how to parallelize a function using ampoule.
Again, I have a function.. I have a for loop.. the function takes in an argument from loop.. but the for loop is extremely parallelizable.
I found couple of use features.
(defer.ensureDeferred
and
)
Can these two be used instead of deferToThread.
When are the above defered used and when to use deferToThread.
Thanks for entertaining my dumb questions.
Hi
If your purpose is merely not to block the reactor, you can run your code in a separate thread by using deferToThread. If you want to benefit from multi cores, you may consider use
https://github.com/twisted/ampoule or other similar tools.
Regards
gelin yan
_______________________________________________
Twisted-Python mailing list
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python