I think I got around the second problem (thanks for sharing the react documentation) How do I use more multiple CPU cores? I guess atleast the fact that twisted app wont be blocking is good, but what would be a good way to use multiple cores. I do see that meejah suggested using multiple python processes while setting up the reactor? My question is that.. say.. I set up multiple twisted based python processes by leveraging the spawnProcess.. And then we setup a resource 'foo'.. When the request is hit to `foo` end point... will the parallelization happen here.. say.. I have spawned 2 processes and one request came and hit the 'foo' end point.. I am guessing.. the fact that 2 processes is running would help responding to multiple requests simultaneously. I think that level of parallelization is good and I will take it.. But here.. what I want is... when you hit "foo".. then there is a function that is parallelizable.. Like.. you hit "foo" endpoint.. I fetch data from somewhere.. (yay treq).. and then there is a function that is called when we hit "foo" end point and that function is extremely parallelizable over for loop. Now, what is great is that with all your help, I am able to make this flow non-blocking. But, would definitely like to exploit parallellism wherever feasible. On Mon, Jun 24, 2019 at 7:37 PM Moshe Zadka <moshez@zadka.club> wrote:
On Mon, Jun 24, 2019, at 16:27, Chengi Liu wrote:
cpu_res *= yield *defer.*gatherResults*(cpus)
Remember: This will not block the reactor (Good!) but will still limit you to one CPU core (in general, some caveats, but true.) If you are CPU bound, this is woefully underutilizing modern CPU resources (again, some caveats here).
if __name__ == '__main__':
In general, the best thing to do is to to use twisted.internet.task.react for main. This would also solve the other problem you point out below.
https://twistedmatrix.com/documents/current/api/twisted.internet.task.html#r... has a good example.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python