Edwin Zimmerman wrote:
On 5/25/2020 5:56 AM, email@example.com wrote:
Edwin Zimmerman wrote: Only if your workload is CPU bound. Python optimizes IO bound workload performance by releasing the GIL while doing IO. Green threads generally do not offer this option. Real threads is not needed in Python: 1) Real threads do not work parallel 2) Real threads only consume resources of OS As I said above, threads in Python provide limited parallelization when do IO. For example, if I have a thread that reads a 1GB file into memory, that thread releases the GIL while reading the file, allowing other threads to run in parallel. 3) Real threads have also as trade off context switching between threads (small but ...) I think considering this trade offs we should switch Real Thread -> Green Thread internally or provide alternative functionality ... I think asyncio already provide functionality of GreenThread ... Okay maybe we should leave Real Threads as it is ... I need to think about it ... But also in my email I have mentioned about interpreters.run_string @Edwin Zimmerman Do you know if it is blocking call ? And if yes than I do not understand how with such API we will have TRUE Parallel execution, because in this case only one Thread at the same time could be executed, but those threads could be executed only one at the time !! Also there are drawback that we need additional thread for run sub-interpretter !! Better would be the something like this in addition interpreters.run_string_async, interpreters.run_async in addition to interpreters.run_string, interpreters.run to remove trad offs
Sub-interpreters are a work in progress. The API is not anywhere near being finalized. True parallel execution will occur only if the GIL is changed to a per interpreter lock, and that requires some rather large changes to Python that haven't happened yet.
Why ? True parallel execution is possible with sub-interpreters !! `Sub-interpretter` should be run in separate thread not controlled by GIL and `sub-interpretter` will notify `main interpreter` with `atomic variable` that it is finished (like worker) !! We just need to provide two APIs: 1) Synchronized: `run`, `run_string`, that will wait until thread notify with setting `atomic variable` in true that it finished 2) Asynchronized (based on async): `run_async`, `run_string_async`, event_loop will wait on `atomic variable` reading it periodically