On 02.02.2021 00:37, Senthil Kumaran wrote:
On Mon, Feb 01, 2021 at 06:34:04PM +0100, M.-A. Lemburg wrote:
As I said: we have those years to wait. The GIL is not causing people to run away from Python anymore, since compute has moved on and no longer relies on threads for scaling up.
Where has it moved to? I see people choosing Go and Rust for performance reasons.
Compute has moved on to distributed computing, quoting an earlier email:
""" Also note that the Python GIL scalability problem is not really that urgent anymore these days. People are moving to distributed computing, splitting workloads across processes, containers, VMs, GPUs and other specialized hardware. Together with async code, the GIL no longer prevents Python applications from scaling easily. """
Have a look at Python Dask for example. There are plenty others as well, e.g. Apache Airflow, Dagster, Prefect, Luigi. In the web app space many people are using Celery or a WSGI server for distributing the load or using memcached for inter-process communication and caching.
Apache Arrow sets out to become the new inter-process comm standard.
For low level computing, several Python libraries are turning to GPUs for speed, e.g. RAPIDS, dask-sql or BlazingSQL.
All this is available in Python, turning to lower level languages and implementations for speed in the same way numpy started this long ago.
The reason for this is simple: threads only work locally, they don't scale beyond the number of cores you have available on your machine.
For many applications I/O has become the main bottleneck and again the only way to scale this up beyond the number of I/O channels you have in the server hardware is by using multiple machines.
Go and Rust are alternatives to get better local performance, but they have the same scalability problems.
Python is well positioned in the distributed computing space, because the C API makes it so easy to interface to other tools out there, even bridging to the Java world, which still dominates a lot of these areas.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Feb 02 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/