Need help in concluding Multi-Thread based model thoughts between Python and PyPy
Hi All, I have tried to perform a CPU intensive operation ( which typically involved calculation of prime numbers from 1-100000 ) using Multi-process and Multi-Thread models provided by C, Python-2.6, PyPy-1.9. Here are results that are obtained from this exercise; *C Program results ( Number of processes = 500 )* CPU utilization = 99.81% (peak) Memory utilization = 14.77 GB (peak) Execution time = *2m16s* (real) ( time taken between invocation to termination ) *C Program results ( Number of Threads = 500 )* CPU utilization = 99.94% (peak) Memory utilization = 14.79 GB (peak) Execution time = *2m10s* (real) *{ Here I came to a conclusion that C gave a comparatively equal good results in terms of both Multi-Threaded as well as Multi-Process }* *Python Program results ( Number of processes = 500 )* CPU utilization = 99.62% (peak) Memory utilization = 17.03 GB (peak) Execution time = *87m42.785s* (real) *Python Program results ( Number of Threads = 500 )* CPU utilization = 18.23% (peak) Memory utilization = 33.074 GB (peak) Execution time = *239m10.210s* (real) *{ Here I came to a conclusion that Python is limited by the Global Interpreter Look (GIL) when used in Multi-Thread mode (that is why it gave poor results when compared with process model }* *PyPy Program results ( Number of processes = 500 )* CPU utilization = 99.19% (peak) Memory utilization = 22.92 GB (peak) Execution time = *6m4.970s* (real) *PyPy Program results ( Number of Threads = 500 )* CPU utilization = 38.88% (peak) Memory utilization = 21.06 GB (peak) Execution time = *59m29s* (real) *{ Here I came to a conclusion that PyPy is better than Python }* * * Now what am I struggling to understand is ; * * 1. Has PyPy optimized / reduced the GIL limitation ? ( what is the progress in PyPy version 1.9 in that when compared with Python's progress ) 2. If PyPy is also suffering from the same GIL limitations, what made the program run faster than Python, is it because of more warm-up time, optimization of loops ? 3. What are your suggestions for me if I wanted to go for Multi-Thread application design ( in-terms of Python / PyPy ) I request your inputs/suggestion techniques towards optimizing Multi threaded load loads. Kindly share your experiences and feedback in this scenario. -- Sasikanth
On 10/30/2012 11:19 AM, Sasikanth Eda wrote:
I have tried to perform a CPU intensive operation ( which typically involved calculation of prime numbers from 1-100000 ) using Multi-process and Multi-Thread models provided by C, Python-2.6, PyPy-1.9.
Note that your code is very slow. I can find all primes up to 100000 in less than 0.3s in CPython and less than 0.2s in PyPy, using a single process with a single thread.
*/{ Here I came to a conclusion that Python is limited by the Global Interpreter Look (GIL) when used in Multi-Thread mode (that is why it gave poor results when compared with process model }/*
Yes.
*{ /Here I came to a conclusion that PyPy is better than Python }/*
Yes, PyPy is faster than CPython for most (not all!) Python programs.
1. Has PyPy optimized / reduced the GIL limitation ? ( what is the progress in PyPy version 1.9 in that when compared with Python's progress ) 2. If PyPy is also suffering from the same GIL limitations, what made the program run faster than Python, is it because of more warm-up time, optimization of loops ?
PyPy has a Just In Time (JIT) compiler that compiles hot loops to machine language.
3. What are your suggestions for me if I wanted to go for Multi-Thread application design ( in-terms of Python / PyPy )
For now, you will not be happy with multiple CPU-bound threads in CPython or PyPy, because of the GIL. Jython is free-threaded but is slow in other ways. If you insist on using threads for CPU-bound work, you will probably be happier with another language. However, Armin Rigo is working on Software Transactional Memory for PyPy, which may someday end up making multi-threaded code fast. Web search for "pypy stm" to find details. -- David Ripton dripton@ripton.net
On Tue, Oct 30, 2012 at 9:33 AM, David Ripton <dripton@ripton.net> wrote:
On 10/30/2012 11:19 AM, Sasikanth Eda wrote:
I have tried to perform a CPU intensive operation ( which typically
involved calculation of prime numbers from 1-100000 ) using Multi-process and Multi-Thread models provided by C, Python-2.6, PyPy-1.9.
For now, you will not be happy with multiple CPU-bound threads in CPython or PyPy, because of the GIL. Jython is free-threaded but is slow in other ways. If you insist on using threads for CPU-bound work, you will probably be happier with another language.
I've actually found Jython rather slow to start up, but decent once it's loaded.
Jython is wayyy much slower than CPython in many cases. I can't even find its use. On Wed, Oct 31, 2012 at 5:23 AM, Dan Stromberg <drsalists@gmail.com> wrote:
On Tue, Oct 30, 2012 at 9:33 AM, David Ripton <dripton@ripton.net> wrote:
On 10/30/2012 11:19 AM, Sasikanth Eda wrote:
I have tried to perform a CPU intensive operation ( which typically
involved calculation of prime numbers from 1-100000 ) using Multi-process and Multi-Thread models provided by C, Python-2.6, PyPy-1.9.
For now, you will not be happy with multiple CPU-bound threads in CPython or PyPy, because of the GIL. Jython is free-threaded but is slow in other ways. If you insist on using threads for CPU-bound work, you will probably be happier with another language.
I've actually found Jython rather slow to start up, but decent once it's loaded.
_______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Hi, On Wed, Oct 31, 2012 at 3:14 PM, Phyo Arkar <phyo.arkarlwin@gmail.com> wrote:
Jython is wayyy much slower than CPython in many cases. I can't even find its use.
If your problem is hard to divide in multiple processes but easy to divide in multiple threads, and if you have a machine with enough CPUs, then Jython wins. Although dividing in multiple raw threads is rarely easy. A bientôt, Armin.
I see that may be the case. My program uses a lot of File IO and it is divided into multiple processess , working good in python.Failing so far in pypy (due to requirement of so many different document parsers) . I tested with Jython and i frustrated (At-least , in jython i can use Tika to parse documents) . On Wed, Oct 31, 2012 at 9:22 PM, Armin Rigo <arigo@tunes.org> wrote:
Hi,
On Wed, Oct 31, 2012 at 3:14 PM, Phyo Arkar <phyo.arkarlwin@gmail.com> wrote:
Jython is wayyy much slower than CPython in many cases. I can't even find its use.
If your problem is hard to divide in multiple processes but easy to divide in multiple threads, and if you have a machine with enough CPUs, then Jython wins. Although dividing in multiple raw threads is rarely easy.
A bientôt,
Armin.
Hi, On Tue, Oct 30, 2012 at 4:19 PM, Sasikanth Eda <sasikanth.ece@gmail.com> wrote:
1. Has PyPy optimized / reduced the GIL limitation ?
Not so far: PyPy has a GIL, similar to CPython. (Note that I'm answering with "CPython" here to make it clearer; "Python" means the language which both CPython and PyPy implement.)
2. If PyPy is also suffering from the same GIL limitations, what made the program run faster than Python
PyPy is faster than CPython in a lot of cases. It has a rather good just-in-time compiler.
3. What are your suggestions for me if I wanted to go for Multi-Thread application design ( in-terms of Python / PyPy )
One of the researchy goals of PyPy is to use STM to make multi-threaded programming easier and still scale to multiple CPUs: http://morepypy.blogspot.ch/2012/08/multicore-programming-in-pypy-and.html . This is still in-development. Note that the speed you'll get is unlikely to be better than using multiple processes, so doing that is the best solution for a few cases. But there are also a few other cases where using multiple processes would be very hard. Moreover, the theory goes that in all remaining intermediate cases, using multiple threads ---not directly but hidden below some interface like thread.atomic--- is actually much easier and cleaner than using multiple processes. A bientôt, Armin.
participants (5)
-
Armin Rigo -
Dan Stromberg -
David Ripton -
Phyo Arkar -
Sasikanth Eda