Accelerate your Python code with parallel processing
I am an Application Engineer at Interactive Supercomputing and we are rolling out a beta version of our Star-P product for Python. We are actively looking for computationally intensive Python application to port to Star-P. Star-P is a parallel application development platform that allows users to tap into the power and memory of supercomputers from the comfort of the favorite desktop applications, in this case Python. Star-P is capable of both fine-grained parallel computation and embarrassingly parallel computation. The fine-grained mode of our Star-P Python implementation has been modeled on the Python NumPy package - for example: x = starp.random.rand(20000,20000) y = starp.linalg.inv(x) instead of x = numpy.random.rand(20000,20000) y = numpy.linalg.inv(x) Where the first couple of lines are executed on the Star-P parallel server in full C/MPI mode and the last couple of lines are executed on the desktop using Python. The embarrassingly parallel mode is capable of executing any Python module, although input and output parameters are currently limited to NumPy arrays, scalars, and strings - for example: y = starp.ppeval(mymodule.dosomething,x) instead of for i in range(0,n): y[:,:,i] = mymodule.dosomething(x[:,i]) Where again in the former example the iterations are spread out over the available CPUs (note the abstraction - user need not worry regarding the number of CPUs) on the Star-P server using Python and in the latter the looping is doing in serial on the client using Python. We are looking for real Python application that you would be willing to share with us that we can port to Star-P. We want to use this experience as a basis for further improvements and development of our Python client. Thanks, Ronnie
Hi, Is there a comparison with parallel libraries that can be branched on numpy like MKL ? (and IPP for random number ?) Matthieu 2007/6/29, Ronnie Hoogerwerf <rhoogerwerf@interactivesupercomputing.com>:
I am an Application Engineer at Interactive Supercomputing and we are rolling out a beta version of our Star-P product for Python. We are actively looking for computationally intensive Python application to port to Star-P. Star-P is a parallel application development platform that allows users to tap into the power and memory of supercomputers from the comfort of the favorite desktop applications, in this case Python.
Star-P is capable of both fine-grained parallel computation and embarrassingly parallel computation. The fine-grained mode of our Star-P Python implementation has been modeled on the Python NumPy package - for example:
x = starp.random.rand(20000,20000) y = starp.linalg.inv(x)
instead of
x = numpy.random.rand(20000,20000) y = numpy.linalg.inv(x)
Where the first couple of lines are executed on the Star-P parallel server in full C/MPI mode and the last couple of lines are executed on the desktop using Python.
The embarrassingly parallel mode is capable of executing any Python module, although input and output parameters are currently limited to NumPy arrays, scalars, and strings - for example:
y = starp.ppeval(mymodule.dosomething,x)
instead of
for i in range(0,n): y[:,:,i] = mymodule.dosomething(x[:,i])
Where again in the former example the iterations are spread out over the available CPUs (note the abstraction - user need not worry regarding the number of CPUs) on the Star-P server using Python and in the latter the looping is doing in serial on the client using Python.
We are looking for real Python application that you would be willing to share with us that we can port to Star-P. We want to use this experience as a basis for further improvements and development of our Python client.
Thanks, Ronnie
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Matthieu, linking in a multi-threading enabled library to numpy, like MKL, will provide speedup to only a limited set of operations and scale only on when you limit yourself to the cores in a single smp- like system. Star-P on the other hand, since it is MPI based, scales on cluster and covers a much larger set of operations that can be performed in parallel plus it provides a simple mechanism to do embarrassingly parallel operations. Ronnie On Jun 29, 2007, at 1:46 PM, Matthieu Brucher wrote:
Hi,
Is there a comparison with parallel libraries that can be branched on numpy like MKL ? (and IPP for random number ?)
Matthieu
2007/6/29, Ronnie Hoogerwerf < rhoogerwerf@interactivesupercomputing.com>: I am an Application Engineer at Interactive Supercomputing and we are rolling out a beta version of our Star-P product for Python. We are actively looking for computationally intensive Python application to port to Star-P. Star-P is a parallel application development platform that allows users to tap into the power and memory of supercomputers from the comfort of the favorite desktop applications, in this case Python.
Star-P is capable of both fine-grained parallel computation and embarrassingly parallel computation. The fine-grained mode of our Star-P Python implementation has been modeled on the Python NumPy package - for example:
x = starp.random.rand (20000,20000) y = starp.linalg.inv(x)
instead of
x = numpy.random.rand(20000,20000) y = numpy.linalg.inv(x)
Where the first couple of lines are executed on the Star-P parallel server in full C/MPI mode and the last couple of lines are executed on the desktop using Python.
The embarrassingly parallel mode is capable of executing any Python module, although input and output parameters are currently limited to NumPy arrays, scalars, and strings - for example:
y = starp.ppeval(mymodule.dosomething,x)
instead of
for i in range(0,n): y[:,:,i] = mymodule.dosomething(x[:,i])
Where again in the former example the iterations are spread out over the available CPUs (note the abstraction - user need not worry regarding the number of CPUs) on the Star-P server using Python and in the latter the looping is doing in serial on the client using Python.
We are looking for real Python application that you would be willing to share with us that we can port to Star-P. We want to use this experience as a basis for further improvements and development of our Python client.
Thanks, Ronnie
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
I didn't find your python prices for Star-P. Or are there any chances for GPL/other free license for Python Star-P? Also, it would be interesting to see comparison numerical results of your product vs stackless python ( http://www.google.com.ua/search?q=stackless+python&ie=utf-8&oe=utf-8&aq=t&rls=com.ubuntu:en-US:official&client=firefox-a ) and/or all those paralles stuff http://www.scipy.org/Topical_Software#head-cf472934357fda4558aafdf558a977c4d... All software developers say "our product is one of the best ones!" but I would prefer to see benchmark results before pay anything. Those mentioned stuff is free - and maybe difference with your Star-P in a cluster and/or for example my AMD Athlon X2 is insufficient? Regards, D. Ronnie Hoogerwerf wrote:
I am an Application Engineer at Interactive Supercomputing and we are rolling out a beta version of our Star-P product for Python. We are actively looking for computationally intensive Python application to port to Star-P. Star-P is a parallel application development platform that allows users to tap into the power and memory of supercomputers from the comfort of the favorite desktop applications, in this case Python.
Star-P is capable of both fine-grained parallel computation and embarrassingly parallel computation. The fine-grained mode of our Star-P Python implementation has been modeled on the Python NumPy package - for example:
x = starp.random.rand(20000,20000) y = starp.linalg.inv(x)
instead of
x = numpy.random.rand(20000,20000) y = numpy.linalg.inv(x)
Where the first couple of lines are executed on the Star-P parallel server in full C/MPI mode and the last couple of lines are executed on the desktop using Python.
The embarrassingly parallel mode is capable of executing any Python module, although input and output parameters are currently limited to NumPy arrays, scalars, and strings - for example:
y = starp.ppeval(mymodule.dosomething,x)
instead of
for i in range(0,n): y[:,:,i] = mymodule.dosomething(x[:,i])
Where again in the former example the iterations are spread out over the available CPUs (note the abstraction - user need not worry regarding the number of CPUs) on the Star-P server using Python and in the latter the looping is doing in serial on the client using Python.
We are looking for real Python application that you would be willing to share with us that we can port to Star-P. We want to use this experience as a basis for further improvements and development of our Python client.
Thanks, Ronnie
------------------------------------------------------------------------
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
2007/6/30, dmitrey <openopt@ukr.net>:
I didn't find your python prices for Star-P. Or are there any chances for GPL/other free license for Python Star-P?
I've found them, several k€. A link would have been great ( http://www.interactivesupercomputing.com/products/starpandpython.php) All software developers say "our product is one of the best ones!" but I
would prefer to see benchmark results before pay anything.
Same for me, but safe for the benchmarks provided on the web page, nothing, and no comparison with other products. Matthieu
participants (3)
-
dmitrey -
Matthieu Brucher -
Ronnie Hoogerwerf