Hi, First, thanks to the authors for opening kwant to the community. In the kwant paper it is mentioned that "a few additional lines of code allow us to parallelize the script for a multi-core workstation or cluster". I was wondering if you could share any tip in this direction. Best regards. Luis.
Hi Luis,
Luis Foa Torres
In the kwant paper it is mentioned that "a few additional lines of code allow us to parallelize the script for a multi-core workstation or cluster". I was wondering if you could share any tip in this direction.
As Kwant is simply a Python library, you can use any of the numerous parallelization approaches that exist for Python. (One restriction is that, as of Kwant 1.0, Kwant systems cannot be “pickled”, i.e. it’s not possible to create a Kwant system object on one node of a cluster and send it over the network to another node. However, this is not a significant restriction in practice, as it is usually anyway preferable to simply construct the same system on all the nodes that need it. Constructing the system only once, pickling, sending, and unpickling it would take more time.) If your goal is to parallelize for a single machine with multiple cores, I recommend the concurrent.futures module. This is what is used in the spin valve example of the paper. The full example can be found in http://downloads.kwant-project.org/examples/kwant-examples-njp.zip. concurrent.futures has been introduced in Python 3.2, but has been backported for Python 2 so that it’s possible to use it with Kwant. If you would like to parallelize a code for a cluster, you could choose a classic approach and use something like mpi4py, or something modern like scoop (https://code.google.com/p/scoop/). For interactive work on a cluster ipython parallel (http://ipython.org/ipython-doc/dev/parallel/) works well, especially in connection with the ipython notebook. Christoph
Hi Christoph,
Thanks a lot for the very detailed answer.
With best regards,
Luis.
On Sat, Aug 23, 2014 at 5:10 AM,
Hi Luis,
Luis Foa Torres
writes: In the kwant paper it is mentioned that "a few additional lines of code
allow us to parallelize the script for a multi-core workstation or cluster". I was wondering if you could share any tip in this direction.
As Kwant is simply a Python library, you can use any of the numerous parallelization approaches that exist for Python.
(One restriction is that, as of Kwant 1.0, Kwant systems cannot be “pickled”, i.e. it’s not possible to create a Kwant system object on one node of a cluster and send it over the network to another node. However, this is not a significant restriction in practice, as it is usually anyway preferable to simply construct the same system on all the nodes that need it. Constructing the system only once, pickling, sending, and unpickling it would take more time.)
If your goal is to parallelize for a single machine with multiple cores, I recommend the concurrent.futures module. This is what is used in the spin valve example of the paper. The full example can be found in http://downloads.kwant-project.org/examples/kwant-examples-njp.zip. concurrent.futures has been introduced in Python 3.2, but has been backported for Python 2 so that it’s possible to use it with Kwant.
If you would like to parallelize a code for a cluster, you could choose a classic approach and use something like mpi4py, or something modern like scoop (https://code.google.com/p/scoop/). For interactive work on a cluster ipython parallel (http://ipython.org/ipython-doc/dev/parallel/) works well, especially in connection with the ipython notebook.
Christoph
participants (3)
-
christoph.groth@cea.fr
-
Luis Foa Torres
-
Luis Foà Torres