Problem when run yt dev in parallel on Blue Waters
Hi all, I have trouble running yt in parallel on Blue Waters. I installed yt using miniconda, the version of yt is ~/miniconda/lib $yt version yt module located at: /u/sciteam/madcpf/miniconda/lib/python2.7/site-packages/yt-3.3.dev0-py2.7-linux-x86_64.egg The current version and changeset for the code is: --- Version = 3.3-dev Changeset = 90f900be7a36+ yt Then with miniconda/bin in PATH I installed mpi4py-2.0.0. But when I tried to make the following simple output in parallel, I get: import yt yt.enable_parallelism() from yt.utilities.parallel_tools.parallel_analysis_interface import\ parallel_objects, communication_system comm = communication_system.communicators[-1] print comm.rank, comm.size 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 ... When I run a similar code but with yt-2.x also on Blue Waters, I get what I expect: 7 16 15 16 6 16 9 16 11 16 8 16 0 16 4 16 ... I'm confused about it. Could anyone give me some suggestions please? Thanks, Pengfei
Hi Pengfei, Can you try to run the following test script in parallel? This will determine if the issue is on the yt side of things or the mpi4py side of things: from mpi4py import MPI print(MPI.COMM_WORLD.rank) -Nathan On Wed, Nov 18, 2015 at 5:36 PM, Pengfei Chen <madcpf@gmail.com> wrote:
Hi all,
I have trouble running yt in parallel on Blue Waters. I installed yt using miniconda, the version of yt is
~/miniconda/lib $yt version yt module located at:
/u/sciteam/madcpf/miniconda/lib/python2.7/site-packages/yt-3.3.dev0-py2.7-linux-x86_64.egg The current version and changeset for the code is: --- Version = 3.3-dev Changeset = 90f900be7a36+ yt
Then with miniconda/bin in PATH I installed mpi4py-2.0.0. But when I tried to make the following simple output in parallel, I get:
import yt yt.enable_parallelism() from yt.utilities.parallel_tools.parallel_analysis_interface import\ parallel_objects, communication_system
comm = communication_system.communicators[-1] print comm.rank, comm.size
0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 ...
When I run a similar code but with yt-2.x also on Blue Waters, I get what I expect:
7 16 15 16 6 16 9 16 11 16 8 16 0 16 4 16 ...
I'm confused about it. Could anyone give me some suggestions please?
Thanks, Pengfei
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Nathan, Thank you very much for your reply! When I run that test script in parallel with 16 cores, I get 16 0s. Any suggestions to fix this? Thanks, Pengfei On Wed, Nov 18, 2015 at 3:40 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi Pengfei,
Can you try to run the following test script in parallel? This will determine if the issue is on the yt side of things or the mpi4py side of things:
from mpi4py import MPI
print(MPI.COMM_WORLD.rank)
-Nathan
On Wed, Nov 18, 2015 at 5:36 PM, Pengfei Chen <madcpf@gmail.com> wrote:
Hi all,
I have trouble running yt in parallel on Blue Waters. I installed yt using miniconda, the version of yt is
~/miniconda/lib $yt version yt module located at:
/u/sciteam/madcpf/miniconda/lib/python2.7/site-packages/yt-3.3.dev0-py2.7-linux-x86_64.egg The current version and changeset for the code is: --- Version = 3.3-dev Changeset = 90f900be7a36+ yt
Then with miniconda/bin in PATH I installed mpi4py-2.0.0. But when I tried to make the following simple output in parallel, I get:
import yt yt.enable_parallelism() from yt.utilities.parallel_tools.parallel_analysis_interface import\ parallel_objects, communication_system
comm = communication_system.communicators[-1] print comm.rank, comm.size
0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 ...
When I run a similar code but with yt-2.x also on Blue Waters, I get what I expect:
7 16 15 16 6 16 9 16 11 16 8 16 0 16 4 16 ...
I'm confused about it. Could anyone give me some suggestions please?
Thanks, Pengfei
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
So it looks like either mpi4py isn't able to properly link against the MPI installation on Blue waters or properly initialize itself at runtime. In either case, this points to the issue being a problem with mpi4py, not yt. Three things to try: * check which version version of mpi4py you have installed in your yt-2.x environment and install that version in your yt-3.x environment. You may have found a bug in mpi4py 2.0, which made major changes to the mpi4py library. * contact the Blue Waters sysadmins. * contact the mpi4py developers Hope that helps, Nathan On Thursday, November 19, 2015, Pengfei Chen <madcpf@gmail.com> wrote:
Hi Nathan,
Thank you very much for your reply! When I run that test script in parallel with 16 cores, I get 16 0s. Any suggestions to fix this?
Thanks, Pengfei
On Wed, Nov 18, 2015 at 3:40 PM, Nathan Goldbaum <nathan12343@gmail.com <javascript:_e(%7B%7D,'cvml','nathan12343@gmail.com');>> wrote:
Hi Pengfei,
Can you try to run the following test script in parallel? This will determine if the issue is on the yt side of things or the mpi4py side of things:
from mpi4py import MPI
print(MPI.COMM_WORLD.rank)
-Nathan
On Wed, Nov 18, 2015 at 5:36 PM, Pengfei Chen <madcpf@gmail.com <javascript:_e(%7B%7D,'cvml','madcpf@gmail.com');>> wrote:
Hi all,
I have trouble running yt in parallel on Blue Waters. I installed yt using miniconda, the version of yt is
~/miniconda/lib $yt version yt module located at:
/u/sciteam/madcpf/miniconda/lib/python2.7/site-packages/yt-3.3.dev0-py2.7-linux-x86_64.egg The current version and changeset for the code is: --- Version = 3.3-dev Changeset = 90f900be7a36+ yt
Then with miniconda/bin in PATH I installed mpi4py-2.0.0. But when I tried to make the following simple output in parallel, I get:
import yt yt.enable_parallelism() from yt.utilities.parallel_tools.parallel_analysis_interface import\ parallel_objects, communication_system
comm = communication_system.communicators[-1] print comm.rank, comm.size
0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 ...
When I run a similar code but with yt-2.x also on Blue Waters, I get what I expect:
7 16 15 16 6 16 9 16 11 16 8 16 0 16 4 16 ...
I'm confused about it. Could anyone give me some suggestions please?
Thanks, Pengfei
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <javascript:_e(%7B%7D,'cvml','yt-users@lists.spacepope.org');> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <javascript:_e(%7B%7D,'cvml','yt-users@lists.spacepope.org');> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Pengfei, Did you use my installation instructions for yt on Blue Waters? If not, I can send those along. B On Thu, Nov 19, 2015 at 10:02 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
So it looks like either mpi4py isn't able to properly link against the MPI installation on Blue waters or properly initialize itself at runtime.
In either case, this points to the issue being a problem with mpi4py, not yt.
Three things to try:
* check which version version of mpi4py you have installed in your yt-2.x environment and install that version in your yt-3.x environment. You may have found a bug in mpi4py 2.0, which made major changes to the mpi4py library.
* contact the Blue Waters sysadmins.
* contact the mpi4py developers
Hope that helps,
Nathan
On Thursday, November 19, 2015, Pengfei Chen <madcpf@gmail.com> wrote:
Hi Nathan,
Thank you very much for your reply! When I run that test script in parallel with 16 cores, I get 16 0s. Any suggestions to fix this?
Thanks, Pengfei
On Wed, Nov 18, 2015 at 3:40 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi Pengfei,
Can you try to run the following test script in parallel? This will determine if the issue is on the yt side of things or the mpi4py side of things:
from mpi4py import MPI
print(MPI.COMM_WORLD.rank)
-Nathan
On Wed, Nov 18, 2015 at 5:36 PM, Pengfei Chen <madcpf@gmail.com> wrote:
Hi all,
I have trouble running yt in parallel on Blue Waters. I installed yt using miniconda, the version of yt is
~/miniconda/lib $yt version yt module located at:
/u/sciteam/madcpf/miniconda/lib/python2.7/site-packages/yt-3.3.dev0-py2.7-linux-x86_64.egg The current version and changeset for the code is: --- Version = 3.3-dev Changeset = 90f900be7a36+ yt
Then with miniconda/bin in PATH I installed mpi4py-2.0.0. But when I tried to make the following simple output in parallel, I get:
import yt yt.enable_parallelism() from yt.utilities.parallel_tools.parallel_analysis_interface import\ parallel_objects, communication_system
comm = communication_system.communicators[-1] print comm.rank, comm.size
0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 ...
When I run a similar code but with yt-2.x also on Blue Waters, I get what I expect:
7 16 15 16 6 16 9 16 11 16 8 16 0 16 4 16 ...
I'm confused about it. Could anyone give me some suggestions please?
Thanks, Pengfei
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Brian, Thank you very much for your reply! No, I followed the instructions here http://yt-project.org/docs/dev/installing.html <http://yt-project.org/docs/dev/installing.html>. I need the yt-development, but failed when using All-in-One Installation Script to install it on BW, so I used miniconda to install. Could you please send me your installation instructions for yt on BW? I will try it out. Thanks, Pengfei On Thu, Nov 19, 2015 at 8:45 AM, Brian O'Shea <bwoshea@gmail.com> wrote:
Hi Pengfei,
Did you use my installation instructions for yt on Blue Waters? If not, I can send those along.
B
On Thu, Nov 19, 2015 at 10:02 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
So it looks like either mpi4py isn't able to properly link against the MPI installation on Blue waters or properly initialize itself at runtime.
In either case, this points to the issue being a problem with mpi4py, not yt.
Three things to try:
* check which version version of mpi4py you have installed in your yt-2.x environment and install that version in your yt-3.x environment. You may have found a bug in mpi4py 2.0, which made major changes to the mpi4py library.
* contact the Blue Waters sysadmins.
* contact the mpi4py developers
Hope that helps,
Nathan
On Thursday, November 19, 2015, Pengfei Chen <madcpf@gmail.com> wrote:
Hi Nathan,
Thank you very much for your reply! When I run that test script in parallel with 16 cores, I get 16 0s. Any suggestions to fix this?
Thanks, Pengfei
On Wed, Nov 18, 2015 at 3:40 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi Pengfei,
Can you try to run the following test script in parallel? This will determine if the issue is on the yt side of things or the mpi4py side of things:
from mpi4py import MPI
print(MPI.COMM_WORLD.rank)
-Nathan
On Wed, Nov 18, 2015 at 5:36 PM, Pengfei Chen <madcpf@gmail.com> wrote:
Hi all,
I have trouble running yt in parallel on Blue Waters. I installed yt using miniconda, the version of yt is
~/miniconda/lib $yt version yt module located at:
/u/sciteam/madcpf/miniconda/lib/python2.7/site-packages/yt-3.3.dev0-py2.7-linux-x86_64.egg The current version and changeset for the code is: --- Version = 3.3-dev Changeset = 90f900be7a36+ yt
Then with miniconda/bin in PATH I installed mpi4py-2.0.0. But when I tried to make the following simple output in parallel, I get:
import yt yt.enable_parallelism() from yt.utilities.parallel_tools.parallel_analysis_interface import\ parallel_objects, communication_system
comm = communication_system.communicators[-1] print comm.rank, comm.size
0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 ...
When I run a similar code but with yt-2.x also on Blue Waters, I get what I expect:
7 16 15 16 6 16 9 16 11 16 8 16 0 16 4 16 ...
I'm confused about it. Could anyone give me some suggestions please?
Thanks, Pengfei
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Pengfei, Sorry for the delay in response. Here are instructions for getting yt-3 to work on Blue Waters using mpi4py. I believe that Rob Sisneros has gotten yt-3 working as a module; however, it is likely that it will not be updated very frequently, at least compared to the pace of yt development. --Brian ----------------------------------------------------------------------- Getting yt-3 to work on Blue Waters: 0) Get the GNU software environment set up: “module swap PrgEnv-cray PrgEnv-gnu” . This will be necessary later. 1) Install the Anaconda python distribution ( https://store.continuum.io/cshop/anaconda/) for 64-bit Linux systems. Refresh your path so the Anaconda python’s in your path. 2) Download mpi4py from https://bitbucket.org/mpi4py/mpi4py/, or grab it from my home directory on Blue Waters (~bwoshea/mpi4py-1.3.1.tar.gz). 3) Once you’ve unpacked it in your home directory, edit the file mpi.cfg, which is in the top-level mpi4py directory, and add the following chunk of text at the end of the file: [cray] mpicc = cc mpicxx = CC extra_link_args = -shared then, in the top-level mpi4py directory, type: python setup.py build --mpi=cray python setp.py install At this point, you should have a functional version of mpi4py that uses the cray MPI libraries. 4) You can then install yt via source in your home directory: hg clone https://bitbucket.org/yt_analysis/yt hg update yt python setup.py develop And it should work, though you’re going to need to modify your LD_LIBRARY_PATH to point to your anaconda /lib directory. You can verify that yt and mpi4py are playing nicely together by running the scripts hw_MPI.py and yt_parallel_test.py, both of which are sitting in my home directory. (Note that you can do this via batch script or through the interactive queue, both with aprun - you can’t just do it from the login node.) 5) and, optionally, if you want to install Rockstar using yt from source, you do the following: hg clone http://bitbucket.org/MatthewTurk/rockstar cd rockstar hg pull hg up -C tip make lib and *then* make a file called rockstar.cfg that has a single line of text: the path of the rockstar source code directory. At that point, you need to run “python setup.py develop” again, and yt will recognize that rockstar has been installed and play nicely with it. You will likely also have to modify your LD_LIBRARY_PATH to point at your rockstar directory so it can find the rockstar library (librockstar.so). On Fri, Nov 20, 2015 at 3:46 AM, Pengfei Chen <madcpf@gmail.com> wrote:
Hi Brian,
Thank you very much for your reply! No, I followed the instructions here http://yt-project.org/docs/dev/installing.html <http://yt-project.org/docs/dev/installing.html>. I need the yt-development, but failed when using All-in-One Installation Script to install it on BW, so I used miniconda to install. Could you please send me your installation instructions for yt on BW? I will try it out.
Thanks, Pengfei
On Thu, Nov 19, 2015 at 8:45 AM, Brian O'Shea <bwoshea@gmail.com> wrote:
Hi Pengfei,
Did you use my installation instructions for yt on Blue Waters? If not, I can send those along.
B
On Thu, Nov 19, 2015 at 10:02 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
So it looks like either mpi4py isn't able to properly link against the MPI installation on Blue waters or properly initialize itself at runtime.
In either case, this points to the issue being a problem with mpi4py, not yt.
Three things to try:
* check which version version of mpi4py you have installed in your yt-2.x environment and install that version in your yt-3.x environment. You may have found a bug in mpi4py 2.0, which made major changes to the mpi4py library.
* contact the Blue Waters sysadmins.
* contact the mpi4py developers
Hope that helps,
Nathan
On Thursday, November 19, 2015, Pengfei Chen <madcpf@gmail.com> wrote:
Hi Nathan,
Thank you very much for your reply! When I run that test script in parallel with 16 cores, I get 16 0s. Any suggestions to fix this?
Thanks, Pengfei
On Wed, Nov 18, 2015 at 3:40 PM, Nathan Goldbaum <nathan12343@gmail.com
wrote:
Hi Pengfei,
Can you try to run the following test script in parallel? This will determine if the issue is on the yt side of things or the mpi4py side of things:
from mpi4py import MPI
print(MPI.COMM_WORLD.rank)
-Nathan
On Wed, Nov 18, 2015 at 5:36 PM, Pengfei Chen <madcpf@gmail.com> wrote:
Hi all,
I have trouble running yt in parallel on Blue Waters. I installed yt using miniconda, the version of yt is
~/miniconda/lib $yt version yt module located at:
/u/sciteam/madcpf/miniconda/lib/python2.7/site-packages/yt-3.3.dev0-py2.7-linux-x86_64.egg The current version and changeset for the code is: --- Version = 3.3-dev Changeset = 90f900be7a36+ yt
Then with miniconda/bin in PATH I installed mpi4py-2.0.0. But when I tried to make the following simple output in parallel, I get:
import yt yt.enable_parallelism() from yt.utilities.parallel_tools.parallel_analysis_interface import\ parallel_objects, communication_system
comm = communication_system.communicators[-1] print comm.rank, comm.size
0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 ...
When I run a similar code but with yt-2.x also on Blue Waters, I get what I expect:
7 16 15 16 6 16 9 16 11 16 8 16 0 16 4 16 ...
I'm confused about it. Could anyone give me some suggestions please?
Thanks, Pengfei
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Thanks for the instruction Brian! I'll try to remember this email in case people have questions about yt on blue waters and other cray machines in the future. On Friday, November 20, 2015, Brian O'Shea <bwoshea@gmail.com> wrote:
Hi Pengfei,
Sorry for the delay in response. Here are instructions for getting yt-3 to work on Blue Waters using mpi4py. I believe that Rob Sisneros has gotten yt-3 working as a module; however, it is likely that it will not be updated very frequently, at least compared to the pace of yt development.
--Brian
-----------------------------------------------------------------------
Getting yt-3 to work on Blue Waters:
0) Get the GNU software environment set up: “module swap PrgEnv-cray PrgEnv-gnu” . This will be necessary later.
1) Install the Anaconda python distribution ( https://store.continuum.io/cshop/anaconda/) for 64-bit Linux systems. Refresh your path so the Anaconda python’s in your path.
2) Download mpi4py from https://bitbucket.org/mpi4py/mpi4py/, or grab it from my home directory on Blue Waters (~bwoshea/mpi4py-1.3.1.tar.gz).
3) Once you’ve unpacked it in your home directory, edit the file mpi.cfg, which is in the top-level mpi4py directory, and add the following chunk of text at the end of the file:
[cray] mpicc = cc mpicxx = CC extra_link_args = -shared
then, in the top-level mpi4py directory, type:
python setup.py build --mpi=cray python setp.py install
At this point, you should have a functional version of mpi4py that uses the cray MPI libraries.
4) You can then install yt via source in your home directory:
hg clone https://bitbucket.org/yt_analysis/yt hg update yt python setup.py develop
And it should work, though you’re going to need to modify your LD_LIBRARY_PATH to point to your anaconda /lib directory. You can verify that yt and mpi4py are playing nicely together by running the scripts hw_MPI.py and yt_parallel_test.py, both of which are sitting in my home directory. (Note that you can do this via batch script or through the interactive queue, both with aprun - you can’t just do it from the login node.)
5) and, optionally, if you want to install Rockstar using yt from source, you do the following:
hg clone http://bitbucket.org/MatthewTurk/rockstar cd rockstar hg pull hg up -C tip make lib
and *then* make a file called rockstar.cfg that has a single line of text: the path of the rockstar source code directory. At that point, you need to run “python setup.py develop” again, and yt will recognize that rockstar has been installed and play nicely with it. You will likely also have to modify your LD_LIBRARY_PATH to point at your rockstar directory so it can find the rockstar library (librockstar.so).
On Fri, Nov 20, 2015 at 3:46 AM, Pengfei Chen <madcpf@gmail.com <javascript:_e(%7B%7D,'cvml','madcpf@gmail.com');>> wrote:
Hi Brian,
Thank you very much for your reply! No, I followed the instructions here http://yt-project.org/docs/dev/installing.html <http://yt-project.org/docs/dev/installing.html>. I need the yt-development, but failed when using All-in-One Installation Script to install it on BW, so I used miniconda to install. Could you please send me your installation instructions for yt on BW? I will try it out.
Thanks, Pengfei
On Thu, Nov 19, 2015 at 8:45 AM, Brian O'Shea <bwoshea@gmail.com <javascript:_e(%7B%7D,'cvml','bwoshea@gmail.com');>> wrote:
Hi Pengfei,
Did you use my installation instructions for yt on Blue Waters? If not, I can send those along.
B
On Thu, Nov 19, 2015 at 10:02 AM, Nathan Goldbaum <nathan12343@gmail.com <javascript:_e(%7B%7D,'cvml','nathan12343@gmail.com');>> wrote:
So it looks like either mpi4py isn't able to properly link against the MPI installation on Blue waters or properly initialize itself at runtime.
In either case, this points to the issue being a problem with mpi4py, not yt.
Three things to try:
* check which version version of mpi4py you have installed in your yt-2.x environment and install that version in your yt-3.x environment. You may have found a bug in mpi4py 2.0, which made major changes to the mpi4py library.
* contact the Blue Waters sysadmins.
* contact the mpi4py developers
Hope that helps,
Nathan
On Thursday, November 19, 2015, Pengfei Chen <madcpf@gmail.com <javascript:_e(%7B%7D,'cvml','madcpf@gmail.com');>> wrote:
Hi Nathan,
Thank you very much for your reply! When I run that test script in parallel with 16 cores, I get 16 0s. Any suggestions to fix this?
Thanks, Pengfei
On Wed, Nov 18, 2015 at 3:40 PM, Nathan Goldbaum < nathan12343@gmail.com> wrote:
Hi Pengfei,
Can you try to run the following test script in parallel? This will determine if the issue is on the yt side of things or the mpi4py side of things:
from mpi4py import MPI
print(MPI.COMM_WORLD.rank)
-Nathan
On Wed, Nov 18, 2015 at 5:36 PM, Pengfei Chen <madcpf@gmail.com> wrote:
> Hi all, > > I have trouble running yt in parallel on Blue Waters. I installed yt > using miniconda, the version of yt is > > ~/miniconda/lib $yt version > yt module located at: > > /u/sciteam/madcpf/miniconda/lib/python2.7/site-packages/yt-3.3.dev0-py2.7-linux-x86_64.egg > The current version and changeset for the code is: > --- > Version = 3.3-dev > Changeset = 90f900be7a36+ yt > > Then with miniconda/bin in PATH I installed mpi4py-2.0.0. But when I > tried to make the following simple output in parallel, I get: > > import yt > yt.enable_parallelism() > from yt.utilities.parallel_tools.parallel_analysis_interface import\ > parallel_objects, communication_system > > comm = communication_system.communicators[-1] > print comm.rank, comm.size > > 0 1 > 0 1 > 0 1 > 0 1 > 0 1 > 0 1 > 0 1 > 0 1 > ... > > When I run a similar code but with yt-2.x also on Blue Waters, I get > what I expect: > > 7 16 > 15 16 > 6 16 > 9 16 > 11 16 > 8 16 > 0 16 > 4 16 > ... > > I'm confused about it. Could anyone give me some suggestions please? > > Thanks, > Pengfei > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <javascript:_e(%7B%7D,'cvml','yt-users@lists.spacepope.org');> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <javascript:_e(%7B%7D,'cvml','yt-users@lists.spacepope.org');> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <javascript:_e(%7B%7D,'cvml','yt-users@lists.spacepope.org');> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Brian, It works perfectly! Thank you so much! I've saved the instructions for later use. Thanks, Pengfei On Fri, Nov 20, 2015 at 3:34 AM, Brian O'Shea <bwoshea@gmail.com> wrote:
Hi Pengfei,
Sorry for the delay in response. Here are instructions for getting yt-3 to work on Blue Waters using mpi4py. I believe that Rob Sisneros has gotten yt-3 working as a module; however, it is likely that it will not be updated very frequently, at least compared to the pace of yt development.
--Brian
-----------------------------------------------------------------------
Getting yt-3 to work on Blue Waters:
0) Get the GNU software environment set up: “module swap PrgEnv-cray PrgEnv-gnu” . This will be necessary later.
1) Install the Anaconda python distribution ( https://store.continuum.io/cshop/anaconda/) for 64-bit Linux systems. Refresh your path so the Anaconda python’s in your path.
2) Download mpi4py from https://bitbucket.org/mpi4py/mpi4py/, or grab it from my home directory on Blue Waters (~bwoshea/mpi4py-1.3.1.tar.gz).
3) Once you’ve unpacked it in your home directory, edit the file mpi.cfg, which is in the top-level mpi4py directory, and add the following chunk of text at the end of the file:
[cray] mpicc = cc mpicxx = CC extra_link_args = -shared
then, in the top-level mpi4py directory, type:
python setup.py build --mpi=cray python setp.py install
At this point, you should have a functional version of mpi4py that uses the cray MPI libraries.
4) You can then install yt via source in your home directory:
hg clone https://bitbucket.org/yt_analysis/yt hg update yt python setup.py develop
And it should work, though you’re going to need to modify your LD_LIBRARY_PATH to point to your anaconda /lib directory. You can verify that yt and mpi4py are playing nicely together by running the scripts hw_MPI.py and yt_parallel_test.py, both of which are sitting in my home directory. (Note that you can do this via batch script or through the interactive queue, both with aprun - you can’t just do it from the login node.)
5) and, optionally, if you want to install Rockstar using yt from source, you do the following:
hg clone http://bitbucket.org/MatthewTurk/rockstar cd rockstar hg pull hg up -C tip make lib
and *then* make a file called rockstar.cfg that has a single line of text: the path of the rockstar source code directory. At that point, you need to run “python setup.py develop” again, and yt will recognize that rockstar has been installed and play nicely with it. You will likely also have to modify your LD_LIBRARY_PATH to point at your rockstar directory so it can find the rockstar library (librockstar.so).
On Fri, Nov 20, 2015 at 3:46 AM, Pengfei Chen <madcpf@gmail.com> wrote:
Hi Brian,
Thank you very much for your reply! No, I followed the instructions here http://yt-project.org/docs/dev/installing.html <http://yt-project.org/docs/dev/installing.html>. I need the yt-development, but failed when using All-in-One Installation Script to install it on BW, so I used miniconda to install. Could you please send me your installation instructions for yt on BW? I will try it out.
Thanks, Pengfei
On Thu, Nov 19, 2015 at 8:45 AM, Brian O'Shea <bwoshea@gmail.com> wrote:
Hi Pengfei,
Did you use my installation instructions for yt on Blue Waters? If not, I can send those along.
B
On Thu, Nov 19, 2015 at 10:02 AM, Nathan Goldbaum <nathan12343@gmail.com
wrote:
So it looks like either mpi4py isn't able to properly link against the MPI installation on Blue waters or properly initialize itself at runtime.
In either case, this points to the issue being a problem with mpi4py, not yt.
Three things to try:
* check which version version of mpi4py you have installed in your yt-2.x environment and install that version in your yt-3.x environment. You may have found a bug in mpi4py 2.0, which made major changes to the mpi4py library.
* contact the Blue Waters sysadmins.
* contact the mpi4py developers
Hope that helps,
Nathan
On Thursday, November 19, 2015, Pengfei Chen <madcpf@gmail.com> wrote:
Hi Nathan,
Thank you very much for your reply! When I run that test script in parallel with 16 cores, I get 16 0s. Any suggestions to fix this?
Thanks, Pengfei
On Wed, Nov 18, 2015 at 3:40 PM, Nathan Goldbaum < nathan12343@gmail.com> wrote:
Hi Pengfei,
Can you try to run the following test script in parallel? This will determine if the issue is on the yt side of things or the mpi4py side of things:
from mpi4py import MPI
print(MPI.COMM_WORLD.rank)
-Nathan
On Wed, Nov 18, 2015 at 5:36 PM, Pengfei Chen <madcpf@gmail.com> wrote:
> Hi all, > > I have trouble running yt in parallel on Blue Waters. I installed yt > using miniconda, the version of yt is > > ~/miniconda/lib $yt version > yt module located at: > > /u/sciteam/madcpf/miniconda/lib/python2.7/site-packages/yt-3.3.dev0-py2.7-linux-x86_64.egg > The current version and changeset for the code is: > --- > Version = 3.3-dev > Changeset = 90f900be7a36+ yt > > Then with miniconda/bin in PATH I installed mpi4py-2.0.0. But when I > tried to make the following simple output in parallel, I get: > > import yt > yt.enable_parallelism() > from yt.utilities.parallel_tools.parallel_analysis_interface import\ > parallel_objects, communication_system > > comm = communication_system.communicators[-1] > print comm.rank, comm.size > > 0 1 > 0 1 > 0 1 > 0 1 > 0 1 > 0 1 > 0 1 > 0 1 > ... > > When I run a similar code but with yt-2.x also on Blue Waters, I get > what I expect: > > 7 16 > 15 16 > 6 16 > 9 16 > 11 16 > 8 16 > 0 16 > 4 16 > ... > > I'm confused about it. Could anyone give me some suggestions please? > > Thanks, > Pengfei > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Brian and Nathan, I am trying to install yt-3 on Blue Waters again, since after updating yt to the current tip it somehow can't run in parallel. When I came to step 3) following Brian's instructions, I get the following error message: ~/mpi4py-1.3.1 $python setup.py build --mpi=cray running build running build_src running build_py creating build creating build/lib.linux-x86_64-2.7 creating build/lib.linux-x86_64-2.7/mpi4py copying src/rc.py -> build/lib.linux-x86_64-2.7/mpi4py copying src/__init__.py -> build/lib.linux-x86_64-2.7/mpi4py creating build/lib.linux-x86_64-2.7/mpi4py/include creating build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py copying src/include/mpi4py/mpi4py.MPI.h -> build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py copying src/include/mpi4py/mpi4py.MPI_api.h -> build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py copying src/include/mpi4py/mpi4py.h -> build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py copying src/include/mpi4py/mpi_c.pxd -> build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py copying src/include/mpi4py/MPI.pxd -> build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py copying src/include/mpi4py/__init__.pxd -> build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py copying src/include/mpi4py/__init__.pyx -> build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py copying src/include/mpi4py/mpi.pxi -> build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py copying src/include/mpi4py/mpi4py.i -> build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py copying src/MPI.pxd -> build/lib.linux-x86_64-2.7/mpi4py copying src/mpi_c.pxd -> build/lib.linux-x86_64-2.7/mpi4py running build_clib MPI configuration: [cray] from 'mpi.cfg' MPI C compiler: /opt/cray/craype/2.5.4/bin/cc MPI C++ compiler: /opt/cray/craype/2.5.4/bin/CC Traceback (most recent call last): File "setup.py", line 588, in <module> main() File "setup.py", line 585, in main run_setup() File "setup.py", line 459, in run_setup **metadata) File "/mnt/a/u/sciteam/madcpf/mpi4py-1.3.1/conf/mpidistutils.py", line 494, in setup return fcn_setup(**attrs) File "/u/sciteam/madcpf/anaconda2/lib/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/u/sciteam/madcpf/anaconda2/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/u/sciteam/madcpf/anaconda2/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/u/sciteam/madcpf/anaconda2/lib/python2.7/distutils/command/build.py", line 127, in run self.run_command(cmd_name) File "/u/sciteam/madcpf/anaconda2/lib/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/u/sciteam/madcpf/anaconda2/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/mnt/a/u/sciteam/madcpf/mpi4py-1.3.1/conf/mpidistutils.py", line 894, in run configure_compiler(self.compiler, config) File "/mnt/a/u/sciteam/madcpf/mpi4py-1.3.1/conf/mpidistutils.py", line 249, in configure_compiler compiler.compiler += cc_args TypeError: coercing to Unicode: need string or buffer, list found Do you have any idea what happens here? Thanks a lot! Pengfei On Mon, Nov 23, 2015 at 12:13 PM, Pengfei Chen <madcpf@gmail.com> wrote:
Hi Brian,
It works perfectly! Thank you so much! I've saved the instructions for later use.
Thanks, Pengfei
On Fri, Nov 20, 2015 at 3:34 AM, Brian O'Shea <bwoshea@gmail.com> wrote:
Hi Pengfei,
Sorry for the delay in response. Here are instructions for getting yt-3 to work on Blue Waters using mpi4py. I believe that Rob Sisneros has gotten yt-3 working as a module; however, it is likely that it will not be updated very frequently, at least compared to the pace of yt development.
--Brian
-----------------------------------------------------------------------
Getting yt-3 to work on Blue Waters:
0) Get the GNU software environment set up: “module swap PrgEnv-cray PrgEnv-gnu” . This will be necessary later.
1) Install the Anaconda python distribution (https://store.continuum.io/ cshop/anaconda/) for 64-bit Linux systems. Refresh your path so the Anaconda python’s in your path.
2) Download mpi4py from https://bitbucket.org/mpi4py/mpi4py/, or grab it from my home directory on Blue Waters (~bwoshea/mpi4py-1.3.1.tar.gz).
3) Once you’ve unpacked it in your home directory, edit the file mpi.cfg, which is in the top-level mpi4py directory, and add the following chunk of text at the end of the file:
[cray] mpicc = cc mpicxx = CC extra_link_args = -shared
then, in the top-level mpi4py directory, type:
python setup.py build --mpi=cray python setp.py install
At this point, you should have a functional version of mpi4py that uses the cray MPI libraries.
4) You can then install yt via source in your home directory:
hg clone https://bitbucket.org/yt_analysis/yt hg update yt python setup.py develop
And it should work, though you’re going to need to modify your LD_LIBRARY_PATH to point to your anaconda /lib directory. You can verify that yt and mpi4py are playing nicely together by running the scripts hw_MPI.py and yt_parallel_test.py, both of which are sitting in my home directory. (Note that you can do this via batch script or through the interactive queue, both with aprun - you can’t just do it from the login node.)
5) and, optionally, if you want to install Rockstar using yt from source, you do the following:
hg clone http://bitbucket.org/MatthewTurk/rockstar cd rockstar hg pull hg up -C tip make lib
and *then* make a file called rockstar.cfg that has a single line of text: the path of the rockstar source code directory. At that point, you need to run “python setup.py develop” again, and yt will recognize that rockstar has been installed and play nicely with it. You will likely also have to modify your LD_LIBRARY_PATH to point at your rockstar directory so it can find the rockstar library (librockstar.so).
On Fri, Nov 20, 2015 at 3:46 AM, Pengfei Chen <madcpf@gmail.com> wrote:
Hi Brian,
Thank you very much for your reply! No, I followed the instructions here http://yt-project.org/docs/dev/installing.html <http://yt-project.org/docs/dev/installing.html>. I need the yt-development, but failed when using All-in-One Installation Script to install it on BW, so I used miniconda to install. Could you please send me your installation instructions for yt on BW? I will try it out.
Thanks, Pengfei
On Thu, Nov 19, 2015 at 8:45 AM, Brian O'Shea <bwoshea@gmail.com> wrote:
Hi Pengfei,
Did you use my installation instructions for yt on Blue Waters? If not, I can send those along.
B
On Thu, Nov 19, 2015 at 10:02 AM, Nathan Goldbaum < nathan12343@gmail.com> wrote:
So it looks like either mpi4py isn't able to properly link against the MPI installation on Blue waters or properly initialize itself at runtime.
In either case, this points to the issue being a problem with mpi4py, not yt.
Three things to try:
* check which version version of mpi4py you have installed in your yt-2.x environment and install that version in your yt-3.x environment. You may have found a bug in mpi4py 2.0, which made major changes to the mpi4py library.
* contact the Blue Waters sysadmins.
* contact the mpi4py developers
Hope that helps,
Nathan
On Thursday, November 19, 2015, Pengfei Chen <madcpf@gmail.com> wrote:
Hi Nathan,
Thank you very much for your reply! When I run that test script in parallel with 16 cores, I get 16 0s. Any suggestions to fix this?
Thanks, Pengfei
On Wed, Nov 18, 2015 at 3:40 PM, Nathan Goldbaum < nathan12343@gmail.com> wrote:
> Hi Pengfei, > > Can you try to run the following test script in parallel? This will > determine if the issue is on the yt side of things or the mpi4py side of > things: > > from mpi4py import MPI > > print(MPI.COMM_WORLD.rank) > > -Nathan > > On Wed, Nov 18, 2015 at 5:36 PM, Pengfei Chen <madcpf@gmail.com> > wrote: > >> Hi all, >> >> I have trouble running yt in parallel on Blue Waters. I installed >> yt using miniconda, the version of yt is >> >> ~/miniconda/lib $yt version >> yt module located at: >> /u/sciteam/madcpf/miniconda/lib/python2.7/site-packages/ >> yt-3.3.dev0-py2.7-linux-x86_64.egg >> The current version and changeset for the code is: >> --- >> Version = 3.3-dev >> Changeset = 90f900be7a36+ yt >> >> Then with miniconda/bin in PATH I installed mpi4py-2.0.0. But when >> I tried to make the following simple output in parallel, I get: >> >> import yt >> yt.enable_parallelism() >> from yt.utilities.parallel_tools.parallel_analysis_interface >> import\ >> parallel_objects, communication_system >> >> comm = communication_system.communicators[-1] >> print comm.rank, comm.size >> >> 0 1 >> 0 1 >> 0 1 >> 0 1 >> 0 1 >> 0 1 >> 0 1 >> 0 1 >> ... >> >> When I run a similar code but with yt-2.x also on Blue Waters, I >> get what I expect: >> >> 7 16 >> 15 16 >> 6 16 >> 9 16 >> 11 16 >> 8 16 >> 0 16 >> 4 16 >> ... >> >> I'm confused about it. Could anyone give me some suggestions please? >> >> Thanks, >> Pengfei >> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi all, I tried to use mpi-2.0.0 instead of mpi-1.3.1, and compiled with the same procedure. Now yt could run in parallel. Thanks, Pengfei On Sun, Aug 21, 2016 at 5:09 PM, Pengfei Chen <madcpf@gmail.com> wrote:
Hi Brian and Nathan,
I am trying to install yt-3 on Blue Waters again, since after updating yt to the current tip it somehow can't run in parallel. When I came to step 3) following Brian's instructions, I get the following error message:
~/mpi4py-1.3.1 $python setup.py build --mpi=cray running build running build_src running build_py creating build creating build/lib.linux-x86_64-2.7 creating build/lib.linux-x86_64-2.7/mpi4py copying src/rc.py -> build/lib.linux-x86_64-2.7/mpi4py copying src/__init__.py -> build/lib.linux-x86_64-2.7/mpi4py creating build/lib.linux-x86_64-2.7/mpi4py/include creating build/lib.linux-x86_64-2.7/mpi4py/include/mpi4py copying src/include/mpi4py/mpi4py.MPI.h -> build/lib.linux-x86_64-2.7/ mpi4py/include/mpi4py copying src/include/mpi4py/mpi4py.MPI_api.h -> build/lib.linux-x86_64-2.7/ mpi4py/include/mpi4py copying src/include/mpi4py/mpi4py.h -> build/lib.linux-x86_64-2.7/ mpi4py/include/mpi4py copying src/include/mpi4py/mpi_c.pxd -> build/lib.linux-x86_64-2.7/ mpi4py/include/mpi4py copying src/include/mpi4py/MPI.pxd -> build/lib.linux-x86_64-2.7/ mpi4py/include/mpi4py copying src/include/mpi4py/__init__.pxd -> build/lib.linux-x86_64-2.7/ mpi4py/include/mpi4py copying src/include/mpi4py/__init__.pyx -> build/lib.linux-x86_64-2.7/ mpi4py/include/mpi4py copying src/include/mpi4py/mpi.pxi -> build/lib.linux-x86_64-2.7/ mpi4py/include/mpi4py copying src/include/mpi4py/mpi4py.i -> build/lib.linux-x86_64-2.7/ mpi4py/include/mpi4py copying src/MPI.pxd -> build/lib.linux-x86_64-2.7/mpi4py copying src/mpi_c.pxd -> build/lib.linux-x86_64-2.7/mpi4py running build_clib MPI configuration: [cray] from 'mpi.cfg' MPI C compiler: /opt/cray/craype/2.5.4/bin/cc MPI C++ compiler: /opt/cray/craype/2.5.4/bin/CC Traceback (most recent call last): File "setup.py", line 588, in <module> main() File "setup.py", line 585, in main run_setup() File "setup.py", line 459, in run_setup **metadata) File "/mnt/a/u/sciteam/madcpf/mpi4py-1.3.1/conf/mpidistutils.py", line 494, in setup return fcn_setup(**attrs) File "/u/sciteam/madcpf/anaconda2/lib/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/u/sciteam/madcpf/anaconda2/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/u/sciteam/madcpf/anaconda2/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/u/sciteam/madcpf/anaconda2/lib/python2.7/distutils/command/build.py", line 127, in run self.run_command(cmd_name) File "/u/sciteam/madcpf/anaconda2/lib/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/u/sciteam/madcpf/anaconda2/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/mnt/a/u/sciteam/madcpf/mpi4py-1.3.1/conf/mpidistutils.py", line 894, in run configure_compiler(self.compiler, config) File "/mnt/a/u/sciteam/madcpf/mpi4py-1.3.1/conf/mpidistutils.py", line 249, in configure_compiler compiler.compiler += cc_args TypeError: coercing to Unicode: need string or buffer, list found
Do you have any idea what happens here?
Thanks a lot! Pengfei
On Mon, Nov 23, 2015 at 12:13 PM, Pengfei Chen <madcpf@gmail.com> wrote:
Hi Brian,
It works perfectly! Thank you so much! I've saved the instructions for later use.
Thanks, Pengfei
On Fri, Nov 20, 2015 at 3:34 AM, Brian O'Shea <bwoshea@gmail.com> wrote:
Hi Pengfei,
Sorry for the delay in response. Here are instructions for getting yt-3 to work on Blue Waters using mpi4py. I believe that Rob Sisneros has gotten yt-3 working as a module; however, it is likely that it will not be updated very frequently, at least compared to the pace of yt development.
--Brian
-----------------------------------------------------------------------
Getting yt-3 to work on Blue Waters:
0) Get the GNU software environment set up: “module swap PrgEnv-cray PrgEnv-gnu” . This will be necessary later.
1) Install the Anaconda python distribution ( https://store.continuum.io/cshop/anaconda/) for 64-bit Linux systems. Refresh your path so the Anaconda python’s in your path.
2) Download mpi4py from https://bitbucket.org/mpi4py/mpi4py/, or grab it from my home directory on Blue Waters (~bwoshea/mpi4py-1.3.1.tar.gz).
3) Once you’ve unpacked it in your home directory, edit the file mpi.cfg, which is in the top-level mpi4py directory, and add the following chunk of text at the end of the file:
[cray] mpicc = cc mpicxx = CC extra_link_args = -shared
then, in the top-level mpi4py directory, type:
python setup.py build --mpi=cray python setp.py install
At this point, you should have a functional version of mpi4py that uses the cray MPI libraries.
4) You can then install yt via source in your home directory:
hg clone https://bitbucket.org/yt_analysis/yt hg update yt python setup.py develop
And it should work, though you’re going to need to modify your LD_LIBRARY_PATH to point to your anaconda /lib directory. You can verify that yt and mpi4py are playing nicely together by running the scripts hw_MPI.py and yt_parallel_test.py, both of which are sitting in my home directory. (Note that you can do this via batch script or through the interactive queue, both with aprun - you can’t just do it from the login node.)
5) and, optionally, if you want to install Rockstar using yt from source, you do the following:
hg clone http://bitbucket.org/MatthewTurk/rockstar cd rockstar hg pull hg up -C tip make lib
and *then* make a file called rockstar.cfg that has a single line of text: the path of the rockstar source code directory. At that point, you need to run “python setup.py develop” again, and yt will recognize that rockstar has been installed and play nicely with it. You will likely also have to modify your LD_LIBRARY_PATH to point at your rockstar directory so it can find the rockstar library (librockstar.so).
On Fri, Nov 20, 2015 at 3:46 AM, Pengfei Chen <madcpf@gmail.com> wrote:
Hi Brian,
Thank you very much for your reply! No, I followed the instructions here http://yt-project.org/docs/dev/installing.html <http://yt-project.org/docs/dev/installing.html>. I need the yt-development, but failed when using All-in-One Installation Script to install it on BW, so I used miniconda to install. Could you please send me your installation instructions for yt on BW? I will try it out.
Thanks, Pengfei
On Thu, Nov 19, 2015 at 8:45 AM, Brian O'Shea <bwoshea@gmail.com> wrote:
Hi Pengfei,
Did you use my installation instructions for yt on Blue Waters? If not, I can send those along.
B
On Thu, Nov 19, 2015 at 10:02 AM, Nathan Goldbaum < nathan12343@gmail.com> wrote:
So it looks like either mpi4py isn't able to properly link against the MPI installation on Blue waters or properly initialize itself at runtime.
In either case, this points to the issue being a problem with mpi4py, not yt.
Three things to try:
* check which version version of mpi4py you have installed in your yt-2.x environment and install that version in your yt-3.x environment. You may have found a bug in mpi4py 2.0, which made major changes to the mpi4py library.
* contact the Blue Waters sysadmins.
* contact the mpi4py developers
Hope that helps,
Nathan
On Thursday, November 19, 2015, Pengfei Chen <madcpf@gmail.com> wrote:
> Hi Nathan, > > Thank you very much for your reply! When I run that test script in > parallel with 16 cores, I get 16 0s. Any suggestions to fix this? > > Thanks, > Pengfei > > > > On Wed, Nov 18, 2015 at 3:40 PM, Nathan Goldbaum < > nathan12343@gmail.com> wrote: > >> Hi Pengfei, >> >> Can you try to run the following test script in parallel? This will >> determine if the issue is on the yt side of things or the mpi4py side of >> things: >> >> from mpi4py import MPI >> >> print(MPI.COMM_WORLD.rank) >> >> -Nathan >> >> On Wed, Nov 18, 2015 at 5:36 PM, Pengfei Chen <madcpf@gmail.com> >> wrote: >> >>> Hi all, >>> >>> I have trouble running yt in parallel on Blue Waters. I installed >>> yt using miniconda, the version of yt is >>> >>> ~/miniconda/lib $yt version >>> yt module located at: >>> /u/sciteam/madcpf/miniconda/lib/python2.7/site-packages/yt- >>> 3.3.dev0-py2.7-linux-x86_64.egg >>> The current version and changeset for the code is: >>> --- >>> Version = 3.3-dev >>> Changeset = 90f900be7a36+ yt >>> >>> Then with miniconda/bin in PATH I installed mpi4py-2.0.0. But when >>> I tried to make the following simple output in parallel, I get: >>> >>> import yt >>> yt.enable_parallelism() >>> from yt.utilities.parallel_tools.parallel_analysis_interface >>> import\ >>> parallel_objects, communication_system >>> >>> comm = communication_system.communicators[-1] >>> print comm.rank, comm.size >>> >>> 0 1 >>> 0 1 >>> 0 1 >>> 0 1 >>> 0 1 >>> 0 1 >>> 0 1 >>> 0 1 >>> ... >>> >>> When I run a similar code but with yt-2.x also on Blue Waters, I >>> get what I expect: >>> >>> 7 16 >>> 15 16 >>> 6 16 >>> 9 16 >>> 11 16 >>> 8 16 >>> 0 16 >>> 4 16 >>> ... >>> >>> I'm confused about it. Could anyone give me some suggestions >>> please? >>> >>> Thanks, >>> Pengfei >>> >>> _______________________________________________ >>> yt-users mailing list >>> yt-users@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >>> >> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> > _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Nathan, Thank you very much for your reply! I've tried to switch from mpi4py 2.0 to 1.2.2, which I used for yt-2.x, but it still didn't work. I've sent Blue Water admins an email about this. Thanks, Pengfei On Thu, Nov 19, 2015 at 7:02 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
So it looks like either mpi4py isn't able to properly link against the MPI installation on Blue waters or properly initialize itself at runtime.
In either case, this points to the issue being a problem with mpi4py, not yt.
Three things to try:
* check which version version of mpi4py you have installed in your yt-2.x environment and install that version in your yt-3.x environment. You may have found a bug in mpi4py 2.0, which made major changes to the mpi4py library.
* contact the Blue Waters sysadmins.
* contact the mpi4py developers
Hope that helps,
Nathan
On Thursday, November 19, 2015, Pengfei Chen <madcpf@gmail.com> wrote:
Hi Nathan,
Thank you very much for your reply! When I run that test script in parallel with 16 cores, I get 16 0s. Any suggestions to fix this?
Thanks, Pengfei
On Wed, Nov 18, 2015 at 3:40 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi Pengfei,
Can you try to run the following test script in parallel? This will determine if the issue is on the yt side of things or the mpi4py side of things:
from mpi4py import MPI
print(MPI.COMM_WORLD.rank)
-Nathan
On Wed, Nov 18, 2015 at 5:36 PM, Pengfei Chen <madcpf@gmail.com> wrote:
Hi all,
I have trouble running yt in parallel on Blue Waters. I installed yt using miniconda, the version of yt is
~/miniconda/lib $yt version yt module located at:
/u/sciteam/madcpf/miniconda/lib/python2.7/site-packages/yt-3.3.dev0-py2.7-linux-x86_64.egg The current version and changeset for the code is: --- Version = 3.3-dev Changeset = 90f900be7a36+ yt
Then with miniconda/bin in PATH I installed mpi4py-2.0.0. But when I tried to make the following simple output in parallel, I get:
import yt yt.enable_parallelism() from yt.utilities.parallel_tools.parallel_analysis_interface import\ parallel_objects, communication_system
comm = communication_system.communicators[-1] print comm.rank, comm.size
0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 ...
When I run a similar code but with yt-2.x also on Blue Waters, I get what I expect:
7 16 15 16 6 16 9 16 11 16 8 16 0 16 4 16 ...
I'm confused about it. Could anyone give me some suggestions please?
Thanks, Pengfei
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
participants (3)
-
Brian O'Shea -
Nathan Goldbaum -
Pengfei Chen