I think it's a lot easier now that the new install script has been merged in: http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2016-April/006350.... -Matt On Wed, Apr 20, 2016 at 4:22 PM, Brian O'Shea wrote:

Yes indeed. It's a gigantic pain. Here are instructions. Good luck!

-----------------

0) Get the GNU software environment set up: “module swap PrgEnv-cray PrgEnv-gnu” . This will be necessary later.

1) Install the Anaconda python distribution (https://store.continuum.io/cshop/anaconda/) for 64-bit Linux systems. Refresh your path so the Anaconda python’s in your path.

2) Download mpi4py from https://bitbucket.org/mpi4py/mpi4py/, or grab it from my home directory on Blue Waters (~bwoshea/mpi4py-1.3.1.tar.gz).

3) Once you’ve unpacked it in your home directory, edit the file mpi.cfg, which is in the top-level mpi4py directory, and add the following chunk of text at the end of the file:

[cray] mpicc = cc mpicxx = CC extra_link_args = -shared

then, in the top-level mpi4py directory, type:

python setup.py build --mpi=cray python setp.py install

At this point, you should have a functional version of mpi4py that uses the cray MPI libraries.

4) You can then install yt via source in your home directory:

hg clone https://bitbucket.org/yt_analysis/yt hg update yt python setup.py develop

And it should work, though you’re going to need to modify your LD_LIBRARY_PATH to point to your anaconda /lib directory. You can verify that yt and mpi4py are playing nicely together by running the scripts hw_MPI.py and yt_parallel_test.py, both of which are sitting in my home directory. (Note that you can do this via batch script or through the interactive queue, both with aprun - you can’t just do it from the login node.)

5) and, optionally, if you want to install Rockstar using yt from source, you do the following:

hg clone http://bitbucket.org/MatthewTurk/rockstar cd rockstar hg pull hg up -C tip make lib

and *then* make a file called rockstar.cfg that has a single line of text: the path of the rockstar source code directory. At that point, you need to run “python setup.py develop” again, and yt will recognize that rockstar has been installed and play nicely with it. You will likely also have to modify your LD_LIBRARY_PATH to point at your rockstar directory so it can find the rockstar library (librockstar.so).

On Wed, Apr 20, 2016 at 5:04 PM, Jonah Miller wrote:

...

Hi yt users,

Does anybody have experience installing the development version of yt on blue waters (as opposed to relying on the module pre-provided)? If so, do you have any tips or gotchas? I will be attempting an installation in the next few days and I'm sort of hoping it's a solved problem.

Best, Jonah Miller _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Hi Jonah, Just make sure you use the from-source option, and you can pull whatever changesets you like into your source directory in your installation. On Wed, Apr 20, 2016 at 4:34 PM, Jonah Miller wrote:

Hi Brian, Matt,

Thanks a lot! These instructions are exactly what I was hoping for! Regarding the all-in-one install script, that won't work if I want to install a fork of yt though, will it? This is to test my carpet frontend on some production data.

Best, Jonah

On 16-04-20 05:24 PM, Matthew Turk wrote:

...

I think it's a lot easier now that the new install script has been merged in:

http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2016-April/006350....

-Matt

On Wed, Apr 20, 2016 at 4:22 PM, Brian O'Shea wrote:

...

Yes indeed. It's a gigantic pain. Here are instructions. Good luck!

-----------------

0) Get the GNU software environment set up: “module swap PrgEnv-cray PrgEnv-gnu” . This will be necessary later.

1) Install the Anaconda python distribution (https://store.continuum.io/cshop/anaconda/) for 64-bit Linux systems. Refresh your path so the Anaconda python’s in your path.

2) Download mpi4py from https://bitbucket.org/mpi4py/mpi4py/, or grab it from my home directory on Blue Waters (~bwoshea/mpi4py-1.3.1.tar.gz).

3) Once you’ve unpacked it in your home directory, edit the file mpi.cfg, which is in the top-level mpi4py directory, and add the following chunk of text at the end of the file:

[cray] mpicc = cc mpicxx = CC extra_link_args = -shared

then, in the top-level mpi4py directory, type:

python setup.py build --mpi=cray python setp.py install

At this point, you should have a functional version of mpi4py that uses the cray MPI libraries.

4) You can then install yt via source in your home directory:

hg clone https://bitbucket.org/yt_analysis/yt hg update yt python setup.py develop

And it should work, though you’re going to need to modify your LD_LIBRARY_PATH to point to your anaconda /lib directory. You can verify that yt and mpi4py are playing nicely together by running the scripts hw_MPI.py and yt_parallel_test.py, both of which are sitting in my home directory. (Note that you can do this via batch script or through the interactive queue, both with aprun - you can’t just do it from the login node.)

5) and, optionally, if you want to install Rockstar using yt from source, you do the following:

hg clone http://bitbucket.org/MatthewTurk/rockstar cd rockstar hg pull hg up -C tip make lib

and *then* make a file called rockstar.cfg that has a single line of text: the path of the rockstar source code directory. At that point, you need to run “python setup.py develop” again, and yt will recognize that rockstar has been installed and play nicely with it. You will likely also have to modify your LD_LIBRARY_PATH to point at your rockstar directory so it can find the rockstar library (librockstar.so).

On Wed, Apr 20, 2016 at 5:04 PM, Jonah Miller wrote:

...

Hi yt users,

Does anybody have experience installing the development version of yt on blue waters (as opposed to relying on the module pre-provided)? If so, do you have any tips or gotchas? I will be attempting an installation in the next few days and I'm sort of hoping it's a solved problem.

Best, Jonah Miller _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Hi Nathan, Matt, Brian, I see. Yes, I'll want to use the parallelism. Cool! I didn't realize the all-in-one script could pull from different repos. Thanks for your help! Best, Jonah On 16-04-20 05:39 PM, Nathan Goldbaum wrote:

You'll probably also want to follow Brian's instructions for mpi4py if you're going t be using yt's parallelism. The install script doesn't set that up since we can't assume people have an MPI installation.

On Wednesday, April 20, 2016, Matthew Turk mailto:matthewturk@gmail.com> wrote:

Hi Jonah,

Just make sure you use the from-source option, and you can pull whatever changesets you like into your source directory in your installation.

On Wed, Apr 20, 2016 at 4:34 PM, Jonah Miller javascript:;> wrote: > Hi Brian, Matt, > > Thanks a lot! These instructions are exactly what I was hoping for! > Regarding the all-in-one install script, that won't work if I want to > install a fork of yt though, will it? This is to test my carpet frontend on > some production data. > > Best, > Jonah > > > On 16-04-20 05:24 PM, Matthew Turk wrote: >> >> I think it's a lot easier now that the new install script has been merged >> in: >> >> >> http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2016-April/006350.... >> >> -Matt >> >> On Wed, Apr 20, 2016 at 4:22 PM, Brian O'Shea javascript:;> wrote: >>> >>> Yes indeed. It's a gigantic pain. Here are instructions. Good luck! >>> >>> ----------------- >>> >>> 0) Get the GNU software environment set up: “module swap PrgEnv-cray >>> PrgEnv-gnu” . This will be necessary later. >>> >>> 1) Install the Anaconda python distribution >>> (https://store.continuum.io/cshop/anaconda/) for 64-bit Linux systems. >>> Refresh your path so the Anaconda python’s in your path. >>> >>> 2) Download mpi4py from https://bitbucket.org/mpi4py/mpi4py/, or grab it >>> from my home directory on Blue Waters (~bwoshea/mpi4py-1.3.1.tar.gz). >>> >>> 3) Once you’ve unpacked it in your home directory, edit the file >>> mpi.cfg, >>> which is in the top-level mpi4py directory, and add the following chunk >>> of >>> text at the end of the file: >>> >>> [cray] >>> mpicc = cc >>> mpicxx = CC >>> extra_link_args = -shared >>> >>> then, in the top-level mpi4py directory, type: >>> >>> python setup.py build --mpi=cray >>> python setp.py install >>> >>> At this point, you should have a functional version of mpi4py that uses >>> the >>> cray MPI libraries. >>> >>> 4) You can then install yt via source in your home directory: >>> >>> hg clone https://bitbucket.org/yt_analysis/yt >>> hg update yt >>> python setup.py develop >>> >>> And it should work, though you’re going to need to modify your >>> LD_LIBRARY_PATH to point to your anaconda /lib directory. You can verify >>> that yt and mpi4py are playing nicely together by running the scripts >>> hw_MPI.py and yt_parallel_test.py, both of which are sitting in my home >>> directory. (Note that you can do this via batch script or through the >>> interactive queue, both with aprun - you can’t just do it from the login >>> node.) >>> >>> 5) and, optionally, if you want to install Rockstar using yt from source, >>> you do the following: >>> >>> hg clone http://bitbucket.org/MatthewTurk/rockstar >>> cd rockstar >>> hg pull >>> hg up -C tip >>> make lib >>> >>> and *then* make a file called rockstar.cfg that has a single line of >>> text: >>> the path of the rockstar source code directory. At that point, you need >>> to >>> run “python setup.py develop” again, and yt will recognize that rockstar >>> has >>> been installed and play nicely with it. You will likely also have to >>> modify >>> your LD_LIBRARY_PATH to point at your rockstar directory so it can find >>> the >>> rockstar library (librockstar.so). >>> >>> >>> On Wed, Apr 20, 2016 at 5:04 PM, Jonah Miller >>> javascript:;> wrote: >>>> >>>> Hi yt users, >>>> >>>> Does anybody have experience installing the development version of yt on >>>> blue waters (as opposed to relying on the module pre-provided)? If so, >>>> do >>>> you have any tips or gotchas? I will be attempting an installation in >>>> the >>>> next few days and I'm sort of hoping it's a solved problem. >>>> >>>> Best, >>>> Jonah Miller >>>> _______________________________________________ >>>> yt-users mailing list >>>> yt-users@lists.spacepope.org javascript:; >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >>> >>> >>> _______________________________________________ >>> yt-users mailing list >>> yt-users@lists.spacepope.org javascript:; >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org javascript:; >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org javascript:; > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org javascript:; http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Ah, perfect. Thanks! Best, Jonah On 16-04-20 06:05 PM, Nathan Goldbaum wrote:

It can, but you'll need to edit the script to pull from your fork:

https://bitbucket.org/yt_analysis/yt/src/248ccc5ef8b46d83682344d249c8de1c964759ab/doc/install_script.sh?at=yt&fileviewer=file-view-default#install_script.sh-1459

On Wed, Apr 20, 2016 at 4:41 PM, Jonah Miller mailto:jonah.maxwell.miller@gmail.com> wrote:

Hi Nathan, Matt, Brian,

I see. Yes, I'll want to use the parallelism. Cool! I didn't realize the all-in-one script could pull from different repos. Thanks for your help!

Best, Jonah

On 16-04-20 05:39 PM, Nathan Goldbaum wrote:

...

You'll probably also want to follow Brian's instructions for mpi4py if you're going t be using yt's parallelism. The install script doesn't set that up since we can't assume people have an MPI installation.

On Wednesday, April 20, 2016, Matthew Turk mailto:matthewturk@gmail.com> wrote:

Hi Jonah,

Just make sure you use the from-source option, and you can pull whatever changesets you like into your source directory in your installation.

On Wed, Apr 20, 2016 at 4:34 PM, Jonah Miller wrote: > Hi Brian, Matt, > > Thanks a lot! These instructions are exactly what I was hoping for! > Regarding the all-in-one install script, that won't work if I want to > install a fork of yt though, will it? This is to test my carpet frontend on > some production data. > > Best, > Jonah > > > On 16-04-20 05:24 PM, Matthew Turk wrote: >> >> I think it's a lot easier now that the new install script has been merged >> in: >> >> >> http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2016-April/006350.... >> >> -Matt >> >> On Wed, Apr 20, 2016 at 4:22 PM, Brian O'Shea mailto:bwoshea@gmail.com> wrote: >>> >>> Yes indeed. It's a gigantic pain. Here are instructions. Good luck! >>> >>> ----------------- >>> >>> 0) Get the GNU software environment set up: “module swap PrgEnv-cray >>> PrgEnv-gnu” . This will be necessary later. >>> >>> 1) Install the Anaconda python distribution >>> (https://store.continuum.io/cshop/anaconda/) for 64-bit Linux systems. >>> Refresh your path so the Anaconda python’s in your path. >>> >>> 2) Download mpi4py from https://bitbucket.org/mpi4py/mpi4py/, or grab it >>> from my home directory on Blue Waters (~bwoshea/mpi4py-1.3.1.tar.gz). >>> >>> 3) Once you’ve unpacked it in your home directory, edit the file >>> mpi.cfg, >>> which is in the top-level mpi4py directory, and add the following chunk >>> of >>> text at the end of the file: >>> >>> [cray] >>> mpicc = cc >>> mpicxx = CC >>> extra_link_args = -shared >>> >>> then, in the top-level mpi4py directory, type: >>> >>> python setup.py build --mpi=cray >>> python setp.py install >>> >>> At this point, you should have a functional version of mpi4py that uses >>> the >>> cray MPI libraries. >>> >>> 4) You can then install yt via source in your home directory: >>> >>> hg clone https://bitbucket.org/yt_analysis/yt >>> hg update yt >>> python setup.py develop >>> >>> And it should work, though you’re going to need to modify your >>> LD_LIBRARY_PATH to point to your anaconda /lib directory. You can verify >>> that yt and mpi4py are playing nicely together by running the scripts >>> hw_MPI.py and yt_parallel_test.py, both of which are sitting in my home >>> directory. (Note that you can do this via batch script or through the >>> interactive queue, both with aprun - you can’t just do it from the login >>> node.) >>> >>> 5) and, optionally, if you want to install Rockstar using yt from source, >>> you do the following: >>> >>> hg clone http://bitbucket.org/MatthewTurk/rockstar >>> cd rockstar >>> hg pull >>> hg up -C tip >>> make lib >>> >>> and *then* make a file called rockstar.cfg that has a single line of >>> text: >>> the path of the rockstar source code directory. At that point, you need >>> to >>> run “python setup.py develop” again, and yt will recognize that rockstar >>> has >>> been installed and play nicely with it. You will likely also have to >>> modify >>> your LD_LIBRARY_PATH to point at your rockstar directory so it can find >>> the >>> rockstar library (librockstar.so). >>> >>> >>> On Wed, Apr 20, 2016 at 5:04 PM, Jonah Miller >>> wrote: >>>> >>>> Hi yt users, >>>> >>>> Does anybody have experience installing the development version of yt on >>>> blue waters (as opposed to relying on the module pre-provided)? If so, >>>> do >>>> you have any tips or gotchas? I will be attempting an installation in >>>> the >>>> next few days and I'm sort of hoping it's a solved problem. >>>> >>>> Best, >>>> Jonah Miller >>>> _______________________________________________ >>>> yt-users mailing list >>>> yt-users@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >>> >>> >>> _______________________________________________ >>> yt-users mailing list >>> yt-users@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org