Hi all,
Geoffery, Dave and I just discovered that in order to run yt on the login nodes on Kraken XT5, you need to run it with the same PrgEnv environment that you built it with. I guess this makes sense, but if you're like me and messing with this PrgEnv stuff (see my earlier post today), it comes as a shock when something that previously worked, doesn't.
Caveot emptor...
_______________________________________________________
sskory(a)physics.ucsd.edu o__ Stephen Skory
http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student
________________________________(_)_\(_)_______________
Hi all,
Along with a massive amount of help from Matt, I've put together detailed instructions on how build a version of Python that will run yt on the compute nodes on a Cray XT5 (Kraken, specifically). yt will run just fine with the standard install on the login nodes, but if you wish to do parallel analysis or be manganimous and run jobs in the batch queue, you need to run with a statically-linked version of Python with all the modules needed built-in. And that's where it becomes complicated:
http://yt.enzotools.org/wiki/CrayXT5Installation
Many thanks to Matt!
_______________________________________________________
sskory(a)physics.ucsd.edu o__ Stephen Skory
http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student
________________________________(_)_\(_)_______________
Here's how it works.
mpi4py is a module like any other. You build it with the python
installation that you built all the other modules with, ala python setup.py
build and install. In order for that to work, you need some mpi libraries
installed. As I said, I prefer openmpi for this because they were the
easiest for me to install and build mpi4py with. Before you do python build
install in the mpi4py directory, you'll need to edit the .cfg file (can't
remember exactly what it's called) so that the installation has the proper
paths to your mpi install.
When you've got mpi4py properly built, you will be able to run some yt
operations in parallel in the following manner.
1. Whatever you want to do needs to be in some python script. As far as I
know, you can't do parallel entering lines directly into the interpreter.
Here's an example:
### Start
from yt.mods import *
from yt.config import ytcfg
pf = EnzoStaticOutput("EnzoRuns/cool_core_rediculous/DD0252/DD0252")
pc = PlotCollection(pf,center=[0.5,0.5,0.5])
pc.add_projection("Density",0)
if ytcfg.getint("yt","__parallel_rank") == 0:
pc.save("DD0252")
### End
That if statement at the end assures that the final image save is done by
the root process only. The nice thing is this script can be run in exactly
the same form in serial, too.
2. Let's say this script is called proj.py. You'll run it like this:
mpirun -np 4 python proj.py --parallel
If you don't unclude the --parallel, you'll see 4 instances of your proj.py
script running separately, but each one doing the entire projection and not
working together.
Hope that helps,
Britton
On Fri, Feb 13, 2009 at 11:15 PM, rsoares <dlleuz(a)xmission.com> wrote:
> What Python do you parallelize to install mpi4py into - or do you build
> /use mpi4py without python, then how?
>
> R.Soares
>
> Britton Smith wrote:
>
>> I recommend using openmpi. I have been able to build openmpi on multiple
>> platforms and then build mpi4py with it without any customization. As Matt
>> has said, though, you won't see any benefit to using parallel until your
>> simulations are at least 256^3 cells or more.
>>
>> On Thu, Feb 12, 2009 at 8:16 PM, Matthew Turk <matthewturk(a)gmail.com<mailto:
>> matthewturk(a)gmail.com>> wrote:
>>
>> Hi again,
>>
>> I just realized that I should say a couple important caveats --
>>
>> 1. We haven't released 'yt-trunk' as 1.5 yet because it's not quite
>> done or stable. It's going well, and many people use it for
>> production-quality work, but it's not really stamped-and-completed.
>> 2. I should *also* note that you won't really get a lot out of
>> parallel yt unless you have relatively large datasets or relatively
>> large amounts of computation on each cell while creating a derived
>> field. It might end up being a bit more work than you're looking for,
>> if you just want to get some plots out quickly.
>>
>> -Matt
>>
>> On Thu, Feb 12, 2009 at 7:12 PM, Matthew Turk
>> <matthewturk(a)gmail.com <mailto:matthewturk@gmail.com>> wrote:
>> > Hi!
>> >
>> > yt-trunk is now parallelized. Not all tasks work in parallel, but
>> > projections, profiles (if done in 'lazy' mode) and halo finding (if
>> > you use the SS_HopOutput module) are now parallelized. Slices are
>> > almost done, and the new covering grid will be. It's not
>> documented,
>> > but those tasks should all run in parallel. We will be rolling
>> out a
>> > 1.5 release relatively soon, likely shortly after I defend my thesis
>> > in April, that will have documentation and so forth.
>> >
>> > I'm surprised you can't compile against the mpich libraries in a
>> > shared fashion. Unfortunately, I'm not an expert on MPI
>> > implementations, so I can't quite help out there. In my personal
>> > experience, using OpenMPI, I have needed to except when running on
>> > some form of linux without a loader -- the previous discussion about
>> > this was related to Kraken, which runs a Cray-specific form of linux
>> > called "Compute Node Linux." I don't actually know offhand (anybody
>> > else?) of any non-Cray machines at supercomputing out there require
>> > static linking as opposed to a standard installation of Python.
>> (I'm
>> > sure they do, I just don't know of them!)
>> >
>> > As for the second part, usually when instantiating you have to
>> run the
>> > executable via mpirun. (On other MPI implementations, this could be
>> > something different.) One option for this -- if you're running off
>> > trunk -- would be to do something like:
>> >
>> > mpirun -np 4 python my_script.py --parallel
>> >
>> > where the file my_script.py has something like:
>> >
>> > --
>> > from yt.mods import *
>> > pf = EnzoStaticOutput("my_output")
>> > pc = PlotCollection(pf, center=[0.5,0.5,0.5])
>> > pc.add_projection("Density",0)
>> > pc.save("hi_there")
>> > --
>> >
>> > The projection would be executed in parallel, in this case.
>> (There is
>> > a command line interface called 'yt' that also works in
>> parallel, but
>> > it's still a bit in flux.) You can't just run "python" because
>> of the
>> > way the stdin and stdout streams work; you have to supply a
>> script, so
>> > that it can proceed without input from the user. (IPython's
>> parallel
>> > fanciness notwithstanding, which we do not use in yt.)
>> >
>> > But, keep in mind, running "mpirun -np 4" by itself, wihtout setting
>> > up a means of distributing tasks (usually via a tasklist) will run
>> > them all on the current machine. I am, unfortunately, not really
>> > qualified to speak to setting up MPI implementations. But please do
>> > let us know if you have problems with the yt aspects of this!
>> >
>> > -Matt
>> >
>> > On Thu, Feb 12, 2009 at 6:59 PM, rsoares <dlleuz(a)xmission.com
>> <mailto:dlleuz@xmission.com>> wrote:
>> >> Hi,
>> >>
>> >> I'm trying to run mpi4py on my 4 machines, but I need a
>> parallelized version
>> >> of Python. Tried to compile one with Python 2.5 and mpich2 but
>> mpich2 won't
>> >> let me built dynamic /shares libraries which it needs. Trying
>> with the
>> >> static ones involves alot of headers errors from both.
>> >> Is yt-trunk capable of doing python in parallel?
>> >>
>> >> Without parallel-python, I mpdboot -n 4 then
>> >>
>> >> python
>> >>>>>import MPI
>> >>>>> rank, size = MPI.COMM_WORLD.rank, MPI.COMM_WORLD.size
>> >>>>> print 'Hello World! I am process', rank, 'of', size
>> >> Hello World! I am process 0 of 1
>> >>>>>
>> >>
>> >> not 4 processes, and mpirun -np 4 python just hangs. mpi4py
>> installed on
>> >> all 4 nodes.
>> >>
>> >> Thanks.
>> >>
>> >> R.Soares
>> >> _______________________________________________
>> >> yt-users mailing list
>> >> yt-users(a)lists.spacepope.org <mailto:yt-users@lists.spacepope.org>
>> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> >>
>> >
>> _______________________________________________
>> yt-users mailing list
>> yt-users(a)lists.spacepope.org <mailto:yt-users@lists.spacepope.org>
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users(a)lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>
Hi there,
yt needs to be installed on all of the compute nodes; additionally,
your script needs to be on all of them as well. This is usually
handled via a networked file system -- if yt is installed onto that
networked file system (just like you would for enzo) then it should
work.
The only component of yt that requires the GUI is the GUI, 'reason.'
All scripting tasks (unless you explicitly request 'reason'
components, which I don't think anyone would...) do not depend on
wxPython, GTK, etc etc.
-Matt
On Sat, Feb 14, 2009 at 11:21 AM, rsoares <dlleuz(a)xmission.com> wrote:
> Is yt installed on all the computer nodes that you run simulations on --
> does it have to be there for you to stream the data and apply the hop
> processing scripts, within yt from the computer you are viewing from.
>
> Probably a moot question as yt depends on some gui that the compute nodes
> don't have?
>
> Matthew Turk wrote:
>>
>> Britton's done a great job here, I just have one note -- inside the
>> actual save process, yt checks if it's on the root processor, so you
>> don't necessarily have to have the if statement here. It absolutely
>> won't hurt, however, to make the check.
>>
>> I should also note, all of this machinery is only in trunk right now.
>>
>> -Matt
>>
>> On Sat, Feb 14, 2009 at 8:03 AM, Britton Smith <brittonsmith(a)gmail.com>
>> wrote:
>>
>>>
>>> Here's how it works.
>>> mpi4py is a module like any other. You build it with the python
>>> installation that you built all the other modules with, ala python
>>> setup.py
>>> build and install. In order for that to work, you need some mpi
>>> libraries
>>> installed. As I said, I prefer openmpi for this because they were the
>>> easiest for me to install and build mpi4py with. Before you do python
>>> build
>>> install in the mpi4py directory, you'll need to edit the .cfg file (can't
>>> remember exactly what it's called) so that the installation has the
>>> proper
>>> paths to your mpi install.
>>>
>>> When you've got mpi4py properly built, you will be able to run some yt
>>> operations in parallel in the following manner.
>>> 1. Whatever you want to do needs to be in some python script. As far as
>>> I
>>> know, you can't do parallel entering lines directly into the interpreter.
>>> Here's an example:
>>>
>>> ### Start
>>> from yt.mods import *
>>> from yt.config import ytcfg
>>> pf = EnzoStaticOutput("EnzoRuns/cool_core_rediculous/DD0252/DD0252")
>>> pc = PlotCollection(pf,center=[0.5,0.5,0.5])
>>> pc.add_projection("Density",0)
>>>
>>> if ytcfg.getint("yt","__parallel_rank") == 0:
>>> pc.save("DD0252")
>>> ### End
>>> That if statement at the end assures that the final image save is done by
>>> the root process only. The nice thing is this script can be run in
>>> exactly
>>> the same form in serial, too.
>>>
>>> 2. Let's say this script is called proj.py. You'll run it like this:
>>> mpirun -np 4 python proj.py --parallel
>>>
>>> If you don't unclude the --parallel, you'll see 4 instances of your
>>> proj.py
>>> script running separately, but each one doing the entire projection and
>>> not
>>> working together.
>>>
>>> Hope that helps,
>>>
>>> Britton
>>>
>>> On Fri, Feb 13, 2009 at 11:15 PM, rsoares <dlleuz(a)xmission.com> wrote:
>>>
>>>>
>>>> What Python do you parallelize to install mpi4py into - or do you build
>>>> /use mpi4py without python, then how?
>>>>
>>>> R.Soares
>>>>
>>>> Britton Smith wrote:
>>>>
>>>>>
>>>>> I recommend using openmpi. I have been able to build openmpi on
>>>>> multiple
>>>>> platforms and then build mpi4py with it without any customization. As
>>>>> Matt
>>>>> has said, though, you won't see any benefit to using parallel until
>>>>> your
>>>>> simulations are at least 256^3 cells or more.
>>>>>
>>>>> On Thu, Feb 12, 2009 at 8:16 PM, Matthew Turk <matthewturk(a)gmail.com
>>>>> <mailto:matthewturk@gmail.com>> wrote:
>>>>>
>>>>> Hi again,
>>>>>
>>>>> I just realized that I should say a couple important caveats --
>>>>>
>>>>> 1. We haven't released 'yt-trunk' as 1.5 yet because it's not quite
>>>>> done or stable. It's going well, and many people use it for
>>>>> production-quality work, but it's not really stamped-and-completed.
>>>>> 2. I should *also* note that you won't really get a lot out of
>>>>> parallel yt unless you have relatively large datasets or relatively
>>>>> large amounts of computation on each cell while creating a derived
>>>>> field. It might end up being a bit more work than you're looking
>>>>> for,
>>>>> if you just want to get some plots out quickly.
>>>>>
>>>>> -Matt
>>>>>
>>>>> On Thu, Feb 12, 2009 at 7:12 PM, Matthew Turk
>>>>> <matthewturk(a)gmail.com <mailto:matthewturk@gmail.com>> wrote:
>>>>> > Hi!
>>>>> >
>>>>> > yt-trunk is now parallelized. Not all tasks work in parallel, but
>>>>> > projections, profiles (if done in 'lazy' mode) and halo finding (if
>>>>> > you use the SS_HopOutput module) are now parallelized. Slices are
>>>>> > almost done, and the new covering grid will be. It's not
>>>>> documented,
>>>>> > but those tasks should all run in parallel. We will be rolling
>>>>> out a
>>>>> > 1.5 release relatively soon, likely shortly after I defend my
>>>>> thesis
>>>>> > in April, that will have documentation and so forth.
>>>>> >
>>>>> > I'm surprised you can't compile against the mpich libraries in a
>>>>> > shared fashion. Unfortunately, I'm not an expert on MPI
>>>>> > implementations, so I can't quite help out there. In my personal
>>>>> > experience, using OpenMPI, I have needed to except when running on
>>>>> > some form of linux without a loader -- the previous discussion
>>>>> about
>>>>> > this was related to Kraken, which runs a Cray-specific form of
>>>>> linux
>>>>> > called "Compute Node Linux." I don't actually know offhand
>>>>> (anybody
>>>>> > else?) of any non-Cray machines at supercomputing out there require
>>>>> > static linking as opposed to a standard installation of Python.
>>>>> (I'm
>>>>> > sure they do, I just don't know of them!)
>>>>> >
>>>>> > As for the second part, usually when instantiating you have to
>>>>> run the
>>>>> > executable via mpirun. (On other MPI implementations, this could
>>>>> be
>>>>> > something different.) One option for this -- if you're running off
>>>>> > trunk -- would be to do something like:
>>>>> >
>>>>> > mpirun -np 4 python my_script.py --parallel
>>>>> >
>>>>> > where the file my_script.py has something like:
>>>>> >
>>>>> > --
>>>>> > from yt.mods import *
>>>>> > pf = EnzoStaticOutput("my_output")
>>>>> > pc = PlotCollection(pf, center=[0.5,0.5,0.5])
>>>>> > pc.add_projection("Density",0)
>>>>> > pc.save("hi_there")
>>>>> > --
>>>>> >
>>>>> > The projection would be executed in parallel, in this case.
>>>>> (There is
>>>>> > a command line interface called 'yt' that also works in
>>>>> parallel, but
>>>>> > it's still a bit in flux.) You can't just run "python" because
>>>>> of the
>>>>> > way the stdin and stdout streams work; you have to supply a
>>>>> script, so
>>>>> > that it can proceed without input from the user. (IPython's
>>>>> parallel
>>>>> > fanciness notwithstanding, which we do not use in yt.)
>>>>> >
>>>>> > But, keep in mind, running "mpirun -np 4" by itself, wihtout
>>>>> setting
>>>>> > up a means of distributing tasks (usually via a tasklist) will run
>>>>> > them all on the current machine. I am, unfortunately, not really
>>>>> > qualified to speak to setting up MPI implementations. But please
>>>>> do
>>>>> > let us know if you have problems with the yt aspects of this!
>>>>> >
>>>>> > -Matt
>>>>> >
>>>>> > On Thu, Feb 12, 2009 at 6:59 PM, rsoares <dlleuz(a)xmission.com
>>>>> <mailto:dlleuz@xmission.com>> wrote:
>>>>> >> Hi,
>>>>> >>
>>>>> >> I'm trying to run mpi4py on my 4 machines, but I need a
>>>>> parallelized version
>>>>> >> of Python. Tried to compile one with Python 2.5 and mpich2 but
>>>>> mpich2 won't
>>>>> >> let me built dynamic /shares libraries which it needs. Trying
>>>>> with the
>>>>> >> static ones involves alot of headers errors from both.
>>>>> >> Is yt-trunk capable of doing python in parallel?
>>>>> >>
>>>>> >> Without parallel-python, I mpdboot -n 4 then
>>>>> >>
>>>>> >> python
>>>>> >>>>>import MPI
>>>>> >>>>> rank, size = MPI.COMM_WORLD.rank, MPI.COMM_WORLD.size
>>>>> >>>>> print 'Hello World! I am process', rank, 'of', size
>>>>> >> Hello World! I am process 0 of 1
>>>>> >>>>>
>>>>> >>
>>>>> >> not 4 processes, and mpirun -np 4 python just hangs. mpi4py
>>>>> installed on
>>>>> >> all 4 nodes.
>>>>> >>
>>>>> >> Thanks.
>>>>> >>
>>>>> >> R.Soares
>>>>> >> _______________________________________________
>>>>> >> yt-users mailing list
>>>>> >> yt-users(a)lists.spacepope.org <mailto:yt-users@lists.spacepope.org>
>>>>> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>> >>
>>>>> >
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users(a)lists.spacepope.org <mailto:yt-users@lists.spacepope.org>
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users(a)lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>
>>>>>
>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users(a)lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>>
>>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users(a)lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>
Matt,
I have a script that runs parallel hop, and then writes out the results to an HDF5 file. I'd like to more efficiently parallelize a certain part of the process, where I retrieve and store the particle particulars, but it doesn't seem to work the way I'd like. This first example works, where I get the various values on all processors, and then store it only on the one that owns the halo:
http://paste.enzotools.org/show/43/
But this one doesn't work, where I only retrieve the values if the proc owns the halo. And by not working, I mean it hangs on halo 0 and doesn't finish after 10 minutes, while the script above takes less than a minute:
http://paste.enzotools.org/show/44/
Can you give me any ideas? In particular, the part of the script where it retrieves the particle_indexes is going serially in the functioning script (I can watch it in the output log) and it's taking far longer than it should. Thanks!
_______________________________________________________
sskory(a)physics.ucsd.edu o__ Stephen Skory
http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student
________________________________(_)_\(_)_______________
Hi,
I'm trying to run mpi4py on my 4 machines, but I need a parallelized
version of Python. Tried to compile one with Python 2.5 and mpich2 but
mpich2 won't let me built dynamic /shares libraries which it needs.
Trying with the static ones involves alot of headers errors from both.
Is yt-trunk capable of doing python in parallel?
Without parallel-python, I mpdboot -n 4 then
python
>>>import MPI
>>> rank, size = MPI.COMM_WORLD.rank, MPI.COMM_WORLD.size
>>> print 'Hello World! I am process', rank, 'of', size
Hello World! I am process 0 of 1
>>>
not 4 processes, and mpirun -np 4 python just hangs. mpi4py installed
on all 4 nodes.
Thanks.
R.Soares
Matt,
> I would say before you try downgrading, that you compile without the
> "--without-threads" option.
I've made some progress using this. I can get numpy and matplotlib to build statically with Python. Now pytables is giving me trouble, see below. I'll take another whack at this stuff tomorrow, but I'm wondering if you have any bright ideas?
sskory@krakenpf14(XT5):~/yt/src/hdf5-1.6.8> python
Python 2.6.1 (r261:67515, Feb 12 2009, 19:41:28)
[GCC 4.2.0 20070514 (quadcore:rpm:113)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
File "/etc/pythonstart", line 7, in <module>
import readline
ImportError: No module named readline
>>> import tables
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/nics/c/home/sskory/yt/lib/python2.6/site-packages/tables/__init__.py", line 56, in <module>
from tables.utilsExtension import getPyTablesVersion, getHDF5Version
ImportError: dynamic module does not define init function (initutilsExtension)
_______________________________________________________
sskory(a)physics.ucsd.edu o__ Stephen Skory
http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student
________________________________(_)_\(_)_______________
I've put the output of my best attempt here, which uses gcc and catamount, but
I'm getting a shared object error:
http://paste.enzotools.org/show/41/
any ideas?
_______________________________________________________
sskory(a)physics.ucsd.edu o__ Stephen Skory
http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student
________________________________(_)_\(_)_______________
Hi all,
I have problems in using yt with the dm-only output of enzo-1.5. If I
run e.g. the hop recipe, than it fails already at:
full_sphere = pf.h.sphere([0.5,0.5,0.5], 1.0)
Using the same script on dm+hydro output works without any problems, so
it seems that yt is confused by the output format.
Does anyone know how to use yt with the dm-only output.
Thanks,
Jean-Claude
--
Jean-Claude Waizmann
Institut für Theoretische Astrophysik (ITA/ZAH)
Albert-Ueberle-Strasse 2
69120 Heidelberg, Germany
Phone: +49-6221-54-8987
Fax: +49-6221-54-4221
E-mail:
waizmann(a)ita.uni-heidelberg.de
Hi everyone,
Okay, the results of the short, two-question survey are in: almost
everybody that uses a Mac uses EPD, everybody else works on remote
systems, and nobody uses funky-shaped/aligned covering grids. This
has two upshots:
1. the GUI may start including components that utilize TraitsUI or
TVTK, both of which are Enthought packages. The first will speed
development time, make for easier expansion of featuresets, and enable
better maintenance of the GUI. The second will allow me to wrap into
the main GUI the VTK stuff that currently sits inside VTKInterface and
that is currently standalone. (It's kind of simple so far; movable
planes, transparent contours and camera path design.) TraitsUI would
likely be wrapped into the installation script with wxPython, but TVTK
would not be.
2. the covering grid will be rewritten to use integer math. This
will speed thing up not just because of the math, but because of the
greater number of assumptions that can be made, and thus the
simplified algorithm for filling in the covering grid.
Thanks, everybody!
Now, for the second part. As some of you may have realized, the
documentation could be improved. I'd like to put out a public call
for feedback. If you could answer (off-list, private to me) the
following three questions, that would be great:
1. Where would you like documentation efforts to be focused: a)
recipes/scripts, b) object model description or c) tutorials?
2. Would you prefer this documentation be on a) the Wiki (
http://yt.enzotools.org/wiki/ ) or b) in the yt source distribution (
http://yt.enzotools.org/doc/ )
3. Do you have any suggestions?
(your response can be as simple a response as: ' a, b, no ' if you like)
Thanks, and I apologize for filling up everyone's inboxes. The 1.5
release is not terribly far away, but an important component of that
is going to be documentation, and it'd be helpful to know where to
focus our efforts.
-Matt
