Hi Britton, all,
That took care of my issues, now it’s run successfully. Just one more question: as I’m using RAMSES I create a DatasetSeries using:
es = yt.load('../output_*/info_*.txt’)
rh = RockstarHaloFinder(es, num_readers=1, num_writers=2)
Is there a way for me to add a dark matter only filter as done here for an enzo dataset:
# create a particle filter to remove star particles
@yt.particle_filter("dark_matter", requires=["creation_time"])
def _dm_filter(pfilter, data):
return data["creation_time"] < 0.0
def setup_ds(ds):
ds.add_particle_filter("dark_matter”)
es = yt.simulation("enzo_tiny_cosmology/32Mpc_32.enzo", "Enzo")
es.get_time_series(setup_function=setup_ds, redshift_data=False)
rh = RockstarHaloFinder(es, num_readers=1, num_writers=2,
particle_type="dark_matter")
I’ve looked at the docs on particle filtering but I can’t add filters to the DatasetSeries object I have.
Many thanks for the help so far!
David
On 12 Aug 2014, at 12:44, yt-users-request(a)lists.spacepope.org wrote:
> Hi Cameron, David,
>
> Cameron, you are correct, on Mac systems you need to set DYLD_LIBRARY_PATH
> instead of LD_LIBRARY_PATH. The fact that this causes vi and other things
> to fail is apparently an OS X bug (I found some discussions of it on the
> internet) and has nothing to do with yt. I went through a similar thing
> and had to resort to having a separate terminal for running rockstar that I
> didn't try to do anything else in. For what it's worth, this behavior
> seems to have been fixed. I am on OS X 10.8.5 and no longer have the issue
> of things crashing in a terminal where I have edited DYLD_LIBRARY_PATH.
>
> On the second issue that you both reported, the problem is the way in which
> one tells yt to run in parallel depends on how you import yt.
>
> If you do "from yt.mods import *", the parallelism is enabled by adding the
> --parallel flag on the command line. However, if you do "import yt", then
> parallelism is enabled by doing "yt.enable_parallelism()" directly below
> the import statement within your script.
>
> I looked at the latest documentation on parallelism (
> http://yt-project.org/doc/analyzing/parallel_computation.html) and is
> discusses the use of yt.enable_parallelism(), but not the old way that
> works with "from yt.mods import *". There should probably be a short
> discussion added of how this has changed. Additionally, let's make sure
> that the yt-rockstar docs get any needed additions there as well.
>
> Britton
Hi all,
I’ve been trying to build yt and mpi4py on the cray machines at oak ridge and came across this old thread in the mailing list:
http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2013-December/0…
I was just wondering if anyone had managed this since? If so, what the best approach is (system python or install script)?
Many thanks,
David
Hi all,
I also get the same error as Cameron when trying to run rockstar with mpi:
http://paste.yt-project.org/show/5019/
I’m using a ramses dataset (not sure if this is supported, but couldn’t see any clear evidence that it isn’t) and the following script:
mpirun -np 4 --mca btl ^openib python run_rockstar.py --parallel
http://paste.yt-project.org/show/5020/
I followed the same install instructions as Cameron, but I’m using a Linux cluster running CentOS. Appreciate any help!
Cheers,
David
On 12 Aug 2014, at 10:04, yt-users-request(a)lists.spacepope.org wrote:
> Message: 3
> Date: Tue, 12 Aug 2014 02:04:10 -0700
> From: Sam Skillman <samskillman(a)gmail.com>
> To: Discussion of the yt analysis package
> <yt-users(a)lists.spacepope.org>
> Subject: Re: [yt-users] Problems using Rockstar in yt
> Message-ID:
> <CAB4=1s1oHK0uPFo88rkSaEa-W6EASMLoyOiEFeniV-xSm6SPaQ(a)mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Cameron,
>
> Are you running in MPI? You need to run with at least 3 MPI tasks for
> Rockstar to work properly.
>
> Sam
>
>
> On Mon, Aug 11, 2014 at 6:41 PM, Cameron Hummels <chummels(a)gmail.com> wrote:
>
>> Hello,
>>
>> I'm trying to use Rockstar with the new HaloCatalog functionality in yt
>> 3.0, but I'm encountering some problems. For reference, this is an OS X
>> box with the "yt" branch. I've tested it on two OS X systems to the same
>> result.
>>
>> I modified my install_script.py to have it install Rockstar as described
>> here:
>>
>>
>> http://yt-project.org/docs/dev/analyzing/analysis_modules/halo_finders.html…
>>
>> Then, after rerunning my install script for the `yt` branch (ie yt-3.1
>> dev) I tried to run the script provided here in order to generate a halo
>> catalog using HOP:
>>
>>
>> http://yt-project.org/docs/dev/analyzing/analysis_modules/halo_transition.h…
>>
>> It can create the catalog fine with finder_method='hop', but when I change
>> it to finder_method='rockstar', I get failures. This is the traceback:
>>
>> http://paste.yt-project.org/show/TE2DW8gc50urchv4JnUk
>>
>> In the traceback, it claims that it cannot access librockstar.so, but this
>> library exists in the yt-x86_64/lib directory, which is explicitly in from
>> my $LD_LIBRARY_PATH. I can seemingly get it to work, if I add that path to
>> my $DYLD_LIBRARY_PATH environment variable, but then a bunch of other
>> things break (like vi). Not sure what to do here.
>>
>> But assuming there is no vi problem here, even then, when I run the script
>> above, I get failures. Here is the traceback:
>>
>> http://paste.yt-project.org/show/rCOT1eVT1szR1T6DEUDb
>>
>> I'm not sure what to do here. I've also tried running rockstar external
>> to yt on an enzo dataset, but it doesn't appear that the version we install
>> as part of yt allows it to work on HDF5 datasets.
>>
>> Has anyone run into any of these problems before? I'm at a bit of a loss
>> as to how to proceed on any of these fronts.
>>
>> Cameron
>>
>>
>> --
>> Cameron Hummels
>> Postdoctoral Researcher
>> Steward Observatory
>> University of Arizona
>> http://chummels.org
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users(a)lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
Hi yt-er's,
I just wanted to email the list and let everyone know about a new package I've developed called YT (yes, in caps), which as of today has its first release as v0.1.0.
YT is a wrapper for yt for the Julia programming language. Julia is a new(-ish) high-level, high-performance dynamic programming language for technical computing. If you want to learn more about Julia, visit http://julialang.org. One of the most important things to note about Julia is that it has a "just-in-time" compiler, so it can be very fast (within a factor of few of C speed in some cases). There is also a lot of interoperability between Julia and Python, including wrappers for Matplotlib and Sympy, and even an extension of the IPython (soon to be Jupyter) notebook!
A short summary of what YT exposes from yt in Julia:
* Datasets
* Some data objects (e.g., spheres, rectangular regions, slices, projections, profiles, etc.)
* YTArrays and YTQuantities
* Simple visualization tools (e.g., SlicePlot, ProjectionPlot, FixedResolutionBuffer)
To find out more about the YT package, visit the documentation:
http://www.jzuhone.com/yt_julia
or the GitHub project page:
http://github.com/jzuhone/YT.jl
Thanks for reading, and sorry for the spam.
Best,
John Z
Hello,
I'm trying to use Rockstar with the new HaloCatalog functionality in yt
3.0, but I'm encountering some problems. For reference, this is an OS X
box with the "yt" branch. I've tested it on two OS X systems to the same
result.
I modified my install_script.py to have it install Rockstar as described
here:
http://yt-project.org/docs/dev/analyzing/analysis_modules/halo_finders.html…
Then, after rerunning my install script for the `yt` branch (ie yt-3.1 dev)
I tried to run the script provided here in order to generate a halo catalog
using HOP:
http://yt-project.org/docs/dev/analyzing/analysis_modules/halo_transition.h…
It can create the catalog fine with finder_method='hop', but when I change
it to finder_method='rockstar', I get failures. This is the traceback:
http://paste.yt-project.org/show/TE2DW8gc50urchv4JnUk
In the traceback, it claims that it cannot access librockstar.so, but this
library exists in the yt-x86_64/lib directory, which is explicitly in from
my $LD_LIBRARY_PATH. I can seemingly get it to work, if I add that path to
my $DYLD_LIBRARY_PATH environment variable, but then a bunch of other
things break (like vi). Not sure what to do here.
But assuming there is no vi problem here, even then, when I run the script
above, I get failures. Here is the traceback:
http://paste.yt-project.org/show/rCOT1eVT1szR1T6DEUDb
I'm not sure what to do here. I've also tried running rockstar external to
yt on an enzo dataset, but it doesn't appear that the version we install as
part of yt allows it to work on HDF5 datasets.
Has anyone run into any of these problems before? I'm at a bit of a loss
as to how to proceed on any of these fronts.
Cameron
--
Cameron Hummels
Postdoctoral Researcher
Steward Observatory
University of Arizona
http://chummels.org
====================================================
DistArray 0.5 release
====================================================
**Documentation:** http://distarray.readthedocs.org
**License:** Three-clause BSD
**Python versions:** 2.7, 3.3, and 3.4
**OS support:** \*nix and Mac OS X
What is DistArray?
------------------
DistArray aims to bring the ease-of-use of NumPy to data-parallel
high-performance computing. It provides distributed multi-dimensional NumPy
arrays, distributed ufuncs, and distributed IO capabilities. It can
efficiently interoperate with external distributed libraries like Trilinos.
DistArray works with NumPy and builds on top of it in a flexible and natural
way.
0.5 Release
-----------
Noteworthy improvements in this release include:
* closer alignment with NumPy's API,
* support for Python 3.4 (existing support for Python 2.7 and 3.3),
* a performance-oriented MPI-only mode for deployment on clusters and
supercomputers,
* a way to register user-defined functions to be callable locally on worker
processes,
* more consistent naming of sub-packages,
* testing with MPICH2 (already tested against OpenMPI),
* improved and expanded examples,
* installed version testable via ``distarray.test()``, and
* performance and scaling improvements.
With this release, DistArray ready for real-world testing and deployment.
The
project is still evolving rapidly and we appreciate the continued input from
the larger scientific-Python community.
Existing features
-----------------
DistArray:
* supports NumPy-like slicing, reductions, and ufuncs on distributed
multidimensional arrays;
* has a client-engine process design -- data resides on the worker
processes,
commands are initiated from master;
* allows full control over what is executed on the worker processes and
integrates transparently with the master process;
* allows direct communication between workers, bypassing the master process
for scalability;
* integrates with IPython.parallel for interactive creation and exploration
of
distributed data;
* supports distributed ufuncs (currently without broadcasting);
* builds on and leverages MPI via MPI4Py in a transparent and user-friendly
way;
* has basic support for unstructured arrays;
* supports user-controllable array distributions across workers (block,
cyclic, block-cyclic, and unstructured) on a per-axis basis;
* has a straightforward API to control how an array is distributed;
* has basic plotting support for visualization of array distributions;
* separates the array’s distribution from the array’s data -- useful for
slicing, reductions, redistribution, broadcasting, and other operations;
* implements distributed random arrays;
* supports ``.npy``-like flat-file IO and hdf5 parallel IO (via ``h5py``);
leverages MPI-based IO parallelism in an easy-to-use and transparent way;
and
* supports the distributed array protocol [protocol]_, which allows
independently developed parallel libraries to share distributed arrays
without copying, analogous to the PEP-3118 new buffer protocol.
Planned features and roadmap
----------------------------
Near-term features and improvements include:
* array re-distribution capabilities;
* lazy evaluation and deferred computation for latency hiding;
* interoperation with Trilinos [Trilinos]_; and
* distributed broadcasting support.
The longer-term roadmap includes:
* Integration with other packages [petsc]_ that subscribe to the distributed
array protocol [protocol]_;
* Distributed fancy indexing;
* Out-of-core computations;
* Support for distributed sorting and other non-trivial distributed
algorithms; and
* End-user control over communication and temporary array creation, and
other
performance aspects of distributed computations.
History and funding
-------------------
Brian Granger started DistArray as a NASA-funded SBIR project in 2008.
Enthought picked it up as part of a DOE Phase II SBIR [SBIR]_ to provide a
generally useful distributed array package. It builds on NumPy, MPI,
MPI4Py,
IPython, IPython.parallel, and interfaces with the Trilinos suite of
distributed HPC solvers (via PyTrilinos [Trilinos]_).
This material is based upon work supported by the Department of Energy under
Award Number DE-SC0007699.
This report was prepared as an account of work sponsored by an agency of the
United States Government. Neither the United States Government nor any
agency
thereof, nor any of their employees, makes any warranty, express or implied,
or assumes any legal liability or responsibility for the accuracy,
completeness, or usefulness of any information, apparatus, product, or
process
disclosed, or represents that its use would not infringe privately owned
rights. Reference herein to any specific commercial product, process, or
service by trade name, trademark, manufacturer, or otherwise does not
necessarily constitute or imply its endorsement, recommendation, or favoring
by the United States Government or any agency thereof. The views and
opinions
of authors expressed herein do not necessarily state or reflect those of the
United States Government or any agency thereof.
.. [protocol]
http://distributed-array-protocol.readthedocs.org/en/rel-0.10.0/
.. [Trilinos] http://trilinos.org/
.. [petsc] http://www.mcs.anl.gov/petsc/
.. [SBIR] http://www.sbir.gov/sbirsearch/detail/410257
--
Kurt W. Smith, Ph.D. <ksmith(a)enthought.com>
Enthought, Inc. <http://www.enthought.com> | 512.536.1057
Hello! I have a vanilla gadget snapshot (one from the lcdm sample run that
the code comes with) that I was able to plot by following the instructions
given here:
http://yt-project.org/docs/3.0/cookbook/tipsy_notebook.html#tipsy-notebook
I then moved on to the 'Making Smoothed Images' section of that page and
tried the SlicePlot command. However, I get the following error:
IndexError: index 32767 is out of bounds for axis 0 with size 32767
Basically, I'm not really sure what that means. It seems like it's
overstepping the bounds of an array, as, for a zero-indexed array of size
32767, index 32767 would be out of bounds by one. I just wanted to see if
anyone else has had this problem, and if so, how they fixed it? Thank you
very much!
-Jared
Dear All,
My yt version is
---
Version = 2.6.1
Changeset = c7bae05dd4ef
---
If I update it to 3.0 with the command: yt update --all , will my old
analysis scripts(written for 2.6 ) works?
Thank you.
--
Reju Sam John
Hi folks,
I'm going back to resimulate some cosmological simulations with a
nested-grid setup, and can't seem to figure out how to get yt to tell me
where all of the dark matter particles in a given halo start out at high
redshift. It seems that something like this used to exist in the Hub
(i.e.,
http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2011-August/001…),
but that is not around any more. Would somebody mind sharing such a tool
if they have already coded it up?
Thanks!
--Brian
Hi all,
I am just worried about the problem associated to the density gradient
field in yt-3.0.
When I was trying to get the gradient of density field along z (actually
any) direction, I got this error message (last few lines) when calling
data[ 'density_gradient_z' ]:
~/lilew/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in
selector(self)
529 if sclass is None:
530 raise YTDataSelectorNotImplemented(self._type_name)
--> 531 self._selector = sclass(self)
532 return self._selector
533
~/lilew/yt-x86_64/src/yt-hg/yt/geometry/selection_routines.so in
yt.geometry.selection_routines.RegionSelector.__init__
(yt/geometry/selection_routines.c:20276)()
RuntimeError: Error: bad Region in non-periodic domain along
dimension 0. Region left edge = -0.01171875 code_length, Region
right edge = 0.13671875 code_lengthDataset left edge = 0.
0 code_length, Dataset right edge = 1.0 code_length
Similar things happen for the pressure gradient.
The data input was generated by Enzo with AMR. In comparison, it works
well with yt-2.6, where I refer to the gradient by data[ 'gradDensityZ' ].
Any ideas are appreciated. Thanks!
Cheers,
Lile
-----------------------------------------------------------
Lile 'Patrick' Wang
Dept. of Astroph. Sci., Princeton University, NJ 08544 USA
Website : http://www.astro.princeton.edu/~lilew/
<http://www.astro.princeton.edu/%7Elilew/>
Tel (Mobile): +1-(734)-834-3936
Tel (GVoice): +1-(510)-926-6831
-----------------------------------------------------------