Yes, apologies, I got that backwards. A larger n_ref means there needs be more particles per octree leaf zone to trigger refinement on that zone, so the you end up with fewer octree leaf nodes over all and you need less memory.

-Nathan

On Tue, Feb 20, 2018 at 7:53 PM, Desika Narayanan <desika.narayanan@gmail.com> wrote:
quick note -- set n_ref=64 or 128 (or bigger than 32) to reduce RAM footprint (instead of 16).   

-d

On Tue, Feb 20, 2018 at 8:37 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi Jared,

Just a comment that for these really big particle simulations the way yt currently supports particle data doesn't scale terribly well, so you might run into either issues with operations taking a very long time or using too much RAM.

You may find it useful to only load in data in a subset of the simulation domain to restrict the number of particles yt is dealing with at any one time. This can be controlled at a coarse level via the "bounding_box" keyword argument to the load() function, in particular by supplying a bounding box that only covers a portion of the domain. It might also help with RAM usage to supply n_ref=16 instead of the default (32). There's more detail about what these parameters do in the docs:

http://yt-project.org/docs/3.4.1/examining/loading_data.html#gadget-data

If you'd like you could also try the "sph-viz" branch on my fork of yt at https://github.com/ngoldbaum/yt. This branch contains an implementation of the next-generation support for particle data in yt that is still under development. This version will scale much better for a dataset as big as yours.

Best,

Nathan

On Tue, Feb 20, 2018 at 7:22 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Jared,

I know you can make the particle IDs work by specifying the dtype, but
my reading of the _validate_header code is the same as yours.  I think
it would be a pretty straightforward change to conditionally make the
'I' read into the struct into a 64-bit variable, but it's not there
now.

-Matt

On Tue, Feb 20, 2018 at 7:16 PM, Jared Coughlin
<Jared.W.Coughlin.29@nd.edu> wrote:
> Hello! I have a gadget snapshot file with 1024^3 particles. When I tried to
> load it with yt, yt fails by saying that it cannot identify the file type. I
> did some digging and found that the way yt validates a gadget snapshot is by
> reading the number of particles from the header and then comparing that
> number to the number derived from the size of the position block. This size
> is 1024^3 *3*4 (the number of particles times 3 positions times 4 bytes per
> position). However, as I just discovered, this number is larger than the max
> value of an int in c, and so gadget writes a value of 0 due to the overflow.
> As such, I need to use gadget's unsigned long long int for the padding, but
> this is 8 bytes, not 4. I was just wondering if yt had some ability to
> detect the need to use something larger than an int? If not, I can add it.
> Thanks!
> -Jared
>
> _______________________________________________
> yt-users mailing list -- yt-users@python.org
> To unsubscribe send an email to yt-users-leave@python.org
>
> _______________________________________________
> yt-users mailing list -- yt-users@python.org
> To unsubscribe send an email to yt-users-leave@python.org
>
_______________________________________________
yt-users mailing list -- yt-users@python.org
To unsubscribe send an email to yt-users-leave@python.org
_______________________________________________
yt-users mailing list -- yt-users@python.org
To unsubscribe send an email to yt-users-leave@python.org


_______________________________________________
yt-users mailing list -- yt-users@python.org
To unsubscribe send an email to yt-users-leave@python.org

_______________________________________________
yt-users mailing list -- yt-users@python.org
To unsubscribe send an email to yt-users-leave@python.org



_______________________________________________
yt-users mailing list -- yt-users@python.org
To unsubscribe send an email to yt-users-leave@python.org

_______________________________________________
yt-users mailing list -- yt-users@python.org
To unsubscribe send an email to yt-users-leave@python.org