Hi Jared,

Just a comment that for these really big particle simulations the way yt currently supports particle data doesn't scale terribly well, so you might run into either issues with operations taking a very long time or using too much RAM.

You may find it useful to only load in data in a subset of the simulation domain to restrict the number of particles yt is dealing with at any one time. This can be controlled at a coarse level via the "bounding_box" keyword argument to the load() function, in particular by supplying a bounding box that only covers a portion of the domain. It might also help with RAM usage to supply n_ref=16 instead of the default (32). There's more detail about what these parameters do in the docs:

http://yt-project.org/docs/3.4.1/examining/loading_data.html#gadget-data

If you'd like you could also try the "sph-viz" branch on my fork of yt at https://github.com/ngoldbaum/yt. This branch contains an implementation of the next-generation support for particle data in yt that is still under development. This version will scale much better for a dataset as big as yours.

Best,

Nathan

On Tue, Feb 20, 2018 at 7:22 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Jared,

I know you can make the particle IDs work by specifying the dtype, but
my reading of the _validate_header code is the same as yours.  I think
it would be a pretty straightforward change to conditionally make the
'I' read into the struct into a 64-bit variable, but it's not there
now.

-Matt

On Tue, Feb 20, 2018 at 7:16 PM, Jared Coughlin
<Jared.W.Coughlin.29@nd.edu> wrote:
> Hello! I have a gadget snapshot file with 1024^3 particles. When I tried to
> load it with yt, yt fails by saying that it cannot identify the file type. I
> did some digging and found that the way yt validates a gadget snapshot is by
> reading the number of particles from the header and then comparing that
> number to the number derived from the size of the position block. This size
> is 1024^3 *3*4 (the number of particles times 3 positions times 4 bytes per
> position). However, as I just discovered, this number is larger than the max
> value of an int in c, and so gadget writes a value of 0 due to the overflow.
> As such, I need to use gadget's unsigned long long int for the padding, but
> this is 8 bytes, not 4. I was just wondering if yt had some ability to
> detect the need to use something larger than an int? If not, I can add it.
> Thanks!
> -Jared
>
> _______________________________________________
> yt-users mailing list -- yt-users@python.org
> To unsubscribe send an email to yt-users-leave@python.org
>
> _______________________________________________
> yt-users mailing list -- yt-users@python.org
> To unsubscribe send an email to yt-users-leave@python.org
>
_______________________________________________
yt-users mailing list -- yt-users@python.org
To unsubscribe send an email to yt-users-leave@python.org
_______________________________________________
yt-users mailing list -- yt-users@python.org
To unsubscribe send an email to yt-users-leave@python.org