Hi Matt,

Sure, I'm doing the following with the rockstar_halos data on yt-project.org/data:

pf = load("rockstar_halos/halos_0.0.bin")

dd = pf.all_data()

print dd["particle_mass"].size

for chunk in dd.chunks([], "io"):

print chunk["particle_mass"].size

This dataset has two files, but running this, I only get one chunk with the same size as the all_data container.

Britton

On Mon, May 12, 2014 at 2:19 PM, Matthew Turk <matthewturk@gmail.com> wrote:

Hi Britton,

On Mon, May 12, 2014 at 2:16 PM, Britton Smith <brittonsmith@gmail.com> wrote:
> Hey all,
>
> I'm working on changing how HaloCatalog objects loop over halos from a model
> where every processor has to hold the entire halo list to a model in which
> we loop over "io" chunks. This will make the HaloCatalog scale much better
> to extremely large catalogs. My understanding was that an "io" chunk was
> essentially one file on disk. However, when I try to get io chunks from any
> one of the various halo catalog frontends, there seems to only ever be a
> single chunk that contains everything, regardless of how many files on disk
> the data is spread over. Is there a way to change this so that an io chunk
> represents the data from a single file?

That sounds like a bug; IO should be single file at a time. Can you
show me how you're checking?

-Matt

>
> Britton
>
> _______________________________________________
> yt-dev mailing list
> yt-dev@lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
_______________________________________________
yt-dev mailing list
yt-dev@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org