Hi Nathan, I actually was doing that before, and it did work. I was hoping that by creating just one projection and then querying the frb for fields that I would get a speed boost by not having to construct the projection object over and over again. Or is that not the case? John

On Dec 4, 2014, at 5:59 PM, Nathan Goldbaum wrote:

On Thu Dec 04 2014 at 2:54:52 PM John ZuHone mailto:jzuhone@gmail.com> wrote: Hi all,

I know that parallelizing a single projection doesn’t really give you that much speed benefit.

But what would happen if I tried to do something like this (please excuse the Python pseudocode):

fields = [big list of fields I want to project]

prj = ds.proj(“ones”, 2) frb = prj.to_frb(…)

storage = {} for sto, fd in parallel_objects(fields, storage=storage): buf = frb[fd]

sto.result_id = fd sto.result = buf

What happens if you do something like:

storage = {} for sto, fd in parallel_objects(fields, storage=storage): prj = ds.proj(fd, 2) frb = prj.to_frb(…) buf = frb[fd]

sto.result_id = fd sto.result = buf

?

I did something like this but it hung at the end, which makes me think that I’ve got some kind of blocking operation here that I’m not aware of.

Best,

John _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org