Parallelizing a bunch of projections
Hi all, I know that parallelizing a single projection doesn’t really give you that much speed benefit. But what would happen if I tried to do something like this (please excuse the Python pseudocode): fields = [big list of fields I want to project] prj = ds.proj(“ones”, 2) frb = prj.to_frb(…) storage = {} for sto, fd in parallel_objects(fields, storage=storage): buf = frb[fd] sto.result_id = fd sto.result = buf I did something like this but it hung at the end, which makes me think that I’ve got some kind of blocking operation here that I’m not aware of. Best, John
On Thu Dec 04 2014 at 2:54:52 PM John ZuHone
Hi all,
I know that parallelizing a single projection doesn’t really give you that much speed benefit.
But what would happen if I tried to do something like this (please excuse the Python pseudocode):
fields = [big list of fields I want to project]
prj = ds.proj(“ones”, 2) frb = prj.to_frb(…)
storage = {} for sto, fd in parallel_objects(fields, storage=storage): buf = frb[fd]
sto.result_id = fd sto.result = buf
What happens if you do something like: storage = {} for sto, fd in parallel_objects(fields, storage=storage): prj = ds.proj(fd, 2) frb = prj.to_frb(…) buf = frb[fd] sto.result_id = fd sto.result = buf ?
I did something like this but it hung at the end, which makes me think that I’ve got some kind of blocking operation here that I’m not aware of.
Best,
John _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
On Thu, Dec 4, 2014 at 4:59 PM, Nathan Goldbaum
On Thu Dec 04 2014 at 2:54:52 PM John ZuHone
wrote: Hi all,
I know that parallelizing a single projection doesn’t really give you that much speed benefit.
But what would happen if I tried to do something like this (please excuse the Python pseudocode):
fields = [big list of fields I want to project]
prj = ds.proj(“ones”, 2) frb = prj.to_frb(…)
storage = {} for sto, fd in parallel_objects(fields, storage=storage): buf = frb[fd]
sto.result_id = fd sto.result = buf
What happens if you do something like:
storage = {} for sto, fd in parallel_objects(fields, storage=storage): prj = ds.proj(fd, 2) frb = prj.to_frb(…) buf = frb[fd]
sto.result_id = fd sto.result = buf
?
I did something like this but it hung at the end, which makes me think that I’ve got some kind of blocking operation here that I’m not aware of.
There's also a barrier option to parallel_objects, which defaults to True.
Best,
John _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Nathan, I actually was doing that before, and it did work. I was hoping that by creating just one projection and then querying the frb for fields that I would get a speed boost by not having to construct the projection object over and over again. Or is that not the case? John
On Dec 4, 2014, at 5:59 PM, Nathan Goldbaum
wrote: On Thu Dec 04 2014 at 2:54:52 PM John ZuHone
mailto:jzuhone@gmail.com> wrote: Hi all, I know that parallelizing a single projection doesn’t really give you that much speed benefit.
But what would happen if I tried to do something like this (please excuse the Python pseudocode):
fields = [big list of fields I want to project]
prj = ds.proj(“ones”, 2) frb = prj.to_frb(…)
storage = {} for sto, fd in parallel_objects(fields, storage=storage): buf = frb[fd]
sto.result_id = fd sto.result = buf
What happens if you do something like:
storage = {} for sto, fd in parallel_objects(fields, storage=storage): prj = ds.proj(fd, 2) frb = prj.to_frb(…) buf = frb[fd]
sto.result_id = fd sto.result = buf
?
I did something like this but it hung at the end, which makes me think that I’ve got some kind of blocking operation here that I’m not aware of.
Best,
John _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org mailto:yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
participants (3)
-
John ZuHone
-
Matthew Turk
-
Nathan Goldbaum