Dear Britton, I tried something like that, but clearly I'm not doing it correctly. I'm noticing two errors, 1) everything is reporting running on the root proc (with a "Yo I'm root!") and 2) I'm getting four copies of my_storage back. Here's the code: ######################################################################### from matplotlib import rc import matplotlib.pyplot as plt import yt from yt import derived_field import numpy as np import yt.units as u import glob # Use YT in parallel. yt.enable_parallelism() # Number of procs on the machine num_procs=4 @derived_field(name = "total_KE", units = "erg") def _KE(field, data): return np.sum(data["kinetic_energy"].in_mks()*data["cell_volume"].in_cgs()) @derived_field(name = "total_PE", units = "erg") def _PE(field, data): return 0.5*np.sum(data["gpot"].in_mks()*data["cell_mass"].in_cgs()) @derived_field(name = "total_Eint", units = "erg") def _Eint(field, data): return np.sum(data["eint"]*data["cell_mass"]).in_cgs() @derived_field(name = "total_energy", units = "erg") def _Etotal(field, data): return data["total_Eint"] + data["total_PE"] + data["total_KE"] @derived_field(name = "total_px", units = "g*cm/s") def _px(field, data): return np.sum(data["cell_mass"]*data["velx"]).in_cgs() # Use glob to gather up the data files. Note here we use plt # and particle files because they are much smaller than # checkpoint files. plot = glob.glob("./SinkMomTest/*plt_cnt_000?") plot.sort() part = glob.glob("./SinkMomTest/*part_000?") part.sort() # Make a list of variables you are collecting to plot/store as # dictionaries. my_storage = {} # Now the parallel loop. i = 0 for sto,p in yt.parallel_objects(plot, num_procs, storage = my_storage): ds = yt.load(plot[i], particle_filename=part[i]) dd = ds.all_data() if (yt.is_root()): print "Yo I'm root!" else: print "Not root!" energy = dd["total_energy"] mom = dd["total_px"] sto.result_id = p[-4:] sto.result = [energy, mom] i=i+1 print "Done!" print my_storage ######################################################################### Is this what you mean? Kindly, Josh On Sun, May 24, 2015 at 12:19 PM Britton Smith <brittonsmith@gmail.com> wrote:

Hi Josh,

The reason that's not working is that there is still only one storage item (the sto variable) per iteration of the loop. You could keep both things if you make sto.result a list object and just append each of those things to that list.

Britton

On Sun, May 24, 2015 at 11:12 AM, Joshua Wall <joshua.e.wall@gmail.com> wrote:

...

Dear users,

This there a simple way to store multiple values at each step using parallel_objects? I'm doing:

######################################################## yt.enable_parallelism() num_procs = 4 plot = glob.glob(./plot*) my_storage = {}

for sto, p in yt.parallel_objects(plots, num_procs, my_storage):

sto.result_id = 'energy' + p sto.resutl = dd["total_energy"]

sto.result_id = 'Px' + p sto.result = dd["total_mom_x"]

######################################################

which is just overwriting my energy with the momentum. I also tried to just set to different dictionaries without passing it to yt.parallel_objects as storage, but of course each proc as only a local copy then.

Cordially,

Josh

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Hi Josh, Glad to hear you've got it working. I have just one comment on your script. I don't think you want to use the i variable to load the datasets from the plot list. Since you are incrementing i by one, each process will be loading datasets starting from 0 and going up to the total divided by the number of processors. Instead, since you're iterating over the plot list, you can just do: ds = yt.load(p, particle_filename=p) Britton On Sun, May 24, 2015 at 5:59 PM, Joshua Wall <joshua.e.wall@gmail.com> wrote:

Okay, I *think* I got it sorted out. Here's my solution (for future use if deemed correct!)

plot = glob.glob("./SinkMomTest/*plt_cnt_000?") plot.sort() part = glob.glob("./SinkMomTest/*part_000?") part.sort() #print plot # Testing

# Make a list of variables you are collecting to plot/store as # dictionaries.

my_storage = {}

# Now the parallel loop.

i = 0

for sto,p in yt.parallel_objects(plot, num_procs, storage = my_storage):

ds = yt.load(plot[i], particle_filename=part[i]) dd = ds.all_data()

if (yt.is_root()): print "Yo I'm root!"

if (~yt.is_root()): print "Not root!"

momx = dd["total_px"] momy = dd["total_py"] partmomx = dd["total_part_px"] partmomy = dd["total_part_py"]

sto.result_id = p[-4:] sto.result = [momx,momy,partmomx,partmomy]

i=i+1

print "Done!"

if yt.is_root():

my_storage = collections.OrderedDict(sorted(my_storage.items())) print my_storage

Thanks again for the help,

Josh

On Sun, May 24, 2015 at 6:33 PM Joshua Wall <joshua.e.wall@gmail.com> wrote:

...

Dear Britton,

I tried something like that, but clearly I'm not doing it correctly. I'm noticing two errors, 1) everything is reporting running on the root proc (with a "Yo I'm root!") and 2) I'm getting four copies of my_storage back. Here's the code:

#########################################################################

from matplotlib import rc import matplotlib.pyplot as plt import yt from yt import derived_field import numpy as np import yt.units as u import glob

# Use YT in parallel. yt.enable_parallelism()

# Number of procs on the machine num_procs=4

@derived_field(name = "total_KE", units = "erg") def _KE(field, data): return np.sum(data["kinetic_energy"].in_mks()*data["cell_volume"].in_cgs())

@derived_field(name = "total_PE", units = "erg") def _PE(field, data): return 0.5*np.sum(data["gpot"].in_mks()*data["cell_mass"].in_cgs())

@derived_field(name = "total_Eint", units = "erg") def _Eint(field, data): return np.sum(data["eint"]*data["cell_mass"]).in_cgs()

@derived_field(name = "total_energy", units = "erg") def _Etotal(field, data): return data["total_Eint"] + data["total_PE"] + data["total_KE"]

@derived_field(name = "total_px", units = "g*cm/s") def _px(field, data): return np.sum(data["cell_mass"]*data["velx"]).in_cgs()

# Use glob to gather up the data files. Note here we use plt # and particle files because they are much smaller than # checkpoint files.

plot = glob.glob("./SinkMomTest/*plt_cnt_000?") plot.sort() part = glob.glob("./SinkMomTest/*part_000?") part.sort()

# Make a list of variables you are collecting to plot/store as # dictionaries. my_storage = {}

# Now the parallel loop.

i = 0

for sto,p in yt.parallel_objects(plot, num_procs, storage = my_storage):

ds = yt.load(plot[i], particle_filename=part[i]) dd = ds.all_data()

if (yt.is_root()): print "Yo I'm root!" else: print "Not root!"

energy = dd["total_energy"] mom = dd["total_px"]

sto.result_id = p[-4:] sto.result = [energy, mom]

i=i+1

print "Done!"

print my_storage

#########################################################################

Is this what you mean?

Kindly,

Josh

On Sun, May 24, 2015 at 12:19 PM Britton Smith <brittonsmith@gmail.com> wrote:

...

Hi Josh,

The reason that's not working is that there is still only one storage item (the sto variable) per iteration of the loop. You could keep both things if you make sto.result a list object and just append each of those things to that list.

Britton

On Sun, May 24, 2015 at 11:12 AM, Joshua Wall <joshua.e.wall@gmail.com> wrote:

...

Dear users,

This there a simple way to store multiple values at each step using parallel_objects? I'm doing:

######################################################## yt.enable_parallelism() num_procs = 4 plot = glob.glob(./plot*) my_storage = {}

for sto, p in yt.parallel_objects(plots, num_procs, my_storage):

sto.result_id = 'energy' + p sto.resutl = dd["total_energy"]

sto.result_id = 'Px' + p sto.result = dd["total_mom_x"]

######################################################

which is just overwriting my energy with the momentum. I also tried to just set to different dictionaries without passing it to yt.parallel_objects as storage, but of course each proc as only a local copy then.

Cordially,

Josh

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Hi Josh, If you want to loop over two or more things, you can something like this: list_a = ... list_b = ... for sto, (a, b) in yt.parallel_objects(zip(list_a, list_b), num_procs, storage=my_storage): print a, b Britton On Wed, May 27, 2015 at 10:26 AM, Joshua Wall <joshua.e.wall@gmail.com> wrote:

Britton,

Indeed, I have noticed in the runs that work was being repeated. I'm not quite sure how to get around it, since the particle filenames are different than the plot file names. Perhaps I can do?

for plt_file, part_file in plot, part:

ds = yt.load(plt_file, particle_filename=part_file)

Thnaks again,

Josh

On Wed, May 27, 2015 at 11:14 AM Britton Smith <brittonsmith@gmail.com> wrote:

...

Hi Josh,

Glad to hear you've got it working. I have just one comment on your script. I don't think you want to use the i variable to load the datasets from the plot list. Since you are incrementing i by one, each process will be loading datasets starting from 0 and going up to the total divided by the number of processors. Instead, since you're iterating over the plot list, you can just do: ds = yt.load(p, particle_filename=p)

Britton

On Sun, May 24, 2015 at 5:59 PM, Joshua Wall <joshua.e.wall@gmail.com> wrote:

...

Okay, I *think* I got it sorted out. Here's my solution (for future use if deemed correct!)

plot = glob.glob("./SinkMomTest/*plt_cnt_000?") plot.sort() part = glob.glob("./SinkMomTest/*part_000?") part.sort() #print plot # Testing

# Make a list of variables you are collecting to plot/store as # dictionaries.

my_storage = {}

# Now the parallel loop.

i = 0

for sto,p in yt.parallel_objects(plot, num_procs, storage = my_storage):

ds = yt.load(plot[i], particle_filename=part[i]) dd = ds.all_data()

if (yt.is_root()): print "Yo I'm root!"

if (~yt.is_root()): print "Not root!"

momx = dd["total_px"] momy = dd["total_py"] partmomx = dd["total_part_px"] partmomy = dd["total_part_py"]

sto.result_id = p[-4:] sto.result = [momx,momy,partmomx,partmomy]

i=i+1

print "Done!"

if yt.is_root():

my_storage = collections.OrderedDict(sorted(my_storage.items())) print my_storage

Thanks again for the help,

Josh

On Sun, May 24, 2015 at 6:33 PM Joshua Wall <joshua.e.wall@gmail.com> wrote:

...

Dear Britton,

I tried something like that, but clearly I'm not doing it correctly. I'm noticing two errors, 1) everything is reporting running on the root proc (with a "Yo I'm root!") and 2) I'm getting four copies of my_storage back. Here's the code:

#########################################################################

from matplotlib import rc import matplotlib.pyplot as plt import yt from yt import derived_field import numpy as np import yt.units as u import glob

# Use YT in parallel. yt.enable_parallelism()

# Number of procs on the machine num_procs=4

@derived_field(name = "total_KE", units = "erg") def _KE(field, data): return np.sum(data["kinetic_energy"].in_mks()*data["cell_volume"].in_cgs())

@derived_field(name = "total_PE", units = "erg") def _PE(field, data): return 0.5*np.sum(data["gpot"].in_mks()*data["cell_mass"].in_cgs())

@derived_field(name = "total_Eint", units = "erg") def _Eint(field, data): return np.sum(data["eint"]*data["cell_mass"]).in_cgs()

@derived_field(name = "total_energy", units = "erg") def _Etotal(field, data): return data["total_Eint"] + data["total_PE"] + data["total_KE"]

@derived_field(name = "total_px", units = "g*cm/s") def _px(field, data): return np.sum(data["cell_mass"]*data["velx"]).in_cgs()

# Use glob to gather up the data files. Note here we use plt # and particle files because they are much smaller than # checkpoint files.

plot = glob.glob("./SinkMomTest/*plt_cnt_000?") plot.sort() part = glob.glob("./SinkMomTest/*part_000?") part.sort()

# Make a list of variables you are collecting to plot/store as # dictionaries. my_storage = {}

# Now the parallel loop.

i = 0

for sto,p in yt.parallel_objects(plot, num_procs, storage = my_storage):

ds = yt.load(plot[i], particle_filename=part[i]) dd = ds.all_data()

if (yt.is_root()): print "Yo I'm root!" else: print "Not root!"

energy = dd["total_energy"] mom = dd["total_px"]

sto.result_id = p[-4:] sto.result = [energy, mom]

i=i+1

print "Done!"

print my_storage

#########################################################################

Is this what you mean?

Kindly,

Josh

On Sun, May 24, 2015 at 12:19 PM Britton Smith <brittonsmith@gmail.com> wrote:

...

Hi Josh,

The reason that's not working is that there is still only one storage item (the sto variable) per iteration of the loop. You could keep both things if you make sto.result a list object and just append each of those things to that list.

Britton

On Sun, May 24, 2015 at 11:12 AM, Joshua Wall <joshua.e.wall@gmail.com

...

wrote:

...

Dear users,

This there a simple way to store multiple values at each step using parallel_objects? I'm doing:

######################################################## yt.enable_parallelism() num_procs = 4 plot = glob.glob(./plot*) my_storage = {}

for sto, p in yt.parallel_objects(plots, num_procs, my_storage):

sto.result_id = 'energy' + p sto.resutl = dd["total_energy"]

sto.result_id = 'Px' + p sto.result = dd["total_mom_x"]

######################################################

which is just overwriting my energy with the momentum. I also tried to just set to different dictionaries without passing it to yt.parallel_objects as storage, but of course each proc as only a local copy then.

Cordially,

Josh

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org