Re: [Yt-dev] [yt-users] parallel data retrieval
Hi guys, I'm moving this to -dev. This should be in the halos themselves; unfortunately we want to avoid double-communicating, both between procs and into the file system. I will work on that, using Stephen's script as a template. However, I've also set up a simple little way to add things to wrap, to get rid of the get_indices, get_velocities, etc functions. I think we can use this to prevent any getter/setter methods. http://paste.enzotools.org/show/46/ Stephen, this is just a simple little mod, but if you could test that this appropriately wraps the __getitem__ method, that'd be awesome. Then we can ditch the getters. -Matt On Fri, Feb 13, 2009 at 10:27 AM, Stephen Skory <stephenskory@yahoo.com> wrote:
Matt,
If you want to try it again, you might
consider manually setting (and then unsetting) the _processing attribute of your HopGroup to True; this should set it to turn off the locking/barrierizing.
To follow up, your suggestion works, and I've pasted the full script below for everyone's edification.
http://paste.enzotools.org/show/45/
Please note that I haven't fixed some things in the version I pasted, in particular "grouped_particles" will be incorrect.
Thanks!
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Matt,
Stephen, this is just a simple little mod, but if you could test that this appropriately wraps the __getitem__ method, that'd be awesome. Then we can ditch the getters.
to be clear, you want me to change things like: gpx = halo.get_particle_positions('x') to gpx = halo["particle_position_x"] ? Also, Ranger is taking a dump right now, so it will have to wait until they sort it out: login3% uptime 12:54:02 up 2 days, 21:48, 152 users, load average: 129.11, 122.11, 96.43 _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
to be clear, you want me to change things like:
gpx = halo.get_particle_positions('x')
to
gpx = halo["particle_position_x"] ?
I believe with this patch you can do that, yes. Also, guys -- I think we're almost ready to move SS_HopOutput.py to replace HopOutput.py. -Matt
Matt, is the logic correct on lines 138-139 in your diff? The way you wrote it is giving me a syntax error. Can you take another look? http://paste.enzotools.org/show/46/ _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
should be "not in extra" On Fri, Feb 13, 2009 at 1:23 PM, Stephen Skory <stephenskory@yahoo.com> wrote:
Matt,
is the logic correct on lines 138-139 in your diff? The way you wrote it is giving me a syntax error. Can you take another look?
http://paste.enzotools.org/show/46/
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Matt, That doesn't work either: http://paste.enzotools.org/show/47/ do you want "if attrname not in extra or attrname.startswith("_") \"? I tried that and got a different, but more encouraging error: http://paste.enzotools.org/show/48/ _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Hi Stephen, When I made reference to removing the getter methods, I did not mean getter methods that are not for particle fields. get_size does not refer to, for instance, a "particle_size" value. So get_size needs to stay. -Matt On Fri, Feb 13, 2009 at 1:43 PM, Stephen Skory <stephenskory@yahoo.com> wrote:
Matt,
That doesn't work either:
http://paste.enzotools.org/show/47/
do you want "if attrname not in extra or attrname.startswith("_") \"?
I tried that and got a different, but more encouraging error:
http://paste.enzotools.org/show/48/
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Matt,
When I made reference to removing the getter methods, I did not mean getter methods that are not for particle fields. get_size does not refer to, for instance, a "particle_size" value. So get_size needs to stay.
I haven't removed any of the get_ methods from SS_HopOutput.py. get_size is still there. _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Alright, I'll take it from here and get back to you. On Fri, Feb 13, 2009 at 1:52 PM, Stephen Skory <stephenskory@yahoo.com> wrote:
Matt,
When I made reference to removing the getter methods, I did not mean getter methods that are not for particle fields. get_size does not refer to, for instance, a "particle_size" value. So get_size needs to stay.
I haven't removed any of the get_ methods from SS_HopOutput.py. get_size is still there. _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I've done it in r1168. On Fri, Feb 13, 2009 at 1:52 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Alright, I'll take it from here and get back to you.
On Fri, Feb 13, 2009 at 1:52 PM, Stephen Skory <stephenskory@yahoo.com> wrote:
Matt,
When I made reference to removing the getter methods, I did not mean getter methods that are not for particle fields. get_size does not refer to, for instance, a "particle_size" value. So get_size needs to stay.
I haven't removed any of the get_ methods from SS_HopOutput.py. get_size is still there. _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Matt,
I've done it in r1168.
It works now without giving an error, but I still need to set _processing = True for it to work. Was that your intention? Without fippping _processing, it just hangs, as it did at the beginning of this thread. Thanks! _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
This was not meant to be relevant to setting _processing. _processing exists so that sub-calls withing a given calculation of halo properties don't block. The original intent of the proxy class was to avoid any per-processor changes in operation; that's why the _processing flag still has to be set manually. I mean, if you put the particle dumping routines *as methods of the class itself* they should work perfectly fine, because they'll be inside the proxy handling. This is the path I would recommend. If PyTables had parallel HDF5 IO, we could look into how to write multiple at a time. It doesn't, so we can simply write a routine that accepts a filename, opens it for append, writes, returns. The proxy class was not designed to handled differently based on which processor each group is on. The idea here is to avoid that, and to make the API independent of processor location; this amounts to a contract between the user and the code: "If you promise to do your best to make it so I don't have to mess with the internals, I promise not to touch them." If you start ignoring the tacit agreement that in the process of executing a script, you're going to have to recognize that the system might break down, and you might have to mess with internals. Things aren't perfect, but they do sort of work. -Matt On Fri, Feb 13, 2009 at 2:54 PM, Stephen Skory <stephenskory@yahoo.com> wrote:
Matt,
I've done it in r1168.
It works now without giving an error, but I still need to set _processing = True for it to work. Was that your intention? Without fippping _processing, it just hangs, as it did at the beginning of this thread.
Thanks!
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Things aren't perfect, but they do sort of work.
That's all I require, Matt. Thanks, again! _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Matt, I think I don't quite understand something fundamental about how parallel python works. In particular, how do I make the loop at line 29 run on all processors at the same time? As an experiment I'm trying to write separate HDF5 files per processor to speed things up, but from the timestamp (line 73) I can see that it's going serially in order of halo.id. For example, the first halo on proc 0 is the sixth, but it isn't written until the first five are written by proc 1, so it's going in strict order. http://paste.enzotools.org/show/49/ wrote halo 0 from proc 1 at 2009-02-14 10:53:05.206974 wrote halo 1 from proc 1 at 2009-02-14 10:53:05.296788 wrote halo 2 from proc 1 at 2009-02-14 10:53:05.406035 wrote halo 3 from proc 1 at 2009-02-14 10:53:05.506907 wrote halo 4 from proc 1 at 2009-02-14 10:53:05.626588 wrote halo 5 from proc 0 at 2009-02-14 10:53:06.474605 wrote halo 6 from proc 0 at 2009-02-14 10:53:06.553487 wrote halo 7 from proc 1 at 2009-02-14 10:53:06.688611 wrote halo 8 from proc 1 at 2009-02-14 10:53:06.790755 wrote halo 9 from proc 0 at 2009-02-14 10:53:06.879232 wrote halo 10 from proc 1 at 2009-02-14 10:53:06.978932 Thanks for answering my naive questions... _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
It blocks inside all calls to get information about halos that aren't on the current processor unless _processing is set -- you can read the code for this in yt/lagos/ParallelTools.py, specifically line 114. All methods of HopGroup, unless they're named in dont_wrap or start with a _, are wrapped in the function single_proc_results. This barriers, calculates, broadcasts, and barriers. So everything inside your if statement in your loop is fine; what's causing the problem is get_size. On Sat, Feb 14, 2009 at 9:08 AM, Stephen Skory <stephenskory@yahoo.com> wrote:
Matt,
I think I don't quite understand something fundamental about how parallel python works. In particular, how do I make the loop at line 29 run on all processors at the same time? As an experiment I'm trying to write separate HDF5 files per processor to speed things up, but from the timestamp (line 73) I can see that it's going serially in order of halo.id. For example, the first halo on proc 0 is the sixth, but it isn't written until the first five are written by proc 1, so it's going in strict order.
http://paste.enzotools.org/show/49/
wrote halo 0 from proc 1 at 2009-02-14 10:53:05.206974 wrote halo 1 from proc 1 at 2009-02-14 10:53:05.296788 wrote halo 2 from proc 1 at 2009-02-14 10:53:05.406035 wrote halo 3 from proc 1 at 2009-02-14 10:53:05.506907 wrote halo 4 from proc 1 at 2009-02-14 10:53:05.626588 wrote halo 5 from proc 0 at 2009-02-14 10:53:06.474605 wrote halo 6 from proc 0 at 2009-02-14 10:53:06.553487 wrote halo 7 from proc 1 at 2009-02-14 10:53:06.688611 wrote halo 8 from proc 1 at 2009-02-14 10:53:06.790755 wrote halo 9 from proc 0 at 2009-02-14 10:53:06.879232 wrote halo 10 from proc 1 at 2009-02-14 10:53:06.978932
Thanks for answering my naive questions...
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Matt,
what's causing the problem is get_size.
That makes sense, and indeed I just confirmed it by commenting out get_size. Thank you very much. _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Awesome! I'm glad to hear it. We should at some point put in this means of both serially and non-serially writing out the halos info. On Sat, Feb 14, 2009 at 9:35 AM, Stephen Skory <stephenskory@yahoo.com> wrote:
Matt,
what's causing the problem is get_size.
That makes sense, and indeed I just confirmed it by commenting out get_size. Thank you very much.
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
participants (2)
-
Matthew Turk -
Stephen Skory