
Hi everyone, Stephen's gotten a lot of great stuff stuck into the SVN trunk, and I'm trying to make sure that the particle support is there for him. The new ParticleIO is getting close to being "ready for primetime", but I'm going to need some help testing it. I've uploaded a patch: http://paste.enzotools.org/show/288/ What this will do is co-opt the particle IO from a standard dict-like access of an AMR3DData instance and, if it's a particle type, pass it through the ParticleIO(.py) object. Right now this will only affect Regions -- everything else will simply operate as normal. (i.e., spheres will just get passed on through.) As a note, the new ParticleIO is set up to do two-passes through a region, first counting the particles that will end up in the object and then reading them. This speeds up IO considerably, as it reduces the number of array creation operations to a minimum. (My sample script for testing is here, but I'd prefer if you could test in a real-world application... http://paste.enzotools.org/show/289/ ) If possible, could a few of you test this out, and let me know if this patch interferes with any normal behavior? I'm running my own tests here and it's looking *okay*, but it should get tested in the field a bit before being committed. After that, I'll add the necessary methods for spheres, and then the ParticleIO should be mostly handled by the new mechanism. Thanks for any ideas and testing! -Matt

Hi everyone (again), I've got a new particle IO patch here: http://paste.enzotools.org/show/292/ It should now handle long double particle positions (I might be the only target audience for that) as well as sphere's. Additionally, it should be okay with handling integer fields like particle type and particle ID. Stephen has said he'd give a go with testing it using his HOP and other stuff, and if that goes okay I think it is ready for prime time -- this should speed up particle access considerably... I think there might be some lingering issues with memory leaks, but I am still looking into that; first step is making sure it doesn't break anything. -Matt On Wed, Jan 6, 2010 at 9:01 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi everyone,
Stephen's gotten a lot of great stuff stuck into the SVN trunk, and I'm trying to make sure that the particle support is there for him. The new ParticleIO is getting close to being "ready for primetime", but I'm going to need some help testing it. I've uploaded a patch:
http://paste.enzotools.org/show/288/
What this will do is co-opt the particle IO from a standard dict-like access of an AMR3DData instance and, if it's a particle type, pass it through the ParticleIO(.py) object. Right now this will only affect Regions -- everything else will simply operate as normal. (i.e., spheres will just get passed on through.) As a note, the new ParticleIO is set up to do two-passes through a region, first counting the particles that will end up in the object and then reading them. This speeds up IO considerably, as it reduces the number of array creation operations to a minimum.
(My sample script for testing is here, but I'd prefer if you could test in a real-world application... http://paste.enzotools.org/show/289/ )
If possible, could a few of you test this out, and let me know if this patch interferes with any normal behavior? I'm running my own tests here and it's looking *okay*, but it should get tested in the field a bit before being committed. After that, I'll add the necessary methods for spheres, and then the ParticleIO should be mostly handled by the new mechanism.
Thanks for any ideas and testing!
-Matt

Matt,
That patch is truncated. Can you post the full patch? [sskory@login-4-0 yt-trunk-svn]$ patch -p0 < IO.patch patching file yt/lagos/DataReadingFuncs.py Hunk #5 succeeded at 391 (offset 85 lines). patching file yt/lagos/HierarchyType.py Hunk #1 succeeded at 590 (offset 1 line). patching file yt/lagos/OutputTypes.py patching file yt/lagos/BaseDataTypes.py patching file yt/lagos/HDF5LightReader.c patching file yt/lagos/ParticleIO.py patch unexpectedly ends in middle of line patch: **** malformed patch at line 532: Thanks! _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

Hi Stephen, I've pasted a new one that I've verified applies cleanly for me: http://paste.enzotools.org/show/294/ There're a couple hunks in here that aren't necessary and that will be trimmed ultimately. Thanks! -Matt On Sun, Jan 10, 2010 at 9:14 AM, Stephen Skory <stephenskory@yahoo.com> wrote:
Matt,
That patch is truncated. Can you post the full patch?
[sskory@login-4-0 yt-trunk-svn]$ patch -p0 < IO.patch patching file yt/lagos/DataReadingFuncs.py Hunk #5 succeeded at 391 (offset 85 lines). patching file yt/lagos/HierarchyType.py Hunk #1 succeeded at 590 (offset 1 line). patching file yt/lagos/OutputTypes.py patching file yt/lagos/BaseDataTypes.py patching file yt/lagos/HDF5LightReader.c patching file yt/lagos/ParticleIO.py patch unexpectedly ends in middle of line patch: **** malformed patch at line 532:
Thanks! _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

Matt, I am getting this error. I see that 'get_args' is defined in the ParticleIOHandlerRegion class, and a '_get_args' for ParticleIOHandlerSphere. I'm guessing that it should be '_get_args' under ParticleIOHandlerRegion. Thoughts? File "/home/sskory/yt/lib/python2.6/site-packages/yt-1.6dev-py2.6-linux-x86_64.egg/yt/lagos/HaloFinding.py", line 1114, in __init__ data = self._data_source.particles["particle_position_x"] File "/home/sskory/yt/lib/python2.6/site-packages/yt-1.6dev-py2.6-linux-x86_64.egg/yt/lagos/ParticleIO.py", line 47, in __getitem__ self.get_data(key) File "/home/sskory/yt/lib/python2.6/site-packages/yt-1.6dev-py2.6-linux-x86_64.egg/yt/lagos/ParticleIO.py", line 74, in get_data rtype, args = self._get_args() AttributeError: 'ParticleIOHandlerPeriodicRegionStrict' object has no attribute '_get_args' _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

Hi Stephen, It should be _get_args. Oddly enough, I ran into and fixed this earlier this weekend, before I generated the patch. I must have gotten confused and made the patch from the wrong source tree. -Matt On Sun, Jan 10, 2010 at 9:50 AM, Stephen Skory <stephenskory@yahoo.com> wrote:
Matt,
I am getting this error. I see that 'get_args' is defined in the ParticleIOHandlerRegion class, and a '_get_args' for ParticleIOHandlerSphere. I'm guessing that it should be '_get_args' under ParticleIOHandlerRegion. Thoughts?
File "/home/sskory/yt/lib/python2.6/site-packages/yt-1.6dev-py2.6-linux-x86_64.egg/yt/lagos/HaloFinding.py", line 1114, in __init__ data = self._data_source.particles["particle_position_x"] File "/home/sskory/yt/lib/python2.6/site-packages/yt-1.6dev-py2.6-linux-x86_64.egg/yt/lagos/ParticleIO.py", line 47, in __getitem__ self.get_data(key) File "/home/sskory/yt/lib/python2.6/site-packages/yt-1.6dev-py2.6-linux-x86_64.egg/yt/lagos/ParticleIO.py", line 74, in get_data rtype, args = self._get_args() AttributeError: 'ParticleIOHandlerPeriodicRegionStrict' object has no attribute '_get_args' _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

Matt, I'm getting problems with particle mass. Specifically, 'particle_mass' is coming back correct, but 'ParticleMassMsun' is not. I'll do some digging and let you know. _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

I'm getting problems with particle mass. Specifically, 'particle_mass' is coming back correct, but 'ParticleMassMsun' is not. I'll do some digging and let you know.
I should add that this is not an AMR dataset. _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

Matt, What's happening is the 'npy_type' is not getting set for each call of 'run_validators()' because it is set under the same 'if' statement that creates the initial array to store the values in. Subsequent calls don't create the array, but they also don't set 'npy_type'. So I think this: if(pv->return_values[i] == NULL){ should be moved to surround: pv->return_values[i] = (PyArrayObject *) PyArray_SimpleNewFromDescr( 1, &dims, PyArray_DescrFromType(npy_type)); Or, 'npy_type' needs to be stored more persistently between calls of 'run_validators()'. Thoughts? _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

Hi Stephen, Wow, good catch. I've updated the particle_validation struct with an npy_types member, which I now set as an array. The new patch is here: http://paste.enzotools.org/show/295/ Can you try this one? Thanks! -Matt On Sun, Jan 10, 2010 at 2:46 PM, Stephen Skory <stephenskory@yahoo.com> wrote:
Matt,
What's happening is the 'npy_type' is not getting set for each call of 'run_validators()' because it is set under the same 'if' statement that creates the initial array to store the values in. Subsequent calls don't create the array, but they also don't set 'npy_type'. So I think this:
if(pv->return_values[i] == NULL){
should be moved to surround:
pv->return_values[i] = (PyArrayObject *) PyArray_SimpleNewFromDescr( 1, &dims, PyArray_DescrFromType(npy_type));
Or, 'npy_type' needs to be stored more persistently between calls of 'run_validators()'. Thoughts? _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

Matt,
Hi Stephen,
Wow, good catch. I've updated the particle_validation struct with an npy_types member, which I now set as an array. The new patch is here:
http://paste.enzotools.org/show/295/
Can you try this one?
This appears to work for me. I ran Parallel HOP and I got the answers I expected. I looked at the 'particle_mass' and 'ParticleMassMsun' arrays on an AMR and non-AMR dataset and they made sense. Any other things to test? Otherwise, I'd say it's good to go! _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

Hi Stephen,
This appears to work for me. I ran Parallel HOP and I got the answers I expected. I looked at the 'particle_mass' and 'ParticleMassMsun' arrays on an AMR and non-AMR dataset and they made sense. Any other things to test? Otherwise, I'd say it's good to go!
Awesome! I've committed it in r1572. I think this leaves only a couple things before release -- a new build of the docs, me documenting the software ray caster ... I think that might be it. -Matt
participants (2)
-
Matthew Turk
-
Stephen Skory