yt Particle Plots & openPMD Frontend
Hi yt-devs,
I am forwarding this conversation to the public list to keep it open for others to join in and to build up a record that might help others.
Our new student Fabian updated our frontend and implemented particle readers. Unfortunately some details of the particle "fields" still cause some bumps that we can not get our head around.
We are now able to read fields and particles, even chunk-wise, and can work with the data we read in python scripts via the all_data() method.
Nevertheless, using the particle scatter plots such as yt.ParticlePlot seems not to work. We have some problems with unions / the "all" group and the fact that we have several particle species (e.g., "hot electrons", "cold electrons", "helium ions", "nitrogen ions", etc.)
Also since the last rebase: did someone recently change the unit system in yt? T for "Tesla" seems not to be understood any more. Is there a changelog somewhere available? Do we have to describe our data in cgs / mks?
Would someone be interested for a quick heads up via e.g., skype / web-rtc so we can ask specific questions?
Fabian is around on Tuesdays and Fridays, something like early CA time, and late GER time usually works great (e.g., 9pm-PDT / 6pm-CEST).
Our current branch HEAD can be found here (->yt->frontends->openPMD): https://bitbucket.org/C0nsultant/openpmd/src?at=yt
And an example script is attached.
Thanks a lot! Axel
On Tue, May 24, 2016 at 11:30 AM, Huebl, Axel a.huebl@hzdr.de wrote:
Hi yt-devs,
I am forwarding this conversation to the public list to keep it open for others to join in and to build up a record that might help others.
Our new student Fabian updated our frontend and implemented particle readers. Unfortunately some details of the particle "fields" still cause some bumps that we can not get our head around.
We are now able to read fields and particles, even chunk-wise, and can work with the data we read in python scripts via the all_data() method.
Nevertheless, using the particle scatter plots such as yt.ParticlePlot seems not to work. We have some problems with unions / the "all" group and the fact that we have several particle species (e.g., "hot electrons", "cold electrons", "helium ions", "nitrogen ions", etc.)
The 'all' particle union should get set up automatically, but maybe something is going wrong with that machinery.
Can you create a test script that illustrates this issue using a publicly available OpenPMD dataset?
Also since the last rebase: did someone recently change the unit system in yt? T for "Tesla" seems not to be understood any more. Is there a changelog somewhere available? Do we have to describe our data in cgs / mks?
I just double-checked, and Tesla should be working:
In [1]: from yt import YTQuantity
In [2]: YTQuantity(1, 'T') Out[2]: 1.0 T
In [3]: YTQuantity(1, 'T').to_equivalent('G', 'CGS') Out[3]: 10000.0 G
Can you explain a little bit more what issue you're having? A test script that illustrates the issue would also help.
Would someone be interested for a quick heads up via e.g., skype / web-rtc so we can ask specific questions?
Fabian is around on Tuesdays and Fridays, something like early CA time, and late GER time usually works great (e.g., 9pm-PDT / 6pm-CEST).
Our current branch HEAD can be found here (->yt->frontends->openPMD): https://bitbucket.org/C0nsultant/openpmd/src?at=yt
And an example script is attached.
Thanks a lot! Axel
Dear Nathan and John,
thank you for the quick answers.
Yes, the resources to reproduce the questions are:
Repo: https://bitbucket.org/C0nsultant/openpmd/src?at=yt
Example Files: PIConGPU > 2D3V > one of the .h5 files https://owncloud.hzdr.de/index.php/s/ouisGZe5M0wCVzj password: openPMD
Example script (attached, like last mail).
Axel
On 24.05.2016 18:37, Nathan Goldbaum wrote:
On Tue, May 24, 2016 at 11:30 AM, Huebl, Axel <a.huebl@hzdr.de mailto:a.huebl@hzdr.de> wrote:
Hi yt-devs, I am forwarding this conversation to the public list to keep it open for others to join in and to build up a record that might help others. Our new student Fabian updated our frontend and implemented particle readers. Unfortunately some details of the particle "fields" still cause some bumps that we can not get our head around. We are now able to read fields and particles, even chunk-wise, and can work with the data we read in python scripts via the all_data() method. Nevertheless, using the particle scatter plots such as yt.ParticlePlot seems not to work. We have some problems with unions / the "all" group and the fact that we have several particle species (e.g., "hot electrons", "cold electrons", "helium ions", "nitrogen ions", etc.)The 'all' particle union should get set up automatically, but maybe something is going wrong with that machinery.
Can you create a test script that illustrates this issue using a publicly available OpenPMD dataset?
Also since the last rebase: did someone recently change the unit system in yt? T for "Tesla" seems not to be understood any more. Is there a changelog somewhere available? Do we have to describe our data in cgs / mks?I just double-checked, and Tesla should be working:
In [1]: from yt import YTQuantity
In [2]: YTQuantity(1, 'T') Out[2]: 1.0 T
In [3]: YTQuantity(1, 'T').to_equivalent('G', 'CGS') Out[3]: 10000.0 G
Can you explain a little bit more what issue you're having? A test script that illustrates the issue would also help.
Would someone be interested for a quick heads up via e.g., skype / web-rtc so we can ask specific questions? Fabian is around on Tuesdays and Fridays, something like early CA time, and late GER time usually works great (e.g., 9pm-PDT / 6pm-CEST). Our current branch HEAD can be found here (->yt->frontends->openPMD): https://bitbucket.org/C0nsultant/openpmd/src?at=yt And an example script is attached. Thanks a lot! Axel
I've opened a pull request to fix the issue you discovered with magnetic units:
https://bitbucket.org/yt_analysis/yt/pull-requests/2213
I'd appreciate it if one of you could give it a test. Let me know if you have questions about how to pull a pull request in locally with mercurial.
I haven't had a time to look at the issue you were having with the 'all' particle union, but I have it on my list to look at it.
On Fri, May 27, 2016 at 2:37 AM, Huebl, Axel a.huebl@hzdr.de wrote:
Dear Nathan and John,
thank you for the quick answers.
Yes, the resources to reproduce the questions are:
Repo: https://bitbucket.org/C0nsultant/openpmd/src?at=yt
Example Files: PIConGPU > 2D3V > one of the .h5 files https://owncloud.hzdr.de/index.php/s/ouisGZe5M0wCVzj password: openPMD
Example script (attached, like last mail).
Axel
On 24.05.2016 18:37, Nathan Goldbaum wrote:
On Tue, May 24, 2016 at 11:30 AM, Huebl, Axel <a.huebl@hzdr.de mailto:a.huebl@hzdr.de> wrote:
Hi yt-devs, I am forwarding this conversation to the public list to keep it openfor
others to join in and to build up a record that might help others. Our new student Fabian updated our frontend and implemented particle readers. Unfortunately some details of the particle "fields" stillcause
some bumps that we can not get our head around. We are now able to read fields and particles, even chunk-wise, andcan
work with the data we read in python scripts via the all_data()method.
Nevertheless, using the particle scatter plots such asyt.ParticlePlot
seems not to work. We have some problems with unions / the "all"group
and the fact that we have several particle species (e.g., "hot electrons", "cold electrons", "helium ions", "nitrogen ions", etc.)The 'all' particle union should get set up automatically, but maybe something is going wrong with that machinery.
Can you create a test script that illustrates this issue using a publicly available OpenPMD dataset?
Also since the last rebase: did someone recently change the unitsystem
in yt? T for "Tesla" seems not to be understood any more. Is there a changelog somewhere available? Do we have to describe our data in cgs / mks?I just double-checked, and Tesla should be working:
In [1]: from yt import YTQuantity
In [2]: YTQuantity(1, 'T') Out[2]: 1.0 T
In [3]: YTQuantity(1, 'T').to_equivalent('G', 'CGS') Out[3]: 10000.0 G
Can you explain a little bit more what issue you're having? A test script that illustrates the issue would also help.
Would someone be interested for a quick heads up via e.g., skype / web-rtc so we can ask specific questions? Fabian is around on Tuesdays and Fridays, something like early CAtime,
and late GER time usually works great (e.g., 9pm-PDT / 6pm-CEST). Our current branch HEAD can be found here (->yt->frontends->openPMD): https://bitbucket.org/C0nsultant/openpmd/src?at=yt And an example script is attached. Thanks a lot! Axel--
Axel Huebl Phone +49 351 260 3582 https://www.hzdr.de/crp Computational Radiation Physics Laser Particle Acceleration Division Helmholtz-Zentrum Dresden - Rossendorf e.V.
Bautzner Landstrasse 400, 01328 Dresden POB 510119, D-01314 Dresden Vorstand: Prof. Dr.Dr.h.c. R. Sauerbrey Prof. Dr.Dr.h.c. P. Joehnk VR 1693 beim Amtsgericht Dresden
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Cheers Nathan,
thank you for the patch. I pulled the code (#2213) into my local copy and gave it as in-depth a test as I could. It works, except for one detail.
I am unsure whether the problem arises from the frontend or through the unit system, but what I am currently facing is: fields from known_other_fields do not have units (ergo are dimensionless) in datasets, no matter what the unit in known_other_fields.
I've attached a script which tries to show the problem by using one of the example files Axel linked to in his last mail.
--Fabian
On 03/06/16 23:05, Nathan Goldbaum wrote:
I've opened a pull request to fix the issue you discovered with magnetic units:
https://bitbucket.org/yt_analysis/yt/pull-requests/2213
I'd appreciate it if one of you could give it a test. Let me know if you have questions about how to pull a pull request in locally with mercurial.
I haven't had a time to look at the issue you were having with the 'all' particle union, but I have it on my list to look at it.
On Fri, May 27, 2016 at 2:37 AM, Huebl, Axel <a.huebl@hzdr.de mailto:a.huebl@hzdr.de> wrote:
Dear Nathan and John, thank you for the quick answers. Yes, the resources to reproduce the questions are: Repo: https://bitbucket.org/C0nsultant/openpmd/src?at=yt Example Files: PIConGPU > 2D3V > one of the .h5 files https://owncloud.hzdr.de/index.php/s/ouisGZe5M0wCVzj password: openPMD Example script (attached, like last mail). Axel On 24.05.2016 18 <tel:24.05.2016%2018>:37, Nathan Goldbaum wrote: > > > On Tue, May 24, 2016 at 11:30 AM, Huebl, Axel <a.huebl@hzdr.de <mailto:a.huebl@hzdr.de> > <mailto:a.huebl@hzdr.de <mailto:a.huebl@hzdr.de>>> wrote: > > Hi yt-devs, > > I am forwarding this conversation to the public list to keep it open for > others to join in and to build up a record that might help others. > > Our new student Fabian updated our frontend and implemented particle > readers. Unfortunately some details of the particle "fields" still cause > some bumps that we can not get our head around. > > We are now able to read fields and particles, even chunk-wise, and can > work with the data we read in python scripts via the all_data() method. > > Nevertheless, using the particle scatter plots such as yt.ParticlePlot > seems not to work. We have some problems with unions / the "all" group > and the fact that we have several particle species (e.g., "hot > electrons", "cold electrons", "helium ions", "nitrogen ions", etc.) > > > The 'all' particle union should get set up automatically, but maybe > something is going wrong with that machinery. > > Can you create a test script that illustrates this issue using a > publicly available OpenPMD dataset? > > > > Also since the last rebase: did someone recently change the unit system > in yt? T for "Tesla" seems not to be understood any more. Is there a > changelog somewhere available? Do we have to describe our data in > cgs / mks? > > > I just double-checked, and Tesla should be working: > > In [1]: from yt import YTQuantity > > In [2]: YTQuantity(1, 'T') > Out[2]: 1.0 T > > In [3]: YTQuantity(1, 'T').to_equivalent('G', 'CGS') > Out[3]: 10000.0 G > > Can you explain a little bit more what issue you're having? A test > script that illustrates the issue would also help. > > > Would someone be interested for a quick heads up via e.g., skype / > web-rtc so we can ask specific questions? > > Fabian is around on Tuesdays and Fridays, something like early CA time, > and late GER time usually works great (e.g., 9pm-PDT / 6pm-CEST). > > Our current branch HEAD can be found here (->yt->frontends->openPMD): > https://bitbucket.org/C0nsultant/openpmd/src?at=yt > > And an example script is attached. > > > Thanks a lot! > Axel > > -- Axel Huebl Phone +49 351 260 3582 <tel:%2B49%20351%20260%203582> https://www.hzdr.de/crp Computational Radiation Physics Laser Particle Acceleration Division Helmholtz-Zentrum Dresden - Rossendorf e.V. Bautzner Landstrasse 400, 01328 Dresden POB 510119, D-01314 Dresden Vorstand: Prof. Dr.Dr.h.c. R. Sauerbrey Prof. Dr.Dr.h.c. P. Joehnk VR 1693 beim Amtsgericht Dresden _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org <mailto:yt-dev@lists.spacepope.org> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hey,
The issue you're having is because you're constructing known_other_fields dynamically. All the other frontends define it statically, so it's filled out by the time it's needed in the `setup_fluid_aliases` function defined in in yt/fields/field_info_container.py. The fix is to change the order of operations - setup known_other_fields and known_particle_fields before calling the __init__ method on the superclass of OpenPMDFieldInfo.
You're also incorrectly setting the units of the field to a Unit object --- the unit entry in the known_other_fields tuple should be a string.
The following diff allows me to set the units properly for your frontend:
http://paste.yt-project.org/show/6606/
Out of curiosity - why build the known_other_fields and known_particle_fields tuples dynamically? The reason they're static for all the other frotends is to make it easy to reason about which fields are associated with a given frontend and so that the fields known by a given frontend are independent of any particular dataset you might be looking at.
By the way, it also looks like the 'all' particle union is working correctly - are you still having issues with that?
-Nathan
On Fri, Jun 10, 2016 at 8:24 AM, Fabian Koller f.koller@hzdr.de wrote:
Cheers Nathan,
thank you for the patch. I pulled the code (#2213) into my local copy and gave it as in-depth a test as I could. It works, except for one detail.
I am unsure whether the problem arises from the frontend or through the unit system, but what I am currently facing is: fields from known_other_fields do not have units (ergo are dimensionless) in datasets, no matter what the unit in known_other_fields.
I've attached a script which tries to show the problem by using one of the example files Axel linked to in his last mail.
--Fabian
On 03/06/16 23:05, Nathan Goldbaum wrote:
I've opened a pull request to fix the issue you discovered with magnetic units:
https://bitbucket.org/yt_analysis/yt/pull-requests/2213
I'd appreciate it if one of you could give it a test. Let me know if you have questions about how to pull a pull request in locally with mercurial.
I haven't had a time to look at the issue you were having with the 'all' particle union, but I have it on my list to look at it.
On Fri, May 27, 2016 at 2:37 AM, Huebl, Axel <a.huebl@hzdr.de mailto:a.huebl@hzdr.de> wrote:
Dear Nathan and John, thank you for the quick answers. Yes, the resources to reproduce the questions are: Repo: https://bitbucket.org/C0nsultant/openpmd/src?at=yt Example Files: PIConGPU > 2D3V > one of the .h5 files https://owncloud.hzdr.de/index.php/s/ouisGZe5M0wCVzj password: openPMD Example script (attached, like last mail). Axel On 24.05.2016 18 <tel:24.05.2016%2018>:37, Nathan Goldbaum wrote: > > > On Tue, May 24, 2016 at 11:30 AM, Huebl, Axel <a.huebl@hzdr.de <mailto:a.huebl@hzdr.de> > <mailto:a.huebl@hzdr.de <mailto:a.huebl@hzdr.de>>> wrote: > > Hi yt-devs, > > I am forwarding this conversation to the public list to keep it open for > others to join in and to build up a record that might help others. > > Our new student Fabian updated our frontend and implemented particle > readers. Unfortunately some details of the particle "fields" still cause > some bumps that we can not get our head around. > > We are now able to read fields and particles, even chunk-wise, and can > work with the data we read in python scripts via the all_data() method. > > Nevertheless, using the particle scatter plots such as yt.ParticlePlot > seems not to work. We have some problems with unions / the "all" group > and the fact that we have several particle species (e.g., "hot > electrons", "cold electrons", "helium ions", "nitrogen ions", etc.) > > > The 'all' particle union should get set up automatically, but maybe > something is going wrong with that machinery. > > Can you create a test script that illustrates this issue using a > publicly available OpenPMD dataset? > > > > Also since the last rebase: did someone recently change the unit system > in yt? T for "Tesla" seems not to be understood any more. Is there a > changelog somewhere available? Do we have to describe our data in > cgs / mks? > > > I just double-checked, and Tesla should be working: > > In [1]: from yt import YTQuantity > > In [2]: YTQuantity(1, 'T') > Out[2]: 1.0 T > > In [3]: YTQuantity(1, 'T').to_equivalent('G', 'CGS') > Out[3]: 10000.0 G > > Can you explain a little bit more what issue you're having? A test > script that illustrates the issue would also help. > > > Would someone be interested for a quick heads up via e.g., skype / > web-rtc so we can ask specific questions? > > Fabian is around on Tuesdays and Fridays, something like early CA time, > and late GER time usually works great (e.g., 9pm-PDT / 6pm-CEST). > > Our current branch HEAD can be found here (->yt->frontends->openPMD): > https://bitbucket.org/C0nsultant/openpmd/src?at=yt > > And an example script is attached. > > > Thanks a lot! > Axel > > -- Axel Huebl Phone +49 351 260 3582 <tel:%2B49%20351%20260%203582> https://www.hzdr.de/crp Computational Radiation Physics Laser Particle Acceleration Division Helmholtz-Zentrum Dresden - Rossendorf e.V. Bautzner Landstrasse 400, 01328 Dresden POB 510119, D-01314 Dresden Vorstand: Prof. Dr.Dr.h.c. R. Sauerbrey Prof. Dr.Dr.h.c. P. Joehnk VR 1693 beim Amtsgericht Dresden _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org <mailto:yt-dev@lists.spacepope.org> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Wow!
Thanks for the speedy and thorough solution. It seems to work flawlessly, based on a quick test.
I can see that the decision to produce known fields on the fly seems a bit odd, but there's a simple reason behind it: The openPMD stadard (in theory) allows for infinitely many different fields to exist, all of them with different unit-dimensions. Very little is actually static: only the basic structure of our data, not its content, not its names, not its units and dimensions. Trying to cover as much of the standard as is feasible, we decided to dynamically generate known fields. (Also, having done it this way has helped me internalize the standard. A lot.)
If you're interested our reasoning more than this brief explanation, I'd refer you to the current (well, upcoming) written standard: https://github.com/openPMD/openPMD-standard/blob/upcoming-1.0.1/STANDARD.md, in particular "unitDimension".
--Fabian
On 10/06/16 17:55, Nathan Goldbaum wrote:
Hey,
The issue you're having is because you're constructing known_other_fields dynamically. All the other frontends define it statically, so it's filled out by the time it's needed in the `setup_fluid_aliases` function defined in in yt/fields/field_info_container.py. The fix is to change the order of operations - setup known_other_fields and known_particle_fields before calling the __init__ method on the superclass of OpenPMDFieldInfo.
You're also incorrectly setting the units of the field to a Unit object --- the unit entry in the known_other_fields tuple should be a string.
The following diff allows me to set the units properly for your frontend:
http://paste.yt-project.org/show/6606/
Out of curiosity - why build the known_other_fields and known_particle_fields tuples dynamically? The reason they're static for all the other frotends is to make it easy to reason about which fields are associated with a given frontend and so that the fields known by a given frontend are independent of any particular dataset you might be looking at.
By the way, it also looks like the 'all' particle union is working correctly - are you still having issues with that?
-Nathan _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Thanks you nathan and fabian,
That sounds really great!
Yes our namings are not fixed nor the number of data sets since our simulations have a high agility on solvers and methods and the openPMD markup is actually not bound to a specific domain of science (e.g., one file could be a sim, an other a CCD image from an experiment, etc. ...). Regarding the link: 1.0.0 is the released standard, the upcoming 1.0.1 just adds some clarifications and examples.
Best, Axel
On June 10, 2016 8:20:53 PM CEST, Fabian Koller f.koller@hzdr.de wrote:
Wow!
Thanks for the speedy and thorough solution. It seems to work flawlessly, based on a quick test.
I can see that the decision to produce known fields on the fly seems a bit odd, but there's a simple reason behind it: The openPMD stadard (in theory) allows for infinitely many different fields to exist, all of them with different unit-dimensions. Very little is actually static: only the basic structure of our data, not its content, not its names, not its units and dimensions. Trying to cover as much of the standard as is feasible, we decided to dynamically generate known fields. (Also, having done it this way has helped me internalize the standard. A lot.)
If you're interested our reasoning more than this brief explanation, I'd refer you to the current (well, upcoming) written standard: https://github.com/openPMD/openPMD-standard/blob/upcoming-1.0.1/STANDARD.md, in particular "unitDimension".
--Fabian
On 10/06/16 17:55, Nathan Goldbaum wrote:
Hey,
The issue you're having is because you're constructing known_other_fields dynamically. All the other frontends define it statically, so it's filled out by the time it's needed in the `setup_fluid_aliases` function defined in in yt/fields/field_info_container.py. The fix is to change the order of operations - setup known_other_fields and known_particle_fields
before
calling the __init__ method on the superclass of OpenPMDFieldInfo.
You're also incorrectly setting the units of the field to a Unit object --- the unit entry in the known_other_fields tuple should be a string.
The following diff allows me to set the units properly for your
frontend:
http://paste.yt-project.org/show/6606/
Out of curiosity - why build the known_other_fields and known_particle_fields tuples dynamically? The reason they're static for all the other frotends is to make it easy to reason about which fields are associated with a given frontend and so that the fields known by a given frontend are independent of any particular dataset you might be looking at.
By the way, it also looks like the 'all' particle union is working correctly - are you still having issues with that?
-Nathan _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
The latest development version of yt now has “unit systems”:
http://yt-project.org/docs/dev/analyzing/units/unit_systems.html http://yt-project.org/docs/dev/analyzing/units/unit_systems.html
But Nathan is correct—this doesn’t affect the Tesla unit per se—it should be fine. If we could get a script and a more specific error that you are seeing, that would help.
On May 24, 2016, at 12:30 PM, Huebl, Axel a.huebl@hzdr.de wrote:
Hi yt-devs,
I am forwarding this conversation to the public list to keep it open for others to join in and to build up a record that might help others.
Our new student Fabian updated our frontend and implemented particle readers. Unfortunately some details of the particle "fields" still cause some bumps that we can not get our head around.
We are now able to read fields and particles, even chunk-wise, and can work with the data we read in python scripts via the all_data() method.
Nevertheless, using the particle scatter plots such as yt.ParticlePlot seems not to work. We have some problems with unions / the "all" group and the fact that we have several particle species (e.g., "hot electrons", "cold electrons", "helium ions", "nitrogen ions", etc.)
Also since the last rebase: did someone recently change the unit system in yt? T for "Tesla" seems not to be understood any more. Is there a changelog somewhere available? Do we have to describe our data in cgs / mks?
Would someone be interested for a quick heads up via e.g., skype / web-rtc so we can ask specific questions?
Fabian is around on Tuesdays and Fridays, something like early CA time, and late GER time usually works great (e.g., 9pm-PDT / 6pm-CEST).
Our current branch HEAD can be found here (->yt->frontends->openPMD): https://bitbucket.org/C0nsultant/openpmd/src?at=yt
And an example script is attached.
Thanks a lot! Axel <read_openPMD.py>_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi there,
I am currently trying to extend the WIP frontend for openPMD.
Let me be more specific about the magnetic unit problem: The already existing frontend was created on top of yt 3.2.1 and appeared to work fine. Recently, the code was migrated onto the yt_analysis/yt branch, which I understand to be your dev version. What appears to be the problem is this line of code in our openPMDDataset class: self.magnetic_unit = self.quan(1.0, "T")
This worked with out noticeable problems with the old codebase. After the migration, loading a oPMD file throws YTUnitsNotReducible:
In [1]: import yt In [2]: f = yt.load('/data/home/koller47/openpmd/example-2d/hdf5/data00000100.h5') Warning: Attribute author (recommended) does NOT exist in `/`! Found 1 iteration(s) Iteration 100 : found 4 meshes Iteration 100 : found 2 particle species Warning: Key particlePatches (recommended) does NOT exist in `/data/100/particles/Hydrogen1+`! Warning: Key particlePatches (recommended) does NOT exist in `/data/100/particles/electrons`! yt : [INFO ] 2016-05-26 19:22:56,315 openPMD: found 5 iterations in directory yt : [WARNING ] 2016-05-26 19:22:56,316 openPMD: only choose to load the first iteration (100) --------------------------------------------------------------------------- YTUnitsNotReducible Traceback (most recent call last) <ipython-input-7-b3e410e545a5> in <module>() ----> 1 f = yt.load('/data/home/koller47/openpmd/example-2d/hdf5/data00000100.h5')
/data/home/koller47/openpmd/yt/convenience.pyc in load(*args, **kwargs) 84 candidates = find_lowest_subclasses(candidates) 85 if len(candidates) == 1: ---> 86 return candidates[0](*args, **kwargs) 87 if len(candidates) == 0: 88 if ytcfg.get("yt", "enzo_db") != '' \
/data/home/koller47/openpmd/yt/frontends/openPMD/data_structures.pyc in __init__(self, filename, dataset_type, storage_filename, units_override) 294 self._setBasePath(self._handle, self._filepath) 295 Dataset.__init__(self, filename, dataset_type, --> 296 units_override=units_override) 297 self.storage_filename = storage_filename 298 self.fluid_types += ('openPMD',)
/data/home/koller47/openpmd/yt/data_objects/static_output.pyc in __init__(self, filename, dataset_type, file_style, units_override, unit_system) 235 self._create_unit_registry() 236 self._parse_parameter_file() --> 237 self.set_units() 238 self._setup_coordinate_handler() 239
/data/home/koller47/openpmd/yt/data_objects/static_output.pyc in set_units(self) 806 self.unit_registry.modify('a', 1/(1+self.current_redshift)) 807 --> 808 self.set_code_units() 809 810 if getattr(self, "cosmological_simulation", False):
/data/home/koller47/openpmd/yt/data_objects/static_output.pyc in set_code_units(self) 843 # If we do not have this set, but some fields come in in 844 # "code_magnetic", this will allow them to remain in that unit. --> 845 self.unit_registry.modify("code_magnetic", self.magnetic_unit) 846 vel_unit = getattr( 847 self, "velocity_unit", self.length_unit / self.time_unit)
/data/home/koller47/openpmd/yt/units/unit_registry.pyc in modify(self, symbol, base_value) 107 108 if hasattr(base_value, "in_base"): --> 109 base_value = base_value.in_base().value 110 111 self.lut[symbol] = (float(base_value), ) + self.lut[symbol][1:]
/data/home/koller47/openpmd/yt/units/yt_array.pyc in in_base(self, unit_system) 559 >>> E_new = E.in_base(unit_system="galactic") 560 """ --> 561 return self.in_units(self.units.get_base_equivalent(unit_system)) 562 563 def in_cgs(self):
/data/home/koller47/openpmd/yt/units/unit_object.pyc in get_base_equivalent(self, unit_system) 421 if unit_system == "cgs": 422 if current_mks in self.dimensions.free_symbols: --> 423 raise YTUnitsNotReducible(self, "cgs") 424 return yt_base_unit 425 else:
YTUnitsNotReducible: The unit 'T' cannot be reduced to a single expression within the cgs base system of units.
With self.magnetic_unit = self.quan(1.0, "gauss") loading the file works fine and as expected.
As for openPMD test data, you can get 2D and 3D ones from here: https://github.com/openPMD/openPMD-example-datasets
On 24/05/16 18:54, John Zuhone wrote:
The latest development version of yt now has “unit systems”:
http://yt-project.org/docs/dev/analyzing/units/unit_systems.html
But Nathan is correct—this doesn’t affect the Tesla unit per se—it should be fine. If we could get a script and a more specific error that you are seeing, that would help.
On May 24, 2016, at 12:30 PM, Huebl, Axel <a.huebl@hzdr.de mailto:a.huebl@hzdr.de> wrote:
Hi yt-devs,
I am forwarding this conversation to the public list to keep it open for others to join in and to build up a record that might help others.
Our new student Fabian updated our frontend and implemented particle readers. Unfortunately some details of the particle "fields" still cause some bumps that we can not get our head around.
We are now able to read fields and particles, even chunk-wise, and can work with the data we read in python scripts via the all_data() method.
Nevertheless, using the particle scatter plots such as yt.ParticlePlot seems not to work. We have some problems with unions / the "all" group and the fact that we have several particle species (e.g., "hot electrons", "cold electrons", "helium ions", "nitrogen ions", etc.)
Also since the last rebase: did someone recently change the unit system in yt? T for "Tesla" seems not to be understood any more. Is there a changelog somewhere available? Do we have to describe our data in cgs / mks?
Would someone be interested for a quick heads up via e.g., skype / web-rtc so we can ask specific questions?
Fabian is around on Tuesdays and Fridays, something like early CA time, and late GER time usually works great (e.g., 9pm-PDT / 6pm-CEST).
Our current branch HEAD can be found here (->yt->frontends->openPMD): https://bitbucket.org/C0nsultant/openpmd/src?at=yt
And an example script is attached.
Thanks a lot! Axel <read_openPMD.py>_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org mailto:yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Fabian,
OK—we now know exactly what is going on.
First, some background on some changes to the way units in general (and electromagnetic units in particular) are handled in yt:
Even though superficially 1 G ~ 1.0e-4 T, it is not strictly true because
1 G = 1 sqrt(g)/(sqrt(cm)*s) —> dimensions of sqrt(mass)/(sqrt(length)*time)
1 T = 1 kg/(A*s**2) —> dimensions mass/(current*time**2)
since SI has a fundamental unit of current and CGS does not, it means that you cannot strictly convert between the two since yt unit conversions are based on converting between objects with the same dimensions.
We ignored this discrepancy in previous versions of yt, and simply allowed conversions between G and T, but in order to make things more self-consistent and better support MKS units we have now implemented the notion of separate unit systems, which means that you cannot convert directly between G and T anymore since they are not the same dimensions.
However, you can associate a specific set of units (such as MKS/SI) with a particular frontend, by adding this keyword in the call to __init__ like this:
class openPMDDataset(Dataset): def __init__(self, filename, …, unit_system=“mks”)
However, as Nathan has pointed out to me, though this is supposed to work it won’t because we need to make sure the dataset’s unit registry itself has a unit system. This will require a patch, which we are happy to do and provide for you to test to make sure we can get this to work for you.
I should note that regardless of what units your magnetic field is in you can make conversions between MKS/SI and CGS EM units using the to_equivalent() method for YTArrays and YTQuantities:
B = YTQuantity(1.0,"T") # magnetic field in Tesla print (B, B.to_equivalent("gauss","CGS")) # convert to Gauss
There is more information about this here:
http://yt-project.org/docs/dev/analyzing/units/unit_equivalencies.html#Elect... http://yt-project.org/docs/dev/analyzing/units/unit_equivalencies.html#Electromagnetic-Equivalencies
You can see the rationale for the new unit system setup here:
http://ytep.readthedocs.io/en/latest/YTEPs/YTEP-0028.html http://ytep.readthedocs.io/en/latest/YTEPs/YTEP-0028.html
and, most relevant for you, the specific language on magnetic fields here:
http://ytep.readthedocs.io/en/latest/YTEPs/YTEP-0028.html#special-handling-f... http://ytep.readthedocs.io/en/latest/YTEPs/YTEP-0028.html#special-handling-for-magnetic-fields
Let me know if you have any other questions about any of this, or if any of it is unclear—I think Nathan wants to work on this.
Best,
John
On May 26, 2016, at 1:59 PM, Fabian Koller f.koller@hzdr.de wrote:
Hi there,
I am currently trying to extend the WIP frontend for openPMD.
Let me be more specific about the magnetic unit problem: The already existing frontend was created on top of yt 3.2.1 and appeared to work fine. Recently, the code was migrated onto the yt_analysis/yt branch, which I understand to be your dev version. What appears to be the problem is this line of code in our openPMDDataset class: self.magnetic_unit = self.quan(1.0, "T")
This worked with out noticeable problems with the old codebase. After the migration, loading a oPMD file throws YTUnitsNotReducible:
In [1]: import yt In [2]: f = yt.load('/data/home/koller47/openpmd/example-2d/hdf5/data00000100.h5') Warning: Attribute author (recommended) does NOT exist in `/`! Found 1 iteration(s) Iteration 100 : found 4 meshes Iteration 100 : found 2 particle species Warning: Key particlePatches (recommended) does NOT exist in `/data/100/particles/Hydrogen1+`! Warning: Key particlePatches (recommended) does NOT exist in `/data/100/particles/electrons`! yt : [INFO ] 2016-05-26 19:22:56,315 openPMD: found 5 iterations in directory yt : [WARNING ] 2016-05-26 19:22:56,316 openPMD: only choose to load the first iteration (100)
YTUnitsNotReducible Traceback (most recent call last) <ipython-input-7-b3e410e545a5> in <module>() ----> 1 f = yt.load('/data/home/koller47/openpmd/example-2d/hdf5/data00000100.h5')
/data/home/koller47/openpmd/yt/convenience.pyc in load(*args, **kwargs) 84 candidates = find_lowest_subclasses(candidates) 85 if len(candidates) == 1: ---> 86 return candidates[0](*args, **kwargs) 87 if len(candidates) == 0: 88 if ytcfg.get("yt", "enzo_db") != '' \
/data/home/koller47/openpmd/yt/frontends/openPMD/data_structures.pyc in __init__(self, filename, dataset_type, storage_filename, units_override) 294 self._setBasePath(self._handle, self._filepath) 295 Dataset.__init__(self, filename, dataset_type, --> 296 units_override=units_override) 297 self.storage_filename = storage_filename 298 self.fluid_types += ('openPMD',)
/data/home/koller47/openpmd/yt/data_objects/static_output.pyc in __init__(self, filename, dataset_type, file_style, units_override, unit_system) 235 self._create_unit_registry() 236 self._parse_parameter_file() --> 237 self.set_units() 238 self._setup_coordinate_handler() 239
/data/home/koller47/openpmd/yt/data_objects/static_output.pyc in set_units(self) 806 self.unit_registry.modify('a', 1/(1+self.current_redshift)) 807 --> 808 self.set_code_units() 809 810 if getattr(self, "cosmological_simulation", False):
/data/home/koller47/openpmd/yt/data_objects/static_output.pyc in set_code_units(self) 843 # If we do not have this set, but some fields come in in 844 # "code_magnetic", this will allow them to remain in that unit. --> 845 self.unit_registry.modify("code_magnetic", self.magnetic_unit) 846 vel_unit = getattr( 847 self, "velocity_unit", self.length_unit / self.time_unit)
/data/home/koller47/openpmd/yt/units/unit_registry.pyc in modify(self, symbol, base_value) 107 108 if hasattr(base_value, "in_base"): --> 109 base_value = base_value.in_base().value 110 111 self.lut[symbol] = (float(base_value), ) + self.lut[symbol][1:]
/data/home/koller47/openpmd/yt/units/yt_array.pyc in in_base(self, unit_system) 559 >>> E_new = E.in_base(unit_system="galactic") 560 """ --> 561 return self.in_units(self.units.get_base_equivalent(unit_system)) 562 563 def in_cgs(self):
/data/home/koller47/openpmd/yt/units/unit_object.pyc in get_base_equivalent(self, unit_system) 421 if unit_system == "cgs": 422 if current_mks in self.dimensions.free_symbols: --> 423 raise YTUnitsNotReducible(self, "cgs") 424 return yt_base_unit 425 else:
YTUnitsNotReducible: The unit 'T' cannot be reduced to a single expression within the cgs base system of units.
With self.magnetic_unit = self.quan(1.0, "gauss") loading the file works fine and as expected.
As for openPMD test data, you can get 2D and 3D ones from here: https://github.com/openPMD/openPMD-example-datasets
On 24/05/16 18:54, John Zuhone wrote:
The latest development version of yt now has “unit systems”:
http://yt-project.org/docs/dev/analyzing/units/unit_systems.html
But Nathan is correct—this doesn’t affect the Tesla unit per se—it should be fine. If we could get a script and a more specific error that you are seeing, that would help.
On May 24, 2016, at 12:30 PM, Huebl, Axel <a.huebl@hzdr.de mailto:a.huebl@hzdr.de> wrote:
Hi yt-devs,
I am forwarding this conversation to the public list to keep it open for others to join in and to build up a record that might help others.
Our new student Fabian updated our frontend and implemented particle readers. Unfortunately some details of the particle "fields" still cause some bumps that we can not get our head around.
We are now able to read fields and particles, even chunk-wise, and can work with the data we read in python scripts via the all_data() method.
Nevertheless, using the particle scatter plots such as yt.ParticlePlot seems not to work. We have some problems with unions / the "all" group and the fact that we have several particle species (e.g., "hot electrons", "cold electrons", "helium ions", "nitrogen ions", etc.)
Also since the last rebase: did someone recently change the unit system in yt? T for "Tesla" seems not to be understood any more. Is there a changelog somewhere available? Do we have to describe our data in cgs / mks?
Would someone be interested for a quick heads up via e.g., skype / web-rtc so we can ask specific questions?
Fabian is around on Tuesdays and Fridays, something like early CA time, and late GER time usually works great (e.g., 9pm-PDT / 6pm-CEST).
Our current branch HEAD can be found here (->yt->frontends->openPMD): https://bitbucket.org/C0nsultant/openpmd/src?at=yt
And an example script is attached.
Thanks a lot! Axel <read_openPMD.py>_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org mailto:yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Axel,
I could do a Google hangout or something on Friday. In the meantime, I agree with Nathan that it looks like some particle fields that are supposed to be set up automatically aren't getting created for some reason, but it's hard to say exactly what's going wrong. If you can reproduce this problem on a public dataset I could download, I'd be happy to take a look.
On Tue, May 24, 2016 at 9:30 AM, Huebl, Axel a.huebl@hzdr.de wrote:
Hi yt-devs,
I am forwarding this conversation to the public list to keep it open for others to join in and to build up a record that might help others.
Our new student Fabian updated our frontend and implemented particle readers. Unfortunately some details of the particle "fields" still cause some bumps that we can not get our head around.
We are now able to read fields and particles, even chunk-wise, and can work with the data we read in python scripts via the all_data() method.
Nevertheless, using the particle scatter plots such as yt.ParticlePlot seems not to work. We have some problems with unions / the "all" group and the fact that we have several particle species (e.g., "hot electrons", "cold electrons", "helium ions", "nitrogen ions", etc.)
Also since the last rebase: did someone recently change the unit system in yt? T for "Tesla" seems not to be understood any more. Is there a changelog somewhere available? Do we have to describe our data in cgs / mks?
Would someone be interested for a quick heads up via e.g., skype / web-rtc so we can ask specific questions?
Fabian is around on Tuesdays and Fridays, something like early CA time, and late GER time usually works great (e.g., 9pm-PDT / 6pm-CEST).
Our current branch HEAD can be found here (->yt->frontends->openPMD): https://bitbucket.org/C0nsultant/openpmd/src?at=yt
And an example script is attached.
Thanks a lot! Axel
participants (7)
-
Andrew Myers -
Axel Huebl -
Fabian Koller -
Huebl, Axel
-
John Zuhone -
John ZuHone
-
Nathan Goldbaum