in-situ visualization possible with yt?
Hello, I'm trying to make in-situ visualization available for a simulation, and I'm considering different softwares for that (VisIt, YT,...). I've never used YT, so my question is: is it possible to add some code in a simulation (written in C++) to have some kind of distributed rendering engine using YT, just like VisIt does with code-instrumentation? Thanks, Matthieu -- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
Hi Matthieu, The short answer is that for some codes, yes, yt works as an embedding system. The only code that this has been used extensively in is Enzo, and it will likely take a bit of work to get it to work. We're eager to test it out on other codes, however; the first and easiest application would be a non-Enzo, patch-based AMR code. The current state is: * In-memory between Enzo and yt works (Britton Smith can speak more to this) * Creation of arbitrary sources (using the Stream frontend) currently works, which can be used from within any Python code * No protocol exists (yet) for MPI inter-communicator data loading Our strategies for the future: * We're working to create simple C libraries that can create the necessary in-memory data objects to link against yt directly * The Stream frontend is usable and should be used by non-Enzo codes as a prototyping mechanism * ParaView can call yt for serial calculations, and we are working to utilize its CoProcessing facilities to analyze and visualize in parallel from running simulations So I think you should be able to add a code to in situ viz/analysis, but it will likely take a bit of work at the current time to make sure everything correctly passes. I'd be able to help you and send you demo code if you would like. Alternately, we intend to instrument yt to use ParaView as a data loader, which will enable the CoProcessing techniques from ParaView to be used as well. Let me know what you think, Matt On Tue, Oct 4, 2011 at 1:28 PM, Matthieu Dorier <matthieu.dorier@gmail.com> wrote:
Hello,
I'm trying to make in-situ visualization available for a simulation, and I'm considering different softwares for that (VisIt, YT,...). I've never used YT, so my question is: is it possible to add some code in a simulation (written in C++) to have some kind of distributed rendering engine using YT, just like VisIt does with code-instrumentation? Thanks,
Matthieu
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hello Matthew, Thank you for your answer. Can YT work in parallel? I intend to embed a python interpreter within my simulation in order to allow some simple data analysis to be performed without having to re-compile the application every time this analysis changes, so maybe it would be possible to call YT from an embedded python interpreter? I guess the problem is transferring raw data from the simulation to the interpreter without too much overhead... My output is actually very simple, consisting in 2D arrays, each MPI rank being responsible for several chunks (equally sized, but not necessarily contiguous). I would like to avoid writing these 2D arrays into HDF5 file and perform visualization on these files, but automatically make the processes synchronize to compute a single image from all these parts. Do you think it is worth investigating the use of YT for that or should I try other softwares? Thanks Matthieu 2011/10/5 Matthew Turk <matthewturk@gmail.com>
Hi Matthieu,
The short answer is that for some codes, yes, yt works as an embedding system. The only code that this has been used extensively in is Enzo, and it will likely take a bit of work to get it to work. We're eager to test it out on other codes, however; the first and easiest application would be a non-Enzo, patch-based AMR code.
The current state is:
* In-memory between Enzo and yt works (Britton Smith can speak more to this) * Creation of arbitrary sources (using the Stream frontend) currently works, which can be used from within any Python code * No protocol exists (yet) for MPI inter-communicator data loading
Our strategies for the future:
* We're working to create simple C libraries that can create the necessary in-memory data objects to link against yt directly * The Stream frontend is usable and should be used by non-Enzo codes as a prototyping mechanism * ParaView can call yt for serial calculations, and we are working to utilize its CoProcessing facilities to analyze and visualize in parallel from running simulations
So I think you should be able to add a code to in situ viz/analysis, but it will likely take a bit of work at the current time to make sure everything correctly passes. I'd be able to help you and send you demo code if you would like. Alternately, we intend to instrument yt to use ParaView as a data loader, which will enable the CoProcessing techniques from ParaView to be used as well.
Let me know what you think,
Matt
On Tue, Oct 4, 2011 at 1:28 PM, Matthieu Dorier <matthieu.dorier@gmail.com> wrote:
Hello,
I'm trying to make in-situ visualization available for a simulation, and I'm considering different softwares for that (VisIt, YT,...). I've never used YT, so my question is: is it possible to add some code in a simulation (written in C++) to have some kind of distributed rendering engine using YT, just like VisIt does with code-instrumentation? Thanks,
Matthieu
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
Hi Matthieu, On Wed, Oct 5, 2011 at 11:45 AM, Matthieu Dorier <matthieu.dorier@gmail.com> wrote:
Hello Matthew,
Thank you for your answer.
Can YT work in parallel?
Yup.
I intend to embed a python interpreter within my simulation in order to allow some simple data analysis to be performed without having to re-compile the application every time this analysis changes, so maybe it would be possible to call YT from an embedded python interpreter? I guess the problem is transferring raw data from the simulation to the interpreter without too much overhead...
That's actually how Enzo works as well. We then construct numpy array wrappers around the already-allocated data, and then feed those to yt in a dict.
My output is actually very simple, consisting in 2D arrays, each MPI rank being responsible for several chunks (equally sized, but not necessarily contiguous). I would like to avoid writing these 2D arrays into HDF5 file and perform visualization on these files, but automatically make the processes synchronize to compute a single image from all these parts.
That's what we do with Enzo, except with 3D. There may be some oddities to work out with 2D data.
Do you think it is worth investigating the use of YT for that or should I try other softwares? Thanks
Matthieu
I think you should give a look at yt -- but also guide your views by reading how we do in situ with Enzo. If you look in the Enzo source code at InitializePythonInterface.C and ExposeDataHierarchy.C you can see our current method, although I believe in the future with better memory-pool allocations in Enzo we'll make this a bit more streamlined. Please don't hesitate to write back with more questions! Best, Matt
2011/10/5 Matthew Turk <matthewturk@gmail.com>
Hi Matthieu,
The short answer is that for some codes, yes, yt works as an embedding system. The only code that this has been used extensively in is Enzo, and it will likely take a bit of work to get it to work. We're eager to test it out on other codes, however; the first and easiest application would be a non-Enzo, patch-based AMR code.
The current state is:
* In-memory between Enzo and yt works (Britton Smith can speak more to this) * Creation of arbitrary sources (using the Stream frontend) currently works, which can be used from within any Python code * No protocol exists (yet) for MPI inter-communicator data loading
Our strategies for the future:
* We're working to create simple C libraries that can create the necessary in-memory data objects to link against yt directly * The Stream frontend is usable and should be used by non-Enzo codes as a prototyping mechanism * ParaView can call yt for serial calculations, and we are working to utilize its CoProcessing facilities to analyze and visualize in parallel from running simulations
So I think you should be able to add a code to in situ viz/analysis, but it will likely take a bit of work at the current time to make sure everything correctly passes. I'd be able to help you and send you demo code if you would like. Alternately, we intend to instrument yt to use ParaView as a data loader, which will enable the CoProcessing techniques from ParaView to be used as well.
Let me know what you think,
Matt
On Tue, Oct 4, 2011 at 1:28 PM, Matthieu Dorier <matthieu.dorier@gmail.com> wrote:
Hello,
I'm trying to make in-situ visualization available for a simulation, and I'm considering different softwares for that (VisIt, YT,...). I've never used YT, so my question is: is it possible to add some code in a simulation (written in C++) to have some kind of distributed rendering engine using YT, just like VisIt does with code-instrumentation? Thanks,
Matthieu
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hello, I come back to this discussion, as I have instrumented my simulation to wrap the C++ arrays into NumPy arrays and call a python script in each process. This currently works and now I would like to actually do something with these NumPy arrays. I can't find any tutorial on the YT homepage on how to create an image from a NumPy array. (I'm not only new to YT, I'm also new to Python and learned the basic syntax while coding the wrapper). Let's imagine I have a 2D NumPy array of floating point values, and I want to plot it as an image, how would the python code look like? Thanks Matthieu 2011/10/5 Matthew Turk <matthewturk@gmail.com>
Hi Matthieu,
On Wed, Oct 5, 2011 at 11:45 AM, Matthieu Dorier <matthieu.dorier@gmail.com> wrote:
Hello Matthew,
Thank you for your answer.
Can YT work in parallel?
Yup.
I intend to embed a python interpreter within my simulation in order to allow some simple data analysis to be performed without having to re-compile the application every time this analysis changes, so maybe it would be possible to call YT from an embedded python interpreter? I guess the problem is transferring raw data from the simulation to the interpreter without too much overhead...
That's actually how Enzo works as well. We then construct numpy array wrappers around the already-allocated data, and then feed those to yt in a dict.
My output is actually very simple, consisting in 2D arrays, each MPI rank being responsible for several chunks (equally sized, but not necessarily contiguous). I would like to avoid writing these 2D arrays into HDF5 file and perform visualization on these files, but automatically make the processes synchronize to compute a single image from all these parts.
That's what we do with Enzo, except with 3D. There may be some oddities to work out with 2D data.
Do you think it is worth investigating the use of YT for that or should I try other softwares? Thanks
Matthieu
I think you should give a look at yt -- but also guide your views by reading how we do in situ with Enzo. If you look in the Enzo source code at InitializePythonInterface.C and ExposeDataHierarchy.C you can see our current method, although I believe in the future with better memory-pool allocations in Enzo we'll make this a bit more streamlined.
Please don't hesitate to write back with more questions!
Best,
Matt
2011/10/5 Matthew Turk <matthewturk@gmail.com>
Hi Matthieu,
The short answer is that for some codes, yes, yt works as an embedding system. The only code that this has been used extensively in is Enzo, and it will likely take a bit of work to get it to work. We're eager to test it out on other codes, however; the first and easiest application would be a non-Enzo, patch-based AMR code.
The current state is:
* In-memory between Enzo and yt works (Britton Smith can speak more to this) * Creation of arbitrary sources (using the Stream frontend) currently works, which can be used from within any Python code * No protocol exists (yet) for MPI inter-communicator data loading
Our strategies for the future:
* We're working to create simple C libraries that can create the necessary in-memory data objects to link against yt directly * The Stream frontend is usable and should be used by non-Enzo codes as a prototyping mechanism * ParaView can call yt for serial calculations, and we are working to utilize its CoProcessing facilities to analyze and visualize in parallel from running simulations
So I think you should be able to add a code to in situ viz/analysis, but it will likely take a bit of work at the current time to make sure everything correctly passes. I'd be able to help you and send you demo code if you would like. Alternately, we intend to instrument yt to use ParaView as a data loader, which will enable the CoProcessing techniques from ParaView to be used as well.
Let me know what you think,
Matt
On Tue, Oct 4, 2011 at 1:28 PM, Matthieu Dorier <matthieu.dorier@gmail.com> wrote:
Hello,
I'm trying to make in-situ visualization available for a simulation, and I'm considering different softwares for that (VisIt, YT,...). I've never used YT, so my question is: is it possible to add some code in a simulation (written in C++) to have some kind of distributed rendering engine using YT, just like VisIt does with code-instrumentation? Thanks,
Matthieu
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
Hi Matthieu, If you already have arrays that you are ready to plot, you are probably better going straight to matplotlib from here. You can find their documentation here: http://matplotlib.sourceforge.net/ This is the package used for most of the plotting and visualization that yt does. For example, you would probably want to use pcolormesh to plot up a 2D numpy array: http://matplotlib.sourceforge.net/api/pyplot_api.html#matplotlib.pyplot.pcol... Britton On Tue, Nov 8, 2011 at 8:50 AM, Matthieu Dorier <matthieu.dorier@gmail.com>wrote:
Hello,
I come back to this discussion, as I have instrumented my simulation to wrap the C++ arrays into NumPy arrays and call a python script in each process. This currently works and now I would like to actually do something with these NumPy arrays. I can't find any tutorial on the YT homepage on how to create an image from a NumPy array. (I'm not only new to YT, I'm also new to Python and learned the basic syntax while coding the wrapper).
Let's imagine I have a 2D NumPy array of floating point values, and I want to plot it as an image, how would the python code look like?
Thanks
Matthieu
2011/10/5 Matthew Turk <matthewturk@gmail.com>
Hi Matthieu,
On Wed, Oct 5, 2011 at 11:45 AM, Matthieu Dorier <matthieu.dorier@gmail.com> wrote:
Hello Matthew,
Thank you for your answer.
Can YT work in parallel?
Yup.
I intend to embed a python interpreter within my simulation in order to allow some simple data analysis to be performed without having to re-compile the application every time this analysis changes, so maybe it would be possible to call YT from an embedded python interpreter? I guess the problem is transferring raw data from the simulation to the interpreter without too much overhead...
That's actually how Enzo works as well. We then construct numpy array wrappers around the already-allocated data, and then feed those to yt in a dict.
My output is actually very simple, consisting in 2D arrays, each MPI
rank
being responsible for several chunks (equally sized, but not necessarily contiguous). I would like to avoid writing these 2D arrays into HDF5 file and perform visualization on these files, but automatically make the processes synchronize to compute a single image from all these parts.
That's what we do with Enzo, except with 3D. There may be some oddities to work out with 2D data.
Do you think it is worth investigating the use of YT for that or should
I
try other softwares? Thanks
Matthieu
I think you should give a look at yt -- but also guide your views by reading how we do in situ with Enzo. If you look in the Enzo source code at InitializePythonInterface.C and ExposeDataHierarchy.C you can see our current method, although I believe in the future with better memory-pool allocations in Enzo we'll make this a bit more streamlined.
Please don't hesitate to write back with more questions!
Best,
Matt
2011/10/5 Matthew Turk <matthewturk@gmail.com>
Hi Matthieu,
The short answer is that for some codes, yes, yt works as an embedding system. The only code that this has been used extensively in is Enzo, and it will likely take a bit of work to get it to work. We're eager to test it out on other codes, however; the first and easiest application would be a non-Enzo, patch-based AMR code.
The current state is:
* In-memory between Enzo and yt works (Britton Smith can speak more to this) * Creation of arbitrary sources (using the Stream frontend) currently works, which can be used from within any Python code * No protocol exists (yet) for MPI inter-communicator data loading
Our strategies for the future:
* We're working to create simple C libraries that can create the necessary in-memory data objects to link against yt directly * The Stream frontend is usable and should be used by non-Enzo codes as a prototyping mechanism * ParaView can call yt for serial calculations, and we are working to utilize its CoProcessing facilities to analyze and visualize in parallel from running simulations
So I think you should be able to add a code to in situ viz/analysis, but it will likely take a bit of work at the current time to make sure everything correctly passes. I'd be able to help you and send you demo code if you would like. Alternately, we intend to instrument yt to use ParaView as a data loader, which will enable the CoProcessing techniques from ParaView to be used as well.
Let me know what you think,
Matt
On Tue, Oct 4, 2011 at 1:28 PM, Matthieu Dorier <matthieu.dorier@gmail.com> wrote:
Hello,
I'm trying to make in-situ visualization available for a simulation, and I'm considering different softwares for that (VisIt, YT,...). I've never used YT, so my question is: is it possible to add some code in a simulation (written in C++) to have some kind of distributed rendering engine using YT, just like VisIt does with code-instrumentation? Thanks,
Matthieu
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Thanks, I managed to make what I wanted with matplotlib. Matthieu 2011/11/8 Britton Smith <brittonsmith@gmail.com>
Hi Matthieu,
If you already have arrays that you are ready to plot, you are probably better going straight to matplotlib from here. You can find their documentation here: http://matplotlib.sourceforge.net/ This is the package used for most of the plotting and visualization that yt does.
For example, you would probably want to use pcolormesh to plot up a 2D numpy array:
http://matplotlib.sourceforge.net/api/pyplot_api.html#matplotlib.pyplot.pcol...
Britton
On Tue, Nov 8, 2011 at 8:50 AM, Matthieu Dorier <matthieu.dorier@gmail.com
wrote:
Hello,
I come back to this discussion, as I have instrumented my simulation to wrap the C++ arrays into NumPy arrays and call a python script in each process. This currently works and now I would like to actually do something with these NumPy arrays. I can't find any tutorial on the YT homepage on how to create an image from a NumPy array. (I'm not only new to YT, I'm also new to Python and learned the basic syntax while coding the wrapper).
Let's imagine I have a 2D NumPy array of floating point values, and I want to plot it as an image, how would the python code look like?
Thanks
Matthieu
2011/10/5 Matthew Turk <matthewturk@gmail.com>
Hi Matthieu,
On Wed, Oct 5, 2011 at 11:45 AM, Matthieu Dorier <matthieu.dorier@gmail.com> wrote:
Hello Matthew,
Thank you for your answer.
Can YT work in parallel?
Yup.
I intend to embed a python interpreter within my simulation in order to allow some simple data analysis to be performed without having to re-compile the application every time this analysis changes, so maybe it would be possible to call YT from an embedded python interpreter? I guess the problem is transferring raw data from the simulation to the interpreter without too much overhead...
That's actually how Enzo works as well. We then construct numpy array wrappers around the already-allocated data, and then feed those to yt in a dict.
My output is actually very simple, consisting in 2D arrays, each MPI
rank
being responsible for several chunks (equally sized, but not necessarily contiguous). I would like to avoid writing these 2D arrays into HDF5 file and perform visualization on these files, but automatically make the processes synchronize to compute a single image from all these parts.
That's what we do with Enzo, except with 3D. There may be some oddities to work out with 2D data.
Do you think it is worth investigating the use of YT for that or
should I
try other softwares? Thanks
Matthieu
I think you should give a look at yt -- but also guide your views by reading how we do in situ with Enzo. If you look in the Enzo source code at InitializePythonInterface.C and ExposeDataHierarchy.C you can see our current method, although I believe in the future with better memory-pool allocations in Enzo we'll make this a bit more streamlined.
Please don't hesitate to write back with more questions!
Best,
Matt
2011/10/5 Matthew Turk <matthewturk@gmail.com>
Hi Matthieu,
The short answer is that for some codes, yes, yt works as an embedding system. The only code that this has been used extensively in is Enzo, and it will likely take a bit of work to get it to work. We're eager to test it out on other codes, however; the first and easiest application would be a non-Enzo, patch-based AMR code.
The current state is:
* In-memory between Enzo and yt works (Britton Smith can speak more
to
this) * Creation of arbitrary sources (using the Stream frontend) currently works, which can be used from within any Python code * No protocol exists (yet) for MPI inter-communicator data loading
Our strategies for the future:
* We're working to create simple C libraries that can create the necessary in-memory data objects to link against yt directly * The Stream frontend is usable and should be used by non-Enzo codes as a prototyping mechanism * ParaView can call yt for serial calculations, and we are working to utilize its CoProcessing facilities to analyze and visualize in parallel from running simulations
So I think you should be able to add a code to in situ viz/analysis, but it will likely take a bit of work at the current time to make sure everything correctly passes. I'd be able to help you and send you demo code if you would like. Alternately, we intend to instrument yt to use ParaView as a data loader, which will enable the CoProcessing techniques from ParaView to be used as well.
Let me know what you think,
Matt
On Tue, Oct 4, 2011 at 1:28 PM, Matthieu Dorier <matthieu.dorier@gmail.com> wrote:
Hello,
I'm trying to make in-situ visualization available for a simulation, and I'm considering different softwares for that (VisIt, YT,...). I've never used YT, so my question is: is it possible to add some code in a simulation (written in C++) to have some kind of distributed rendering engine using YT, just like VisIt does with code-instrumentation? Thanks,
Matthieu
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
Hi Matthieu, There's been quite a bit of interest in providing data to yt from in-memory datasets. You can try something like this: http://hg.yt-project.org/yt/wiki/InterfacingWithYT The idea would be that if you *have* the arrays, you should be able to describe their mesh to yt very easily. Unfortunately, the API is as you can see clunky. But I must warn you, the interface is in flux. Fixing this up is a goal for the month of December, both documentation and API-wise. Would you be interested in helping us make this process more streamlined? -Matt On Tue, Nov 8, 2011 at 2:31 PM, Matthieu Dorier <matthieu.dorier@gmail.com> wrote:
Thanks, I managed to make what I wanted with matplotlib.
Matthieu
2011/11/8 Britton Smith <brittonsmith@gmail.com>
Hi Matthieu,
If you already have arrays that you are ready to plot, you are probably better going straight to matplotlib from here. You can find their documentation here: http://matplotlib.sourceforge.net/ This is the package used for most of the plotting and visualization that yt does.
For example, you would probably want to use pcolormesh to plot up a 2D numpy array:
http://matplotlib.sourceforge.net/api/pyplot_api.html#matplotlib.pyplot.pcol...
Britton
On Tue, Nov 8, 2011 at 8:50 AM, Matthieu Dorier <matthieu.dorier@gmail.com> wrote:
Hello, I come back to this discussion, as I have instrumented my simulation to wrap the C++ arrays into NumPy arrays and call a python script in each process. This currently works and now I would like to actually do something with these NumPy arrays. I can't find any tutorial on the YT homepage on how to create an image from a NumPy array. (I'm not only new to YT, I'm also new to Python and learned the basic syntax while coding the wrapper). Let's imagine I have a 2D NumPy array of floating point values, and I want to plot it as an image, how would the python code look like? Thanks Matthieu 2011/10/5 Matthew Turk <matthewturk@gmail.com>
Hi Matthieu,
On Wed, Oct 5, 2011 at 11:45 AM, Matthieu Dorier <matthieu.dorier@gmail.com> wrote:
Hello Matthew,
Thank you for your answer.
Can YT work in parallel?
Yup.
I intend to embed a python interpreter within my simulation in order to allow some simple data analysis to be performed without having to re-compile the application every time this analysis changes, so maybe it would be possible to call YT from an embedded python interpreter? I guess the problem is transferring raw data from the simulation to the interpreter without too much overhead...
That's actually how Enzo works as well. We then construct numpy array wrappers around the already-allocated data, and then feed those to yt in a dict.
My output is actually very simple, consisting in 2D arrays, each MPI rank being responsible for several chunks (equally sized, but not necessarily contiguous). I would like to avoid writing these 2D arrays into HDF5 file and perform visualization on these files, but automatically make the processes synchronize to compute a single image from all these parts.
That's what we do with Enzo, except with 3D. There may be some oddities to work out with 2D data.
Do you think it is worth investigating the use of YT for that or should I try other softwares? Thanks
Matthieu
I think you should give a look at yt -- but also guide your views by reading how we do in situ with Enzo. If you look in the Enzo source code at InitializePythonInterface.C and ExposeDataHierarchy.C you can see our current method, although I believe in the future with better memory-pool allocations in Enzo we'll make this a bit more streamlined.
Please don't hesitate to write back with more questions!
Best,
Matt
2011/10/5 Matthew Turk <matthewturk@gmail.com>
Hi Matthieu,
The short answer is that for some codes, yes, yt works as an embedding system. The only code that this has been used extensively in is Enzo, and it will likely take a bit of work to get it to work. We're eager to test it out on other codes, however; the first and easiest application would be a non-Enzo, patch-based AMR code.
The current state is:
* In-memory between Enzo and yt works (Britton Smith can speak more to this) * Creation of arbitrary sources (using the Stream frontend) currently works, which can be used from within any Python code * No protocol exists (yet) for MPI inter-communicator data loading
Our strategies for the future:
* We're working to create simple C libraries that can create the necessary in-memory data objects to link against yt directly * The Stream frontend is usable and should be used by non-Enzo codes as a prototyping mechanism * ParaView can call yt for serial calculations, and we are working to utilize its CoProcessing facilities to analyze and visualize in parallel from running simulations
So I think you should be able to add a code to in situ viz/analysis, but it will likely take a bit of work at the current time to make sure everything correctly passes. I'd be able to help you and send you demo code if you would like. Alternately, we intend to instrument yt to use ParaView as a data loader, which will enable the CoProcessing techniques from ParaView to be used as well.
Let me know what you think,
Matt
On Tue, Oct 4, 2011 at 1:28 PM, Matthieu Dorier <matthieu.dorier@gmail.com> wrote: > Hello, > > I'm trying to make in-situ visualization available for a > simulation, > and I'm considering different softwares for that (VisIt, YT,...). > I've never used YT, so my question is: is it possible to add > some code in a simulation (written in C++) to have some kind of > distributed rendering engine using YT, just like VisIt does with > code-instrumentation? > Thanks, > > Matthieu > > -- > Matthieu Dorier > ENS Cachan, antenne de Bretagne > Département informatique et télécommunication > http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > > _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Matthew, It depends, what do you mean by "helping us"? As I said I'm new to python (and to visualization in general, my work is more focused on I/O). Yet I can tell you what my use cases are, what I would like to be able to do with YT, and give feedbacks on the API and documentation when they are ready. I should mention also that my current work involves building a simulation's side library to uniformly and easily integrate multiple visualization backends (YT, matplotlib, VisIt, etc.) in simulations. VisIt provides a way to instrument simulations to act as visualization engines when required, and by embedding python like Enzo does one can transfert data from the simulation to YT or matplotlib. The goal would be to make this data exposure more transparent and more generic. Thus I'm following your progress with a lot of interest, as the way YT can work from in-memory data immediately impacts the design of my library. Matthieu 2011/11/8 Matthew Turk <matthewturk@gmail.com>
Hi Matthieu,
There's been quite a bit of interest in providing data to yt from in-memory datasets. You can try something like this:
http://hg.yt-project.org/yt/wiki/InterfacingWithYT
The idea would be that if you *have* the arrays, you should be able to describe their mesh to yt very easily. Unfortunately, the API is as you can see clunky. But I must warn you, the interface is in flux. Fixing this up is a goal for the month of December, both documentation and API-wise. Would you be interested in helping us make this process more streamlined?
-Matt
On Tue, Nov 8, 2011 at 2:31 PM, Matthieu Dorier <matthieu.dorier@gmail.com> wrote:
Thanks, I managed to make what I wanted with matplotlib.
Matthieu
2011/11/8 Britton Smith <brittonsmith@gmail.com>
Hi Matthieu,
If you already have arrays that you are ready to plot, you are probably better going straight to matplotlib from here. You can find their documentation here: http://matplotlib.sourceforge.net/ This is the package used for most of the plotting and visualization that yt does.
For example, you would probably want to use pcolormesh to plot up a 2D numpy array:
http://matplotlib.sourceforge.net/api/pyplot_api.html#matplotlib.pyplot.pcol...
Britton
On Tue, Nov 8, 2011 at 8:50 AM, Matthieu Dorier <matthieu.dorier@gmail.com> wrote:
Hello, I come back to this discussion, as I have instrumented my simulation to wrap the C++ arrays into NumPy arrays and call a python script in each process. This currently works and now I would like to actually do something with these NumPy arrays. I can't find any tutorial on the YT homepage on
how to
create an image from a NumPy array. (I'm not only new to YT, I'm also new to Python and learned the basic syntax while coding the wrapper). Let's imagine I have a 2D NumPy array of floating point values, and I want to plot it as an image, how would the python code look like? Thanks Matthieu 2011/10/5 Matthew Turk <matthewturk@gmail.com>
Hi Matthieu,
On Wed, Oct 5, 2011 at 11:45 AM, Matthieu Dorier <matthieu.dorier@gmail.com> wrote:
Hello Matthew,
Thank you for your answer.
Can YT work in parallel?
Yup.
I intend to embed a python interpreter within my simulation in order to allow some simple data analysis to be performed without having to
re-compile
the application every time this analysis changes, so maybe it would be possible to call YT from an embedded python interpreter? I guess the problem is transferring raw data from the simulation to the interpreter without too much overhead...
That's actually how Enzo works as well. We then construct numpy array wrappers around the already-allocated data, and then feed those to yt in a dict.
My output is actually very simple, consisting in 2D arrays, each MPI rank being responsible for several chunks (equally sized, but not necessarily contiguous). I would like to avoid writing these 2D arrays into HDF5 file and perform visualization on these files, but automatically make the processes synchronize to compute a single image from all these parts.
That's what we do with Enzo, except with 3D. There may be some oddities to work out with 2D data.
Do you think it is worth investigating the use of YT for that or should I try other softwares? Thanks
Matthieu
I think you should give a look at yt -- but also guide your views by reading how we do in situ with Enzo. If you look in the Enzo source code at InitializePythonInterface.C and ExposeDataHierarchy.C you can see our current method, although I believe in the future with better memory-pool allocations in Enzo we'll make this a bit more streamlined.
Please don't hesitate to write back with more questions!
Best,
Matt
2011/10/5 Matthew Turk <matthewturk@gmail.com> > > Hi Matthieu, > > The short answer is that for some codes, yes, yt works as an > embedding > system. The only code that this has been used extensively in is > Enzo, > and it will likely take a bit of work to get it to work. We're
eager
> to test it out on other codes, however; the first and easiest > application would be a non-Enzo, patch-based AMR code. > > The current state is: > > * In-memory between Enzo and yt works (Britton Smith can speak more > to > this) > * Creation of arbitrary sources (using the Stream frontend) > currently > works, which can be used from within any Python code > * No protocol exists (yet) for MPI inter-communicator data loading > > Our strategies for the future: > > * We're working to create simple C libraries that can create the > necessary in-memory data objects to link against yt directly > * The Stream frontend is usable and should be used by non-Enzo codes > as a prototyping mechanism > * ParaView can call yt for serial calculations, and we are working > to > utilize its CoProcessing facilities to analyze and visualize in > parallel from running simulations > > So I think you should be able to add a code to in situ viz/analysis, > but it will likely take a bit of work at the current time to make > sure > everything correctly passes. I'd be able to help you and send you > demo code if you would like. Alternately, we intend to instrument yt > to use ParaView as a data loader, which will enable the CoProcessing > techniques from ParaView to be used as well. > > Let me know what you think, > > Matt > > On Tue, Oct 4, 2011 at 1:28 PM, Matthieu Dorier > <matthieu.dorier@gmail.com> wrote: > > Hello, > > > > I'm trying to make in-situ visualization available for a > > simulation, > > and I'm considering different softwares for that (VisIt, YT,...). > > I've never used YT, so my question is: is it possible to add > > some code in a simulation (written in C++) to have some kind of > > distributed rendering engine using YT, just like VisIt does with > > code-instrumentation? > > Thanks, > > > > Matthieu > > > > -- > > Matthieu Dorier > > ENS Cachan, antenne de Bretagne > > Département informatique et télécommunication > > http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/ > > > > _______________________________________________ > > yt-users mailing list > > yt-users@lists.spacepope.org > > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > > > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Matthieu Dorier ENS Cachan, antenne de Bretagne Département informatique et télécommunication http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
participants (3)
-
Britton Smith -
Matthew Turk -
Matthieu Dorier