Real-time plotting and data storage format questions
Greetings, A small team of us are developing a new simulation package from the ground up. Our legacy approach relied on MATLAB and other proprietary software. A hope pf ours is to be able to shed the use of MATLAB for the analysis of our simulation results and instead use python with scipy/numpy/matplotlib etc. I've successfully installed and compiled optimized numpy/scipy and all the supporting packages (ATLAS, FFTW, etc). So far so good. To the point - I have two questions: 1) We would like to have a "scope" to monitor simulation outputs in real time. We're using one tool that can take data over a tcp/ip port, but is clunky and only works on a single platform. Does such a thing exists within the python realm for plotting data in real time? 2) Our simulation creates large (1-4 GB) data sets. Since we're writing this simulation ourselves (C++) we can save the data in any format. Does anyone have a suggestion for a specific format or API that's been found to be optimal in terms of memory usage and ability to import into python for analysis and plotting? Thank you for any suggestions. We're still new with Python, so I apologize if these questions seem mundane.
1. Will this project produce code that you will be sharing? If so, please post updates to this list! 2. Is the following of interest? Last example at <URL:http://code.enthought.com/chaco/gallery/index.shtml> Code at <URL:http://pyserial.sourceforge.net/> Comments at: <URL:https://svn.enthought.com/enthought/browser/Chaco/trunk/examples/advanced/spectrum.py> Cheers, Alan Isaac
Hi there, For the data format I strongly recommend NetCDF and especially NetCDF4 (allows for compression) In addition of NetCDF4 I strongly recommend you look at the CF conventions, which is a standard and allows software to learn about the data you're storing, see: http://cf-pcmdi.llnl.gov/ As far as an all in 1 python based software to store/analyze/plot your data, you can look at CDAT (which happens to redistribute scipy ) Hope this helps, C. heytrent@gmail.com wrote:
Greetings,
A small team of us are developing a new simulation package from the ground up. Our legacy approach relied on MATLAB and other proprietary software. A hope pf ours is to be able to shed the use of MATLAB for the analysis of our simulation results and instead use python with scipy/numpy/matplotlib etc. I've successfully installed and compiled optimized numpy/scipy and all the supporting packages (ATLAS, FFTW, etc).
So far so good.
To the point - I have two questions:
1) We would like to have a "scope" to monitor simulation outputs in real time. We're using one tool that can take data over a tcp/ip port, but is clunky and only works on a single platform. Does such a thing exists within the python realm for plotting data in real time?
2) Our simulation creates large (1-4 GB) data sets. Since we're writing this simulation ourselves (C++) we can save the data in any format. Does anyone have a suggestion for a specific format or API that's been found to be optimal in terms of memory usage and ability to import into python for analysis and plotting?
Thank you for any suggestions. We're still new with Python, so I apologize if these questions seem mundane. ------------------------------------------------------------------------
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
On Tue, Apr 01, 2008 at 10:06:35AM -0700, Charles Doutriaux wrote:
For the data format I strongly recommend NetCDF and especially NetCDF4 (allows for compression)
Actually, why not hdf5, which seems to be used by NetCDF4, but is also largely used across many scientific comunities, and very well supported under python (pytables)? My 2 cents, Gaël
For the data format I strongly recommend NetCDF and especially NetCDF4 (allows for compression)
Actually, why not hdf5, which seems to be used by NetCDF4, but is also largely used across many scientific comunities, and very well supported under python (pytables)?
My 2 cents,
Gaël
Seconded. We use HDF5 [1] in a parallel simulation code written in C+ + and Fortran, and find it to be incredibly useful. I personally use PyTables to access the data for analysis and plotting. If you have any questions, feel free to email me. --Rick [1] - http://hdfgroup.org/ ------------------------------------------------------------------------ - Rick Wagner, Graduate Student Researcher UCSD Physics 9500 Gilman Drive La Jolla, CA 92093-0424 Email: rwagner@physics.ucsd.edu WWW: http://lca.ucsd.edu/projects/rpwagner (858) 822-4784 Phone ------------------------------------------------------------------------ - Measuring programming progress by lines of code is like measuring aircraft building progress by weight. --Bill Gates ------------------------------------------------------------------------ -
I too recommend hdf5 as data format. hdf5 has become my standard data format for pretty much all my work with scientific data because pytables makes it very easy to use, and yet offers the possibility of using advanced features at the same time. Have a look at their website http://www.pytables.org . Best, Gabriel On Tue, 2008-04-01 at 19:09 +0200, Gael Varoquaux wrote:
On Tue, Apr 01, 2008 at 10:06:35AM -0700, Charles Doutriaux wrote:
For the data format I strongly recommend NetCDF and especially NetCDF4 (allows for compression)
Actually, why not hdf5, which seems to be used by NetCDF4, but is also largely used across many scientific comunities, and very well supported under python (pytables)?
My 2 cents,
Gaël _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
netCDF is mostly used for geophysical, in particular atmospheric data, although I also use it for more general physical model results. There are a wealth of command line tools for inspecting and operating on netCDF files. I found the AMBER (an MD package - http://amber.scripps.edu/netcdf/nctraj.html) convention useful in working out how to lay out my particle data in a netCDF file. I've never used HDF, so I can't give a comparison. On Wed, Apr 2, 2008 at 7:52 PM, Gabriel J.L. Beckers <beckers@orn.mpg.de> wrote:
I too recommend hdf5 as data format. hdf5 has become my standard data format for pretty much all my work with scientific data because pytables makes it very easy to use, and yet offers the possibility of using advanced features at the same time. Have a look at their website http://www.pytables.org .
Best, Gabriel
On Tue, 2008-04-01 at 19:09 +0200, Gael Varoquaux wrote:
On Tue, Apr 01, 2008 at 10:06:35AM -0700, Charles Doutriaux wrote:
For the data format I strongly recommend NetCDF and especially NetCDF4 (allows for compression)
Actually, why not hdf5, which seems to be used by NetCDF4, but is also largely used across many scientific comunities, and very well supported under python (pytables)?
My 2 cents,
Gaël _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
-- ------------------------- Andrew Charles
I second the pytables recommendation, but if your simulations are in C and want to avoid additional dependencies, you might simply just want to write binary data direct to disk in your own format. Numpy's memmap function can then use them directly if your data are in same-length rows. Note that the term "rows" does not hint at what is actually possible. For example, an n-dimensional array can be one "column" of each such "row". -Andrew Gael Varoquaux wrote:
On Tue, Apr 01, 2008 at 10:06:35AM -0700, Charles Doutriaux wrote:
For the data format I strongly recommend NetCDF and especially NetCDF4 (allows for compression)
Actually, why not hdf5, which seems to be used by NetCDF4, but is also largely used across many scientific comunities, and very well supported under python (pytables)?
My 2 cents,
Gaël _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
On 03/04/2008, Andrew Straw <strawman@astraw.com> wrote:
I second the pytables recommendation, but if your simulations are in C and want to avoid additional dependencies, you might simply just want to write binary data direct to disk in your own format. Numpy's memmap function can then use them directly if your data are in same-length rows. Note that the term "rows" does not hint at what is actually possible. For example, an n-dimensional array can be one "column" of each such "row".
If you follow this route, you may want to take a look at the new NumPy .npy format. Regards Stéfan
I second the pytables recommendation, but if your simulations are in C and want to avoid additional dependencies, you might simply just want to write binary data direct to disk in your own format. Numpy's memmap function can then use them directly if your data are in same-length rows. Note that the term "rows" does not hint at what is actually possible. For example, an n-dimensional array can be one "column" of each such "row".
-Andrew
I have to disagree with using raw binary data, if data portability is at all a concern. I use binary formats often, but only as a quick and dirty way get data out of C. If you go with binary data, you will soon find yourself defining a custom format that includes array sizes, endianess, precision, etc. Which is why formats like HDF, NetCDF and FITS were invented. And HDF5's simple API makes it very convenient to write datasets from C. Plus, you get the HDF5 command line tools which allow you to inspect your data without needing to write your own custom tools. Also, to clarify the PyTables aspect, I think HDF5 is useful, and PyTables provides a convenient Python API to HDF5 data. However, I would not suggest using PyTables for storing the data, unless you were writing everything in Python. OK, that was longer and more opinionated than I intended. There must be a bad experience with raw binary data somewhere in my past (or several). --Rick
Gael Varoquaux wrote:
On Tue, Apr 01, 2008 at 10:06:35AM -0700, Charles Doutriaux wrote:
For the data format I strongly recommend NetCDF and especially NetCDF4 (allows for compression)
Actually, why not hdf5, which seems to be used by NetCDF4, but is also largely used across many scientific comunities, and very well supported under python (pytables)?
My 2 cents,
Gaël _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
------------------------------------------------------------------------ - Rick Wagner, Graduate Student Researcher UCSD Physics 9500 Gilman Drive La Jolla, CA 92093-0424 Email: rwagner@physics.ucsd.edu WWW: http://lca.ucsd.edu/projects/rpwagner (858) 822-4784 Phone ------------------------------------------------------------------------ - Measuring programming progress by lines of code is like measuring aircraft building progress by weight. --Bill Gates ------------------------------------------------------------------------ -
On Thu, Apr 03, 2008 at 01:19:13AM -0700, Rick Wagner wrote:
Also, to clarify the PyTables aspect, I think HDF5 is useful, and PyTables provides a convenient Python API to HDF5 data. However, I would not suggest using PyTables for storing the data, unless you were writing everything in Python.
+1. Use HDF5, but do not use anything pytables-specific (though in my experience, this is quite easy to avoid, it is even what you do naturaly, if you are not trying to make fancy stuff). My 2 cents, Ga�l
Thanks to everyone for the recommendations. I investigated HDF5 and given everyone's thoughts and the information I've read it seems to be the best route for us to take. I'm a little overwhelmed with it right now (Rick I may take you up on your offer for Q&A!) but excited about the prospects of it. Between the platform independence, the fact that it can be read in by python (pytables), matlab and others I can't think of a better choice. I just hope we can execute it properly! Thanks for everyone's suggestions.
On Tue, Apr 1, 2008 at 12:36 PM, heytrent@gmail.com <heytrent@gmail.com> wrote:
Greetings,
A small team of us are developing a new simulation package from the ground up. Our legacy approach relied on MATLAB and other proprietary software. A hope pf ours is to be able to shed the use of MATLAB for the analysis of our simulation results and instead use python with scipy/numpy/matplotlib etc. I've successfully installed and compiled optimized numpy/scipy and all the supporting packages (ATLAS, FFTW, etc).
So far so good.
To the point - I have two questions:
1) We would like to have a "scope" to monitor simulation outputs in real time. We're using one tool that can take data over a tcp/ip port, but is clunky and only works on a single platform. Does such a thing exists within the python realm for plotting data in real time?
Not sure of your detailed needs, but I can point out some options. One typical problem in python 'scope applications is speed. Most current graphics solutions weren't designed with realtime use in mind. Nonetheless, the ones that are being actively developed (e.g. matplotlib) are better than they used to be. Pmw-Blt. This is pretty fast, but development has all but stopped. But not entirely stopped. Too bad. It's the fastest solution that I found in a not-exhaustive search. Look out: the Linux version of the current release has a library bug, and it's necessary to set an environment variable to get it to work. This is not documented, AFAIK I'm currently writing a realtime data acquisition/display application, but speed is not critical. I'm using Enthought Tool Suite (ETS) and Traits, and Chaco for data display. IMHO, I think that in five years ETS/Traits will be the most commonly used framework for scientific applications.
2) Our simulation creates large (1-4 GB) data sets. Since we're writing this simulation ourselves (C++) we can save the data in any format. Does anyone have a suggestion for a specific format or API that's been found to be optimal in terms of memory usage and ability to import into python for analysis and plotting?
Thank you for any suggestions. We're still new with Python, so I apologize if these questions seem mundane.
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
hi Gary, Gary Pajer wrote:
On Tue, Apr 1, 2008 at 12:36 PM, heytrent@gmail.com <heytrent@gmail.com> wrote:
Greetings,
A small team of us are developing a new simulation package from the ground up. Our legacy approach relied on MATLAB and other proprietary software. A hope pf ours is to be able to shed the use of MATLAB for the analysis of our simulation results and instead use python with scipy/numpy/matplotlib etc. I've successfully installed and compiled optimized numpy/scipy and all the supporting packages (ATLAS, FFTW, etc).
So far so good.
To the point - I have two questions:
1) We would like to have a "scope" to monitor simulation outputs in real time. We're using one tool that can take data over a tcp/ip port, but is clunky and only works on a single platform. Does such a thing exists within the python realm for plotting data in real time?
Not sure of your detailed needs, but I can point out some options. One typical problem in python 'scope applications is speed. Most current graphics solutions weren't designed with realtime use in mind. Nonetheless, the ones that are being actively developed (e.g. matplotlib) are better than they used to be.
Pmw-Blt. Where can I find information about Pwm-Blt ? This is pretty fast, but development has all but stopped. But not entirely stopped. Too bad. It's the fastest solution that I found in a not-exhaustive search. Look out: the Linux version of the current release has a library bug, and it's necessary to set an environment variable to get it to work. This is not documented, AFAIK
I'm currently writing a realtime data acquisition/display application, but speed is not critical. I'm using Enthought Tool Suite (ETS) and Traits, and Chaco for data display. IMHO, I think that in five years ETS/Traits will be the most commonly used framework for scientific applications.
I'm too writing one/two scope-like units (rewrite of PyPlot), because MatPlotLib was too slow ;-) I'm aiming at 16 channels 1 kHz per channel. cheers, Stef
2) Our simulation creates large (1-4 GB) data sets. Since we're writing this simulation ourselves (C++) we can save the data in any format. Does anyone have a suggestion for a specific format or API that's been found to be optimal in terms of memory usage and ability to import into python for analysis and plotting?
Thank you for any suggestions. We're still new with Python, so I apologize if these questions seem mundane.
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
participants (12)
-
Alan G Isaac -
Andrew Charles -
Andrew Straw -
Charles Doutriaux -
Gabriel J.L. Beckers -
Gael Varoquaux -
Gary Pajer -
heytrent＠gmail.com -
Rick Wagner -
Stef Mientki -
Stéfan van der Walt -
Trent