TimeSeries and Milliseconds for aircraft data
Hi All, I've got lots of timeseries data from aircraft sensors (such as altitude, temperatures, landing gear up/down, airspeed..) which I was hoping to load into a timeseries array. However, I've just read that it only supports time resolutions down to a second. Is there a solution for my data which contains samples from hundreds of different parameters at different frequencies from 8Hz to 1/64Hz over an 8 hour period? I need to be able to interpolate between values, mask values when samples are corrupt / outside of operational boundaries and store values as accurately to the nearest 1/16th of a second if possible. Your help is much appreciated! Thanks, Chris
On Tue, Feb 23, 2010 at 3:35 AM, Chris Jesse <chris.jesse@flightdataservices.com> wrote:
Hi All,
I've got lots of timeseries data from aircraft sensors (such as altitude, temperatures, landing gear up/down, airspeed..) which I was hoping to load into a timeseries array. However, I've just read that it only supports time resolutions down to a second. Is there a solution for my data which contains samples from hundreds of different parameters at different frequencies from 8Hz to 1/64Hz over an 8 hour period?
I need to be able to interpolate between values, mask values when samples are corrupt / outside of operational boundaries and store values as accurately to the nearest 1/16th of a second if possible.
Your help is much appreciated! Thanks,
Chris
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
Hi Chris, I analyze both 10 and 20 Hz sonic anemometer data regularly without the use of a timeseries array (I assume you mean a TimeSeries object from scikits.timeseries). I've had a lot of success using datetime objects from the standard datetime module that ships with python to work with my time stamps, and numpy masked arrays to handle the data. The scikits.timeseries module does look nice though, but I too run up against the limit on the sampling frequency. Sean University of Oklahoma School of Meteorology PhD Candidate
Hi, For the kind of thing you're doing, is there any need for the timeseries machinery? I'd just represent all my times in terms of seconds since observation start, at which point you can just use the ordinary numpy/masked array machinery. In particular, the decision not to support milliseconds was made on the explicit assumption that more complicated timing tasks - up to and including pulsar timing, which deals with nanosecond measurements spanning decades - would simply use numpy directly, rather than try to make TimeSeries support everything anyone might ever want from it. Anne On 23 February 2010 04:35, Chris Jesse <chris.jesse@flightdataservices.com> wrote:
Hi All,
I've got lots of timeseries data from aircraft sensors (such as altitude, temperatures, landing gear up/down, airspeed..) which I was hoping to load into a timeseries array. However, I've just read that it only supports time resolutions down to a second. Is there a solution for my data which contains samples from hundreds of different parameters at different frequencies from 8Hz to 1/64Hz over an 8 hour period?
I need to be able to interpolate between values, mask values when samples are corrupt / outside of operational boundaries and store values as accurately to the nearest 1/16th of a second if possible.
Your help is much appreciated! Thanks,
Chris
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
On Tue, Feb 23, 2010 at 3:35 AM, Chris Jesse < chris.jesse@flightdataservices.com> wrote:
Hi All,
I've got lots of timeseries data from aircraft sensors (such as altitude, temperatures, landing gear up/down, airspeed..) which I was hoping to load into a timeseries array. However, I've just read that it only supports time resolutions down to a second. Is there a solution for my data which contains samples from hundreds of different parameters at different frequencies from 8Hz to 1/64Hz over an 8 hour period?
I need to be able to interpolate between values, mask values when samples are corrupt / outside of operational boundaries and store values as accurately to the nearest 1/16th of a second if possible.
Your help is much appreciated! Thanks,
Chris
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
Hi Chris, Here at the University of North Dakota we work with aircraft measurements as well. Similar to your case depends on the campaign [1] sometimes the number of measured parameters reach over hundred. We use a QNX 4.25 based data acquisition system called M300 where all the probe data are connected in. M300 saves everything in a binary file which we later process using our ADPAA package [2] It is mostly IDL coded and mainly developed by Dr. David Delene [3] He is also the main responsible person for the aircraft measurements/deployments in our department. ADPAA produces files using a pre-determined ASCII data structure shown here [4] The system supports frequencies up to 100 Hz but typically the highest we sample is at 25 Hz. There might be exceptional cases where we might need higher sample rates. However 1-4 Hz is sufficiently enough when sampling clouds with cloud-aerosol microphysical instruments. (PCASP, FSSP, CCN) (I am disregarding instruments like holographic particle detectors for the simplicity of my response :) which we don't often get to fly with.) Anyway after some advertisement back to your question: in our system all the data are time-stamped with a main "Time [seconds]; UT seconds from midnight on day aircraft flight started" value. It simplifies data analysis a lot in my view where the Python comes into action actually. Once the ASCII files are created (we also output to NetCDF containers but I find easy to work on simple text files.) I read the data using NumPy and I use the MaskedArray module to deal with the missing values in the data. I publish my code along with data on the ccnworks [5] page where I mostly commit for my thesis using ground and airborne measurements and scientific Python ecosystem. For instance to read the ASCII data files (also known as NASA formatted) I use the generic script [6] As a part of my thesis I am working on this script [7] recently. I use the lab data [8] so far. In logn-fit.py I have a section where I do masking on the data like: mask02 = (pcasp.data['Time'] > 13800) & (pcasp.data['Time'] <= 14400) This is one of the most important part of the code --to extract the right portion of the required data. In this case I am interested in time periods based on the supersaturation values from the DMT-CCNC instrument. The next step will be to analyse the cloud base data. As far as I know there is no easy solution for this. What do PCASP probe shows when the aircraft is levelling below a cloud deck :) The only way I know is to mask manually and most logically using the common time-stamp in all data files. I remember Berkeley guys work on a project called NiPy but it is for neuro-scientists mostly and not much design consideration in it for airborne measurement scientists in mind :) I hope extra information doesn't bother you and let me know your opinions. [1] http://atmoswiki.aero.und.edu/atmos/citation/home [2] http://sourceforge.net/projects/adpaa/ [3] http://aerosol.atmos.und.edu/ [4] http://aerosol.atmos.und.edu/ADPAA/fileheader.html [5] http://code.google.com/p/ccnworks/ [6] http://code.google.com/p/ccnworks/source/browse/trunk/nasafile.py [7] http://code.google.com/p/ccnworks/source/browse/trunk/thesis/part1/logn-fit.... [8] http://code.google.com/p/ccnworks/source/browse/trunk/thesis/part1/20090225_... -- Gökhan
Hi everyone, Gokhan is referring to this: http://nipy.sourceforge.net/nitime/ For an exposition, see our Scipy conference proceedings paper (a pdf of which can be found here: http://argentum.ucbso.berkeley.edu/papers/Rokem2009Nitime.pdf). We are still working on it. The intention of the library is to support analysis of data from neuroscience experiments, because we are neuroscientists, but so far, I don't think that we have made any design decisions that would preclude other scientists from using our time-series objects. In fact, the time-series objects we have designed support temporal resolutions as fast as picoseconds (the representation of time is done in int64, in order to avoid float-precision issues). It is still under development and we have yet to make a release of this, but the code (in development) is already available on github and the tests therein can direct you on the possible usage: http://github.com/fperez/nitime Cheers, Ariel On Tue, Feb 23, 2010 at 7:23 AM, Gökhan Sever <gokhansever@gmail.com> wrote:
On Tue, Feb 23, 2010 at 3:35 AM, Chris Jesse < chris.jesse@flightdataservices.com> wrote:
Hi All,
I've got lots of timeseries data from aircraft sensors (such as altitude, temperatures, landing gear up/down, airspeed..) which I was hoping to load into a timeseries array. However, I've just read that it only supports time resolutions down to a second. Is there a solution for my data which contains samples from hundreds of different parameters at different frequencies from 8Hz to 1/64Hz over an 8 hour period?
I need to be able to interpolate between values, mask values when samples are corrupt / outside of operational boundaries and store values as accurately to the nearest 1/16th of a second if possible.
Your help is much appreciated! Thanks,
Chris
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
Hi Chris,
Here at the University of North Dakota we work with aircraft measurements as well. Similar to your case depends on the campaign [1] sometimes the number of measured parameters reach over hundred. We use a QNX 4.25 based data acquisition system called M300 where all the probe data are connected in. M300 saves everything in a binary file which we later process using our ADPAA package [2] It is mostly IDL coded and mainly developed by Dr. David Delene [3] He is also the main responsible person for the aircraft measurements/deployments in our department.
ADPAA produces files using a pre-determined ASCII data structure shown here [4] The system supports frequencies up to 100 Hz but typically the highest we sample is at 25 Hz. There might be exceptional cases where we might need higher sample rates. However 1-4 Hz is sufficiently enough when sampling clouds with cloud-aerosol microphysical instruments. (PCASP, FSSP, CCN) (I am disregarding instruments like holographic particle detectors for the simplicity of my response :) which we don't often get to fly with.)
Anyway after some advertisement back to your question: in our system all the data are time-stamped with a main "Time [seconds]; UT seconds from midnight on day aircraft flight started" value. It simplifies data analysis a lot in my view where the Python comes into action actually. Once the ASCII files are created (we also output to NetCDF containers but I find easy to work on simple text files.) I read the data using NumPy and I use the MaskedArray module to deal with the missing values in the data. I publish my code along with data on the ccnworks [5] page where I mostly commit for my thesis using ground and airborne measurements and scientific Python ecosystem. For instance to read the ASCII data files (also known as NASA formatted) I use the generic script [6] As a part of my thesis I am working on this script [7] recently. I use the lab data [8] so far. In logn-fit.py I have a section where I do masking on the data like:
mask02 = (pcasp.data['Time'] > 13800) & (pcasp.data['Time'] <= 14400)
This is one of the most important part of the code --to extract the right portion of the required data. In this case I am interested in time periods based on the supersaturation values from the DMT-CCNC instrument. The next step will be to analyse the cloud base data. As far as I know there is no easy solution for this. What do PCASP probe shows when the aircraft is levelling below a cloud deck :) The only way I know is to mask manually and most logically using the common time-stamp in all data files.
I remember Berkeley guys work on a project called NiPy but it is for neuro-scientists mostly and not much design consideration in it for airborne measurement scientists in mind :)
I hope extra information doesn't bother you and let me know your opinions.
[1] http://atmoswiki.aero.und.edu/atmos/citation/home [2] http://sourceforge.net/projects/adpaa/ [3] http://aerosol.atmos.und.edu/ [4] http://aerosol.atmos.und.edu/ADPAA/fileheader.html [5] http://code.google.com/p/ccnworks/ [6] http://code.google.com/p/ccnworks/source/browse/trunk/nasafile.py [7] http://code.google.com/p/ccnworks/source/browse/trunk/thesis/part1/logn-fit.... [8] http://code.google.com/p/ccnworks/source/browse/trunk/thesis/part1/20090225_...
-- Gökhan
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
-- Ariel Rokem Helen Wills Neuroscience Institute University of California, Berkeley http://argentum.ucbso.berkeley.edu/ariel
On Wed, Feb 24, 2010 at 12:32 AM, Ariel Rokem <arokem@berkeley.edu> wrote:
Hi everyone,
Gokhan is referring to this:
http://nipy.sourceforge.net/nitime/
For an exposition, see our Scipy conference proceedings paper (a pdf of which can be found here: http://argentum.ucbso.berkeley.edu/papers/Rokem2009Nitime.pdf).
We are still working on it. The intention of the library is to support analysis of data from neuroscience experiments, because we are neuroscientists, but so far, I don't think that we have made any design decisions that would preclude other scientists from using our time-series objects. In fact, the time-series objects we have designed support temporal resolutions as fast as picoseconds (the representation of time is done in int64, in order to avoid float-precision issues). It is still under development and we have yet to make a release of this, but the code (in development) is already available on github and the tests therein can direct you on the possible usage:
http://github.com/fperez/nitime
Cheers,
Ariel
Hi Ariel, What kind of interface do you use to measure pico-second resolutions? Are you talking measurements from only one instrument at a time? In our work even at 1 Hz levels we encounter issues like time-syncing different measurements since we interface many different instruments with one main data acquisition unit. This is mainly due to one instrument sits under the far edge of a wing the other one is inside the cabin sampling air from outside. It is usually a good idea to sample fastest the system and probes permits, in the end they would be easily averaged to a lower acceptable resolution range. Your job should be very hard indeed if you are dealing with a couple different instruments at that high measurement rates. -- Gökhan
Hi Gokhan - easy - we are not dealing with the measurement part at all, only with the analysis of data that has already been measured, so if you don't think a pico-second representation is appropriate for your data (and it probably is appropriate for very few kinds of data...), then use the right temporal resolution for your data. Best - Ariel On Wed, Feb 24, 2010 at 9:10 AM, Gökhan Sever <gokhansever@gmail.com> wrote:
On Wed, Feb 24, 2010 at 12:32 AM, Ariel Rokem <arokem@berkeley.edu> wrote:
Hi everyone,
Gokhan is referring to this:
http://nipy.sourceforge.net/nitime/
For an exposition, see our Scipy conference proceedings paper (a pdf of which can be found here: http://argentum.ucbso.berkeley.edu/papers/Rokem2009Nitime.pdf).
We are still working on it. The intention of the library is to support analysis of data from neuroscience experiments, because we are neuroscientists, but so far, I don't think that we have made any design decisions that would preclude other scientists from using our time-series objects. In fact, the time-series objects we have designed support temporal resolutions as fast as picoseconds (the representation of time is done in int64, in order to avoid float-precision issues). It is still under development and we have yet to make a release of this, but the code (in development) is already available on github and the tests therein can direct you on the possible usage:
http://github.com/fperez/nitime
Cheers,
Ariel
Hi Ariel,
What kind of interface do you use to measure pico-second resolutions? Are you talking measurements from only one instrument at a time?
In our work even at 1 Hz levels we encounter issues like time-syncing different measurements since we interface many different instruments with one main data acquisition unit. This is mainly due to one instrument sits under the far edge of a wing the other one is inside the cabin sampling air from outside. It is usually a good idea to sample fastest the system and probes permits, in the end they would be easily averaged to a lower acceptable resolution range.
Your job should be very hard indeed if you are dealing with a couple different instruments at that high measurement rates.
-- Gökhan
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
-- Ariel Rokem Helen Wills Neuroscience Institute University of California, Berkeley http://argentum.ucbso.berkeley.edu/ariel
participants (5)
-
Anne Archibald -
Ariel Rokem -
Chris Jesse -
Gökhan Sever -
Sean Arms