On Tue, Feb 23, 2010 at 3:35 AM, Chris Jesse <chris.jesse@flightdataservices.com> wrote:
Hi All,

I've got lots of timeseries data from aircraft sensors (such as altitude, temperatures, landing gear up/down, airspeed..) which I was hoping to load into a timeseries array. However, I've just read that it only supports time resolutions down to a second. Is there a solution for my data which contains samples from hundreds of different parameters at different frequencies from 8Hz to 1/64Hz over an 8 hour period?

I need to be able to interpolate between values, mask values when samples are corrupt / outside of operational boundaries and store values as accurately to the nearest 1/16th of a second if possible.

Your help is much appreciated! Thanks,

Chris

_______________________________________________
SciPy-User mailing list
SciPy-User@scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user


Hi Chris,

Here at the University of North Dakota we work with aircraft measurements as well. Similar to your case depends on the campaign [1] sometimes the number of measured parameters reach over hundred. We use a QNX 4.25 based data acquisition system called M300 where all the probe data are connected in. M300 saves everything in a binary file which we later process using our ADPAA package [2] It is mostly IDL coded and mainly developed by Dr. David Delene [3] He is also the main responsible person for the aircraft measurements/deployments in our department.

ADPAA produces files using a pre-determined ASCII data structure shown here [4] The system supports frequencies up to 100 Hz but typically the highest we sample is at 25 Hz. There might be exceptional cases where we might need higher sample rates. However 1-4 Hz is sufficiently enough when sampling clouds with cloud-aerosol microphysical instruments. (PCASP, FSSP, CCN) (I am disregarding instruments like holographic particle detectors for the simplicity of my response :) which we don't often get to fly with.)

Anyway after some advertisement back to your question: in our system all the data are time-stamped with a main "Time [seconds]; UT seconds from midnight on day aircraft flight started" value. It simplifies data analysis a lot in my view where the Python comes into action actually. Once the ASCII files are created (we also output to NetCDF containers but I find easy to work on simple text files.) I read the data using NumPy and I use the MaskedArray module to deal with the missing values in the data. I publish my code along with data on the ccnworks [5] page where I mostly commit for my thesis using ground and airborne measurements and scientific Python ecosystem. For instance to read the ASCII data files (also known as NASA formatted) I use the generic script [6] As a part of my thesis I am working on this script [7] recently. I use the lab data [8] so far. In logn-fit.py I have a section where I do masking on the data like:

mask02 = (pcasp.data['Time'] > 13800) & (pcasp.data['Time'] <= 14400)

This is one of the most important part of the code --to extract the right portion of the required data. In this case I am interested in time periods based on the supersaturation values from the DMT-CCNC instrument. The next step will be to analyse the cloud base data. As far as I know there is no easy solution for this. What do PCASP probe shows when the aircraft is levelling below a cloud deck :) The only way I know is to mask manually and most logically using the common time-stamp in all data files.

I remember Berkeley guys work on a project called NiPy but it is for neuro-scientists mostly and not much design consideration in it for airborne measurement scientists in mind :)

I hope extra information doesn't bother you and let me know your opinions.


[1] http://atmoswiki.aero.und.edu/atmos/citation/home
[2] http://sourceforge.net/projects/adpaa/
[3] http://aerosol.atmos.und.edu/
[4] http://aerosol.atmos.und.edu/ADPAA/fileheader.html
[5] http://code.google.com/p/ccnworks/
[6] http://code.google.com/p/ccnworks/source/browse/trunk/nasafile.py
[7] http://code.google.com/p/ccnworks/source/browse/trunk/thesis/part1/logn-fit.py
[8] http://code.google.com/p/ccnworks/source/browse/trunk/thesis/part1/20090225_034214/09_02_25_03_42_14.lab.clean

--
Gökhan