[Tutor] Help for Python Beginner with extracting and manipulating data from thousands of ASCII files

[Alan Gauld]

On 30/09/12 23:07, Cecilia Chavana-Bryant wrote:
> Hola again Python Tutor!
>
> With a friend's help I have the following code to extract reflectance
> data from an ASCII data file, do a bit of data manipulation to calibrate
> the data and then write the calibrated file into an out file.


<snip>

> I have successfully calibrated one ASCII file at a time with this code.
> However, I have 1,000s of files that I need to calibrate so I would like
> some help to modify this code so it can:
>
> 1. Use one calibration file (Cal_FileP17.txt) on data files created from
> July to the 18th Sep and a different calibration file (Cal_FileP19.txt)
> for data files created from the 19th of Sep onwards.
>
> 2. Find all the .txt files in a folder called ASCII_files, which is
> subdivided into 12 different folders and calibrate all these files


Number 2 is easier to solve and the os.walk() and glob.glob()
functions should provide all the tools you need.

Number 1 is more tricky since there is no obvious way to determine the 
arbitrary start/stop dates you specify. So I'd suggest you need to 
generalise the requirement to take a start/stop date as well as the 
calibration file name and the input data file pattern. Use those as 
input parameters to a function that generates the list of files to 
process and then calls your existing code (wrapped in a new function) 
and possibly provide default values for all/some of the parameters.

Another option is to add the start/end dates to the calibration file if 
you have control of that, but personally I'd stick with input parameters...


-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/