<br><br><div class="gmail_quote">On Mon, Oct 1, 2012 at 1:16 AM, Dave Angel <span dir="ltr"><<a href="mailto:d@davea.name" target="_blank">d@davea.name</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On 09/30/2012 06:07 PM, Cecilia Chavana-Bryant wrote:<br>
> Hola again Python Tutor!<br>
><br>
> With a friend's help I have the following code to extract reflectance data<br>
> from an ASCII data file, do a bit of data manipulation to calibrate the<br>
> data and then write the calibrated file into an out file.<br>
><br>
> import numpy<br>
> # import glob - use if list_of_files is used<br>
><br>
><br>
> dataFile = "1SH0109.001.txt"<br>
> #list_of_files = glob.glob('./*.txt') to replace dataFile to search for all<br>
> text files in ASCII_files folder?<br>
<br>
</div>First, an important observation. This code has no functions defined in<br>
it. Thus it's not reusable. So every time you make a change, you'll be<br>
breaking the existing code and then trying to make the new version work.<br>
<br>
The decision of one file versus many is usually handled by writing a<br>
function that deals with one file. Test it with a single file. Then<br>
write another function that uses glob to build a list of files, and call<br>
the original one in a loop.<br>
<br>
As you work on it, you should discover that there are a half dozen other<br>
functions that you need, rather than one big one.<br>
<div class="im"><br></div></blockquote><div>Many thanks for this advise this helps me to get started with trying to write functions for the different procedures and then think about many files.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">
> caliFile1 = "Cal_File_P17.txt" # calibration file to be used on data files<br>
> created from July to 18 Sep<br>
> caliFile2 = "Cal_File_P19.txt" # calibration file to be used on data files<br>
> created from 19 Sep onwards<br>
> outFile = "Cal_" + dataFile # this will need to change if list_of_files is<br>
> used<br>
> fileDate = data[6][16:26] # location of the creation date on the data files<br>
<br>
</div>Show us the full traceback from the runtime error you get on this line.<br>
<div class="im"><br></div></blockquote><div>The option of using 2 different calibration files is an idea that I haven't tested yet as I am a bit lost in how to do this. I have gotten as far as adding start and end dates on both calibration files as part of the header information for each file.</div>
<div><br></div><div>#extract data from data file</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
> fdData = open(dataFile,"rt")<br>
> data = fdData.readlines()<br>
> fdData.close()<br>
><br>
> #extract data from calibration file<br>
> fdCal = open(caliFile,"rt")<br>
<br>
</div>Show us the full traceback from the runtime error here, as well.<br>
<div><div class="h5"><br></div></div></blockquote><div>In the original code which uses only one calibration file this and the rest of the code works without error.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div class="h5">
> calibration = fdCal.readlines()<br>
> fdCal.close()<br>
><br>
> #create data table<br>
> k=0 #just a counter<br>
> dataNum = numpy.ndarray((2151,2))<br>
><br>
> #the actual data (the numbers) in the data file begin at line 30<br>
> for anItem in data[30:]:<br>
> theNums = anItem.replace("\r\n","").split("\t")<br>
> dataNum[k,0] = int(theNums[0])<br>
> dataNum[k,1] = float(theNums[1])<br>
> k+=1 #advance the counter<br>
><br>
> #create the calibration table<br>
> k = 0<br>
> calNum = numpy.ndarray((2151,2))<br>
> for anItem in calibration[5:]:<br>
> theNums = anItem.replace("\r\n","").split("\t")<br>
> calNum[k,0] = int(theNums[0])<br>
> calNum[k,1] = float(theNums[1])<br>
> k+=1<br>
><br>
> #calibrate the data<br>
> k=0<br>
> calibratedData = numpy.ndarray((2151,2))<br>
> for aNum in dataNum:<br>
> calibratedData[k,0] = aNum[0] #first column is the wavelength<br>
> calibratedData[k,1] = (aNum[1] * dataNum[k,1]) * 100.0 #second column<br>
> is the measurement to be calibrated.<br>
> k+=1<br>
><br>
> #write the calibrated data<br>
> fd = open(outFile,"wt")<br>
</div></div>Error traceback ?<br>
<div class="im">> #prior to writing the calibrated contents, write the headers for data files<br>
> and calibration files<br>
> fd.writelines(data[0:30])<br>
> fd.writelines(calibration[0:5])<br>
> for aNum in calibratedData:<br>
> #Write the data in the file in the following format:<br>
> # "An integer with 3 digits", "tab character", "Floating point number"<br>
> fd.write("%03d\t%f\n" % (aNum[0],aNum[1]))<br>
><br>
> #close the file<br>
> fd.close()<br>
><br>
<br>
</div>Are the individual files small? By doing readlines() on them, you're<br>
assuming you can hold all of both the data file and the calibration file<br>
in memory.<br>
<div class="im"><br></div></blockquote><div>Both the calibration and the data files are small. The original excel calibration files have been saved as "Tab delimited text files" and the data files are ASCII files with 2151 rows and 2 columns.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
> I have successfully calibrated one ASCII file at a time with this code.<br>
</div>Unless I'm missing something, this code does not run. I didn't try it,<br>
though, just inspected it quickly.<br>
<div class="HOEnZb"><div class="h5">> However, I have 1,000s of files that I need to calibrate so I would like<br>
> some help to modify this code so it can:<br>
><br>
> 1. Use one calibration file (Cal_FileP17.txt) on data files created from<br>
> July to the 18th Sep and a different calibration file (Cal_FileP19.txt) for<br>
> data files created from the 19th of Sep onwards.<br>
><br>
> 2. Find all the .txt files in a folder called ASCII_files, which is<br>
> subdivided into 12 different folders and calibrate all these files<br>
><br>
> I have googled and tried thinking about how to make changes and I've<br>
> managed to get myself a bit more confused. Thus, I would like some guidance<br>
> on how to tackle/think about this process and how to get started. Please, I<br>
> am not asking for someone to do my work and write the code for me, I would<br>
> like some guidance on how to approach this and get started.<br>
><br>
> Many thanks in advance for your help,<br>
> Cecilia<br>
><br>
><br>
<br>
<br>
</div></div><span class="HOEnZb"><font color="#888888">--<br>
<br>
DaveA<br>
<br>
</font></span></blockquote></div><br>