processing input from multiple files

[Christopher Steele]

Thanks,

The issue with the times is now sorted, however I'm running into a problem
towards the end of the script:

 File "sortoutsynop2.py", line 131, in <module>
    newline =
message_type+c+str(station_id)+c+newtime+c+lat+c+lon+c+c+"-9999"+c+ "002"
+c+"-9999"+c+"-9999"+c+str(pressure)+c
TypeError: cannot concatenate 'str' and 'list' objects


I think I can see the issue here, but I'm not entirely sure how to get
around it. Several of my variables change either from one file to the next
or from each line. Time and pressure would be examples of both of these
types.Yet others, such as message_type, are constant. As a result I have a
mixture of both lists and strings. Should I then create a list of the
constant values? I'm a little confused, I'll send you the script that works
for a single file and I'll see if I can come up with a more logical way
around it.

#!/usr/bin/python

import sys
import os
import re

#foutname = 'test.txt'
#filelist = os.system('ls
fname = "datalist_201081813.txt"
foutname1 = 'prestest.txt'
foutname2 = 'temptest.txt'
foutname3 = 'tempdtest.txt'
foutname4 = 'wspeedtest.txt'
foutname5 = 'winddtest.txt'

time = fname.split('_')[1].split('.')[0]
year = time[:4]
month = time[4:6]
day = time[6:8]
hour = time[-2:]

newtime = year+month+day+'_'+hour+'0000'
c = ','
file1 = open(fname,"r")


file2 = open("uk_stations.txt","r")
stations = file2.readlines()
ids=[]
names=[]
lats=[]
lons=[]
for item in stations:
    item_list = item.strip().split(',')
    ids.append(item_list[0])
    names.append(item_list[1])
    lats.append(item_list[2])
    lons.append(item_list[3])


st = file1.readlines()
print st
data=[item[:item.find(' 333 ')] for item in st]
#data=st[split:]
print data

pres_out = ''
temp_out = ''
dtemp_out = ''
dir_out = ''
speed_out = ''

for line in data:
    elements=line.split(' ')
    station_id = elements[0]
    try:
        index = ids.index(station_id)
        lat = lats[index]
        lon = lons[index]
        message_type = 'blah'
    except:
        print 'Station ID',station_id,'not in list!'
        lat = lon = 'NaN'
        message_type = 'Bad_station_id'

    try:
        temp = [item for item in elements if item.startswith('1')][0]
        temperature = float(temp[2:])/10
        sign = temp[1]
        if sign == 1:
            temperature=-temperature
    except:
        temperature='NaN'

    try:
        dtemp = [item for item in elements if item.startswith('2')][0]
        dtemperature = float(dtemp[2:])/10
        sign = dtemp[1]
        if sign == 1:
            dtemperature=-dtemperature
    except:
        detemperature='NaN'
    try:
        press = [item for item in elements[2:] if item.startswith('4')][0]
        if press[1]=='9':
            pressure = float(press[1:])/10
        else:
            pressure = float(press[1:])/10+1000
    except:
        pressure = 'NaN'

    try:
        wind = elements[elements.index(temp)-1]
        direction = float(wind[1:3])*10
        speed = float(wind[3:])*0.514444444
    except:
        direction=speed='NaN'



    newline =
message_type+c+str(station_id)+c+newtime+c+lat+c+lon+c+'-9999'+c+'002'+c+'-9999'+c+'-9999'+c+str(pressure)+c
    print newline
    pres_out+=newline+'\n'


    newline2 =
message_type+c+str(station_id)+c+newtime+c+lat+c+lon+c+c+"-9999"+c+ "011"
+c+"-9999"+c+"-9999"+c+str(temperature)+c
    print newline2
    temp_out+=newline2+'\n'
    fout = open(foutname2,'w')
    fout.writelines(temp_out)
    fout.close()




    newline3 =
message_type+c+str(station_id)+c+newtime+c+lat+c+lon+c+c+"-9999"+c+ "017"
+c+"-9999"+c+"-9999"+c+str(dtemperature)+c
    print newline3
    dtemp_out+=newline3+'\n'
    fout = open(foutname3,'w')
    fout.writelines(dtemp_out)
    fout.close()


    newline4 =
message_type+c+str(station_id)+c+newtime+c+lat+c+lon+c+c+"-9999"+c+ "031"
+c+"-9999"+c+"-9999"+c+str(direction)+c
    print newline4
    dir_out+=newline4+'\n'
    fout = open(foutname4,'w')
    fout.writelines(dir_out)
    fout.close()


    newline5 =
message_type+c+str(station_id)+c+newtime+c+lat+c+lon+c+c+"-9999"+c+
"032"+c+"-9999"+c+"-9999"+c+str(speed)+c
    print newline5
    speed_out+=newline5+'\n'


fout = open(foutname1,'w')
fout.writelines(pres_out)
fout.close()
fout = open(foutname2,'w')
fout.writelines(temp_out)
fout.close()
fout = open(foutname3,'w')
fout.writelines(dtemp_out)
fout.close()
fout = open(foutname4,'w')
fout.writelines(dir_out)
fout.close()
fout = open(foutname5,'w')
fout.writelines(speed_out)
fout.close()


cheers

Chris












On Thu, Oct 14, 2010 at 8:15 PM, John Posner <jjposner at optimum.net> wrote:

> On 10/14/2010 10:44 AM, Christopher Steele wrote:
>
>> The issue is that I need to be able to both, split the names of the files
>> so that I can extract the relevant times, and open each individual file and
>> process each line individually. Once I have achieved this I need to append
>> the sorted files onto one another in one long file so that I can pass them
>> into a verification package. I've tried changing the name to textline and I
>> get the same result
>>
>
> I'm very happy to hear that changing the name of a variable did not affect
> the way the program works! Anything else would be worrisome.
>
>
>
>  - the sorted files overwrite one another.
>>
>
> Variable *time* names a list, with one member for each input file. But
> variable *newtime* names a scalar value, not a list. That looks like a
> problem to me. Either of the following changes might help:
>
> Original:
>
>
>  for x in time:
>      hour= x[:2]
>      print hour
>      newtime = year+month+day+'_'+hour+'00'
>
> Alternative #1:
>
>  newtime = []
>
>  for x in time:
>      hour= x[:2]
>      print hour
>      newtime.append(year+month+day+'_'+hour+'00')
>
> Alternative #2:
>  newtime = [year + month + day + '_' + x[:2] + '00' for x in time]
>
>
> HTH,
> John
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20101015/fffadbfe/attachment.html>