[Tutor] 2016-02-03 Filter STRINGS from urllib and Pass as VARAIBLE within PYTHON script

[Peter Otten]

knnleow GOOGLE wrote:

> advice on how to filter VARIABLES from output below.
> 
> 01. need to strip out
>        BEGIN_STRINGS  b'{  and
>        END_STRINGS }\n'

In b'{...}' the b prefix indicates that you are dealing with a byte string.
You have to decode it and then you can use the json module to parse the 
data:

>>> import urllib.request, json
>>> data = urllib.request.urlopen(
... "http://freegeoip.net/json/222.187.222.220").read()
>>> data
b'{"ip":"222.187.222.220","country_code":"CN","country_name":"China","region_code":"31","region_name":"Shanghai 
Shi","city":"Shanghai","zip_code":"","time_zone":"Asia/Shanghai","latitude":31.0456,"longitude":121.3997,"metro_code":0}\n'
>>> data = data.decode()

If the above fails with a UnicodeDecodeError you have to provide the actual 
encoding.

>>> data
'{"ip":"222.187.222.220","country_code":"CN","country_name":"China","region_code":"31","region_name":"Shanghai 
Shi","city":"Shanghai","zip_code":"","time_zone":"Asia/Shanghai","latitude":31.0456,"longitude":121.3997,"metro_code":0}\n'

> 02. need to filter out these FIELDS
> FIELD 1   "ip":"222.187.222.220"
> FIELD 2   "country_code":"CN"
> FIELD 3  "country_name":"China"
> FIELD 6  "city":"Shanghai"
> FIELD 8  "time_zone":"Asia/Shanghai"

>>> data = json.loads(data)
>>> data["city"]
'Shanghai'
 
> 03.  need to feed this into python script so as to generate a HTML File
> by date script was run. example "/fail2ban/2016-01-31.html"

Instead of doing this manually you should pick one of the many templating 
languages out there.
 
> i am doing this using shell script today. will like to get this working
> with python.
> 
> regards,
> kuenn
> 
> 
> SCRIPT:
>                  banIP_addrs = set()
>                  with open("/var/log/fail2ban.log") as fail_log:
>                          for line in fail_log:
>                                  if("Ban" in line and "fail2ban.actions:
> WARNING" in line and myDate in line):
>                                          words = line.split()
>                                          word6 = words[6]
>                                          #print("word6:" , word6)
>                                          banIP_addrs.add(word6)
>                  #print("banIP_addrs:" , banIP_addrs)
> 
>                  for i in banIP_addrs:
>                          #print("i:" , i)
>                          myGeoip =
> urllib.request.urlopen("http://freegeoip.net/json/" + i).read()
>                          print(myGeoip)
> 
> OUTPUT:
> 
> 
b'{"ip":"222.187.222.220","country_code":"CN","country_name":"China","region_code":"31","region_name":"Shanghai_Shi","city":"Shanghai","zip_code":"","time_zone":"Asia/Shanghai","latitude":31.0456,"longitude":121.3997,"metro_code":0}\n'
> 
b'{"ip":"185.130.5.184","country_code":"LT","country_name":"Republic_of_Lithuania","region_code":"","region_name":"","city":"","zip_code":"","time_zone":"Europe/Vilnius","latitude":56,"longitude":24,"metro_code":0}\n'
> _______________________________________________