how to handle network failures

[Diez B. Roggisch]

harryos <oswald.harry at gmail.com> writes:

> hi
> I  am trying to write a DataGrabber which reads some data from given
> url..I made DataGrabber as a Thread and want to wait for some interval
> of time in case there is a network failure that prevents read().
> I am not very sure how to implement this
>
> class DataGrabber(threading.Thread):
>     def __init__(self,url):
>         threading.Thread.__init__(self)
>         self.url=url
>     def run(self):
>         data=self.get_page_data()
>         process_data(data)
>
>     def get_page_data():
>         try:
>             f=urllib.urlopen(self.url)
>             data=f.read(1024)
>         except IOError:
>             #wait for some time and try again
>             time.sleep(120)
>             data=self.get_page_data()
>         return data
>
> Is this the way to  implement the part where the thread waits and
> reads the  data again? Will this handle network failures?Can somebody
> please help?

This only works if your page is always 1024 bytes long. Which I
doubt. So don't pass the 1024 to read.

Also, you need a loop to re-read the data. Like this:


for n in xrange(max_number_of_retries):
    try:
        f=urllib.urlopen(self.url)
        data = f.read()
        break # exist the loop if all
    except IOError:
        pass

        
self.process_data(data)


Diez