[Tutor] open a webpage which may be unavailable
alan.gauld at btinternet.com
Thu Oct 18 13:43:29 CEST 2007
"pileux systeme" <nogentstanford at yahoo.fr> wrote
> I am trying to retrieve data from several webpages.
> My problem is the following: after a random number of requests,
> the page I'm trying to open is unavailable (and I get an IOError).
> Note that the page may become available if I try again
This is similar to the problem faced by marketeers doing
outbound phone calls. The customer may not be home so
you have to try again later (maybe several times).
The normal solution to this is to buoild a list of failed addresses
and then after processing the original list go back and
process the exceptions list. Repeat for N iterations or
until exceptions list is empty.
This approach saves multiple attempts in clise proximitry
to a faulty address when you could be processing productive
addresses so is time efficient.
The only snag is if you have to process the addresses in
a given sequence. In that case you may need to create
a list of groups and if any address within a group fails
put the entire group in the exceptions list.
Coincidentally, given my recent message on Mainframe
technologies, this is the same technique used in high
volume data processing (on mainframes) where millions
of orders or bills etc need to be processed quickly. Rather than
delay the multitude to fix errors on a case by case basis
a quick pass is made collecting faulty records and a
second pass (maybe with a more rigorouds algorithm)
is made of the exceptions.
Author of the Learn to Program web site
More information about the Tutor