[Tutor] breaking out of a function that takes too long

[Alan Gauld]

"Serdar Tumgoren" <zstumgoren at gmail.com> wrote 

> Is there a way to break out of a function if it exceeds a certain time
> limit for execution?

Yes, you can put the call in a thread then in the main thread 
start a timer. If the timer expuires before the thread returns 
kill the thread.thread 

> There's a Website I'm scraping on a regular basis, and for some reason
> that I can't divine the site has radically varying response times. 

Could be a lot of people have written scrapers and they all 
schedule them to run at reulat intervals... If the clocks all suddenly 
align you will get a massive demand on the server (its effectively 
like a denial of service attack on the server!). Or it could even be
real users if it happens when some significant change occurs - like 
stock/share price web sites often get big peaks at market 
open and close times.

> try it again a few moments later, it will sometimes work just fine
> ("sometimes" being the key word). I'm not certain, but I'm guessing
> this inconsistent behavior has something to do with the site that I'm
> scraping.

It could be anywhere in your network connection, including your 
ISP. If its a local operator there may be peak periods when people 
go online based on local patterns of usage, especially if its ot a big 
company - they might only have a couple of 2M pipes say to the core 
network. 

HTH,


-- 
Alan Gauld
Author of the Learn to Program web site
http://www.alan-g.me.uk/