Repeatedly crawl website every 1 min
Steve D'Aprano
steve+python at pearwood.info
Thu May 11 10:05:08 EDT 2017
On Thu, 11 May 2017 12:18 pm, liyucun2012 at gmail.com wrote:
> Hi Everyone,
>
> Thanks for stoping by. I am working on a feature to crawl website content
> every 1 min. I am curious to know if there any good open source project
> for this specific scenario.
I agree with Iuri: crawling a website every minute is abuse. Unless it is
your own website, once a month is more appropriate -- and even then, you
should be very careful to restrict the rate at which you make requests.
--
Steve
Emoji: a small, fuzzy, indistinct picture used to replace a clear and
perfectly comprehensible word.
More information about the Python-list
mailing list