Repeatedly crawl website every 1 min

liyucun2012 at gmail.com liyucun2012 at gmail.com
Wed May 10 22:18:13 EDT 2017


Hi Everyone,

Thanks for stoping by. I am working on a feature to crawl website content every 1 min. I am curious to know if there any good open source project for this specific scenario.

Specifically, I have many urls, and I want to maintain a thread pool so that each thread will repeatedly crawl content from the given url. It could be a hundreds thread at the same time.

Your help is greatly appreciated.

;)



More information about the Python-list mailing list