Repeatedly crawl website every 1 min
liyucun2012 at gmail.com
liyucun2012 at gmail.com
Wed May 10 22:18:13 EDT 2017
Hi Everyone,
Thanks for stoping by. I am working on a feature to crawl website content every 1 min. I am curious to know if there any good open source project for this specific scenario.
Specifically, I have many urls, and I want to maintain a thread pool so that each thread will repeatedly crawl content from the given url. It could be a hundreds thread at the same time.
Your help is greatly appreciated.
;)
More information about the Python-list
mailing list