Concurrent threads to pull web pages?
Gilles Ganault
nospam at nospam.com
Thu Oct 1 05:28:09 EDT 2009
Hello
I recently asked how to pull companies' ID from an SQLite database,
have multiple instances of a Python script download each company's web
page from a remote server, eg. www.acme.com/company.php?id=1, and use
regexes to extract some information from each page.
I need to run multiple instances to save time, since each page takes
about 10 seconds to be returned to the script/browser.
Since I've never written a multi-threaded Python script before, to
save time investigating, I was wondering if someone already had a
script that downloads web pages and save some information into a
database.
Thank you for any tip.
More information about the Python-list
mailing list