Concurrent threads to pull web pages?

[Kyle Terry]

On Thu, Oct 1, 2009 at 6:33 PM, <exarkun at twistedmatrix.com> wrote:

> On 1 Oct, 09:28 am, nospam at nospam.com wrote:
>
>> Hello
>>
>>        I recently asked how to pull companies' ID from an SQLite database,
>> have multiple instances of a Python script download each company's web
>> page from a remote server, eg. www.acme.com/company.php?id=1, and use
>> regexes to extract some information from each page.
>>
>> I need to run multiple instances to save time, since each page takes
>> about 10 seconds to be returned to the script/browser.
>>
>> Since I've never written a multi-threaded Python script before, to
>> save time investigating, I was wondering if someone already had a
>> script that downloads web pages and save some information into a
>> database.
>>
>
> There's no need to use threads for this.  Have a look at Twisted:
>
>  http://twistedmatrix.com/trac/
>
> Here's an example of how to use the Twisted HTTP client:
>
>  http://twistedmatrix.com/projects/web/documentation/examples/getpage.py
>

I don't think he was looking for a framework... Specifically a framework
that you work on.


>
> Jean-Paul
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20091001/2b7f801c/attachment.html>