Tips on Speeding up Python Execution

Raymond Hettinger python at
Fri Apr 8 19:59:41 CEST 2011

On Apr 8, 12:25 am, Chris Angelico <ros... at> wrote:
> On Fri, Apr 8, 2011 at 5:04 PM, Abhijeet Mahagaonkar
> <abhijeet.mano... at> wrote:
> > I was able to isolate that major chunk of run time is eaten up in opening a
> > webpages, reading from them and extracting text.
> > I wanted to know if there is a way to concurrently calling the functions.
> So, to clarify: you have code that's loading lots of separate pages,
> and the time is spent waiting for the internet? If you're saturating
> your connection, then this won't help, but if they're all small pages
> and they're coming over the internet, then yes, you certainly CAN
> fetch them concurrently. As the Perl folks say, There's More Than One
> Way To Do It; one is to spawn a thread for each request, then collect
> up all the results at the end. Look up the 'threading' module for
> details:

The docs for Python3.2 have a nice example for downloading multiple
webpages in parallel:


More information about the Python-list mailing list