urllib2 slow for multiple requests

Tomas Svarovsky svarovsky.tomas at gmail.com
Wed May 13 14:57:47 CEST 2009


Hello everybody, really new to python, so bear with me. I am trying to
do some very basic scraping tool. Bascally it just grabs a page xy
times and tells me how long it took. When I do this once, it is
blazingly fast, but when I increase the number of repetitions, it is
slowing down considerably (1 is like 3 ms, 100 takes 6 seconds). I
have done implementations in couple more languages (php, ruby) and
none of them seems to suffer from a similar problem and it seems, that
it behaves linearly. Maybe it is a known issue in urllib2, or I am
simply using it badly. I am using python 2.4.3, machine has CentOS,
below is the sc. Thanks in advance

import urllib2
from datetime import datetime

def application():
    start = datetime.now()
    req = urllib2.Request("http://127.0.0.1/gdc/about", None,
{'Accept': 'application/json'})
    for number in range(100):
      response = urllib2.urlopen(req)
    end = datetime.now()
    output = end - start
    print output

application()



More information about the Python-list mailing list