urllib2 slow for multiple requests

Tomas Svarovsky svarovsky.tomas at gmail.com
Thu May 14 05:33:29 EDT 2009


One more thing, since I am stuck with 2.4 (and if this is really 2.4
issue), is there some substitution for urllib2?

On May 14, 11:00 am, Tomas Svarovsky <svarovsky.to... at gmail.com>
wrote:
> On May 13, 4:55 pm, cgoldberg <cgoldb... at gmail.com> wrote:
>
> > > Bascally it just grabs a page xy
> > > times and tells me how long it took.
>
> > you aren't doing a read(), so technically you are just connecting to
> > the web server and sending the request but never reading the content
> > back from the socket.  So your timing wouldn't be accurate.
>
> > try this instead:
> > response = urllib2.urlopen(req).read()
>
> > But that is not the problem you are describing...
>
> Thanks for this pointer, didn't come to my mind.
>
> > > when I increase the number of repetitions, it is
> > > slowing down considerably (1 is like 3 ms, 100 takes 6 seconds).
> > > Maybe it is a known issue in urllib2
>
> > I ran your code and can not reproduce that behavior.  No matter how
> > many repetitions, I still get a similar response time per transaction.
>
> > any more details or code samples you can provide?
>
> I don;t know, I have tried the program on my local MacOs, where I have
> several python runtimes installed and there is huge dfference between
> result after running at 2.6 and 2.4. So this might be the problem.
> When ran on the 2.6 result are comparable to php and better than ruby,
> which is what I expect.
>
> The problem is, that CentOS is running on the server and there is only
> 2.4 available. On wich version did you ran these tests?
>
> Thanks
>
> > -Corey Goldberg




More information about the Python-list mailing list