[python-uk] urllib latency

[Matt Hamilton]

On 20 Dec 2010, at 16:26, Doug Winter wrote:

> Matt Hamilton wrote:
>> I'm off to go dig in the urllib code and see if I can see anything
>> there. I'm wondering if urllib is taking some time to process the
>> data after it receives it before doing anything.
>> 
>> This is on OSX, but I'm going to go try it on a FreeBSD box as I can
>> then use ktrace to see what might be happening.
> 
> Could it be SO_REUSEADDR?  Seems unlikely but I can't think of anything
> else.  If ab is specifying this but python isn't you might get a
> reconnection delay.

Close :) It appears to be some interaction between Nagle and delayed acks. The server is FreeBSD, and if I disable delayed acks on the server then I notice a bit of a speedup (seems there is a deadlock that occurs in the TCP sessions between my varnish and haproxy processes on that box). 

I then have a similar issue for the request I mentioned in this thread, that happens to be cached by varnish. If I then disable delayed acks on the OSX box running the python tests, that speeds up again and I lose the 100ms extra latency and it is on par with ab. ie. about 6ms to retrieve the request.

Question is why does python suffer from this deadlock, yet ab on the same machine doesn't? I thought it could be something with TCP_NODELAY, but setting that on the socket on python doesn't seem to help. I'm just wondering if there is a particular order in which the packets and acks are being sent, but from my wireshark trace yesterday they did seem to be exactly the same.

-Matt

-- 
Matt Hamilton                                         matth at netsight.co.uk
Netsight Internet Solutions, Ltd.          Business Vision on the Internet
http://www.netsight.co.uk                               +44 (0)117 9090901
Web Design | Zope/Plone Development and Consulting | Co-location | Hosting