Any clues to source of this delay?

[Gordon McMillan]

Markus Stenberg wrote:

> Ok, as I've noted in some earlier posts, I have nasty tendency of
> prototyping some protocols in Python for occassional implementation
> in more 'robust' 'commercially feasible' languages. I've ran at some
> oddity this time, though:
> 
> Background:
> 
> The protocol involves encapsulating some data as 'packets' on TCP
> connection somewhat like the Record layer of TLSv1.  The simple
> protocol in running top of TCP; this, in itself, is nothing
> particularly new or fancy. The _problem_, however, seems to be.
> 
> Problem:
> 
> On Linux, single connections have 'glass ceiling' of 50
> roundtrips/second; no matter what I kludge, it seems to stay there.
> Only way to fix this was to send 'empty' packets after processed
> ones, thus increasing throughput by about 5X.
> 
> On NT, the glass ceiling is same, just 1/10 of Linux's (5
> roundtrips/second).
> 
> I _assume_ this is some TCP/Python-related feature; no other traffic
> than the one mentioned occurs during the test period. 

I have also run into similar (though not identical) oddities. (In my
case I was stuck with the protocol, but had to get the servers
working better). I found that Linux (AMD K6, 200Mhz) would max out
at 90 "messages" / sec, and NT (P100) at 50 / sec.

As my first cut, I wrote pure Python client and server, and ignored 
their protocol. I was maxing out the LAN connection from any box.

Then I used their protocol and their client (which can't send more 
than 40 / sec on the Linux box) and ran into these limits. Then I 
rewrote in C. The amount of CPU used dropped, but the speed did not 
increase.

So, at least in my case, there's no blame on Python. I still don't 
understand what it is about their protocol that causes this slowdown. 
It appears that select just takes longer than it "should".

- Gordon