urllib slow on FreeBSD 4.7? sockets too

[Bengt Richter]

On Fri, 22 Nov 2002 09:13:30 +0000 (UTC), Jarkko Torppa <torppa at staff.megabaud.fi> wrote:

>In article <3ddd8419_3 at omega.dimensional.com>, Mike Brown wrote:
>> As was pointed out, all the delay is in the reading, and it really only
>> happens when dealing with these file-wrapped sockets as returned by
>> urlopen(). They can be on the localhost, even.
>
>Seems that stdio is somehow confused, try this
>
I'm not sure it's a good experiment to eliminate too many things at once.
I.e., how do you know how much you gained by going to os.read and how much
you gained by buffering via bytea.append instead of having urllib do it internally?

I don't have a real handy way to duplicate your experiment (not having BSD on the other box),
but if it's easy for you, I'd be interested in how fast/slow this mod of your test runs:
(change file an url to taste)

=====
#!/usr/bin/python
# turlread.py
import urllib, time

def main():
    starttime = time.time()
    u = urllib.urlopen('http://localhost/4m')
    f = file('file_4m.html','wb')
    length = 0
    while 1:
        bytes = u.read(16 * 1024)
        if bytes == '':
            break
        f.write(bytes)
        length += len(bytes)
    u.close()
    f.close()
    endtime = time.time()
    elapsed = endtime - starttime
    print "bytes: %.1fK; time: %0.3fs (%0d KB/s)" % (
        length / 1024.0, elapsed, length / 1024.0 / elapsed
    )

if __name__ == '__main__':
    main()

=====

>import urllib, time, os
>
>starttime = time.time()
>u = urllib.urlopen('http://localhost/4m')
>fn = u.fp.fileno()
>bytea = [ ]
>while 1:
>    bytes = os.read(fn, 16 * 1024)
>    if bytes == '':
>        break
>    bytea.append(bytes)
>bytes = ''.join(bytea)
>u.close()
>endtime = time.time()
>elapsed = endtime - starttime
>length = len(bytes)
>print "bytes: %.1fK; time: %0.3fs (%0d KB/s)" % (length / 1024.0, elapsed, length / 1024.0 / elapsed )
>
>On my NetBSD box this gives
>
>bytes: 4096.0K; time: 0.450s (9105 KB/s)
>0.15s user 0.08s system 44% cpu 0.513 total
>vs
>bytes: 4096.0K; time: 14.616s (280 KB/s)
>2.52s user 11.57s system 96% cpu 14.676 total
>

Regards,
Bengt Richter