iirc, I've been bitten by this a couple of times too (before I switched to asyncore...)
any special reason why the input socket is unbuffered by default?
</F>
----- Original Message ----- From: "Andy Bond" bond@dstc.edu.au Newsgroups: comp.lang.python Sent: Thursday, August 31, 2000 8:41 AM Subject: SocketServer and makefile()
I've been working with BaseHTTPServer which in turn uses SocketServer to write a little web server. It is used to accept PUT requests of 30MB chunks of data. I was having a problem where data was flowing at the rate of something like 64K per second over a 100MB network. Weird. Further tracing showed that the rfile variable from SocketServer (used to suck in data to the http server) was created using makefile on the original socket descriptor. It was created with an option of zero for buffering (see SocketServer.py) which means unbuffered.
Now some separate testing with socket.py showed that I could whip a 30MB file across using plain sockets and send/recv but if I made the receivor use makefile on the socket and then read, it slowed down to my 1 sec per 64K. If I specify a buffer (something big but less than 64K ... IP packet size?) then I am back in speedy territory. The unbuffered mode seems almost like it is sending the data 1 char at a time AND this is the default mode used in SocketServer and subsequently BaseHTTPServer ...
This is on solaris 7, python 1.5.2. Anyone else found this to be a problem or am I doing something wrong?
andy
I ran into this same problem on the client side.
The server does a makefile() so that it can do readline() to fetch the HTTP request line and then the MIME headers. The *problem* is that if you do something like:
f = sock.makefile() line = f.readline() data = sock.recv(1000)
You're screwed if you have buffering enabled. "f" will read in a bunch of data -- past the end of the line. That data now sits inside f's buffer and is not available to the sock.recv() call.
If you forget about sock and just stick to f, then you'd be okay. But SocketServer and/or BaseHTTPServer doesn't -- it uses both objects to do the reading.
Solution? Don't use rfile for reading, but go for the socket itself. Or revamp the two classes to forget about the socket once the files (wfile and rfile) are created. The latter might not be possible, tho.
Dunno why the unbuffered reading would be slow. I'd think it would still read large chunks at a time when you request it.
Cheers, -g
On Thu, Aug 31, 2000 at 07:47:45PM +0200, Fredrik Lundh wrote:
iirc, I've been bitten by this a couple of times too (before I switched to asyncore...)
any special reason why the input socket is unbuffered by default?
</F>
----- Original Message ----- From: "Andy Bond" bond@dstc.edu.au Newsgroups: comp.lang.python Sent: Thursday, August 31, 2000 8:41 AM Subject: SocketServer and makefile()
I've been working with BaseHTTPServer which in turn uses SocketServer to write a little web server. It is used to accept PUT requests of 30MB chunks of data. I was having a problem where data was flowing at the rate of something like 64K per second over a 100MB network. Weird. Further tracing showed that the rfile variable from SocketServer (used to suck in data to the http server) was created using makefile on the original socket descriptor. It was created with an option of zero for buffering (see SocketServer.py) which means unbuffered.
Now some separate testing with socket.py showed that I could whip a 30MB file across using plain sockets and send/recv but if I made the receivor use makefile on the socket and then read, it slowed down to my 1 sec per 64K. If I specify a buffer (something big but less than 64K ... IP packet size?) then I am back in speedy territory. The unbuffered mode seems almost like it is sending the data 1 char at a time AND this is the default mode used in SocketServer and subsequently BaseHTTPServer ...
This is on solaris 7, python 1.5.2. Anyone else found this to be a problem or am I doing something wrong?
andy
Python-Dev mailing list Python-Dev@python.org http://www.python.org/mailman/listinfo/python-dev
I ran into this same problem on the client side.
The server does a makefile() so that it can do readline() to fetch the HTTP request line and then the MIME headers. The *problem* is that if you do something like:
f = sock.makefile() line = f.readline() data = sock.recv(1000)You're screwed if you have buffering enabled. "f" will read in a bunch of data -- past the end of the line. That data now sits inside f's buffer and is not available to the sock.recv() call.
If you forget about sock and just stick to f, then you'd be okay. But SocketServer and/or BaseHTTPServer doesn't -- it uses both objects to do the reading.
Solution? Don't use rfile for reading, but go for the socket itself. Or revamp the two classes to forget about the socket once the files (wfile and rfile) are created. The latter might not be possible, tho.
I was about to say that you have it backwards, and that you should only use rfile & wfile, when I realized that CGIHTTPServer.py needs this! The subprocess needs to be able to read the rest of the socket, for POST requests. So you're right.
Solution? The buffer size should be an instance or class variable. Then SocketServer can set it to buffered by default, and CGIHTTPServer can set it to unbuffered.
Dunno why the unbuffered reading would be slow. I'd think it would still read large chunks at a time when you request it.
System call overhead? I had the same complaint about Windows, where apparently winsock makes you pay more of a performance penalty than Unix does in the same case.
--Guido van Rossum (home page: http://www.pythonlabs.com/%7Eguido/)
On Thu, Aug 31, 2000 at 03:48:16PM -0500, Guido van Rossum wrote:
I wrote: ...
Solution? Don't use rfile for reading, but go for the socket itself. Or revamp the two classes to forget about the socket once the files (wfile and rfile) are created. The latter might not be possible, tho.
I was about to say that you have it backwards, and that you should only use rfile & wfile, when I realized that CGIHTTPServer.py needs this! The subprocess needs to be able to read the rest of the socket, for POST requests. So you're right.
Ooh! I hadn't considered that case. Yes: you can't transfer the contents of a FILE's buffer to the CGI, but you can pass a file descriptor (the socket).
Solution? The buffer size should be an instance or class variable. Then SocketServer can set it to buffered by default, and CGIHTTPServer can set it to unbuffered.
Seems reasonable.
Dunno why the unbuffered reading would be slow. I'd think it would still read large chunks at a time when you request it.
System call overhead? I had the same complaint about Windows, where apparently winsock makes you pay more of a performance penalty than Unix does in the same case.
Shouldn't be. There should still be an rfile.read(1000) in that example app (with the big transfers). That read() should be quite fast -- the buffering should have almost no effect.
So... what is the underlying problem?
[ IOW, there are two issues: the sock vs file thing; and why rfile is so darn slow; I have no insights on the latter. ]
Cheers, -g
Dunno why the unbuffered reading would be slow. I'd think it would still read large chunks at a time when you request it.
System call overhead? I had the same complaint about Windows, where apparently winsock makes you pay more of a performance penalty than Unix does in the same case.
Shouldn't be. There should still be an rfile.read(1000) in that example app (with the big transfers). That read() should be quite fast -- the buffering should have almost no effect.
So... what is the underlying problem?
[ IOW, there are two issues: the sock vs file thing; and why rfile is so darn slow; I have no insights on the latter. ]
Should, shouldn't...
It's a quality of implementation issue in stdio. If stdio, when seeing a large read on an unbuffered file, doesn't do something smart but instead calls getc() for each character, that would explain this. It's dumb, but not illegal.
--Guido van Rossum (home page: http://www.pythonlabs.com/%7Eguido/)
-
Fredrik Lundh -
Greg Stein -
Guido van Rossum