[Python-bugs-list] [ python-Bugs-211710 ] socket.send() can do partial writes on some systems.
noreply@sourceforge.net
noreply@sourceforge.net
Sat, 22 Dec 2001 18:26:02 -0800
Bugs item #211710, was opened at 2000-08-11 13:25
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=211710&group_id=5470
Category: Python Library
Group: Platform-specific
Status: Open
Resolution: None
Priority: 5
Submitted By: scott cotton (scottc)
Assigned to: Jeremy Hylton (jhylton)
Summary: socket.send() can do partial writes on some systems.
Initial Comment:
0811 15:57 chronis:~% uname -a
FreeBSD chronis.pobox.com 4.0-STABLE FreeBSD 4.0-STABLE #0: Tue Mar 21 01:05:14 EST 2000 root@chronis.pobox.com:/usr/src/sys/compile/MYBSD i386
0811 15:57 chronis:~%
0811 15:57 chronis:~% cat scratch/sendtstsrv.py
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind("chronis.pobox.com", 8010)
while 1:
s.listen(1)
conn, addr = s.accept()
print "connected from", addr
while 1:
data = conn.recv(1024)
if not data: break
print "done"
conn.close()
0811 15:58 chronis:~% python scratch/sendtstsrv.py &
[2] 76562
0811 15:58 chronis:~% cat scratch/sendtst.py
#!/usr/bin/env python
s = "0" * 10000000
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect("chronis.pobox.com", 8010)
sent = sock.send(s)
if sent != len(s):
print "sent %d/%d chars" % (sent, len(s))
else:
print "sent all chars"
0811 15:58 chronis:~% python !$
python scratch/sendtst.py
connected from ('208.210.124.49', 1258)
sent 17520/10000000 chars
done
0811 15:58 chronis:~%
NOTE: there is nothing in any man page for send() under linux (RH 6.1), Solaris (SunOS 5.5.1), OSF1 V4.0, or FreeBSD{3,4} that states that send() must not perform a
partial write, though of those systems, it only seems
to reproducibly do partial writes under FreeBSD4.0 Stable.
Additionally, W.Richard Stevens' Unix Network Programming Vol 1, second edition states that """
A read or write on a stream socket might input or output fewer bytes than requested, but this is not an error condition.""" (page 77). Later, Stevens says of send() and recv() "These two functions are similar to the standard read and write functions, but one additional argument is required". (page 354).
scott
----------------------------------------------------------------------
>Comment By: Anthony Baxter (anthonybaxter)
Date: 2001-12-22 18:26
Message:
Logged In: YES
user_id=29957
Ah - should have gone back and closed this. The eventual
design that was taken was to add a socket.sendall() call,
and make the various libs call this. This was added to 2.2,
and is also in 2.1.2-to-be.
----------------------------------------------------------------------
Comment By: Mike Brown (mike_j_brown)
Date: 2001-12-22 18:13
Message:
Logged In: YES
user_id=371366
I disagree with changing the behavior of socket.py. The API
is clear and Python is not in the wrong for being a very
thin wrapper over the underlying implementation. Failing to
check the return value of send() is a common socket
programming error in all languages. Just change the libs
that make this mistake. There is probably a use case for
seeing the returns from partial writes.
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2001-10-21 18:02
Message:
Logged In: YES
user_id=6380
Please go slowly here.
----------------------------------------------------------------------
Comment By: Anthony Baxter (anthonybaxter)
Date: 2001-10-21 00:46
Message:
Logged In: YES
user_id=29957
On second (or is this third?) thoughts, patching
socket.py to make it wrap the _socketmodule send
code is an alternative approach that may be safer.
Pity about the _fileobject, tho - it needs considerably
more help.
----------------------------------------------------------------------
Comment By: Anthony Baxter (anthonybaxter)
Date: 2001-10-21 00:07
Message:
Logged In: YES
user_id=29957
The two approaches would seem to be:
patch the socketmodule to make sure send() actually
sends all the bytes.
or
patch the offending std library calls.
The latter preserves the std unix semantics, but possibly
violates the principle of Least Suprise.
For 2.1.2, I'm patching the std library.
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2001-10-19 05:20
Message:
Logged In: YES
user_id=6380
Reopened. You have a point. (I think we closed it becase we
couldn't reproduce it. Our fault for not having access to
FreeBSD. :-)
----------------------------------------------------------------------
Comment By: Richard Jones (richard)
Date: 2001-10-18 23:43
Message:
Logged In: YES
user_id=6405
This problem is only seen on FreeBSD, it seems. Not even
OSX, the BSD derivative, has this problem. However, my
FreeBSD (4.3-RELEASE) socket(2) man page states:
If no messages space is available at the socket to
hold the message to be
transmitted, then send() normally blocks, unless the
socket has been
placed in non-blocking I/O mode. The select(2) call
may be used to de-
termine when it is possible to send more data.
Linux (2.4.7) has a similar message:
When the message does not fit into the send buffer of
the
socket, send normally blocks, unless the socket has
been
placed in non-blocking I/O mode. In non-blocking mode
it
would return EAGAIN in this case. The select(2) call
may
be used to determine when it is possible to send
more
data.
So "normally" is being interpreted loosly by FreeBSD in
this case, it seems.
A solution might be to have socket.send() try to send all
the data. At a minimum, the standard library should be
fixed so it works on FreeBSD.
>From the Lib directory of 2.1.1:
[richard@ike /usr/local/src/Python-2.1.1]% grep -r
'\.send(' Lib
Lib/ftplib.py: self.sock.send(line)
Lib/ftplib.py: self.sock.send(line, MSG_OOB)
Lib/ftplib.py: conn.send(buf)
Lib/ftplib.py: conn.send(buf)
Lib/gopherlib.py: s.send(selector + CRLF)
Lib/httplib.py: self.sock.send(str)
Lib/httplib.py: self.send(str)
Lib/httplib.py: self.send(str)
Lib/httplib.py: self.send(str)
Lib/httplib.py: self.send('\r\n')
Lib/httplib.py: self.send(body)
Lib/imaplib.py: self.sock.send('%s%s' % (data,
CRLF))
Lib/imaplib.py: self.sock.send(literal)
Lib/imaplib.py: self.sock.send(CRLF)
Lib/nntplib.py: self.sock.send(line)
Lib/poplib.py: self.sock.send('%s%s' % (line, CRLF))
Lib/smtplib.py: sendptr = sendptr +
self.sock.send(str[sendptr:])
Lib/smtplib.py: self.send(str)
Lib/smtplib.py: self.send(q)
Lib/socket.py: self._sock.send(self._wbuf)
Lib/telnetlib.py: self.sock.send(buffer)
Lib/telnetlib.py: self.sock.send(IAC +
WONT + opt)
Lib/telnetlib.py: self.sock.send(IAC +
DONT + opt)
Lib/urllib.py: h.send(data)
Lib/urllib.py: h.send(data)
Lib/urllib2.py: h.send(data)
Lib/webbrowser.py: s.send(action)
Lib/test/test_asynchat.py: n = conn.send(buffer)
Lib/test/test_socket.py: conn.send(data)
Lib/test/test_socket.py: s.send(msg)
Lib/test/test_socketserver.py: s.send(teststring)
Note that smtplib handles send()'s return, where nothing
else does. Note also that the above grep does not include
our patch.
----------------------------------------------------------------------
Comment By: Richard Jones (richard)
Date: 2001-10-18 23:04
Message:
Logged In: YES
user_id=6405
We have patched httplib.HTTPConnection.send() so it does
the following:
sent = 0
while sent < len(str):
sent = sent + self.sock.send(str[sent:])
Could this be the default behaviour in the socket module
perhaps?
----------------------------------------------------------------------
Comment By: Richard Jones (richard)
Date: 2001-10-18 22:57
Message:
Logged In: YES
user_id=6405
Why is this bug closed? The standard library uses send()
in a number of places - we've just run up against it in
httplib. Again, on FreeBSD, we're seeing that send() is
only sending a part of the data we want it to.
----------------------------------------------------------------------
Comment By: Jeremy Hylton (jhylton)
Date: 2000-09-15 11:50
Message:
scott-- Is there a bug here? It seems clear that the send system call is not required to send all the bytes you asked it to send. It returns the number of bytes it did send. If you want to make sure all your data is sent, you can wrap send in a while loop that calls send until all the data is consumed.
----------------------------------------------------------------------
Comment By: Jeremy Hylton (jhylton)
Date: 2000-09-07 15:01
Message:
Please do triage on this bug.
----------------------------------------------------------------------
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=211710&group_id=5470