[Patches] [ python-Patches-516715 ] HTTPConnection.send() fails large reques

Sat, 16 Feb 2002 07:41:42 -0800

Patches item #516715, was opened at 2002-02-12 15:44
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=305470&aid=516715&group_id=5470

Category: Library (Lib)
Group: Python 2.1.2
Status: Open
Resolution: None
Priority: 7
Submitted By: James Rucker (jamesrucker)
Assigned to: Nobody/Anonymous (nobody)
Summary: HTTPConnection.send() fails large reques

Initial Comment:
The present implementation of HTTPConnection.send() in
httplib.py does not handle the case of less than the
number of requested bytes being sent via socket.send().

Indeed, this condition occurs for fairly large
requests.  In my environment, socket.send() will max
out at transferring 33011 bytes.

The routine needs to check how many bytes where
actually  transferred and make subsequent socket.send()
calls until all of the data makes it through.  In the
patch I've supplied, I send slices of a particular
maximum size in each iteration, rather than attempting
to send the entire remainder of data, for efficiency
(it proved expensive to have big slices being created
per call, which would happen otherwise).

I noticed that in the description of recent patches it
appears that an attempt has made to address this, but
the version of httplib.py that supposedly provides a
fix appears to make invalid references to socket.py. 
If what's there is actually suitable, please do disregard.

The submitted diff is against version 1.34.2.1

----------------------------------------------------------------------

>Comment By: James Rucker (jamesrucker)
Date: 2002-02-16 07:41

Message:
Logged In: YES 
user_id=351540

Using sendall() proposed in the diff of socketmodule.c
provided by the patch nascheme referenced does make sense. 
Note that there is an extraneous variable 'total' in the
PySocketSock_sendall() (not sure that you care to know that
here, but I thought I should point it out).

Also, while I see the revision of socketmodule.c specifying
sendall() in certain versions, I didn't see versions of
httplib.py that used it (you said that 2.1.1/2 does).  Can
you provide a CVS revision number of httplib.py that uses
sendall() (forgive me, but I'm trying to make sure I'm
understanding your naming conventions, and I'd also like to
apply the patch here with code from the actual repository).

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2002-02-16 01:41

Message:
Logged In: YES 
user_id=21627

I believe it should use sendall throughout; in fact, 2.1.1/2
does that. I've ported Anthony's patch from

http://aspn.activestate.com/ASPN/Mail/Message/Python-checkins/956422

to 2.2, please find this attached. I'll do a quick check as
to why that wasn't applied to 2.2, and then commit it.

----------------------------------------------------------------------

Comment By: Neil Schemenauer (nascheme)
Date: 2002-02-15 21:14

Message:
Logged In: YES 
user_id=35752

See this patch for more details on the problem:
http://sourceforge.net/tracker/index.php?func=detail&aid=474307&group_id=5470&atid=305470

Should httplib be using sendall() instead of send()?

----------------------------------------------------------------------

Comment By: James Rucker (jamesrucker)
Date: 2002-02-15 17:47

Message:
Logged In: YES 
user_id=351540

When I say 'max out' I mean that only the first 33011 (in 
my environment) bytes get through on a call.  You don't 
receive an error, per se, as the client things it's send 
is successful.  The problem is that the server doesn't get 
all the bytes expected.  To be clear, this isn't a problem 
with socket.send(); just httplib.py's use of socket.send
() - in particular, the expectation that a lack of an 
error condition being presumed to mean that all bytes were 
sent.

In general, there isn't a problem passing arbitrarily 
large blocks of data, however, clients of socket.send() 
are to check the result of the call, which is an integer 
specifying the number of bytes that were actually 
transmitted.  Note that this number is not guaranteed to 
be the same as the number of bytes specified.  I believe 
that the number of bytes that make it across in a single 
call depend on the underlying buffering and the rate at 
which the client is reading.

I just found the following url which talks in more detail 
about the issue:
http://py-howto.sourceforge.net/sockets/node6.html

Btw, I'm using freebsd 4.4.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2002-02-15 16:16

Message:
Logged In: YES 
user_id=21627

I fail to see a problem in the current code. What does "max
out" mean exactly? There shouldn't be a problem passing an
arbitrarily large block of data to the send system call,
atleast not over a TCP connection. What operating system are
you using, and what is the precise error that you get with
the  current code?

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=305470&aid=516715&group_id=5470