[Patches] [ python-Patches-517256 ] poor performance in xmlrpc response

[noreply@sourceforge.net]

Patches item #517256, was opened at 2002-02-13 15:48
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=305470&aid=517256&group_id=5470

Category: Library (Lib)
Group: Python 2.1.2
Status: Open
Resolution: Accepted
Priority: 5
Submitted By: James Rucker (jamesrucker)
Assigned to: Fredrik Lundh (effbot)
Summary: poor performance in xmlrpc response

Initial Comment:
xmlrpclib.Transport.parse_response() (called from 
xmlrpclib.Transport.request()) is exhibiting poor 
performance - approx. 10x slower than expected.

I investigated based on using a simple app that sent a 
msg to a server, where all the server did was return 
the message back to the caller.  From profiling, it 
became clear that the return trip was taken 10x the 
time consumed by the client->server trip, and that the 
time was spent getting things across the wire.

parse_response() reads from a file object created via 
socket.makefile(), and as a result exhibits 
performance that is about an order of magnitude worse 
than what it would be if socket.recv() were used on 
the socket.  The patch provided uses socket.recv() 
when possible, to improve performance.

The patch provided is against revision 1.15.  Its use 
provides performance for the return trip that is more 
or less equivalent to that of the forward trip.


----------------------------------------------------------------------

>Comment By: James Rucker (jamesrucker)
Date: 2002-03-19 11:47

Message:
Logged In: YES 
user_id=351540

HTTPConnection.getresponse() will close the socket and set
self.sock to null after instantiating response_class (by
default, this is HTTPResponse; note that HTTPResponse does a
makefile() and stores the result in self.fp) iff the newly
created response class instance's 'will_close' attribute is
true.   

My server is setting the Keep-alive header with a value of 1
(it is based on xmlrpcserver.py), which causes will_close to
evaluate to false.  In your case, I'm presuming that
will_close is being evaluated as false and thus the socket
(accessed via h._conn.sock) has been set to <None>.  Note
that when I removed the Keep-alive header, I witness the
behaviour you're seeing.

Thus, it seems that as it stands, the beneift of the change
will only be realized if Keep-alive is set or HTTP/1.1 is
used (and Keep-alive is either not specified or is set to
non-zero).

The following from httplib.py shows and explains how
'will_close' will be set (from httplib.py):

        conn = self.msg.getheader('connection')
        if conn:
            conn = conn.lower()
            # a "Connection: close" will always close the
connection. if we
            # don't see that and this is not HTTP/1.1, then
the connection will
            # close unless we see a Keep-Alive header.
            self.will_close = conn.find('close') != -1 or \
                              ( self.version != 11 and \
                                not
self.msg.getheader('keep-alive') )
        else:
            # for HTTP/1.1, the connection will always
remain open
            # otherwise, it will remain open IFF we see a
Keep-Alive header
            self.will_close = self.version != 11 and \
                              not
self.msg.getheader('keep-alive')

----------------------------------------------------------------------

Comment By: Fredrik Lundh (effbot)
Date: 2002-03-19 01:24

Message:
Logged In: YES 
user_id=38376

What server did you use?  In all my test setups, 
h._conn.sock is None at the time parse_response
is called...

----------------------------------------------------------------------

Comment By: James Rucker (jamesrucker)
Date: 2002-03-17 08:13

Message:
Logged In: YES 
user_id=351540

The problem was discovered under FreeBSD 4.4.


----------------------------------------------------------------------

Comment By: Fredrik Lundh (effbot)
Date: 2002-03-17 05:30

Message:
Logged In: YES 
user_id=38376

James, what platform(s) did you use?

I'm not sure changing the parse_response() interface is
a good idea, but if this is a Windows-only problem, there
may be a slightly cleaner way to get the same end result.

</F>

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-03-01 08:14

Message:
Logged In: YES 
user_id=6380

My guess makefile() isn't buffering properly. This has been
a long-standing problem on Windows; I'm not sure if it's an
issue on Unix.

----------------------------------------------------------------------

Comment By: Fredrik Lundh (effbot)
Date: 2002-03-01 06:34

Message:
Logged In: YES 
user_id=38376

looks fine to me.  I'll merge it with SLAB changes,
and will check it into the 2.3 codebase asap.

(we probably should try to figure out why makefile
causes a 10x slowdown too -- xmlrpclib isn't exactly
the only client library reading from a buffered
socket)

</F>

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-02-28 15:23

Message:
Logged In: YES 
user_id=6380

Fredrik, does this look OK to you?


----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=305470&aid=517256&group_id=5470