ANN: socketrecvall-1.0 extension module
Thilo.Ernst at dlr.de
Fri Jan 31 20:08:52 CET 2003
This module might help if you observe
degrading performance when receiving large
chunks of data from a TCP socket.
Add-on module for 'socket' or 'timeoutsocket'.
Implements a function
recvall(socket, total_size) -> string
which synchronously receives a large, known-length
chunk of data in one go. recvall() behaves like
recv(..., flags=MSG_WAITALL) - its main reason of
existence is that some platforms (notably Win32/
Winsock2) do not support the MSG_WAITALL flag.
In practice this means that big transmissions end up
getting splitted into many small chunks which get
combined only on Python level (even when resorting
to a read() on the result of makefile(), that's what
happens). This means lots of overhead for allocating,
concatenating and finally deallocating many small
intermediary receive buffers. So implementing the
'receive loop' in C rather than Python and employing
one large receive buffer (inside a Python string
object allocated for the purpose) buys us a
substantial speed gain.
Also, overlapping communication and computation is
improved as recvall() releases the global interpreter
lock for the duration of the entire transfer rather
then 'on and off' for all individual recv()'s.
In our application (which employs a pure-Python server
sending an endless stream of ~2MB data blocks over TCP
and a Python client now using this extension, both
running on Windows), we achieved a speedup of 15 and were
finally able to closely approach the nominal throughput
of the underlying network hardware (test settings:
768KBit/s Bluetooth, 10MBit/s and 100MBit/s Ethernet).
If this can be achieved on Windows with just the standard
socket module (e.g. by tuning TCP/IP parameters in the
registry) we'd like to learn how.
The recvall() function works both with socket objects as
returned from socket.socket() and with the wrapper objects
employed by the 'timeoutsocket' module.
A second function
recvall_cycles() -> int
can be called after recvall() and returns the number of
internal recv() cycles that occurred during the recvall().
It would be possible to integrate this code more closely
with the standard socket module (e.g. having it dynamically
patch the socket.socket class on import) but it works, and
the clean solution would be to move _recvall() into
This module was tested on Windows (98SE, 2000 and XP) and
Linux (RH7.1). As Linux does support the MSG_WAITALL
flag the extension is not really needed there; the port
is meant merely as a first step towards cross-platformness
since there probably are more OSes that lack MSG_WAITALL.
License: Python License
Python 2.2 (tested, earlier versions should work, too)
Find distutils-based source and binary (win32, linux2) versions at:
T. Ernst - te0006<that_special_character>web<dot>de
More information about the Python-list