ANN: socketrecvall-1.0 extension module

Thilo Ernst Thilo.Ernst at
Fri Jan 31 20:08:52 CET 2003

This module might help if you observe 
degrading performance when receiving large 
chunks of data from a TCP socket.


    Add-on module for 'socket' or 'timeoutsocket'.
    Implements a function

         recvall(socket, total_size) -> string

    which synchronously receives a large, known-length
    chunk of data in one go. recvall() behaves like
    recv(..., flags=MSG_WAITALL) - its main reason of
    existence is that some platforms (notably Win32/
    Winsock2) do not support the MSG_WAITALL flag.

    In practice this means that big transmissions end up
    getting splitted into many small chunks which get
    combined only on Python level (even when resorting
    to a read() on the result of makefile(), that's what
    happens). This means lots of overhead for allocating,
    concatenating and finally deallocating many small
    intermediary receive buffers. So implementing the
    'receive loop' in C rather than Python and employing
    one large receive buffer (inside a Python string
    object allocated for the purpose) buys us a
    substantial speed gain.

    Also, overlapping communication and computation is
    improved as recvall() releases the global interpreter
    lock for the duration of the entire transfer rather
    then 'on and off' for all individual recv()'s.

    In our application (which employs a pure-Python server
    sending an endless stream of ~2MB data blocks over TCP
    and a Python client now using this extension, both
    running on Windows), we achieved a speedup of 15 and were
    finally able to closely approach the nominal throughput
    of the underlying network hardware (test settings:
    768KBit/s Bluetooth, 10MBit/s and 100MBit/s Ethernet).
    If this can be achieved on Windows with just the standard
    socket module (e.g. by tuning TCP/IP parameters in the
    registry) we'd like to learn how.

    The recvall() function works both with socket objects as
    returned from socket.socket() and with the wrapper objects
    employed by the 'timeoutsocket' module.

    A second function

            recvall_cycles() -> int

    can be called after recvall() and returns the number of
    internal recv() cycles that occurred during the recvall().

    It would be possible to integrate this code more closely
    with the standard socket module (e.g. having it dynamically
    patch the socket.socket class on import) but it works, and
    the clean solution would be to move _recvall() into
    socketmodule.c anyway.

    This module was tested on Windows (98SE, 2000 and XP) and
    Linux (RH7.1). As Linux does support the MSG_WAITALL
    flag the extension is not really needed there; the port
    is meant merely as a first step towards cross-platformness
    since there probably are more OSes that lack MSG_WAITALL.

    License: Python License


    Python 2.2 (tested, earlier versions should work, too)


    Find distutils-based source and binary (win32, linux2) versions at:

T. Ernst - te0006<that_special_character>web<dot>de

More information about the Python-list mailing list