[Python-bugs-list] [ python-Bugs-774751 ] slow socket binding & netinfo lookups

[SourceForge.net]

Bugs item #774751, was opened at 2003-07-20 18:57
Message generated for change (Comment added) made by montanaro
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=774751&group_id=5470

Category: Macintosh
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: Stuart Bishop (zenzen)
Assigned to: Skip Montanaro (montanaro)
Summary: slow socket binding & netinfo lookups

Initial Comment:
The following code takes < 1 second to run under Python 
2.1, but 20 seconds to run under Python 2.2 or 2.3 on a top-
end PowerBook running OSX 10.2. It is part of the Zope 
startup routine.

def max_server_sockets():
    sl = []
    while 1:
        try:
            s = socket.socket (socket.AF_INET, 
socket.SOCK_STREAM)
            s.bind (('',0))
            s.listen(5)
            sl.append (s)
        except:
            break
    num = len(sl)
    for s in sl:
        s.close()
    del sl
    return num



----------------------------------------------------------------------

>Comment By: Skip Montanaro (montanaro)
Date: 2003-07-21 21:42

Message:
Logged In: YES 
user_id=44345

I think we've concluded that it's okay to ship 2.3 with this problem
unresolved, however I'm completely befuddled at the moment.  I 
wrote this
simple program:

    #include <sys/time.h>
    #include <sys/types.h>
    #include <sys/socket.h>
    #include <netdb.h>
    #include <stdio.h>

    int
    main(int argc, char **argv) {
	    int i;
	    struct addrinfo hints, *res;
	    int error;
	    struct timeval t, u;

	    hints.ai_family = AF_INET;
	    hints.ai_socktype = SOCK_DGRAM;	/*dummy*/
	    hints.ai_flags = AI_PASSIVE;

	    printf("start\n");
	    for (i=0; i<100; i++) {
		    gettimeofday(&t, NULL);
		    error = getaddrinfo(0, "0", &hints, &res);
		    gettimeofday(&u, NULL);
		    fprintf(stderr, "gtod: %.6f\n",
			    u.tv_sec-t.tv_sec+(u.tv_usec-t.tv_usec)*1e-6);
		    freeaddrinfo(res);
	    }
	    printf("finish\n");
    }

When run on my Mac, it takes between 2 and 7 microseconds per 
getaddrinfo
call.  The exact same instrumented call inside 
socketmodule.c:setipaddr at
line 700 takes about 150 *milli*seconds per call.  I tried 
eliminating the
Py_{BEGIN,END}_ALLOW_THREADS calls as well as the ACQUIRE 
and RELEASE of the
getaddrinfo lock.  That had no effect on the call.

I also tweaked the compile like to run just the C preprocessor (gcc 
-E
instead of gcc -g) and checked the output:

	        ...
                hints.ai_family = af;
                hints.ai_socktype = 2;
                hints.ai_flags = 0x00000001;
                { PyThreadState *_save; _save = 
PyEval_SaveThread();
                PyThread_acquire_lock(netdb_lock, 1);
                gettimeofday(&t, 0);
                error = getaddrinfo(0, "0", &hints, &res);
                gettimeofday(&u, 0);
                fprintf((&__sF[2]), "gtod: %.6f\n",
                        u.tv_sec-t.tv_sec+(u.tv_usec-t.tv_usec)*1e-6);
                PyEval_RestoreThread(_save); }
		...

otool -L run against both the test program and _socket.so indicate 
that both
refer to just one shared library (/usr/lib/libSystem.B.dylib), so 
both
should be calling the same routine.


----------------------------------------------------------------------

Comment By: Jack Jansen (jackjansen)
Date: 2003-07-21 15:52

Message:
Logged In: YES 
user_id=45365

I have absolutely nothing to contribute wrt. this bug. As Just is 
away I guess Skip is the best target for the hot patato. 

----------------------------------------------------------------------

Comment By: Skip Montanaro (montanaro)
Date: 2003-07-21 11:33

Message:
Logged In: YES 
user_id=44345

Digging a bit deeper, it appears on Mac OS X that
fake_getaddrinfo is used.  That's deemed not to be thread-safe.
I see three static or global variables acessed by this function:

    firsttime - simple guard
    translate - set inside the guard - readonly after that
    faith_prefix - same

Why not push the guard and initialization code into init_socket
and protect it with the getaddrinfo lock?  Fake_getaddrinfo
would thus be thread-safe and could be accessed without
acquiring a lock.


----------------------------------------------------------------------

Comment By: Skip Montanaro (montanaro)
Date: 2003-07-21 11:14

Message:
Logged In: YES 
user_id=44345

whoops...  my apologies for not posting here.


----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2003-07-21 10:50

Message:
Logged In: YES 
user_id=31435

Attaching copious info from Skip Montanaro (skip.txt).

----------------------------------------------------------------------

Comment By: Anthony Baxter (anthonybaxter)
Date: 2003-07-21 02:05

Message:
Logged In: YES 
user_id=29957

Is this something that should be looked at for 2.3rc2?


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=774751&group_id=5470