[Python-bugs-list] [ python-Bugs-614791 ] gethostbyname() blocks when threaded
noreply@sourceforge.net
noreply@sourceforge.net
Sun, 29 Sep 2002 16:45:00 -0700
Bugs item #614791, was opened at 2002-09-25 23:44
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=614791&group_id=5470
Category: Python Library
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Rob Green (rbgrn)
Assigned to: Nobody/Anonymous (nobody)
Summary: gethostbyname() blocks when threaded
Initial Comment:
I know this is because it is simply a wrapper for the libc
gethostbyname, but it really, really hurts my app to have
it block like this. I am requesting that by any means
necessary, gethostbyname() not block when multiple
threads call it simultaneously. It seriously ruins the
ability to write effective network daemons in python, as it
can take a very, very long time to get through even a
small list of hostnames if one or a few of them need to
timeout to be resolved.
----------------------------------------------------------------------
>Comment By: Rob Green (rbgrn)
Date: 2002-09-29 18:45
Message:
Logged In: YES
user_id=590105
http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libc/net/name6.c
It appears as though this was made thread-safe in revision
1.17 of the file which was Jan 24, 2001. Unfortunately there
are two branches of code here, and I have no idea what's
going on with them. I apparently have code from 2000 that
doesn't appear to be thread-safe but the test worked so I'm
gonna send around some emails and see if I can't clear
anything up.
----------------------------------------------------------------------
Comment By: Rob Green (rbgrn)
Date: 2002-09-29 18:19
Message:
Logged In: YES
user_id=590105
http://orange.kame.net/dev/query-pr.cgi?pr=277
There's the bug. It was submitted over 2 years ago, and I'm
willing to bet it has been fixed but their bug database isn't up-
to-date. I ran the test code and it passed on my machine,
meaning it's fixed in my version, so I emailed the author to
find out what version this was fixed in.
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2002-09-29 16:31
Message:
Logged In: YES
user_id=21627
As for FreeBSD bugs: Please consult the BUGS section of
getaddrinfo(3). It may be that this man page is incorrect, I
don't know.
----------------------------------------------------------------------
Comment By: Rob Green (rbgrn)
Date: 2002-09-29 14:05
Message:
Logged In: YES
user_id=590105
I've done some testing and I partially see what you are talking
about. I wrote a test case in C using pthreads and in
FreeBSD 4.5 I was able to resolve tens of hostnames
simultaneously with no errors or problems. Python of course
has the lock wrapped around the call, and Java appears to as
well. Should I expect problems with this? Or might this be an
indication that somewhere along the lines in freebsd
development this issue was resolved but never
tested/incorporated with Python/Java? If this is the case, then
I suppose we would need to figure out at what version this
changed and have the python lock act accordingly.
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2002-09-26 13:14
Message:
Logged In: YES
user_id=21627
socketmodule.c contains a number of calls to getaddrinfo and
getnameinfo. On systems where those calls are not
thread-safe, you should add a lock around them, see Python
2.1, socketmodule.c, gethostbyname_lock. On all systems, you
should use Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS;
again, see Python 2.1.
There probably should be a single lock around both
getaddrinfo and getnameinfo. By default, those should be
considered thread-safe, giving explicit exceptions: *BSD,
and the fallback implementations (i.e. if either
getaddrinfo.c or getnameinfo.c is included).
----------------------------------------------------------------------
Comment By: Rob Green (rbgrn)
Date: 2002-09-26 12:28
Message:
Logged In: YES
user_id=590105
This is a little over my head, but I'll take a look to see what
you mean. Could you give me any more info to point me in
the right direction? Thanks
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2002-09-26 11:29
Message:
Logged In: YES
user_id=21627
Bad choice of system :-) In general, it would be the easiest
modification of the interpreter to just release the GIL
before getaddrinfo (which gethostbyname uses); this is a
simple change and makes everything free running.
Unfortunately, on BSD, getaddrinfo is not thread-safe. So
for this system, a much more complicated change must be
implemented. I have semi-approved any change that releases
the GIL unconditionally on "good" systems (Solaris, Linux,
Tru64). That won't help you, though.
If you can contribute a patch that adds a getaddrinfo lock
for "bad" systems, and avoids this lock on "good" systems,
there is a small chance that this can get included in 2.2.2.
Without anybody acting quickly, this may or may not get
fixed in 2.3.
----------------------------------------------------------------------
Comment By: Rob Green (rbgrn)
Date: 2002-09-26 11:09
Message:
Logged In: YES
user_id=590105
FreeBSD 4.5, although I don't think it's platform specific, as
I've seen some messages from guido on certain mailing lists
talking about this problem and how it will require a large
rewrite to fix correctly. In all honesty, I'd be happy with a
simple hack to make this work and then a rewrite of whatever
needs it in the future to have it written correctly.
----------------------------------------------------------------------
Comment By: Martin v. Löwis (loewis)
Date: 2002-09-26 11:06
Message:
Logged In: YES
user_id=21627
Can you please report what platform you are using?
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=614791&group_id=5470