[Python-bugs-list] [ python-Bugs-614791 ] gethostbyname() blocks when threaded

[noreply@sourceforge.net]

Bugs item #614791, was opened at 2002-09-25 23:44
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=614791&group_id=5470

Category: Python Library
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Rob Green (rbgrn)
Assigned to: Nobody/Anonymous (nobody)
Summary: gethostbyname() blocks when threaded

Initial Comment:
I know this is because it is simply a wrapper for the libc 
gethostbyname, but it really, really hurts my app to have 
it block like this. I am requesting that by any means 
necessary, gethostbyname() not block when multiple 
threads call it simultaneously. It seriously ruins the 
ability to write effective network daemons in python, as it 
can take a very, very long time to get through even a 
small list of hostnames if one or a few of them need to 
timeout to be resolved.

----------------------------------------------------------------------

>Comment By: Rob Green (rbgrn)
Date: 2002-09-29 18:19

Message:
Logged In: YES 
user_id=590105

http://orange.kame.net/dev/query-pr.cgi?pr=277

There's the bug. It was submitted over 2 years ago, and I'm 
willing to bet it has been fixed but their bug database isn't up-
to-date. I ran the test code and it passed on my machine, 
meaning it's fixed in my version, so I emailed the author to 
find out what version this was fixed in. 

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2002-09-29 16:31

Message:
Logged In: YES 
user_id=21627

As for FreeBSD bugs: Please consult the BUGS section of
getaddrinfo(3). It may be that this man page is incorrect, I
don't know.

----------------------------------------------------------------------

Comment By: Rob Green (rbgrn)
Date: 2002-09-29 14:05

Message:
Logged In: YES 
user_id=590105

I've done some testing and I partially see what you are talking 
about. I wrote a test case in C using pthreads and in 
FreeBSD 4.5 I was able to resolve tens of hostnames 
simultaneously with no errors or problems. Python of course 
has the lock wrapped around the call, and Java appears to as 
well. Should I expect problems with this? Or might this be an 
indication that somewhere along the lines in freebsd 
development this issue was resolved but never 
tested/incorporated with Python/Java? If this is the case, then 
I suppose we would need to figure out at what version this 
changed and have the python lock act accordingly.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2002-09-26 13:14

Message:
Logged In: YES 
user_id=21627

socketmodule.c contains a number of calls to getaddrinfo and
getnameinfo. On systems where those calls are not
thread-safe, you should add a lock around them, see Python
2.1, socketmodule.c, gethostbyname_lock. On all systems, you
should use Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS;
again, see Python 2.1.

There probably should be a single lock around both
getaddrinfo and getnameinfo. By default, those should be
considered thread-safe, giving explicit exceptions: *BSD,
and the fallback implementations (i.e. if either
getaddrinfo.c or getnameinfo.c is included).

----------------------------------------------------------------------

Comment By: Rob Green (rbgrn)
Date: 2002-09-26 12:28

Message:
Logged In: YES 
user_id=590105

This is a little over my head, but I'll take a look to see what 
you mean. Could you give me any more info to point me in 
the right direction? Thanks

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2002-09-26 11:29

Message:
Logged In: YES 
user_id=21627

Bad choice of system :-) In general, it would be the easiest
modification of the interpreter to just release the GIL
before getaddrinfo (which gethostbyname uses); this is a
simple change and makes everything free running.

Unfortunately, on BSD, getaddrinfo is not thread-safe. So
for this system, a much more complicated change must be
implemented. I have semi-approved any change that releases
the GIL unconditionally on "good" systems (Solaris, Linux,
Tru64). That won't help you, though.

If you can contribute a patch that adds a getaddrinfo lock
for "bad" systems, and avoids this lock on "good" systems,
there is a small chance that this can get included in 2.2.2.
Without anybody acting quickly, this may or may not get
fixed in 2.3.

----------------------------------------------------------------------

Comment By: Rob Green (rbgrn)
Date: 2002-09-26 11:09

Message:
Logged In: YES 
user_id=590105

FreeBSD 4.5, although I don't think it's platform specific, as 
I've seen some messages from guido on certain mailing lists 
talking about this problem and how it will require a large 
rewrite to fix correctly. In all honesty, I'd be happy with a 
simple hack to make this work and then a rewrite of whatever 
needs it in the future to have it written correctly.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2002-09-26 11:06

Message:
Logged In: YES 
user_id=21627

Can you please report what platform you are using?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=614791&group_id=5470