michael at stroeder.com
Wed Jun 26 22:58:24 CEST 2002
Leif Hedstrom wrote:
> Well, I don't know if this is the same problem I had with Python LDAP v1.x,
> we haven't tested v2.x yet. But, the result() function in Python LDAP can go
> into a very tight poll loop, with extreme effects if the Python process is
> running on the same machines as the LDAP server. Python will almost
> completely starve slapd for any CPU time ...
> Adding a short sleep() in the polling loop of ldapobject.result() helps, a
Yes, you're right. A time.sleep(0.0001) right before the inner
result() call makes python-ldap hand over the CPU to the OS.
> while all:
> while ldap_result is None:
> if (timeout>=0) and (time.time()-start_time>timeout):
> raise _ldap.TIMELIMIT_EXCEEDED(
> "LDAP time limit (%d secs) exceeded." % (timeout)
> ldap_result = self._ldap_call(self._l.result,msgid,0,0)
> if ldap_result is None:
> if ldap_result is None:
It's possible to make it somewhat simpler since we have a first
result() call before the while loops.
while ldap_result is None:
if (timeout>=0) and (time.time()-start_time>timeout):
"LDAP time limit (%d secs) exceeded." % (timeout)
ldap_result = self._ldap_call(self._l.result,msgid,0,0)
if ldap_result is None:
ldap_result = None,None
> Alternatively, adding
> an (arbitrarily) long timeout in the call to ldap_result() also accomplishes
> the same thing, like:
> ldap_result = self._ldap_call(self._l.result,msgid,0,15 * 60)
Which is a bad idea in a multi-threaded environment.
> I haven't dug deep into this problem yet, to figure out if this is an
> OpenLDAP library problem, or a Python LDAP problem.
The main problem here is that the OpenLDAP libs are not
thread-safe. Therefore a module-wide lock is needed to serialize
all calls into OpenLDAP libs. But just wrapping ldap_result() with
locking around it would lead to blocking threads if one thread is
waiting for large search results. That's why I implemented
LDAPObject.result() in Python like it is today. Unfortunately we
cannot deal with waiting for data at the socket level which leads
to higher-level polling loop.
Everybody is encouraged to try the time.sleep(0.0001) hack and
look how it "feels" now. BTW: it makes the simple benchmark I
posted yesterday slower. It always depends what you wanna
I suspect this still might not fully explain the
30s-vs.-immediate-results reports. There may be more issues with
other effects like Mauro described.
I guess the best solution would be if somebody with some spare
cycles digs into OpenLDAP's libldap_r to check if it's ready for
use with python-ldap. This would make it possible to do a finer
grained locking on LDAP connections instead of module-wide locking
allowing different threads with different LDAPObject instances to
run without blocking each other.
More information about the python-ldap