defect

[Michael Ströder]

Leif Hedstrom wrote:
> Well, I don't know if this is the same problem I had with Python LDAP v1.x,
> we haven't tested v2.x yet. But, the result() function in Python LDAP can go
> into a very tight poll loop, with extreme effects if the Python process is
> running on the same machines as the LDAP server. Python will almost
> completely starve slapd for any CPU time ...
> 
> Adding a short sleep() in the polling loop of ldapobject.result() helps, a
> lot.

Yes, you're right. A time.sleep(0.0001) right before the inner 
result() call makes python-ldap hand over the CPU to the OS.

>     while all:
>       while ldap_result[0] is None:
>         if (timeout>=0) and (time.time()-start_time>timeout):
>           self._ldap_call(self._l.abandon,msgid)
>           raise _ldap.TIMELIMIT_EXCEEDED(
>             "LDAP time limit (%d secs) exceeded." % (timeout)
>           )
>         ldap_result = self._ldap_call(self._l.result,msgid,0,0)
>         if ldap_result[0] is None:
>           time.sleep(.01)
>       if ldap_result[1] is None:
>         break

It's possible to make it somewhat simpler since we have a first 
result() call before the while loops.

     while all:
       while ldap_result[0] is None:
         if (timeout>=0) and (time.time()-start_time>timeout):
           self._ldap_call(self._l.abandon,msgid)
           raise _ldap.TIMELIMIT_EXCEEDED(
             "LDAP time limit (%d secs) exceeded." % (timeout)
           )
         time.sleep(0.0001)
         ldap_result = self._ldap_call(self._l.result,msgid,0,0)
       if ldap_result[1] is None:
         break
       all_results.extend(ldap_result[1])
       ldap_result = None,None
     return all_results

> Alternatively, adding
> an (arbitrarily) long timeout in the call to ldap_result() also accomplishes
> the same thing, like:
> 
>      ldap_result = self._ldap_call(self._l.result,msgid,0,15 * 60)

Which is a bad idea in a multi-threaded environment.

> I haven't dug deep into this problem yet, to figure out if this is an
> OpenLDAP library problem, or a Python LDAP problem.

The main problem here is that the OpenLDAP libs are not 
thread-safe. Therefore a module-wide lock is needed to serialize 
all calls into OpenLDAP libs. But just wrapping ldap_result() with 
locking around it would lead to blocking threads if one thread is 
waiting for large search results. That's why I implemented 
LDAPObject.result() in Python like it is today. Unfortunately we 
cannot deal with waiting for data at the socket level which leads 
to higher-level polling loop.

Everybody is encouraged to try the time.sleep(0.0001) hack and 
look how it "feels" now. BTW: it makes the simple benchmark I 
posted yesterday slower. It always depends what you wanna 
optimize... ;-)

I suspect this still might not fully explain the 
30s-vs.-immediate-results reports. There may be more issues with 
other effects like Mauro described.

I guess the best solution would be if somebody with some spare 
cycles digs into OpenLDAP's libldap_r to check if it's ready for 
use with python-ldap. This would make it possible to do a finer 
grained locking on LDAP connections instead of module-wide locking 
allowing different threads with different LDAPObject instances to 
run without blocking each other.

Ciao, Michael.