[python-ldap] syncrepl_poll timeout causing extra IndexError exception

[Karl Kornel]

Sorry for the delay!  It’s really hard doing intensive work on this during the week.

On 5/29/17, 1:38 AM, "Michael Ströder" <michael at stroeder.com> wrote:
>
>Karl Kornel wrote:
>> I would like to report a syncrepl-related issue with python-ldap 2.4.38, running in
>> Python 2.7.13, on Mac OS X.  The OpenLDAP library is 2.4.44.  It seems like the
>> `timeout` parameter in syncrepl_poll() is causing an unexpected exception (that is,
>> not a timeout).
>
>Sigh … the inconsistent exception raising really sucks.
>
>Was the ldap.TIMEOUT caused by a syncrepl provider not reachable in time?

In this case, the ldap.TIMEOUT was expected to happen.  It happens because of the way syncrepl works.

One of the modes of syncrepl operation is “refresh and persist” mode.  In this mode, once your client is brought up-to-date, you will have a long-running search operation open.  The behavior of the search operation is performed using ldap.syncrepl.SyncreplConsumer.syncrepl_poll, and it depends on the parameter “timeout”.

If the syncrepl_poll “timeout” parameter is set to a number, and no changes come in before the timeout, then an ldap.TIMEOUT exception will be raised.  This is useful for several cases, including:

* In single-threaded code, it lets us do extra work when the LDAP server has no changes pending.  If we left “timeout” at “None”, then our thread would pause forever (or until the connection dies for some reason).

* If we want to cancel the syncrepl search (using the cancel method), then we need to wait for the poll to timeout, because I don’t think we can call the cancel method on an LDAP connection that has an operation in progress.  So, we need to wait for the operation to return a result (if something on the server updates), or we need to wait for the timeout.

>
>Could you come up with a simple test code showing the problem?
>Ideally as patch to Tests/t_ldapobject.py.

Sure!  Since ldap.syncrepl is a separate module, I have created a new test script that you can add into the Tests directory.

I am attaching two files to this email:

* slapdtest.py.patch is a unified diff of changes that need to be made to slapdtest.py, so that the OpenLDAP server (slapd) is able to generate and store the information needed for syncrepl to work.

* t_syncrepl.py is a test script that can go directly into the Tests directory.  I tried to follow your style, in terms of how to set up the test.  Feel free to make changes!

I think it is safe to apply these changes, for two reasons:

1. The syncprov module is part of a normal OpenLDAP distribution, so It should be safe to load and configure.

2. Although ldap.syncrepl requires pyasn1, the test suite checks for this: If pyasn1 is not present, all of the syncrepl tests are skipped.

NOTE: As I mentioned in the test script, there are still five tests to implement.  At least, five more tests that I can think of so far.  If the test script looks good, then I can add the rest of the tests.  Otherwise, if you want to make changes, then I will wait until you check t_syncrepl.py into CVS, then I will modify that file.

>Could you please test whether the patch attached below solves this?
>(It simply masks the IndexError probably raised by e.args[0] not fully populated for the
>ldap.TIMEOUT exception.)
>

I ran the attached tests on python-syncrepl 2.4.39, and unfortunately the timeout test still failed.  But now, you should be able to run the tests yourself! 

~ Karl

-------------- next part --------------
A non-text attachment was scrubbed...
Name: slapdtest.py.patch
Type: application/octet-stream
Size: 526 bytes
Desc: slapdtest.py.patch
URL: <http://mail.python.org/pipermail/python-ldap/attachments/20170605/b1c73e43/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: t_syncrepl.py
Type: text/x-python-script
Size: 12812 bytes
Desc: t_syncrepl.py
URL: <http://mail.python.org/pipermail/python-ldap/attachments/20170605/b1c73e43/attachment-0001.bin>