[python-ldap] LDAP pagination cookie not working across AD server restart

[Ritesh Nadhani]

On Thu, Jan 5, 2017 at 10:56 AM, Michael Ströder <michael at stroeder.com> wrote:
> Ritesh Nadhani wrote:
>> I am writing a script that would poll an AD with 250K user
>> information. It looks something like (from which I took code
>> inspiration):
>>
>> https://gist.github.com/mattfahrner/c228ead9c516fc322d3a
>
> So you're using the Simple Paged Control (see RFC 2696):
>
> https://tools.ietf.org/html/rfc2696
>

Yes.

>> To be resilient of server restarts and my app crashing, I am writing
>> the cookie into a file and reading it from there in the next
>> iteration.
>
> Hmm...
>
>> Everything works and even my script crashes, on restart it picks up
>> the right cookie file from storage and starts from the correct offset
>> (ignoring users that have already been fetched).
>
> I wonder why this works in your case.
> In general I doubt that it's guaranteed to work.

Well, I will be honest and this is not something that I found over the
docs or anything. I just wrote the implementation and during my tests,
it seemed to work so I just assumed. My bad.

>
>> The problem seems to be that, while the operation is going on if I
>> restart the server, I get SERVER_DOWN exception, on which case I just
>> re-setup the connection again after the server is back and continue on
>> the loop. Unfortunately, in this case, using the older cookie gives
>> me:
>>
>> UNAVAILABLE_CRITICAL_EXTENSION: {'info': '00000057: LdapErr:
>> DSID-0C0907B8, comment: Error processing control, data 0, v2580',
>> 'desc': 'Critical extension is unavailable'}
>>
>> ..
>>
>> and I cannot figure out the issue. The cookie way works brilliant if
>> my app and the network IO behaves correctly. If my app crashes, I can
>> restart and reuse the cookie with a new connection object.
>>
>> But if I restart the AD server itself, reconnecting and using the
>> cookie does not work.
>
> In general I would never expect this (both cases) to work since the server can
> throw away any context of your former stale LDAP connection. The first case may
> work with AD but likely does not work with other LDAP servers. AFAICS there is
> no text in RFC 2696 clarifying this.
>
> I wonder why AD returns UNAVAILABLE_CRITICAL_EXTENSION though. But hey, RFC 2696
> uses pretty blurry text like "it SHOULD return the appropriate error".
>

So I guess my question becomes:

If i have to efficiently pull all user information from a directory
service where the network IO can be a bit sketchy, whats the best to
query when I want to ignore already fetched users information?

As far as I can see, there is no way to specifiy ordering, offseting
during the query option.

Or I am just out of luck?


> Ciao, Michael.
>
>



-- 
Ritesh