python-ldap as replication client

[Eric Brunson]

On 03/11/2011 05:40 AM, Michael Ströder wrote:
> Jeroen van Meeuwen (Kolab Systems) wrote:
>> I'm looking to implement LDAP_CONTROL_SYNC(*) capabilities to
>> python-ldap's ldap.controls, and while I do have some experience in
>> several areas, admittedly compared to you I'm probably the most
>> under-qualified programmer to actually do it.
> You're always welcome to send demo code and get it commented here.
>
>> That said, I first wanted to ask whether something like python-ldap
>> becoming a replication client (through server controls) was feasible in
>> your opinion(s).
> No matter which sync protocol you implement it's very likely that you need
> python-LDAP from CVS HEAD (will be python 2.4) since this version contains
> code to extract response controls from intermediate responses. Beware that
> this may still be subject of API changes especially regarding ldap.controls
> and ldap.extop.
>
> Some additional ASN.1 work for encoding/decoding controls is needed too. I'm
> currently using pyasn1.sf.net for that which is outside python-ldap.
>
>> I think RFC 3928[1] is the corresponding standard.
>> Another standard was proposed in RFC 4533[2] but that one bounced in
>> favor of the former.
> Which sync protocol standard suits your needs depends on the LDAP server your
> application is talking to.
>
> If you use the OpenLDAP server the OpenLDAP developers strongly recommend
> syncrepl. There were already some people here implementing syncrepl (RFC 4533)
> based on python-ldap.

I'm currently working on a project that requires me to do a syncrepl 
from python and after much, much reading I'm afraid that the python-ldap 
library does not implement 4533 correctly.

Sync cookies are only retrieved by python-ldap if they are returned in a 
server control, however this is only the case in an 
LDAP_RES_SEARCH_RESULT or an LDAP_RES_SEARCH_ENTRY packets.  The 
protocol passes both deletes and presence records in 
LDAP_RES_INTERMEDIATE packets, which don't get returned to the python 
caller as they don't have LDAP entries in them, and cookies are also 
returned in these intermediate result packets, but not in a server 
control, so those are missed.

To see the proper handling of the syncrepl protocol it's instructional 
to read through the do_syncrep2 found in the file 
servers/slapd/syncrepl.c of the openldap source.  I'm working on moving 
that code over to a new function in the python-ldap module, but I'm not 
sure whether my company is going to allow me to release the code back to 
the project.  If I do it, they probably will.  If we pay someone else to 
do it, possibly not.

In any case, a python-LDAP syncrepl client can be made to work, but if 
there are deletes during a period when the client is not connected I 
believe they will be lost during the catchup phase of the sync.

I'm certainly not an OpenLDAP nor a python-LDAP expert, so if I'm 
mistaken about anything I've said above, please feel free to set me 
straight.  I just thought it would be good to share the caveat as I 
understand it.

Sincerely,
e.

> Personally I'm currently using LDAP persistent search retrieving data from a
> Novell eDirectory server since this is the control this server supports.
>
> Other LDAP servers have other sync controls, e.g. MS AD implemented the
> proprietary DirSync control, etc.
>
> Ciao, Michael.
>
>