[Twisted-Python] newbie -- trying to do async ldap operations
First to set my perspective -- I never heard of Twisted before two days ago. I have an XMLRPC server built with the Medusa framework. I am looking at moving the methods to Twisted in order to take advantage of Deferreds, SSL support, and based on my brief exposure, the overall framework seems cleaner. This is a simplification, but to keep this short I have an XMLRPC method that returns results from an LDAP search. Currently the searches are synchronous but they are fast enough that it has not been a problem. Now I need to expand the search filter options which will (could) result in a slower search. The OpenLDAP API (and python-ldap) supports asynchronus searching. Sounds perfect. A call to ldap.search_s() returns a message ID which is later used in a call to result() to get the results. ldap.result() takes the message ID, and a timeout argument which if zero is in effect a poll. So what I want is for my XMLRPC method to invoke search_s and get the message ID. Then create a defered and provide its callback and the message ID to something that will poll ldap.result() until it returns results (if results are not ready it will return [None, None]) and then pass those to the callback. Here's where I get confused; I think the call to ldap.result() needs to be done by reactor somehow since ldap.result() may need to be called repeatedly before results are ready) but I'm not sure. The whole deferred light bulb has not quite come on for me yet, so I appreciate any guidance. The defer HOWTO examples all use something like reactor.callLater() to represent a "delayed result" which is not really helping me. The XMLRPC example of returning a defered just returns defer.succeed("hello"), again not quite a rich enough example. I thought about adapting the adbapi stuff for LDAP, but the notion is in my head that I should be able to do this without threads since the ldap API supports async operations. Many thanks, Allan -- "If you understand what you're doing, you're not learning anything" - Anon
On 2003.04.22 19:19, Allan Streib wrote:
Eek, polling is terrible. You should check out Tv's Ldaptor for Twisted: http://twistedmatrix.com/users/tv/ldaptor/ It's a natively-Twisted LDAP client library. Also (it's moot if you'll use ldaptor, which I recommend, but..), returning the message ID as the result of the Deferred, and requiring the caller to poll, is a bad idea. You would do the polling yourself (every iteration of the mainloop, with a reactor.callLater(0, pollIt) or somesuch), and once the result is available, trigger the Deferred with it. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://twistedmatrix.com/users/radix.twistd/
On Tue, 22 Apr 2003, Christopher Armstrong wrote:
I saw that and downloaded the latest .tar.gz, but it was not immediately obvious how it worked. There seemed to be no documentation -- did I miss it? I'll peruse through the code a bit more; I'm all for not reinventing the wheel, but on the other hand I'm very happy with the python-ldap wrappers for the OpenLDAP API -- they've worked flawlessly for me so far.
Yes, that's what I was trying to say. I probably did not say it well. Thanks. Allan -- "If you understand what you're doing, you're not learning anything." -- Anonymous
On Tue, Apr 22, 2003 at 10:30:38PM -0500, Allan Streib wrote:
There's no HOWTO-style documentation (because *I* don't usually want to read such, so I don't want to write it), but there current head of the tree actually has decent docstrings. Here's a few links: API Docs: http://tv.debian.net/software/ldaptor/api/ Main site: http://tv.debian.net/software/ldaptor/ See the latest code: http://ldaptor.bkbits.net/ There's also *13* example client apps, as of now. The search utility should be reasonably obvious: http://ldaptor.bkbits.net:8080/ldaptor/anno/bin/ldaptor-search@1.12?nav=index.html|src/|src/bin You also have this great mailing list to ask questions on. -- :(){ :|:&};:
On Tue, Apr 22, 2003 at 06:19:28PM -0500, Allan Streib wrote:
Here's something that might help (note that I am not familiar with the python-ldap API, so I'm just guessing at how it is laidd out): from twisted.internet import reactor, defer class LDAPQueryTracker: def __init__(self): self._ids = {} # Wrapper around python-ldap's search function # returns a Deferred whose callback will be invoked # with the result def search(self, args): id = ldap.search_s(args) self._ids[id] = defer.Deferred() reactor.callLater(0, self.poll) return self._ids[id] # A method to poll any "live" queries, # and to schedule another poll if there are any # remaining outstanding queries def poll(self): for (id, d) in self._ids.items(): r = ldap.result(id, timeout=0): if r is not None: del self._ids[id] d.callback(r) if self._ids: reactor.callLater(0, self.poll) # Two trivial callbacks to handle the result def printResult(result): print 'LDAP result:', result def printFailure(failure): print 'LDAP failed!' failure.printTraceback() # Make a tracker, perform a query, and add some callbacks qt = LDAPQueryTracker() d = qt.search("humm, I have no idea what an ldap query looks like") d.addCallbacks(printResult, printFailure) # Do it. reactor.run() Hope this helps, Jp -- "If you find a neighbor in need, you're responsible for serving that neighbor in need, you're responsible for loving a neighbor just like you'd like to love yourself." -- George W. Bush, Sept. 16, 2002 -- up 33 days, 21:03, 5 users, load average: 2.01, 1.90, 1.90
Thanks, Jp, your example is helpful. I will give it a try. (Note to other readers -- ldap.search_s() is the synchronous (blocking) method. The async method is ldap.search(). I've been using ldap.search_s() up until now and my fingers automatically typed that.) Allan
On Tue, 22 Apr 2003, Jp Calderone wrote:
Here's something that might help (note that I am not familiar with the python-ldap API, so I'm just guessing at how it is laidd out):
Your example works nicely with only minor changes. Here's what I wrote just to verify it works. Many thanks! Allan from twisted.internet import reactor, defer import ldap class LDAPQueryTracker: def __init__(self): self._ids = {} self.ldapCon = ldap.initialize('ldap://localhost/') self.ldapCon.set_option(ldap.OPT_PROTOCOL_VERSION, 3) self.ldapCon.simple_bind_s('cn=Manager,dc=iu,dc=edu', 'xxxxxx') def search(self, searchargs): id = self.ldapCon.search(searchargs['base'], ldap.SCOPE_SUBTREE, searchargs['filter']) self._ids[id] = defer.Deferred() reactor.callLater(0, self.checkResults) return self._ids[id] def checkResults(self): for (id, deferedResult) in self._ids.items(): resultType, resultData = self.ldapCon.result(msgid=id, all=1, timeout=0) if resultType is not None: del self._ids[id] deferedResult.callback(resultData) if self._ids: reactor.callLater(0, self.checkResults) #A simple callback def printResults(results): for result in results: print result def printFailure(failure): print "LDAP Operation failed" failure.printTraceback() qt = LDAPQueryTracker() deferedResult = qt.search({'base': 'ou=people,dc=iu,dc=edu', 'filter': '(uid=astreib)'}) deferedResult.addCallbacks(printResults, printFailure) reactor.run()
On Wed, Apr 23, 2003 at 05:00:52PM -0500, Allan Streib wrote:
This blocks right here; maybe it blocks in the ldap.initialize part, too. And it's error return mechanism seems to be by way of exceptions, so making it async will change the API, too.
Doing the same with Ldaptor (not tested): class Search(ldapclient.LDAPClient): def connectionMade(self): d=self.bind() d.addCallback(self._handle_bind_success) d.addErrback(self.factory.deferred.errback) def _handle_bind_success(self, x): matchedDN, serverSaslCreds = x o=ldapsyntax.LDAPObject(client=self, dn='ou=people,dc=iu,dc=edu') d=o.search(filterText='(uid=astreib)') d.chainDeferred(self.factory.deferred) class SearchFactory(protocol.ClientFactory): protocol = Search def __init__(self, deferred): self.deferred=deferred def clientConnectionFailed(self, connector, reason): self.deferred.errback(reason) d=defer.Deferred() s=SearchFactory(d) d.addCallbacks(printResults, printFailure) d.addBoth(lambda x: reactor.stop()) # service discovery from DNS, no need to know host name of server dn = distinguishedname.DistinguishedName(stringValue='ou=people,dc=iu,dc=edu') c = ldapconnector.LDAPConnector(reactor, dn, s) c.connect() reactor.run() -- :(){ :|:&};:
On 2003.04.22 19:19, Allan Streib wrote:
Eek, polling is terrible. You should check out Tv's Ldaptor for Twisted: http://twistedmatrix.com/users/tv/ldaptor/ It's a natively-Twisted LDAP client library. Also (it's moot if you'll use ldaptor, which I recommend, but..), returning the message ID as the result of the Deferred, and requiring the caller to poll, is a bad idea. You would do the polling yourself (every iteration of the mainloop, with a reactor.callLater(0, pollIt) or somesuch), and once the result is available, trigger the Deferred with it. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://twistedmatrix.com/users/radix.twistd/
On Tue, 22 Apr 2003, Christopher Armstrong wrote:
I saw that and downloaded the latest .tar.gz, but it was not immediately obvious how it worked. There seemed to be no documentation -- did I miss it? I'll peruse through the code a bit more; I'm all for not reinventing the wheel, but on the other hand I'm very happy with the python-ldap wrappers for the OpenLDAP API -- they've worked flawlessly for me so far.
Yes, that's what I was trying to say. I probably did not say it well. Thanks. Allan -- "If you understand what you're doing, you're not learning anything." -- Anonymous
On Tue, Apr 22, 2003 at 10:30:38PM -0500, Allan Streib wrote:
There's no HOWTO-style documentation (because *I* don't usually want to read such, so I don't want to write it), but there current head of the tree actually has decent docstrings. Here's a few links: API Docs: http://tv.debian.net/software/ldaptor/api/ Main site: http://tv.debian.net/software/ldaptor/ See the latest code: http://ldaptor.bkbits.net/ There's also *13* example client apps, as of now. The search utility should be reasonably obvious: http://ldaptor.bkbits.net:8080/ldaptor/anno/bin/ldaptor-search@1.12?nav=index.html|src/|src/bin You also have this great mailing list to ask questions on. -- :(){ :|:&};:
On Tue, Apr 22, 2003 at 06:19:28PM -0500, Allan Streib wrote:
Here's something that might help (note that I am not familiar with the python-ldap API, so I'm just guessing at how it is laidd out): from twisted.internet import reactor, defer class LDAPQueryTracker: def __init__(self): self._ids = {} # Wrapper around python-ldap's search function # returns a Deferred whose callback will be invoked # with the result def search(self, args): id = ldap.search_s(args) self._ids[id] = defer.Deferred() reactor.callLater(0, self.poll) return self._ids[id] # A method to poll any "live" queries, # and to schedule another poll if there are any # remaining outstanding queries def poll(self): for (id, d) in self._ids.items(): r = ldap.result(id, timeout=0): if r is not None: del self._ids[id] d.callback(r) if self._ids: reactor.callLater(0, self.poll) # Two trivial callbacks to handle the result def printResult(result): print 'LDAP result:', result def printFailure(failure): print 'LDAP failed!' failure.printTraceback() # Make a tracker, perform a query, and add some callbacks qt = LDAPQueryTracker() d = qt.search("humm, I have no idea what an ldap query looks like") d.addCallbacks(printResult, printFailure) # Do it. reactor.run() Hope this helps, Jp -- "If you find a neighbor in need, you're responsible for serving that neighbor in need, you're responsible for loving a neighbor just like you'd like to love yourself." -- George W. Bush, Sept. 16, 2002 -- up 33 days, 21:03, 5 users, load average: 2.01, 1.90, 1.90
Thanks, Jp, your example is helpful. I will give it a try. (Note to other readers -- ldap.search_s() is the synchronous (blocking) method. The async method is ldap.search(). I've been using ldap.search_s() up until now and my fingers automatically typed that.) Allan
On Tue, 22 Apr 2003, Jp Calderone wrote:
Here's something that might help (note that I am not familiar with the python-ldap API, so I'm just guessing at how it is laidd out):
Your example works nicely with only minor changes. Here's what I wrote just to verify it works. Many thanks! Allan from twisted.internet import reactor, defer import ldap class LDAPQueryTracker: def __init__(self): self._ids = {} self.ldapCon = ldap.initialize('ldap://localhost/') self.ldapCon.set_option(ldap.OPT_PROTOCOL_VERSION, 3) self.ldapCon.simple_bind_s('cn=Manager,dc=iu,dc=edu', 'xxxxxx') def search(self, searchargs): id = self.ldapCon.search(searchargs['base'], ldap.SCOPE_SUBTREE, searchargs['filter']) self._ids[id] = defer.Deferred() reactor.callLater(0, self.checkResults) return self._ids[id] def checkResults(self): for (id, deferedResult) in self._ids.items(): resultType, resultData = self.ldapCon.result(msgid=id, all=1, timeout=0) if resultType is not None: del self._ids[id] deferedResult.callback(resultData) if self._ids: reactor.callLater(0, self.checkResults) #A simple callback def printResults(results): for result in results: print result def printFailure(failure): print "LDAP Operation failed" failure.printTraceback() qt = LDAPQueryTracker() deferedResult = qt.search({'base': 'ou=people,dc=iu,dc=edu', 'filter': '(uid=astreib)'}) deferedResult.addCallbacks(printResults, printFailure) reactor.run()
On Wed, Apr 23, 2003 at 05:00:52PM -0500, Allan Streib wrote:
This blocks right here; maybe it blocks in the ldap.initialize part, too. And it's error return mechanism seems to be by way of exceptions, so making it async will change the API, too.
Doing the same with Ldaptor (not tested): class Search(ldapclient.LDAPClient): def connectionMade(self): d=self.bind() d.addCallback(self._handle_bind_success) d.addErrback(self.factory.deferred.errback) def _handle_bind_success(self, x): matchedDN, serverSaslCreds = x o=ldapsyntax.LDAPObject(client=self, dn='ou=people,dc=iu,dc=edu') d=o.search(filterText='(uid=astreib)') d.chainDeferred(self.factory.deferred) class SearchFactory(protocol.ClientFactory): protocol = Search def __init__(self, deferred): self.deferred=deferred def clientConnectionFailed(self, connector, reason): self.deferred.errback(reason) d=defer.Deferred() s=SearchFactory(d) d.addCallbacks(printResults, printFailure) d.addBoth(lambda x: reactor.stop()) # service discovery from DNS, no need to know host name of server dn = distinguishedname.DistinguishedName(stringValue='ou=people,dc=iu,dc=edu') c = ldapconnector.LDAPConnector(reactor, dn, s) c.connect() reactor.run() -- :(){ :|:&};:
participants (5)
-
Allan Streib -
Christopher Armstrong -
Jp Calderone -
Tommi Virtanen -
Tommi Virtanen