[Python-Dev] Pythonic concurrency - cooperative MT

[Christopher Armstrong]

On 10/3/05, Martin Blais <blais at furius.ca> wrote:
> On 10/1/05, Antoine <solipsis at pitrou.net> wrote:
> >
> > > like this with their "deferred objects", no?  I figure they would
> > > need to do something like this too.  I will have to check.)
> >
> > A Deferred object is just the abstraction of a callback - or, rather, two
> > callbacks: one for success and one for failure. Twisted is architected
> > around an event loop, which calls your code back when a registered event
> > happens (for example when an operation is finished, or when some data
> > arrives on the wire). Compared to generators, it is a different way of
> > expressing cooperative multi-threading.
>
> So, the question is, in Twisted, if I want to defer on an operation
> that is going to block, say I'm making a call to run a database query
> that I'm expecting will take much time, and want to yield ("defer")
> for other events to be processed while the query is executed, how do I
> do that?  As far as I remember the Twisted docs I read a long time ago
> did not provide a solution for that.

Deferreds don't make blocking code non-blocking; they're just a way to
make it nicer to write non-blocking code. There are utilities in
Twisted for wrapping a blocking function call in a thread and having
the result returned in a Deferred, though (see deferToThread). There
is also a lightweight and complete wrapper for DB-API2 database
modules in twisted.enterprise.adbapi, which does the threading
interaction for you.

So, since this then exposes a non-blocking API, you can do stuff like

d = pool.runQuery('SELECT User_ID FROM Users')
d.addCallback(gotDBData)
d2 = ldapfoo.getUser('bob')
d2.addCallback(gotLDAPData)

And both the database call and the ldap request will be worked on concurrently.

--
  Twisted   |  Christopher Armstrong: International Man of Twistery
   Radix    |    -- http://radix.twistedmatrix.com
            |  Release Manager, Twisted Project
  \\\V///   |    -- http://twistedmatrix.com
   |o O|    |
w----v----w-+