[Python-Dev] threadsafe patch for asynchat

[Andrew Bennetts]

Donovan Baarda wrote:
> On Wed, 2006-02-08 at 02:33 -0500, Steve Holden wrote:
> > Martin v. Löwis wrote:
> > > Tim Peters wrote:
> [...]
> > > What is the reason that people want to use threads when they can have
> > > poll/select-style message processing? Why does Zope require threads?
> > > IOW, why would anybody *want* a "threadsafe patch for asynchat"?
> > > 
> > In case the processing of events needed to block? If I'm processing web 
> > requests in an async* dispatch loop and a request needs the results of a 
> > (probably lengthy) database query in order to generate its output, how 
> > do I give the dispatcher control again to process the next asynchronous 
> > network event?
> > 
> > The usual answer is "process the request in a thread". That way the 
> > dispatcher can spring to life for each event as quickly as needed.
> 
> I believe that Twisted does pretty much this with it's "deferred" stuff.
> It shoves slow stuff off for processing in a separate thread that
> re-syncs with the event loop when it's finished.

Argh!  No.  Threading is completely orthogonal to Deferreds.

Deferreds are just an abstraction for managing callbacks for an asychronous
operation.  They don't magically invoke threads, or otherwise turn synchronous
code into asynchronous code for you.

This seems to be a depressingly common misconception.  I wish I knew how to stop
it.

They're much simpler than people seem to think.  They're an object a function
returns to say "I don't have a result for you yet, but if you attach callbacks
to this I'll run those when I do."  We've do this because it's much nicer than
having to pass callbacks into functions, particularly when you want to deal with
chains of callbacks and error handling.

There is a single utility function in Twisted called "deferToThread" that will
run a function in a threadpool, and arrange for a Deferred to be fired with the
result (in the event loop thread, of course).  This is just one of many possible
uses for Deferreds, and not an especially common one.

I'm happy to provide pointers to several Twisted docs if anyone is at all
unclear on this.

While they are very useful, I don't think they're an essential part of a minimal
Twisted replacement for asyncore/asynchat -- in fact, they'd work just fine with
asyncore/asynchat, because they do so little.

-Andrew.