On Sat, Jul 26, 2003 at 08:48:15AM -0500, John Goerzen wrote:
Hello,
I am the author of OfflineIMAP, a multi-threaded multi-account bi-directional synchronization tool (http://quux.org/devel/offlineimap) written in Python.
Ooh, nifty :) [...]
My questions, then, are these:
1. How does Twisted interact with things like threads and connection pooling?
I currently let the user control the maximum number of connections that are open to a given server, and whenever the code needs to access an IMAP server, it grabs a connection from the appropriate pool and uses it (using threading primitives to block if no connection is available.) Can Twisted handle multiple connections to multiple servers doing multiple things at once? Can it do connection pools? If it needs threading to do these things, does it play nice with it?
Twisted uses non-blocking sockets, so it can do an arbitrary amount of connections (both client and server) without any threads. Running multiple clients (or servers) at a time is as easy as calling reactor.connectTCP more than once. [What happened to question 2?]
3. I notice the "Deferred" object and the benefits it provides when a server response is expected to take some time. I assume that some sort of internal threading is taking place here?
No threading. Deferreds aren't magical at all. Deferreds are simply an object a function can return to say "I don't have a result ready yet; as soon as I've got it, I'll let you know." The caller can then register callbacks and errbacks that will be called when the result does become available. How the Deferred eventually gets called is up to the code that originally created it; it might be the result of query sent to the network being answered (e.g. twisted.web.client.getPage), a spawned process terminating (e.g. twisted.internet.utils.getProcessOutput), a function call in another thread completing (e.g. twisted.internet.threads.deferToThread), and so on.
What if the server *request* takes some time -- say, uploading a 2MB e-mail over a dialup link. Are we still OK with handling that in a non-blocking fashion?
Absolutely! Twisted handles large transfers via a consumer/producer interface, though, rather than with Deferreds (which are one-shot results, rather than streaming).
4. The howto "book" alludes to pending improvements on the mail infrastructure. Anything I should be aware of here?
Jp has been doing a lot of great work on this recently -- you should definitely be working with CVS rather than 1.0.6 if you can. I don't know much about the details, though.
5. Is there any sort of unified mail API in Twisted (like JavaMail) that would present me with a single API to both IMAP and Maildir repositories, or is that something I need to do on my own? (I've already done it, so it's no big deal to do that again)
I've no idea -- Jp?
6. OfflineIMAP supports several different user interfaces (two written with Tkinter, 1 using Curses, and three plain console ones.) I would like to reimplement the Tkinter ones with wxPython, and am glad to see that Twisted works well with this.
One concern, though, is that the users can supply a list of UIs to try in the config file: for instance, Tk, Curses, Noninteractive. The system will try each one in turn until it reaches one that works. (For instance, Tk may not work if the user is not in X, and Curses may not work if stdout is not a tty). It looks like this may be problematic with Twisted since the UI selection always seems to be known in advance with the import commands related to the reactor. Any thoughts here?
I suspect this can be dealt with, but I'm not sure off the top of my head how... hopefully someone else can help you here. -Andrew.