[Python-Dev] generic async io (was: microthreading vs. async io)

[dustin at v.igoro.us]

On Thu, Feb 15, 2007 at 04:28:17PM +0100, Joachim K?nig-Baltes wrote:
> No, I'd like to have:
> 
> - An interface for a task to specifiy the events it's interested in, and 
>   waiting for at least one of the events (with a timeout).
> - an interface for creating a task (similar to creating a thread)
> - an interface for a schedular to manage the tasks

I think this discussion would be facilitated by teasing the first
bullet-point from the latter two: the first deals with async IO, while
the latter two deal with cooperative multitasking.

It's easy to write a single package that does both, but it's much harder
to write *two* fairly generic packages with a clean API between them,
given the varied platform support for async IO and the varied syntax and
structures (continuations vs. microthreads, in my terminology) for
multitasking.  Yet I think that division is exactly what's needed.

Since you asked (I'll assume the check for $0.02 is in the mail), I
think a strictly-async-IO library would offer the following:

 - a sleep queue object to which callables can be added
 - wrappers for all/most of the stdlib blocking IO operations which
   add the operation to the list of outstanding operations and return
   a sleep queue object
   - some relatively easy method of extending that for new IO operations
 - a poll() function (for multitasking libraries) and a serve_forever()
   loop (for asyncore-like uses, where all the action is IO-driven)

The mechanisms for accomplishing all of that on the chosen platform
would be an implementation detail, possibly with some intitialization
"hinting" from the application.  The library would also need to expose
its platform-based limitations (can't wait on thd.join(), can only wait
on 64 fd's, etc.) to the application for compatibility-checking
purposes.

Thoughts?

Dustin