[Python-ideas] The async API of the future: Reactors

[Rene Nejsum]

On Oct 14, 2012, at 9:22 PM, Guido van Rossum <guido at python.org> wrote:

> On Sun, Oct 14, 2012 at 10:51 AM, Rene Nejsum <rene at stranden.com> wrote:
>> On the high level (Python) basically what you need is that the queue.get()
>> can handle:
>> 1) Python objects (as today)
>> 2) timeout (as today, maybe in mills instead of seconds)
>> 3) Network (socket input/state change)
>> 4) File desc input/state change
>> 5) Other I/O changes like serial comm, etc.
>> 6) Maybe also yield based coroutine support ?
>> 
>> This requires support from the underlaying
>> OS. A support which is probably not there today ?
>> 
>> As far as I can see, having this one extended queue.get() would nicely enable
>> all high level concurrency issues in Python.
> 
> [...]
> 
>> I believe a "super" queue.get() would solve all use cases.
>> 
>> I have no idea on how difficult it would be to implement in
>> a cross platform manner.
> 
> Hm. I know that a common (and often right!) recommendation for thread
> communication is to use the queue module. But that module is meant to
> work with threads. I think that the correct I/O primitives are more
> likely to come by looking at what Tornado and Twisted have done than
> by trying to "pimp up" the queue module -- it's good for what it does,
> but trying to add all that new functionality to it doesn't sound like
> a good fit.

You are probably right about the queue class. Maybe it should be a new class,
but I still believe I would be an excellent fit for doing concurrent stuff if Python
had a multiplexer message queue, Python is high-level enough to be able to 
hide thread/select/read etc.

A while ago I implemented pyworks (bitbucket.org/raindog/pyworks) which
is a kind of Erlang implementation for Python, making objects concurrent and return
values Futures, without adding much new code. Methods are sent asynchronous, simply
by doing standard obj.method(). obj is a proxy for the real object sending method() as a
message to the real object running in a separate thread. Return value is a Future. So 
you can do

	val = obj.method()
	… continue async with method()
	… and do some other stuff, until:
	print val

which will hang waiting for the Future to complete, if it's not.

It has been used in a couple of projects, making it much easier to do concurrent systems.
But, it would be great if the object/task could wait for more events than queue.get()

br
/Rene

> 
> -- 
> --Guido van Rossum (python.org/~guido)