[Python-ideas] The async API of the future: Reactors

[Guido van Rossum]

On Sun, Oct 14, 2012 at 4:08 PM, Rene Nejsum <rene at stranden.com> wrote:
>
> On Oct 15, 2012, at 12:05 AM, Guido van Rossum <guido at python.org> wrote:
[...]
>> That sounds like implicit futures (to use the Wikipedia article's
>> terminology). I'm not a big fan of that. In fact, I'm proposing an API
>> where all task switching is explicit, using the yield keyword (or
>> yield from), and accessing the value of a future is also explicit in
>> such a system.
>
> You are right, it's implicit. An I think I understand your concern, how
> much should be hidden/implicit and how much should be left to the
> programmer. IMHO Python is such an excellent tool, mainly
> because it hides a lot of details. Things like Memory management, GC,
> threads and concurrency should be (and - I believe - can be hidden for
> the developer.

I don't think you can hide threads or concurrency. You can offer
different APIs to work with them that have different advantages and
disadvantages, but I don't think you can *hide* them any more than you
can hide language constructs like classes or sequences.

>> I still think you're focused more on concurrent CPU activity than
>> async I/O. These are quire different fields, even though they often
>> use similar terminology (like future, task/thread/process,
>> concurrent/parallel, spawn/join, queue). I think the keyword that most
>> distinguishes them is "event". If you hear people talk about events
>> they are probably multiplexing I/O, not CPU activities.
>
> Yes and No. My field of concurrency and IO is process control, like
> controlling high speed sorting machines with a lot of IO from 24V inputs,
> scanners, scales, OCR, serial ports, etc. So for me it's a combination of concurrent IO,
> state and parallelism (concurrent CPU). when you have an async (I/O) event,
> you need some kind of concurrency to handle it at the next level.
> It is difficult to do concurrent CPU activity without events, even if
> they are only signal events on a semaphore.

Can you do it with threads? Because if threads serve your purpose,
they are probably easier to use than the async API we're considering
here, especially given your desire to hide unnecessary details. The
async APIs under consideration (Twisted, Tornado, coroutines) all
intentionally makes task switching explicit. You may also consider
greenlets/gevent, which is a compromise that makes task-switching
semi-explicit -- only certain calls cause task switches, but those
calls may be hidden inside other calls (or even overloaded operations
like __getattr__).

> One difference from ex. web servers is that we at design time, knows
> exactly who many tasks we need and what the maximum load is going
> to be. Typical between 50 to 100 tasks/threads sending messages to
> each other.

That does sound like threads are just fine for you. Of course you may
have to craft your own synchronization primitives out of the
lower-level locks and queues offered by the stdlib...

-- 
--Guido van Rossum (python.org/~guido)