[Python-Dev] PEP 492: async/await in Python; version 4

[Wolfgang Langner]

Hi Yury,


On 05.05.2015 20:25, Yury Selivanov wrote:

>>
>> We forget to address the major problems here. How can someone in a
>> "sync" script use this async stuff easy. How can async and sync stuff
>> cooperate and we don't need to rewrite the world for async stuff.
>> How can a normal user access the power of async stuff without rewriting
>> all his code. So he can use a simple asyc request library in his code.
>> How can a normal user learn and use all this in an easy way.
> 
> asyncio and twisted answered these questions ;) The answer is
> that you have to write async implementations.
> 
> gevent has a different answer, but greenlents/stackless is
> something that will never be merged in CPython and other
> implementations.
> 

I think monkeypatching and the gevent way is wrong. And explicit is better
than implicit.

I have to clear this. I meant how can we make this async stuff more accessible
to the average sync user. Sometime even without writing or knowing how
to write coroutines or other async stuff.

Let me explain it with a deeper example (Some of it is related to Python 2 and
twisted):

I had the same problem for a server application using twisted. Provide
easy interfaces to my users most not aware of async stuff.

My solution was to write my own decorator similar to twisted's
@inlineCallbacks. On top of it I added one more level to the decorator
and distinguish if it was called from the main thread (also running the
mainloop) and other threads. This object usable also as decorator
is called "task" and has some utility methods. And returns a "deferred".
(in asyncio this would be a Future)

Resulting in code like:

@task
def myAsyncFunction(webaddress):
  result = yield fetch(webaddress)
  # only to show sleep example
  yield task.sleep(0.1)
  task.Return(result)

Usable in a sync context (extra script thread):

def do():
  content = myAsyncFunction("http://xyz")

or async context:

@task
def ado():
  content = yield myAsyncFunction("http://xyz")


The task decorator has functionality to check if something is
called from the main thread (-> also a task and async)
or it is called from another thread (-> sync or script).

So this async interface is usable from both worlds. If someone
operates async he/she must only know the task decorator and when to yield.
If he/she uses it in sync mode nothing special has to be done.

To allow all this the server starts the async main loop in the main thread
and executes the script in an extra script thread. The user has every time his
own thread, also for rpc stuff. The only way to switch into the main loop
is to decorate a function as @task, every task is a coroutine and executed
in the main thread (also thread of main loop).

Benefit of all this:

- Easy to write a async task it is marked as one and special stuff belongs
  to the task object. (task.Return is needed because we are in Python 2)
- The normal user executes his stuff in his own thread and he/she
  can program in sync mode. No problem it is an extra thread and the main loop
  does not block.
- A network client or other stuff has to be written only once, most time this
  can be a @task in the async world. But this should not care the end user.
  We don't have to implement all twice once for async and once for the sync
  world. -> Less overhead

This is what I mean if I say we must address the bridging problem between the
worlds.
It think it is the wrong way to divide it in async and sync stuff and
duplicate all networking libraries in the sync and async ones.

For me the answer is to write one async netowrk library and use it in both,
a sync script and in an async main loop. With an easy interface and not
forcing the user to know this is an async library I have to do something special.

And in future go one step further and use all this combined with PyParallel
to solve the multiprocessing problem in Python.
(Script in extra thread, mainloop in main thread, executed and managed via
PyParallel avoiding the gil)
But this is only a vision/dream of mine.


>>
>> And for all this we still can't tell them "oh the async stuff solves
>> the multiprocessing problem of Python learn it and switch to version
>> 3.5". It does not and it is only most useful for networking stuff
>> nothing more.
> 
> "networking stuff", and in particular, web, is a huge
> part of current Python usage.  Please don't underestimate
> that.

I do not. But for most they want only use it as a client
and the main concern for most is "I want to get this web page"
and not "I will implement a web client have to do this async and get it".


Regards,

Wolfgang