[Web-SIG] WSGI 2.0 Round 2: requirements and call for interest

[Amber "Hawkie" Brown]

> On 6 Jan 2016, at 20:34, Andrew Svetlov <andrew.svetlov at gmail.com> wrote:
> 
> When we are talking about "async WSGI" it's not clear for me what
> concrete async implementation we are discussing.
> In Python world there are at least 3 major async approaches (twisted,
> tornado, asyncio) and many minor libraries.

Any, and all. There is no reason that such an approach could not work for all three, and any other library that uses an event loop. And I'm not going to be playing Twisted favouritism here (I'm a core contributor and release manager, and one of the main forces behind Twisted's Python 3 porting), because the ideal case is each working using their own APIs, on top of any three, via shared event loops and better interop.

> Let's assume we are talking about asyncio way -- it's standardized by
> PEP, Guido promotes it and tornado supports asyncio-based code as
> well.
> 
> In this case "async WSGI" cannot be implemented in Python 2 ---
> asyncio itself is Python 3.3+ only.
> trollius is not "asyncio compatible library for python 2" -- it's a
> port with slightly different API. Supporting both asyncio and trollius
> in a non-trivial third-party library makes a mess.
> That's why we decided don't make python2 compatible aiohttp implementation.

Async WSGI can be perfectly implemented in both Tornado and Twisted, and it is possible to write 2/3 compatible (if not fully idiomatic) asyncio+trollius. Autobahn is one such library that is compatible with Twisted, Trollius, and asyncio, in one codebase.

> Having both synchronous and asyncio API in the same WSGI standard is
> also impractical.
> There are several tries to make WSGI-based frameworks asynchronous.
> The only real successful approach is combination of gunicorn,
> gevent/eventlet and total sockets monkeypatching. I don't want to
> encourage the way -- but  it don't need WSGI protocol changing.
> Monkeypatching just doing own dirty work.

I believe it is possible, and I think it will work out quite well - synchronous code can call functions with results as easily as asynchronous code.

> As aiohttp author I can see several approaches to make adapters for
> running flask, pyramid, django in asyncio way. The most of them uses
> aiohttp as a tool for bringing asynchronous into well-known
> synchronous framework.
> 
> I can state it just doesn't work well. The result is too fragile: yes,
> it is asynchronous but has many subtle limitations which may break a
> program easy.
> It's impossible to write web framework which is synchronous and
> asynchronous at the same time.
> asyncio poisons any code that it touches, everything tends to became
> asynchronous too.

My personal experiments in this area have lead to a fully asynchronous Django core, with semi-async ORM -- I go into it in my Django Under The Hood talk (available at https://opbeat.com/events/duth/ ). And yes, everything needs to be asynchronous, or ran in a thread. Cooperative multitasking systems only work when everything cooperates.

> aiohttp is a quite successful asyncio library, it doesn't need WSGI
> standard at all and works with raw sockets.
> The same is true for tornado and twisted.
> 
> I very doubt aiohttp can get benefits from new WSGI.
> Please keep it simple and small.
> Async web developers need totally different standard not compatible
> with old good WSGI (if they need a standard at all at current stage).

I don't think WSGI as it is now with added async magic is what I'd propose, since it won't work.

What we do need in the async world is a standard API that means that "parsing HTTP", and "doing web logic" are not intermingled. There is currently no way of running a Twisted Web server on top of asyncio, even though it is entirely technically possible -- if aiohttp were to gain this theoretical standard support (which I'm nicknaming "Albany"), then it would mean a Twisted Web service could run on top of aiohttp, or an asyncio-using service could run on top of Twisted. The things for this to become a reality are:

- Twisted needs to gain an asyncio reactor in core, and asyncio needs a way of piggy backing on Twisted's reactors
- Such a standard calling interface needs to be created.

Secondly, there is the issue of running async code in, say, a nginx worker -- which requires something like crochet, which is more or less a well-working hack (which runs a new Twisted reactor in a thread). For long-living Twisted, asyncio, or Tornado services to interact with standalone web servers like nginx or Apache, some sort of gateway protocol needs to exist, that means that you can run an asyncio-API-using service, in a Twisted container, which gets web requests parsed by nginx.

I am not sure if aiohttp can get any benefits from it, but Twisted could get some fantastic benefits, as well as any web framework (such a calling convention could mean Tornado could rip out all their ioloop stuff). This would also theoretically work well for Django (as ASGI works along the same lines, from my reading) and for any framework that wants to go from synchronous to asynchronous (which, honestly, is fairly easy, if you have proper separation of concerns).

- Amber Brown

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/web-sig/attachments/20160106/c7d45c57/attachment.sig>