@cory I didn't want to say "re-implementing requests in asyncio way is impossible". I just want to say: it's hard. Requires massive code rewriting. Moreover, it's pain to support both Python 2 and 3 with asyncio inthe same code base. Really only after dropping python 2 support you are able to adopt requests for asyncio. Hopefully after 2020. Requests library (as well as sqlalchemy) are very crucial scaffolds for python community, I suspect you will be forced to support Python 2 up to the official death of branch at least. On Wed, Aug 3, 2016 at 5:27 PM Cory Benfield <cory@lukasa.co.uk> wrote:
On 2 Aug 2016, at 19:39, Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
Long story short: in async world every IO library should use asyncio. Unfortunately for the most popular libraries (requests, django, flask, sqlalchemy etc.) it's impossible to rewrite the code keeping backward compatibility. Much easier to create a new library than rewrite existing one.
Removing my hyper hat for a moment and replacing it with my Requests hat: we disagree. Some of the discussion has happened on GitHub at [1], but I can summarise it again here.
One of the nice things with an event loop is that you can retrofit synchronous code on top of it. That is, if you have an event loop, you can always turn your evented code (code that returned a Deferred (or Deferred-alike) or a coroutine) into synchronous code by simply running the event loop in the synchronous call. That is, given a method signature like this:
async def get(url): return await do_the_work(url)
It can be turned into this:
def sync_get(url): return some_event_loop.run(get(url))
The reason Requests has not previously done this is that it required us to do one of two things. In the first instance, we could bundle or depend on an event loop (e.g. Twisted’s reactors). That’s annoying: it’s a lot of code that we don’t really care about in order to achieve a task that is fundamentally an implementation detail. The other option is to write backends for all possible event loops *and* a kind of event-loop-alike synchronous approach that can be run as a series of coroutines, but that is actually entirely synchronous under the covers (essentially, ship a coroutine runner that never waits for anything). Both of these were a lot of work for fairly minimal gain.
However, with asyncio’s event loop becoming shipped with the language, and with the other event loop implementations shimming in compatibility layers, libraries like Requests have a perfect escape hatch. Now we only need *two* backends: one for the asyncio event loop, and one synchronous one. And if we’re prepared to depend on Python 3.4+, we only need *one* backend: asyncio, because we can rely on it being present.
This drastic reduction in the space of event loops we need to support suddenly makes it much more viable to consider adjusting the way we do I/O. There’s still a *lot* of work there, and no-one has actually started sitting down to do the hard stuff, but it’s certainly not impossible any longer.
Cory
[1]: https://github.com/kennethreitz/requests/issues/1390#issuecomment-224772923
--
Thanks, Andrew Svetlov