APIs for high-bandwidth large I/O?
Hi, I am currently looking into ways to optimize large data transfers for a distributed computing framework (https://github.com/dask/distributed/). We are using Tornado but the question is more general, as it turns out that certain kinds of API are an impediment to such optimizations. To put things short, there are a couple benchmarks discussed here: https://github.com/tornadoweb/tornado/issues/2147#issuecomment-337187960 - for Tornado, this benchmark: https://gist.github.com/pitrou/0f772867008d861c4aa2d2d7b846bbf0 - for asyncio, this benchmark: https://gist.github.com/pitrou/719e73c1df51e817d618186833a6e2cc Both implement a trivial form of framing using the "preferred" APIs of each framework (IOStream for Tornado, Protocol for asyncio), and then benchmark it over 100 MB frames using a simple echo client/server. The results (on Python 3.6) are interesting: - vanilla asyncio achieves 350 MB/s - vanilla Tornado achieves 400 MB/s - asyncio + uvloop achieves 600 MB/s - an optimized Tornado IOStream with a more sophisticated buffering logic (https://github.com/tornadoweb/tornado/pull/2166) achieves 700 MB/s The latter result is especially interesting. uvloop uses hand-crafted Cython code + the C libuv library, still, a pure Python version of Tornado does better thanks to an improved buffering logic in the streaming layer. Even the Tornado result is not ideal. When profiling, we see that 50% of the runtime is actual IO calls (socket.send and socket.recv), but the rest is still overhead. Especially, buffering on the read side still has costly memory copies (b''.join calls take 22% of the time!). For a framed layer, you shouldn't need so many copies. Once you've read the frame length, you can allocate the frame upfront and read into it. It is at odds, however, with the API exposed by asyncio's Protocol: data_received() gives you a new bytes object as soon as data arrives. It's already too late: a spurious memory copy will have to occur. Tornado's IOStream is less constrained, but it supports too many read schemes (including several types of callbacks). So I crafted a limited version of IOStream (*) that supports little functionality, but is able to use socket.recv_into() when asked for a given number of bytes. When benchmarked, this version achieves 950 MB/s. This is still without C code! (*) see https://github.com/tornadoweb/tornado/compare/master...pitrou:stream_readint... When profiling that limited version of IOStream, we see that 68% of the runtime is actual IO calls (socket.send and socket.recv_into). Still, 21% of the total runtime is spent allocating a 100 MB buffer for each frame! That's 70% of the non-IO overhead! Whether or not there are smart ways to reuse that writable buffer depends on how the application intends to use data: does it throw it away before the next read or not? It doesn't sound easily doable in the general case. So I'm wondering which kind of APIs async libraries could expose to make those use cases faster. I know curio and trio have socket objects which would probably fit the bill. I don't know if there are higher-level concepts that may be as adequate for achieving the highest performance. Also, since asyncio is the de facto standard now, I wonder if asyncio might grow such a new API. That may be troublesome: asyncio already has Protocols and Streams, and people often complain about its extensive API surface that's difficult for beginners :-) Addendum: asyncio streams ------------------------- I didn't think asyncio streams would be a good solution, but I still wrote a benchmark variant for them out of curiosity, and it turns out I was right. The results: - vanilla asyncio streams achieve 300 MB/s - asyncio + uvloop streams achieve 550 MB/s The benchmark script is at https://gist.github.com/pitrou/202221ca9c9c74c0b48373ac89e15fd7 Regards Antoine.
I adapted this benchmark to Curio using streams and Curio's support for readinto(). Code is at https://gist.github.com/dabeaz/999dc7d08ddd2c0dea790de67948e756 Support for readinto() is somewhat recent in Curio so for testing, you will need the latest version from Github (https://github.com/dabeaz/curio). However, here are the results I got on my machine: - vanilla asyncio archieves 145 MB/s - asyncio + uvloop achieves 340 MB/s - Curio achieves 550 MB/s Asyncio tests were run using: https://gist.github.com/pitrou/719e73c1df51e817d618186833a6e2cc Cheers, Dave
On Oct 18, 2017, at 1:04 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Hi,
I am currently looking into ways to optimize large data transfers for a distributed computing framework (https://github.com/dask/distributed/). We are using Tornado but the question is more general, as it turns out that certain kinds of API are an impediment to such optimizations.
To put things short, there are a couple benchmarks discussed here: https://github.com/tornadoweb/tornado/issues/2147#issuecomment-337187960
- for Tornado, this benchmark: https://gist.github.com/pitrou/0f772867008d861c4aa2d2d7b846bbf0 - for asyncio, this benchmark: https://gist.github.com/pitrou/719e73c1df51e817d618186833a6e2cc
Both implement a trivial form of framing using the "preferred" APIs of each framework (IOStream for Tornado, Protocol for asyncio), and then benchmark it over 100 MB frames using a simple echo client/server.
The results (on Python 3.6) are interesting: - vanilla asyncio achieves 350 MB/s - vanilla Tornado achieves 400 MB/s - asyncio + uvloop achieves 600 MB/s - an optimized Tornado IOStream with a more sophisticated buffering logic (https://github.com/tornadoweb/tornado/pull/2166) achieves 700 MB/s
The latter result is especially interesting. uvloop uses hand-crafted Cython code + the C libuv library, still, a pure Python version of Tornado does better thanks to an improved buffering logic in the streaming layer.
Even the Tornado result is not ideal. When profiling, we see that 50% of the runtime is actual IO calls (socket.send and socket.recv), but the rest is still overhead. Especially, buffering on the read side still has costly memory copies (b''.join calls take 22% of the time!).
For a framed layer, you shouldn't need so many copies. Once you've read the frame length, you can allocate the frame upfront and read into it. It is at odds, however, with the API exposed by asyncio's Protocol: data_received() gives you a new bytes object as soon as data arrives. It's already too late: a spurious memory copy will have to occur.
Tornado's IOStream is less constrained, but it supports too many read schemes (including several types of callbacks). So I crafted a limited version of IOStream (*) that supports little functionality, but is able to use socket.recv_into() when asked for a given number of bytes. When benchmarked, this version achieves 950 MB/s. This is still without C code!
(*) see https://github.com/tornadoweb/tornado/compare/master...pitrou:stream_readint...
When profiling that limited version of IOStream, we see that 68% of the runtime is actual IO calls (socket.send and socket.recv_into). Still, 21% of the total runtime is spent allocating a 100 MB buffer for each frame! That's 70% of the non-IO overhead! Whether or not there are smart ways to reuse that writable buffer depends on how the application intends to use data: does it throw it away before the next read or not? It doesn't sound easily doable in the general case.
So I'm wondering which kind of APIs async libraries could expose to make those use cases faster. I know curio and trio have socket objects which would probably fit the bill. I don't know if there are higher-level concepts that may be as adequate for achieving the highest performance.
Also, since asyncio is the de facto standard now, I wonder if asyncio might grow such a new API. That may be troublesome: asyncio already has Protocols and Streams, and people often complain about its extensive API surface that's difficult for beginners :-)
Addendum: asyncio streams -------------------------
I didn't think asyncio streams would be a good solution, but I still wrote a benchmark variant for them out of curiosity, and it turns out I was right. The results: - vanilla asyncio streams achieve 300 MB/s - asyncio + uvloop streams achieve 550 MB/s
The benchmark script is at https://gist.github.com/pitrou/202221ca9c9c74c0b48373ac89e15fd7
Regards
Antoine.
_______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
On Fri, 20 Oct 2017 14:31:22 -0500 David Beazley <dave@dabeaz.com> wrote:
I adapted this benchmark to Curio using streams and Curio's support for readinto(). Code is at https://gist.github.com/dabeaz/999dc7d08ddd2c0dea790de67948e756 Support for readinto() is somewhat recent in Curio so for testing, you will need the latest version from Github (https://github.com/dabeaz/curio). However, here are the results I got on my machine:
- vanilla asyncio archieves 145 MB/s - asyncio + uvloop achieves 340 MB/s - Curio achieves 550 MB/s
Thank you Dave! I ran it on my machine and get roughly 910 MB/s, i.e. more or less the same speed as a tweaked Tornado using readinto. I also wrote a variant of the benchmark using socketserver and plain sockets. Its gets 1150 MB/s, and most time seems spent in the kernel, so I'm not quite sure if it's possible to improve over that: https://gist.github.com/pitrou/3ac31e82b4461cbc9b4eee151a47bfee (note that running the server in a separate process doesn't improve things; neither does using writev() or sendmsg() to send the two buffers at once) Regards Antoine.
Asyncio tests were run using: https://gist.github.com/pitrou/719e73c1df51e817d618186833a6e2cc
Cheers, Dave
On Oct 18, 2017, at 1:04 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Hi,
I am currently looking into ways to optimize large data transfers for a distributed computing framework (https://github.com/dask/distributed/). We are using Tornado but the question is more general, as it turns out that certain kinds of API are an impediment to such optimizations.
To put things short, there are a couple benchmarks discussed here: https://github.com/tornadoweb/tornado/issues/2147#issuecomment-337187960
- for Tornado, this benchmark: https://gist.github.com/pitrou/0f772867008d861c4aa2d2d7b846bbf0 - for asyncio, this benchmark: https://gist.github.com/pitrou/719e73c1df51e817d618186833a6e2cc
Both implement a trivial form of framing using the "preferred" APIs of each framework (IOStream for Tornado, Protocol for asyncio), and then benchmark it over 100 MB frames using a simple echo client/server.
The results (on Python 3.6) are interesting: - vanilla asyncio achieves 350 MB/s - vanilla Tornado achieves 400 MB/s - asyncio + uvloop achieves 600 MB/s - an optimized Tornado IOStream with a more sophisticated buffering logic (https://github.com/tornadoweb/tornado/pull/2166) achieves 700 MB/s
The latter result is especially interesting. uvloop uses hand-crafted Cython code + the C libuv library, still, a pure Python version of Tornado does better thanks to an improved buffering logic in the streaming layer.
Even the Tornado result is not ideal. When profiling, we see that 50% of the runtime is actual IO calls (socket.send and socket.recv), but the rest is still overhead. Especially, buffering on the read side still has costly memory copies (b''.join calls take 22% of the time!).
For a framed layer, you shouldn't need so many copies. Once you've read the frame length, you can allocate the frame upfront and read into it. It is at odds, however, with the API exposed by asyncio's Protocol: data_received() gives you a new bytes object as soon as data arrives. It's already too late: a spurious memory copy will have to occur.
Tornado's IOStream is less constrained, but it supports too many read schemes (including several types of callbacks). So I crafted a limited version of IOStream (*) that supports little functionality, but is able to use socket.recv_into() when asked for a given number of bytes. When benchmarked, this version achieves 950 MB/s. This is still without C code!
(*) see https://github.com/tornadoweb/tornado/compare/master...pitrou:stream_readint...
When profiling that limited version of IOStream, we see that 68% of the runtime is actual IO calls (socket.send and socket.recv_into). Still, 21% of the total runtime is spent allocating a 100 MB buffer for each frame! That's 70% of the non-IO overhead! Whether or not there are smart ways to reuse that writable buffer depends on how the application intends to use data: does it throw it away before the next read or not? It doesn't sound easily doable in the general case.
So I'm wondering which kind of APIs async libraries could expose to make those use cases faster. I know curio and trio have socket objects which would probably fit the bill. I don't know if there are higher-level concepts that may be as adequate for achieving the highest performance.
Also, since asyncio is the de facto standard now, I wonder if asyncio might grow such a new API. That may be troublesome: asyncio already has Protocols and Streams, and people often complain about its extensive API surface that's difficult for beginners :-)
Addendum: asyncio streams -------------------------
I didn't think asyncio streams would be a good solution, but I still wrote a benchmark variant for them out of curiosity, and it turns out I was right. The results: - vanilla asyncio streams achieve 300 MB/s - asyncio + uvloop streams achieve 550 MB/s
The benchmark script is at https://gist.github.com/pitrou/202221ca9c9c74c0b48373ac89e15fd7
Regards
Antoine.
_______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
_______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
Hi Antoine, First, thanks a lot to share with us the results of your research, it's interesting, even if, for me, I'm not the first concerned with that: Most of our network data we handle with AsyncIO have enough space with a simple UDP packet ;-) Anyway, I have questions about your suggestions, see below: 2017-10-18 20:04 GMT+02:00 Antoine Pitrou <solipsis@pitrou.net>:
Also, since asyncio is the de facto standard now, I wonder if asyncio might grow such a new API. That may be troublesome: asyncio already has Protocols and Streams, and people often complain about its extensive API surface that's difficult for beginners :-)
From my point of view, I like a lot to code TCP servers with the Streams API of AsyncIO. When I see your piece of code for Tornado: https://gist.github.com/pitrou/0f772867008d861c4aa2d2d7b846bbf0 It looks very similar of the Streams API.
To my understanding, some AsyncIO libraries, like aiohttp, don't use Streams API of AsyncIO and implement a specific implementation, especially to have a full control on the buffers: based on the information provided inside the protocol, you are able to know if a small payload or a big payload will arrive on the wire, like Content-Length header with HTTP. My question is: Do you think it's possible to have a simple API to fit at the same time small payloads and big payloads, without impacts on efficiency for all use cases ? Or it's too much integrated with the protocol implementation itself to converge until a simple solution for all use cases ? Maybe before to think about an AsyncIO improvement, it might be a new library at the first step ? To let decide people to use this new buffering algorithm ? Thanks for your responses.
Hi Ludovic, Le 22/10/2017 à 18:51, Ludovic Gasc a écrit :
To my understanding, some AsyncIO libraries, like aiohttp, don't use Streams API of AsyncIO and implement a specific implementation, especially to have a full control on the buffers: based on the information provided inside the protocol, you are able to know if a small payload or a big payload will arrive on the wire, like Content-Length header with HTTP.
My question is: Do you think it's possible to have a simple API to fit at the same time small payloads and big payloads, without impacts on efficiency for all use cases ?
On the write side: I think it's always possible, as my PR for Tornado shows (*). You "just" have to implement a smart buffer management scheme. (*) https://github.com/tornadoweb/tornado/pull/2169 On the read side: you need a readinto()-like API for large buffers. Small buffers can still use a read()-like API for convenience. That means a bit of complication to switch from one mode to the other internally, but it looks doable. On the topic of asyncio, however, asyncio Streams are currently layered over the Transport / Protocol abstraction, and the data_received() paradigm means data is *already* copied at least once when read from the socket to a bytes object whose length isn't controlled by the application, so it's a lot battle unless Streams are reimplemented differently. Regards Antoine.
Hi Antoine, Thanks for posting this, and sorry for the delayed reply! I've known about a possibility to optimize asyncio Protocols for a while. I noticed that `Protocol.data_received()` requires making one extra copy of the received data when I was working on the initial version of uvloop. Back then my main priority was to make uvloop fully compatible with asyncio, so I wasn't really thinking about improving asyncio design. Let me explain the current flaw of `Protocol.data_received()` so that other people on the list can catch up with the discussion: 1. Currently, when a Transport is reading data, it uses `sock.recv()` call, which returns a `bytes` object, which is then pushed to `Protocol.data_received()`. Every time `sock.recv()` is called, a new bytes object is allocated. 2. Typically, protocols need to accumulate bytes objects they receive until they have enough buffered data to be parsed. Usually a `deque` is used for that, less optimized code just concatenates all bytes objects into one. 3. When enough data is gathered and a protocol message can be parsed out of it, usually there's a need to concatenate a few buffers from the `deque` or get a slice of the concatenated buffer. At this point, we've copied the received data two times. I propose to add another Protocol base class to asyncio: BufferedProtocol. It won't have the 'data_received()' method, instead it will have 'get_buffer()' and 'buffer_updated(nbytes)' methods: class asyncio.BufferedProtocol: def get_buffer(self) -> memoryview: pass def buffer_updated(self, nbytes: int): pass When the protocol's transport is ready to receive data, it will call `protocol.get_buffer()`. The latter must return an object that implements the buffer protocol. The transport will request a writable buffer over the returned object and receive data *into* that buffer. When the `sock.recv_into(buffer)` call is done, `protocol.buffer_updated(nbytes)` method will be called. The number of bytes received into the buffer will be passed as a first argument. I've implemented the proposed design in uvloop (branch 'get_buffer', [1]) and adjusted your benchmark [2] to use it. Here are benchmark results from my machine (macOS): vanilla asyncio: 120-135 Mb/s uvloop: 320-330 Mb/s uvloop/get_buffer: 600-650 Mb/s. The benchmark is quite unstable, but it's clear that Protocol.get_buffer() allows to implement framing way more efficiently. I'm also working on porting asyncpg library to use get_buffer(), as it has a fairly good benchmark suite. So far I'm seeing 5-15% speed boost on all benchmarks. What's more important is that get_buffer() makes asyncpg buffer implementation simpler! I'm quite happy with these results and Ipropose to implement the get_buffer() API (or its equivalent) in Python 3.7. I've opened an issue [3] to discuss the implementation details. [1] https://github.com/MagicStack/uvloop/tree/get_buffer [2] https://gist.github.com/1st1/1c606e5b83ef0e9c41faf21564d75ad7 Thanks, Yury On Oct 18, 2017, 2:31 PM -0400, Antoine Pitrou <solipsis@pitrou.net>, wrote:
Hi,
I am currently looking into ways to optimize large data transfers for a distributed computing framework (https://github.com/dask/distributed/). We are using Tornado but the question is more general, as it turns out that certain kinds of API are an impediment to such optimizations.
To put things short, there are a couple benchmarks discussed here: https://github.com/tornadoweb/tornado/issues/2147#issuecomment-337187960
- for Tornado, this benchmark: https://gist.github.com/pitrou/0f772867008d861c4aa2d2d7b846bbf0 - for asyncio, this benchmark: https://gist.github.com/pitrou/719e73c1df51e817d618186833a6e2cc
Both implement a trivial form of framing using the "preferred" APIs of each framework (IOStream for Tornado, Protocol for asyncio), and then benchmark it over 100 MB frames using a simple echo client/server.
The results (on Python 3.6) are interesting: - vanilla asyncio achieves 350 MB/s - vanilla Tornado achieves 400 MB/s - asyncio + uvloop achieves 600 MB/s - an optimized Tornado IOStream with a more sophisticated buffering logic (https://github.com/tornadoweb/tornado/pull/2166) achieves 700 MB/s
The latter result is especially interesting. uvloop uses hand-crafted Cython code + the C libuv library, still, a pure Python version of Tornado does better thanks to an improved buffering logic in the streaming layer.
Even the Tornado result is not ideal. When profiling, we see that 50% of the runtime is actual IO calls (socket.send and socket.recv), but the rest is still overhead. Especially, buffering on the read side still has costly memory copies (b''.join calls take 22% of the time!).
For a framed layer, you shouldn't need so many copies. Once you've read the frame length, you can allocate the frame upfront and read into it. It is at odds, however, with the API exposed by asyncio's Protocol: data_received() gives you a new bytes object as soon as data arrives. It's already too late: a spurious memory copy will have to occur.
Tornado's IOStream is less constrained, but it supports too many read schemes (including several types of callbacks). So I crafted a limited version of IOStream (*) that supports little functionality, but is able to use socket.recv_into() when asked for a given number of bytes. When benchmarked, this version achieves 950 MB/s. This is still without C code!
(*) see https://github.com/tornadoweb/tornado/compare/master...pitrou:stream_readint...
When profiling that limited version of IOStream, we see that 68% of the runtime is actual IO calls (socket.send and socket.recv_into). Still, 21% of the total runtime is spent allocating a 100 MB buffer for each frame! That's 70% of the non-IO overhead! Whether or not there are smart ways to reuse that writable buffer depends on how the application intends to use data: does it throw it away before the next read or not? It doesn't sound easily doable in the general case.
So I'm wondering which kind of APIs async libraries could expose to make those use cases faster. I know curio and trio have socket objects which would probably fit the bill. I don't know if there are higher-level concepts that may be as adequate for achieving the highest performance.
Also, since asyncio is the de facto standard now, I wonder if asyncio might grow such a new API. That may be troublesome: asyncio already has Protocols and Streams, and people often complain about its extensive API surface that's difficult for beginners :-)
Addendum: asyncio streams -------------------------
I didn't think asyncio streams would be a good solution, but I still wrote a benchmark variant for them out of curiosity, and it turns out I was right. The results: - vanilla asyncio streams achieve 300 MB/s - asyncio + uvloop streams achieve 550 MB/s
The benchmark script is at https://gist.github.com/pitrou/202221ca9c9c74c0b48373ac89e15fd7
Regards
Antoine.
_______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
On Dec 7, 2017, 9:03 PM -0500, Yury Selivanov <yselivanov@gmail.com>, wrote: [..]
I'm quite happy with these results and Ipropose to implement the get_buffer() API (or its equivalent) in Python 3.7. I've opened an issue [3] to discuss the implementation details.
Issue: https://bugs.python.org/issue32251 Yury
On Dec 7, 2017, 9:03 PM -0500, Yury Selivanov <yselivanov@gmail.com>, wrote:
I'm quite happy with these results and Ipropose to implement the get_buffer() API (or its equivalent) in Python 3.7. I've opened an issue [3] to discuss the implementation details.
I apologize for the noise, my email client is acting weird. Here's the url of the CPython issue I created to discuss the proposed design: https://bugs.python.org/issue32251. Yury
participants (5)
-
Antoine Pitrou -
Antoine Pitrou -
David Beazley -
Ludovic Gasc -
Yury Selivanov