[Twisted-Python] reactor for Linux io_uring
In investigating async file I/O I came across this. In a nutshell it's the new epoll() It's marginally more efficient although this is only apparent at very high loads. What's more interesting is that io_uring accepts files as well as network/pipe handles: avoiding the need for threads. Here's a good intro: https://unixism.net/loti/index.html If people think an IoUringReactor is worthwhile I'll open a ticket and make a start. However it will need a reviewer... :-) Ian
I suspect that this may require somewhat less... cognitive surface area than your other contributions :). And hey, we have a vaccine now, which means that maybe things will go back to normal or close enough to it that I'll have enough capacity to get back to it myself :) -g
On Jan 3, 2021, at 8:01 PM, Ian Haywood <ian@haywood.id.au> wrote:
In investigating async file I/O I came across this. In a nutshell it's the new epoll()
It's marginally more efficient although this is only apparent at very high loads. What's more interesting is that io_uring accepts files as well as network/pipe handles: avoiding the need for threads.
Here's a good intro: https://unixism.net/loti/index.html
If people think an IoUringReactor is worthwhile I'll open a ticket and make a start.
However it will need a reviewer... :-)
Ian
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Proprietary protocols like SMB tend to be over-eng On 4/01/2021 4:41 pm, Glyph wrote:
I suspect that this may require somewhat less... cognitive surface area than your other contributions :).
And hey, we have a vaccine now, which means that maybe things will go back to normal or close enough to it that I'll have enough capacity to get back to it myself :)
-g
On Jan 3, 2021, at 8:01 PM, Ian Haywood <ian@haywood.id.au> wrote:
In investigating async file I/O I came across this. In a nutshell it's the new epoll()
It's marginally more efficient although this is only apparent at very high loads. What's more interesting is that io_uring accepts files as well as network/pipe handles: avoiding the need for threads.
Here's a good intro: https://unixism.net/loti/index.html
If people think an IoUringReactor is worthwhile I'll open a ticket and make a start.
However it will need a reviewer... :-)
Ian
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
over engineered but don't exactly break new ground in a CS sense. On 6/01/2021 11:35 am, Ian Haywood wrote:
Proprietary protocols like SMB tend to be over-eng
On 4/01/2021 4:41 pm, Glyph wrote:
I suspect that this may require somewhat less... cognitive surface area than your other contributions :).
And hey, we have a vaccine now, which means that maybe things will go back to normal or close enough to it that I'll have enough capacity to get back to it myself :)
-g
On Jan 3, 2021, at 8:01 PM, Ian Haywood <ian@haywood.id.au> wrote:
In investigating async file I/O I came across this. In a nutshell it's the new epoll()
It's marginally more efficient although this is only apparent at very high loads. What's more interesting is that io_uring accepts files as well as network/pipe handles: avoiding the need for threads.
Here's a good intro: https://unixism.net/loti/index.html
If people think an IoUringReactor is worthwhile I'll open a ticket and make a start.
However it will need a reviewer... :-)
Ian
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Monday, 4 January 2021 04:01:42 GMT Ian Haywood wrote:
In investigating async file I/O I came across this. In a nutshell it's the new epoll()
It's marginally more efficient although this is only apparent at very high loads.
What's more interesting is that io_uring accepts files as well as network/pipe handles: avoiding the need for threads.
What threads? Why do you call out file FDs different from socket FDs? The point of io_uring is to avoid transitions between user and kernel right? Nothing to do with thread. In current twisted you can run complex network code without threads already.
Here's a good intro: https://unixism.net/loti/index.html
Also there is full coverage of io_uring on lwn.net. Its a fast evolving kernel API.
If people think an IoUringReactor is worthwhile I'll open a ticket and make a start.
I'm guessing that you will need to write a Python extension to get at io_uring or use ctypes. Is that what you where expecting?
However it will need a reviewer... :-)
Yes this is going to be complex code that few people have any experience with. Barry
Ian
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Tue, Jan 5, 2021 at 6:49 AM Barry Scott <barry.scott@forcepoint.com> wrote:
On Monday, 4 January 2021 04:01:42 GMT Ian Haywood wrote:
In investigating async file I/O I came across this. In a nutshell it's the new epoll()
It's marginally more efficient although this is only apparent at very high loads.
What's more interesting is that io_uring accepts files as well as network/pipe handles: avoiding the need for threads.
What threads? Why do you call out file FDs different from socket FDs?
The point of io_uring is to avoid transitions between user and kernel right? Nothing to do with thread.
In current twisted you can run complex network code without threads already.
Here's a good intro: https://unixism.net/loti/index.html
Also there is full coverage of io_uring on lwn.net. Its a fast evolving kernel API.
If people think an IoUringReactor is worthwhile I'll open a ticket and make a start.
I'm guessing that you will need to write a Python extension to get at io_uring or use ctypes. Is that what you where expecting?
However it will need a reviewer... :-)
Yes this is going to be complex code that few people have any experience with.
Just so there's a counter-perspective out there, I would like to suggest that reactors are neither magical nor particular complex in the grand scheme of software and that if you start with the assumption that a piece of code is going to be complex and hard to review then you will almost certainly create a piece of software that is complex and hard to review. My recommendation would be to instead start with the assumption that it's just a bit more mundane code binding a relatively small and straightforward C API related to copying bytes from one location to another. Yea, maybe there will be some hassles getting it to compile smoothly in all the desired environments or maybe there will be a few tricky areas for memory management or some other kind of low-level, localized bookkeeping - but those kinds of issues don't have to make for a complex piece of code. Starting from this assumption, try to produce an implementation that is straightforward, well-factored, and easily reviewed. What do you have to lose? Jean-Paul
Barry
Ian
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Tuesday, 5 January 2021 13:31:49 GMT Jean-Paul Calderone wrote:
On Tue, Jan 5, 2021 at 6:49 AM Barry Scott <barry.scott@forcepoint.com> wrote:
On Monday, 4 January 2021 04:01:42 GMT Ian Haywood wrote:
In investigating async file I/O I came across this. In a nutshell it's the new epoll()
It's marginally more efficient although this is only apparent at very high loads.
What's more interesting is that io_uring accepts files as well as network/pipe handles: avoiding the need for threads.
What threads? Why do you call out file FDs different from socket FDs?
The point of io_uring is to avoid transitions between user and kernel right? Nothing to do with thread.
In current twisted you can run complex network code without threads already.
Here's a good intro: https://unixism.net/loti/index.html
Also there is full coverage of io_uring on lwn.net. Its a fast evolving kernel API.
If people think an IoUringReactor is worthwhile I'll open a ticket and make a start.
I'm guessing that you will need to write a Python extension to get at io_uring or use ctypes. Is that what you where expecting?
However it will need a reviewer... :-)
Yes this is going to be complex code that few people have any experience with.
Just so there's a counter-perspective out there, I would like to suggest that reactors are neither magical nor particular complex in the grand scheme of software and that if you start with the assumption that a piece of code is going to be complex and hard to review then you will almost certainly create a piece of software that is complex and hard to review.
My recommendation would be to instead start with the assumption that it's just a bit more mundane code binding a relatively small and straightforward C API related to copying bytes from one location to another. Yea, maybe there will be some hassles getting it to compile smoothly in all the desired environments or maybe there will be a few tricky areas for memory management or some other kind of low-level, localized bookkeeping - but those kinds of issues don't have to make for a complex piece of code. Starting from this assumption, try to produce an implementation that is straightforward, well-factored, and easily reviewed. What do you have to lose?
I'm not against doing this and would love to see a PoC. I've been following the io_uring work with great interest. But my experience with things like this inside kernels leads me to expect complexity. Barry
Jean-Paul
Barry
Ian
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Tue, 5 Jan 2021 at 13:44, Jean-Paul Calderone <exarkun@twistedmatrix.com> wrote:
On Tue, Jan 5, 2021 at 6:49 AM Barry Scott <barry.scott@forcepoint.com> wrote:
On Monday, 4 January 2021 04:01:42 GMT Ian Haywood wrote:
In investigating async file I/O I came across this. In a nutshell it's the new epoll()
It's marginally more efficient although this is only apparent at very high loads.
What's more interesting is that io_uring accepts files as well as network/pipe handles: avoiding the need for threads.
What threads? Why do you call out file FDs different from socket FDs?
The point of io_uring is to avoid transitions between user and kernel right? Nothing to do with thread.
In current twisted you can run complex network code without threads already.
Here's a good intro: https://unixism.net/loti/index.html
Also there is full coverage of io_uring on lwn.net. Its a fast evolving kernel API.
If people think an IoUringReactor is worthwhile I'll open a ticket and make a start.
I'm guessing that you will need to write a Python extension to get at io_uring or use ctypes. Is that what you where expecting?
However it will need a reviewer... :-)
Yes this is going to be complex code that few people have any experience with.
Just so there's a counter-perspective out there, I would like to suggest that reactors are neither magical nor particular complex in the grand scheme of software and that if you start with the assumption that a piece of code is going to be complex and hard to review then you will almost certainly create a piece of software that is complex and hard to review.
My recommendation would be to instead start with the assumption that it's just a bit more mundane code binding a relatively small and straightforward C API related to copying bytes from one location to another. Yea, maybe there will be some hassles getting it to compile smoothly in all the desired environments or maybe there will be a few tricky areas for memory management or some other kind of low-level, localized bookkeeping - but those kinds of issues don't have to make for a complex piece of code. Starting from this assumption, try to produce an implementation that is straightforward, well-factored, and easily reviewed. What do you have to lose?
Jean-Paul
Hi, There is this wrapper https://pypi.org/project/liburing/ I can give it a try and review a possible implementation. My suggestion is to find a real world / production use case for the new reactor so that we can run more than unit / functional tests. -- Adi Roiban
On 6/01/2021 1:04 am, Adi Roiban wrote:
On Tue, 5 Jan 2021 at 13:44, Jean-Paul Calderone <exarkun@twistedmatrix.com <mailto:exarkun@twistedmatrix.com>> wrote:
On Tue, Jan 5, 2021 at 6:49 AM Barry Scott <barry.scott@forcepoint.com <mailto:barry.scott@forcepoint.com>> wrote:
What threads? Why do you call out file FDs different from socket FDs?
If you give epoll()/select() a file FD it lies and says the FD is always "ready" even when, as you point out, file operations can block.
My suggestion is to find a real world / production use case for the new reactor so that we can run more than unit / functional tests.
-- Adi Roiban
As a "standard" reactor any twisted app would stress-test it for network/pipe operations, no? For async file I/O my plan would be to export a new IFilesystem (which is closely based on conch.interfaces.ISFTPServer) apps would be have to be written to use it, conch could with minimal tweaking, and unsurprisingly the SMB server will. Apps can fallback to thread-based or aio(7) based implementations that already exists as PoC: https://github.com/twisted/twisted/pull/1420 Ian
On Jan 7, 2021, at 3:24 AM, Ian Haywood <ian@haywood.id.au> wrote:
For async file I/O my plan would be to export a new IFilesystem (which is closely based on conch.interfaces.ISFTPServer) apps would be have to be written to use it, conch could with minimal tweaking, and unsurprisingly the SMB server will.
Apps can fallback to thread-based or aio(7) based implementations that already exists as PoC:https://github.com/twisted/twisted/pull/1420 <https://github.com/twisted/twisted/pull/1420>
The mess of ctypes stuff seems somewhat irrelevant to the "asynchronous filesystem" part of this PR - do you think you could do a smaller version of this which decouples it from smb and ctypes? It would be very nice for twisted.web.static.File to be able to use this as well. And it would be much easier to review in isolation. (And also easier to review as concrete "clear asynchronous file I/O interface" functionality, rather than tying it to the term "VFS" which has a fairly checkered past within Twisted :)). (It would also be nice to have an interface that acts as an IProducer to integrate more natively with Twisted's support for backpressure rather than only having a custom readChunk method. I think readChunk is still necessary for completeness since you need to be able to seek and offset, though.) -g
On 8/01/2021 7:23 am, Glyph wrote:
The mess of ctypes stuff
seems somewhat irrelevant to the "asynchronous filesystem" part of this PR - do you think you could do a smaller version of this which decouples it from smb and ctypes? but yes both can be spun out easily, so it's just the interface and a
unclear what you mean: either the aio implementation or statx.py plain portable threads-based implementation. Where in the twisted tree should the interface and implementation go?
(It would also be nice to have an interface that acts as an IProducer to integrate more natively with Twisted's support for backpressure rather than only having a custom readChunk method. I think readChunk is still necessary for completeness since you need to be able to seek and offset, though.)
certainly will look at this. It fits well with FTP's filesystem interaction. Ian
On Jan 10, 2021, at 4:42 PM, Ian Haywood <ian@haywood.id.au> wrote:
On 8/01/2021 7:23 am, Glyph wrote:
The mess of ctypes stuff
unclear what you mean: either the aio implementation or statx.py
Both, really, but I was mainly talking about statx.py.
seems somewhat irrelevant to the "asynchronous filesystem" part of this PR - do you think you could do a smaller version of this which decouples it from smb and ctypes? but yes both can be spun out easily, so it's just the interface and a plain portable threads-based implementation. Where in the twisted tree should the interface and implementation go?
I'd say probably twisted.internet? (This is something that might get modified as folks review & comment)
(It would also be nice to have an interface that acts as an IProducer to integrate more natively with Twisted's support for backpressure rather than only having a custom readChunk method. I think readChunk is still necessary for completeness since you need to be able to seek and offset, though.) certainly will look at this. It fits well with FTP's filesystem interaction.
Thank you so much! -g
On 11/01/2021 8:26 pm, Glyph wrote:
seems somewhat irrelevant to the "asynchronous filesystem" part of this PR - do you think you could do a smaller version of this which decouples it from smb and ctypes?
but yes both can be spun out easily, so it's just the interface and a plain portable threads-based implementation. Where in the twisted tree should the interface and implementation go?
(It would also be nice to have an interface that acts as an IProducer to integrate more natively with Twisted's support for backpressure rather than only having a custom readChunk method. I think readChunk is still necessary for completeness since you need to be able to seek and offset, though.)
certainly will look at this. It fits well with FTP's filesystem interaction.
OK this is done. I have changed the name to "asyncfs" in the hope this is more informative.
sorry, I missed the beginning of the thread / discussion: are you working on a new, full Twisted reactor on top of io_uring? and async filesystem is "just" one part / additional effort in that? Am 12.03.21 um 05:42 schrieb Ian Haywood:
On 11/01/2021 8:26 pm, Glyph wrote:
seems somewhat irrelevant to the "asynchronous filesystem" part of this PR - do you think you could do a smaller version of this which decouples it from smb and ctypes?
but yes both can be spun out easily, so it's just the interface and a plain portable threads-based implementation. Where in the twisted tree should the interface and implementation go?
(It would also be nice to have an interface that acts as an IProducer to integrate more natively with Twisted's support for backpressure rather than only having a custom readChunk method. I think readChunk is still necessary for completeness since you need to be able to seek and offset, though.)
certainly will look at this. It fits well with FTP's filesystem interaction.
OK this is done. I have changed the name to "asyncfs" in the hope this is more informative.
https://twistedmatrix.com/trac/ticket/10079
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
sorry, I missed the beginning of the thread / discussion: are you working on a new, full Twisted reactor on top of io_uring?
and async filesystem is "just" one part / additional effort in that?
On 12/03/2021 7:21 pm, Tobias Oberstein wrote: the advantage of io_uring is being able to combine file and network I/O, so this PR is a precondition to working on the reactor.
interesting. thanks! Am 12.03.21 um 09:59 schrieb Ian Haywood:
sorry, I missed the beginning of the thread / discussion: are you working on a new, full Twisted reactor on top of io_uring?
and async filesystem is "just" one part / additional effort in that?
On 12/03/2021 7:21 pm, Tobias Oberstein wrote: the advantage of io_uring is being able to combine file and network I/O, so this PR is a precondition to working on the reactor.
yeah, the advantages of io_uring are obvious, combining disk and network IO is one. even a network IO only twisted reactor for io_uring would be awesome! disk IO, for me, would be nice to have, but depends, eg LMDB mmaps the DB file, so it wouldn't go through io_uring anyways. one more aspect: io_uring is also about zero-copy (besides reducing context switch overhead / reduce syscalls) twisted protocol base classes always first buffer outgoing bytes in userspace. in a list that is later flattened, before pushing to kernel in the actual socket write. the latter only happens once the FD being written becomes writable of course. since with io_uring the queue/ring to append to never blocks for writes, that buffering in twisted would lead to double buffering (as the ring already buffers) then, io_uring follows a completion IO model (as IOCP). quite different from the "ready to write/read" model (as in epoll etc). I am wondering, do you already have a design for these aspects of io_uring / twisted?
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Tobias Oberstein - phone +49 176 2375 2055 - tobias.oberstein@crossbario.com Crossbar.io GmbH - Waldstrasse 18 - 91054 Erlangen HRB 15870 - Amtsgericht Fuerth - Geschäftsfuehrer/CEO - Tobias Oberstein https://crossbar.io https://crossbario.com
since with io_uring the queue/ring to append to never blocks for writes, that buffering in twisted would lead to double buffering (as the ring already buffers)
then, io_uring follows a completion IO model (as IOCP). quite different from the "ready to write/read" model (as in epoll etc).
I am wondering, do you already have a design for these aspects of io_uring / twisted? I haven't thought about it in detail, but python presents mmap as bytearray, so I would have thought it fairly easy to write directly to
On 12/03/2021 8:29 pm, Tobias Oberstein wrote: the ring from Protocol.write
Am 13.03.21 um 01:01 schrieb Ian Haywood:
since with io_uring the queue/ring to append to never blocks for writes, that buffering in twisted would lead to double buffering (as the ring already buffers)
then, io_uring follows a completion IO model (as IOCP). quite different from the "ready to write/read" model (as in epoll etc).
I am wondering, do you already have a design for these aspects of io_uring / twisted? I haven't thought about it in detail, but python presents mmap as bytearray, so I would have thought it fairly easy to write directly to
On 12/03/2021 8:29 pm, Tobias Oberstein wrote: the ring from Protocol.write
fwiw, couple of thoughts: to make it "zero copy" right up into python, a couple of things would be needed (there might be other ways of course ..): the memory to write into on the producer side must be allocated and managed via liburing that memory might for example actually reside inside a NIC and be DMA'ed into main memory ("zero CPU copy") or even directly mapped into the user level address space ("true zero copy") even when taking the very basic case of just transmitting an unstructed bunch of bytes, to make it truly zero copy up into python, sth like https://zero-buffer.readthedocs.io might be needed rather than standard python str/bytes. then, for structured data, if zero-buffer would be used in python lib of flatbuffers for str/bytes https://google.github.io/flatbuffers/ this would result in a truly zero-copy transfer _and_ serialization for native structured objects. all of that with zero/minimal syscalls. in a way, the holy grail;) cheers, /Tobias alternatives for flatbuffers: https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html
What's more interesting is that io_uring accepts files as well as network/pipe handles: avoiding the need for threads.
What threads? Why do you call out file FDs different from socket FDs?
The point of io_uring is to avoid transitions between user and kernel right? Nothing to do with thread.
In current twisted you can run complex network code without threads already.
Because before io_uring there was no reliable way on Linux to do async file I/O and everybody is forced to use thread pools for it.
On Tuesday, 5 January 2021 13:38:11 GMT Hynek Schlawack wrote:
What's more interesting is that io_uring accepts files as well as network/pipe handles: avoiding the need for threads.
What threads? Why do you call out file FDs different from socket FDs?
The point of io_uring is to avoid transitions between user and kernel right? Nothing to do with thread.
In current twisted you can run complex network code without threads already.
Because before io_uring there was no reliable way on Linux to do async file I/O and everybody is forced to use thread pools for it.
You mean like the way that opening a file can block as the kernel walks the directories? Barry
On 2021-01-04 10:15, Barry Scott wrote:
On Monday, 4 January 2021 04:01:42 GMT Ian Haywood wrote:
If people think an IoUringReactor is worthwhile I'll open a ticket and make a start.
I'm guessing that you will need to write a Python extension to get at io_uring or use ctypes. Is that what you where expecting?
FYI, there is https://github.com/YoSTEALTH/liburing. I have no basis to say if it is good or not, just that it's there. Cheers, -kyle
participants (9)
-
Adi Roiban -
Barry Scott -
Glyph -
Hynek Schlawack -
Ian Haywood -
Jean-Paul Calderone -
Kyle Altendorf -
Tobias Oberstein -
Tobias Oberstein