On Monday, 4 January 2021 04:01:42 GMT Ian Haywood wrote:
> In investigating async file I/O I came across this. In a nutshell it's
> the new epoll()
> It's marginally more efficient although this is only apparent at very
> high loads.
> What's more interesting is that io_uring accepts files as
> well as network/pipe handles: avoiding the need for threads.
What threads? Why do you call out file FDs different from socket FDs?
The point of io_uring is to avoid transitions between user and kernel right?
Nothing to do with thread.
In current twisted you can run complex network code without threads
already.
> Here's a good intro: https://unixism.net/loti/index.html
Also there is full coverage of io_uring on lwn.net.
Its a fast evolving kernel API.
> If people think an IoUringReactor is worthwhile I'll open a ticket and
> make a start.
I'm guessing that you will need to write a Python extension to get at
io_uring or use ctypes. Is that what you where expecting?
> However it will need a reviewer... :-)
Yes this is going to be complex code that few people have any experience with.
Just so there's a counter-perspective out there, I would like to suggest that reactors are neither magical nor particular complex in the grand scheme of software and that if you start with the assumption that a piece of code is going to be complex and hard to review then you will almost certainly create a piece of software that is complex and hard to review.
My recommendation would be to instead start with the assumption that it's just a bit more mundane code binding a relatively small and straightforward C API related to copying bytes from one location to another. Yea, maybe there will be some hassles getting it to compile smoothly in all the desired environments or maybe there will be a few tricky areas for memory management or some other kind of low-level, localized bookkeeping - but those kinds of issues don't have to make for a complex piece of code. Starting from this assumption, try to produce an implementation that is straightforward, well-factored, and easily reviewed. What do you have to lose?
Jean-Paul