Glyph Lefkowitz wrote:
If you only have one disk, you can only really get a benefit from two, maybe three file I/O slave processes, and that's a fairly small amount of resources to manage. Granted, it's tricky to really identify how many "disks" you've got in a system, and the performance characteristics change radically based on what kind of disk technology is involved, but generally speaking a few worker threads and a queue of I/O operations would cover the vast majority of use-cases. I'm working with parallelized build servers. We often have raided disks, solid state disks, servers with huge amounts of disk cache specifically so that an entire build happens in memory, etc. File io is our bottleneck.
I think you probably are also forgetting about NFS. NFS isn't slower than native disk in terms of throughput, only in terms of latency, which is a fabulous opportunity for asyncronous file io. (Granted, NFS seems to have fallen out of fashion recently.) I think twisted already has everything that's required. It could probably use a slightly more friendly interface paradigm so the user doesn't have to do his own os.open, but really, even that wouldn't save much. Reactor core was enough to sell me on twisted. That's probably all I'll even be using. And compared to writing my own, that's enough to be useful. --rich