On Feb 12, 2010, at 9:51 PM, K. Richard Pixley wrote:
I'm working with parallelized build servers. We often have raided
disks, solid state disks, servers with huge amounts of disk cache
specifically so that an entire build happens in memory, etc. File io is
our bottleneck.
Yeah, this level of disk manipulation is past the point where a little bit less blocking on I/O will help... I imagine you've already got some kind of process/thread pooling solution already, or at least you'll need one.
I think you probably are also forgetting about NFS. NFS isn't slower
than native disk in terms of throughput, only in terms of latency, which
is a fabulous opportunity for asyncronous file io. (Granted, NFS seems
to have fallen out of fashion recently.)
No, I'm not forgetting about it: I'm just saying that once you've got an API that applications can start using that gives *some* performance benefit (non-blocking disk I/O at the expense of spinning up a few threads / processes behind the scenes), you can always optimize it for other use-cases later, without necessarily changing the API.
I think twisted already has everything that's required. It could
probably use a slightly more friendly interface paradigm so the user
doesn't have to do his own os.open, but really, even that wouldn't save
much.
It would allow us to do it more portably, I think. os.open()'s behavior can vary a lot depending on what you do with it.
Reactor core was enough to sell me on twisted. That's probably all I'll
even be using. And compared to writing my own, that's enough to be useful.