On 3/25/07, Josiah Carlson <jcarlson@uci.edu> wrote:
But really, transferring little bits of data back and forth isn't what is of my concern in terms of speed. My real concern is transferring nontrivial blocks of data; I usually benchmark blocks of sizes: 1k, 4k, 16k, 64k, 256k, 1M, 4M, 16M, and 64M. Those are usually pretty good to discover the "sweet spot" for a particular implementation, and also allow a person to discover whether or not their system can be used for nontrivial processor loads.
Not directly relevant to the discussion, but I attended recently a talk from the main developer of STXXL (http://stxxl.sourceforge.net/), an STL-compatible library for handling huge volumes of data. The keys to efficient processing are support for parallel disks, explicit overlapping between I/O and computation, and I/O pipelining. More details are available at http://i10www.ira.uka.de/dementiev/stxxl/report/. George