[Python-ideas] Python and Concurrency

[George Sakkis]

On 3/25/07, Josiah Carlson <jcarlson at uci.edu> wrote:

> But really, transferring little bits of data back and forth isn't what
> is of my concern in terms of speed.  My real concern is transferring
> nontrivial blocks of data; I usually benchmark blocks of sizes: 1k, 4k,
> 16k, 64k, 256k, 1M, 4M, 16M, and 64M. Those are usually pretty good to
> discover the "sweet spot" for a particular implementation, and also
> allow a person to discover whether or not their system can be used for
> nontrivial processor loads.

Not directly relevant to the discussion, but I attended recently a
talk from the main developer of STXXL (http://stxxl.sourceforge.net/),
an STL-compatible library for handling huge volumes of data. The keys
to efficient processing are support for parallel disks, explicit
overlapping between I/O and computation, and I/O pipelining. More
details are available at
http://i10www.ira.uka.de/dementiev/stxxl/report/.

George