On Tue, Jan 27, 2009 at 6:15 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:

It's some arbitrary text composed of 95% ASCII characters and 5% non-ASCII. On
this specific example, utf8 decodes at around 250 MB/s, latin1 at almost 1 GB/s
(on the same machine on which I ran the benchmarks).

For the "10MB whole contents at once" test, we then have:
(assuming the code does no pipelining of disk I/O with decoding)

10MB / 980MB/s to read from disk = 10 ms
10MB / 250MB/s to decode to utf8 = 40 ms
10MB / (10ms + 40ms) = 200 MB/s

In practice, your results shows around 90 MB/s. That's at least vaguely in the same ballpark.

--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC