python disk i/o speed

[Karl Vogel]

>> On 7 Aug 2002 13:40:01 -0700, 
>> pruebauno at latinmail.com (nnes) said:

   >> I susoect this last point is the most valuable data in the post, and
   >> I look forward to seeing the sample code.

N> In the end I figure It would have been better to do a giant post, anyway
N> here comes the gory details:

N> ###############gener1.py: generates the sample data
N> ##############bench1.py: python implementation
N> ################bench1.c: implementation in ANSI C

##############bench2.py: better python implementation

import sys,xreadlines
if len(sys.argv)<3:
    sys.exit("bench input output")

input=open(sys.argv[1])
output=open(sys.argv[2],"w")

for line in input.xreadlines():
    part=line.split("\",\"")
    part0=part[0][1:]
    part1=part[1]
    part2=part[2][:-2]

    # joining
    out = [
         part0,
         part1,
         part2,
         str(int(part0)+int(part1)+int(part2)),
    ]

    out = '"' + '","'.join(out) + '"\n'
    output.write(out)

##############bench3.py: better python implementation

import sys,xreadlines
if len(sys.argv)<3:
    sys.exit("bench input output")

input=open(sys.argv[1])
output=open(sys.argv[2],"w")

for line in input.xreadlines():
    part=line.split("\",\"")
    part0=part[0][1:]
    part1=part[1]
    part2=part[2][:-2]

    # formatting
    out = '"%s","%s","%s","%s"\n' % (part0, part1, part2,
    int(part0)+int(part1)+int(part2),)

    output.write(out)

##############bench1.pl: perl implementation

#!/usr/local/bin/perl
$input = shift (@ARGV) || die;
$output = shift (@ARGV) || die;

open (IN, "$input") || die "$input";
open (OUT, "> $output") || die "$output";

while (<IN>)
{
    chomp;
    if (/"(\d+)","(\d+)","(\d+)"/)
    {
        ($part0, $part1, $part2) = ($1, $2, $3);
        $sum = int($part0) + int($part1) + int($part2);
        print OUT "\"$part0\",\"$part1\",\"$part2\",\"$sum\"\n";
    }
}

exit (0);

##############results

Python version is 2.1.3 on all systems.  Zope won't work with 2.2.
Perl version is 5.6.1 on all systems.

Test file created by gener1.py, /tmp directory:

    me% ls -l in
    -rw-r--r--   1 vogelke  mis       6881082 Aug  9 13:24 in

    me% wc in
     332999  332999 6881082 in

Benchmarks on Sparc Ultra 5/10, /tmp is swap:

    me% df /tmp
    Filesystem     1024-blocks   Used Available Capacity  Mounted on
    swap               360432   15896    344536       4%  /tmp

    me% time ./bench.py in out
    ./bench.py in out  66.35s user 0.45s system 59% cpu 1:52.13 total

    me% time ./bench.pl in out
    ./bench.pl in out  39.33s user 0.36s system 98% cpu 40.146 total

    me% time ./bench.c in out
    ./bench in out  3.47s user 0.30s system 100% cpu 3.763 total

Benchmarks on Pentium-200 FreeBSD, /tmp is RAM disk:

    me% df /tmp
    Filesystem  1K-blocks     Used    Avail Capacity  Mounted on
    mfs:34         257998    19234   218126     8%    /tmp

    me% time ./bench.py in out
    ./bench.py in out  145.90s user 0.72s system 99% cpu 2:27.97 total

    me% time ./bench2.py in out
    ./bench2.py in out  143.74s user 0.73s system 97% cpu 2:28.66 total

    me% time ./bench3.py in out 
    ./bench3.py in out  130.80s user 0.54s system 99% cpu 2:12.13 total

    me% time ./bench.pl in out
    ./bench.pl in out  98.31s user 0.70s system 99% cpu 1:39.81 total

    me% time ./bench.c in out
    ./bench in out  6.48s user 0.55s system 96% cpu 7.259 total

Benchmarks on Sun Enterprise-450, /tmp is swap:

    me% df /tmp     
    Filesystem  1M-blocks      Used Available Use% Mounted on
    swap             2304        28      2276   2% /tmp

    me% time ./bench.py in out
    ./bench.py in out  29.95s user 0.14s system 99% cpu 30.123 total

    me% time ./bench.pl in out
    ./bench.pl in out  20.29s user 0.08s system 98% cpu 20.665 total

    me% time ./bench.c in out
    ./bench in out  2.36s user 0.11s system 99% cpu 2.475 total

-- 
Karl Vogel               ASC/YCOA, Wright-Patterson AFB, OH 45433
vogelke at dnaco.net                   http://www.dnaco.net/~vogelke

You know you're a redneck if you've ever used a toilet brush as a
back scratcher.