Python too slow for real world

[Christian Tismer]

Arne Mueller wrote:
> 
> Hi All,
> 
> I can't read in the whole file as a single block, it's too big, if
> readline/write is slow the program will never get realy fast :-(

Please try this one.
For me, it was about 6-7 times faster than the first one.
I don't read by line, also I don't read all in one.
Let me know how it performs on your machine.
I think I'm down to measuring I/O time.
Well, the code is a bit long.

But fast :-)

I believe nothing more can be done, but to
use P2C to get the interpreter overhead away.

def read_write_bulk(input, output, exclude):

    bufsize = 1 << 16
    splitter = ">"

    ignore=exclude.has_key
    split=string.split
    No = None

    buffer = input.read(bufsize) 
    got = len(buffer)
    while len(buffer)>1 :
        pieces = split(buffer, splitter)
        idx = 0
        inner = pieces[1:-1]
        for piece in inner:
            idx = idx+1 ; key = split(piece, No, 1)[0]
            if ignore(key): 
                del inner[idx] ; idx = idx-1
        output.write("<")
        output.write(string.join(inner, splitter))
        if got==0:
            break
        chunk = input.read(bufsize)
        buffer = splitter+pieces[-1] + chunk
        got = len(chunk)
        if got==0:
            buffer = buffer+splitter # spill last one

#:-) end of hack

-- 
Christian Tismer             :^)   <mailto:tismer at appliedbiometrics.com>
Applied Biometrics GmbH      :     Have a break! Take a ride on Python's
Kaiserin-Augusta-Allee 101   :    *Starship* http://starship.python.net
10553 Berlin                 :     PGP key -> http://wwwkeys.pgp.net
PGP Fingerprint       E182 71C7 1A9D 66E9 9D15  D3CC D4D7 93E2 1FAE F6DF
     we're tired of banana software - shipped green, ripens at home