[Python-Dev] file.readinto performance regression in Python 3.2 vs. 2.7?

[Eli Bendersky]

Hi there,

I was doing some experiments with the buffer interface of bytearray today,
for the purpose of quickly reading a file's contents into a bytearray which
I can then modify. I decided to do some benchmarking and ran into
surprising results. Here are the functions I was timing:

def justread():
    # Just read a file's contents into a string/bytes object
    f = open(FILENAME, 'rb')
    s = f.read()

def readandcopy():
    # Read a file's contents and copy them into a bytearray.
    # An extra copy is done here.
    f = open(FILENAME, 'rb')
    b = bytearray(f.read())

def readinto():
    # Read a file's contents directly into a bytearray,
    # hopefully employing its buffer interface
    f = open(FILENAME, 'rb')
    b = bytearray(os.path.getsize(FILENAME))
    f.readinto(b)

FILENAME is the name of a 3.6MB text file. It is read in binary mode,
however, for fullest compatibility between 2.x and 3.x

Now, running this under Python 2.7.2 I got these results ($1 just reflects
the executable name passed to a bash script I wrote to automate these runs):

$1 -m timeit -s'import fileread_bytearray' 'fileread_bytearray.justread()'
1000 loops, best of 3: 461 usec per loop
$1 -m timeit -s'import fileread_bytearray'
'fileread_bytearray.readandcopy()'
100 loops, best of 3: 2.81 msec per loop
$1 -m timeit -s'import fileread_bytearray' 'fileread_bytearray.readinto()'
1000 loops, best of 3: 697 usec per loop

Which make sense. The readinto() approach is much faster than copying the
read buffer into the bytearray.

But with Python 3.2.2 (built from the 3.2 branch today):

$1 -m timeit -s'import fileread_bytearray' 'fileread_bytearray.justread()'
1000 loops, best of 3: 336 usec per loop
$1 -m timeit -s'import fileread_bytearray'
'fileread_bytearray.readandcopy()'
100 loops, best of 3: 2.62 msec per loop
$1 -m timeit -s'import fileread_bytearray' 'fileread_bytearray.readinto()'
100 loops, best of 3: 2.69 msec per loop

Oops, readinto takes the same time as copying. This is a real shame,
because readinto in conjunction with the buffer interface was supposed to
avoid the redundant copy.

Is there a real performance regression here, is this a well-known issue, or
am I just missing something obvious?

Eli

P.S. The machine is quad-core i7-2820QM, running 64-bit Ubuntu 10.04
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20111124/5f8e79c8/attachment.html>