[New-bugs-announce] [issue10160] operator.attrgetter slower than lambda after adding dotted names ability
Χρήστος Γεωργίου (Christos Georgiou)
report at bugs.python.org
Thu Oct 21 02:10:21 CEST 2010
New submission from Χρήστος Γεωργίου (Christos Georgiou) <tzot at users.sourceforge.net>:
(Discovered in that StackOverflow answer: http://stackoverflow.com/questions/3940518/3942509#3942509 ; check the comments too)
operator.attrgetter in its simplest form (i.e. with a single non-dotted name) needs more time to execute than an equivalent lambda expression.
Attached file so3940518.py runs a simple benchmark comparing: a list comprehension of plain attribute access; attrgetter; and lambda. I will append sample benchmark times at the end of the comment.
Browsing Modules/operator.c, I noticed that the dotted_getattr function was using PyUnicode_Check and (possibly) splitting on dots on *every* call of the attrgetter, which I thought to be most inefficient.
I changed the py3k-daily-snapshot source to make the PyUnicode_Check calls in the attrgetter_new function; also, I modified the algorithm to pre-parse the operator.attrgetter functions for possible dots in the names, in order for the dotted_getattr function to become speedier.
The only “drawback” is that now operator.attrgetter raises a TypeError on creation, not on subsequent calls of the attrgetter object; this shouldn't be a compatibility problem. However, I obviously had to update both Doc/library/operator.rst and Lib/test/test_operator.py .
I am not sure whether I should attach a zip/tar file with both the attachments (the sample benchmark and the diff); so I'll attach the diff in a further comment.
On the Ubuntu server 9.10 where I made the changes, I ran the so3940518.py sample benchmark before and after the changes.
Run before the changes (third column is seconds, less is better):
list comp 0.40999999999999925 1000000
map attrgetter 1.3899999999999997 1000000
map lambda 1.0099999999999998 1000000
Run after the changes:
list comp 0.40000000000000036 1000000
map attrgetter 0.5199999999999996 1000000
map lambda 0.96 1000000
assignee: docs at python
components: Documentation, Library (Lib), Tests
nosy: docs at python, tzot
title: operator.attrgetter slower than lambda after adding dotted names ability
versions: Python 3.2
Added file: http://bugs.python.org/file19310/so3940518.py
Python tracker <report at bugs.python.org>
More information about the New-bugs-announce