memcmp performance
Hi, This is my first time on Python-dev, so I apologize for my newbie-ness. I have been doing some performance experiments with memcmp, and I was surprised that memcmp wasn't faster than it was in Python. I did a whole, long analysis and came up with some very simple results. Before I put in a tracker bug report, I wanted to present my findings and make sure they were repeatable to others (isn't that the nature of science? ;) as well as offer discussion. The analysis is a pdf and is here: http://www.picklingtools.com/study.pdf The testcases are a tarball here: http://www.picklingtools.com/PickTest5.tar.gz I have three basic recommendations in the study: I am curious what other people think. Gooday, Richie
Hello,
I have been doing some performance experiments with memcmp, and I was surprised that memcmp wasn't faster than it was in Python. I did a whole, long analysis and came up with some very simple results.
Before I put in a tracker bug report, I wanted to present my findings and make sure they were repeatable to others (isn't that the nature of science? ;) as well as offer discussion.
Thanks for the analysis. Non-bugfix work now happens on Python 3, where the str type is Python 2's unicode type. Your recommendations would have to be revisited under that light. Have you reported gcc's "outdated optimization" issue to them? Or is it already solved in newer gcc versions? Under glibc-based systems, it seems we can't go wrong with the system memcpy function. If gcc doesn't get in the way, that is. Regards Antoine.
On 10/20/2011 5:08 PM, Antoine Pitrou wrote:
Have you reported gcc's "outdated optimization" issue to them? Or is it already solved in newer gcc versions?
I checked this on gcc 4.6, and it still optimizes memcmp/strcmp into a "repz cmpsb" instruction on x86. This has been known to be a problem since at least 2002[1][2]. There are also some alternative implementations available on their mailing list. It seems the main objection to removing the optimization was that gcc isn't always compiling against an optimized libc, so they didn't want to drop the optimization. Beyond that, I think nobody was willing to put in the effort to change the optimization itself. [1] http://gcc.gnu.org/ml/gcc/2002-10/msg01616.html [2] http://gcc.gnu.org/ml/gcc/2003-04/msg00166.html -- Scott Dial scott@scottdial.com
Antoine Pitrou, 20.10.2011 23:08:
I have been doing some performance experiments with memcmp, and I was surprised that memcmp wasn't faster than it was in Python. I did a whole, long analysis and came up with some very simple results.
Thanks for the analysis. Non-bugfix work now happens on Python 3, where the str type is Python 2's unicode type. Your recommendations would have to be revisited under that light.
Well, Py3 is quite a bit different now that PEP393 is in. It appears to use memcmp() or strcmp() a lot less than before, but I think unicode_compare() should actually receive an optimisation to use a fast memcmp() if both string kinds are equal, at least when their character unit size is less than 4 (i.e. especially for ASCII strings). Funny enough, tailmatch() has such an optimisation. Stefan
On Fri, 21 Oct 2011 08:24:44 +0200
Stefan Behnel
Antoine Pitrou, 20.10.2011 23:08:
I have been doing some performance experiments with memcmp, and I was surprised that memcmp wasn't faster than it was in Python. I did a whole, long analysis and came up with some very simple results.
Thanks for the analysis. Non-bugfix work now happens on Python 3, where the str type is Python 2's unicode type. Your recommendations would have to be revisited under that light.
Well, Py3 is quite a bit different now that PEP393 is in. It appears to use memcmp() or strcmp() a lot less than before, but I think unicode_compare() should actually receive an optimisation to use a fast memcmp() if both string kinds are equal, at least when their character unit size is less than 4 (i.e. especially for ASCII strings). Funny enough, tailmatch() has such an optimisation.
Yes, unicode_compare() probably deserves optimizing. Patches welcome, by the way :) Regards Antoine.
participants (4)
-
Antoine Pitrou -
Richard Saunders -
Scott Dial -
Stefan Behnel