comparative performance benchmark Python 1.5.2 - 2.0 - 2.1

Tim Peters tim.one at home.com
Sun Oct 21 15:07:25 EDT 2001


[Frederic Giacometti]
> I have been puzzled this week when shown results of a simple benchmarck
> between python 1.5.2, 2.0 and 2.1.
> I am wondering; Has there been some official research on the issue ?
> (pointers....?)

You won't get anywhere with this unless you analyze it yourself in *great*
detail, or post the code (e.g., attached to a SourceForge report) and hope
someone else can dig into it (I will if I can make time for it, but gotta
say that looks unlikely to happen "soon").

There are dozens of subsystems in Python, and across releases some slow
down, some speed up, some a little and some a lot, all of that often varies
across platforms, and it's exceedingly rare for any two "benchmarks" to
tickle the same set of timing changes.  So if you're looking for a simple,
general answer, the odds of getting one are approximately nil.  Cases of
large slowdowns are usually traced to one or two specific things a program
does, and this can be as specific as "creates millions of dicts of size 3
but almost none of any other size (else it would have sped up instead)".
Sometimes they're traced to platform-specific compiler or C library changes
too.  (Did you use exactly the same C compiler, libc, compiler and linker
options, and OS, for all runs?  C compilers and libraries can also vary
wildly across releases in the speed of generated code in ultra-specific
cases, and Python performance inherits all that uncertainty too.)

> The benchmarking program was only basic python objects (strings,
> dictionaries, string/flot conversions...).

Doesn't really help to know that.

> In my haste to try do defend Python 2.1, we had done the changes
> I was aware of:  turn off garbage collection (20% gain),

I expect that's a record.  People who have bothered to measure this usually
report about 3% for gc overhead, and IIRC the largest I heard of before was
about 10%.

> switch sre to pre (though this program does not use 're'0;

How do you know?  The Python libraries can use regexps too.  BTW, sre is
much faster than pre in many cases.

> replace string.xxx() calls with o.xxx() string methods.

Good -- that can help.

Can you try it under 2.2b1 too (was released on Friday)?

> ...
> As result, I'm getting in the situation where I'm being asked to
> retroceed all our developments from Python 2.1 to Python 1.5.2.

Unless that benchmark is a true indicator of performance for the real apps
your site runs (and very few artificial benchmarks are), that doesn't make
sense.

War story:  A major slowdown in Zope between 1.5.2 and 2.1 was traced to a
one-line change in the core to improve an error msg.  "The problem" was that
Zope triggered this error a *lot*, but always caught the exception.  The
routine raising the error simply spent a lot more time building a nice error
msg than it did in 1.5.2.  Is that the smoking gun for your app too?  It's
impossible to say without tracing thru your code in detail, but it's
unlikely -- every benchmark is unique in some way that even its author can
almost never guess.

"fast-enough?"-is-the-only-question-that-actually-matters-ly y'rs  - tim





More information about the Python-list mailing list