[Python-Dev] collections module

[Kurt B. Kaiser]

Guido van Rossum <guido at python.org> writes:

>> before: 15773 pystones, 29.9 mPB
>> after:  12690 pystones, 31.4 mPB
>
> What's the mPB number?  If its the time to complete parrotbench, this
> is actually a slowdown, just as for pystone.
>
>> Switch back to CVS:
>> 
>> 19531, 19379, 19531, 19379, 19305, 19084 pystones, 31.4 mPB
>> 
>> Switch back to Tim's patch:
>> 
>> 15015, 15480, 14588 pystones
>> 
>> should not need, but:
>> make clean && make  (just make on previous, only listobject.c being recompiled)
>> 
>> Still patched: 
>> 13158, 14006, 13193; restart python; 15923, 16722, 16023
>> restart python; 12953, 13698, 12953, 13513
>> 
>> Well, pystones are all over the map on this box.  But the
>> improvement is there for punchit and the parrotbench benchmark is
>> solid across the change.
>
> Unclear.  Please clarify mPB.

milliParrotBenches/sec.   1000/(time to complete Parrotbench [sec])

In a few years we'll be spec'ing kPB  :-)

So the data so far is
29.9 mPB unpatched
31.4 mPB patched
31.4 mPB unpatched
This morning:
30.7, 30.4, 30.1, 32.1, 32.2, 31.9 mPB patched (using make time)

rm Objects/listobject.c && cvs up Objects/listobject.c
make clean && make

30.1, 30.2, 30.3, 32.1, 29.6, 29.9, 28.2 mPB unpatched

Average unpatched: 30.2 mPB 
Average patched: 31.3 mPB

So a 3 - 4% improvement, maybe.  Needs more data.

I reported the equivalent of 31.1 mPB on 2Jan04, which was the best
of several trials.  The 'best of 3' pystones at that time was 17182.

Pystone
-------
My impression is that Pystone isn't a good benchmark for this box
unless run many times over a long period with python restarts, perhaps
a cron at night.  It may be the 128kB cache, but the results are very
variable.

This morning, with patched listobject.c, it's 
15015, 15060, 14881; restart python; 16891, 17182, 17361
restart python; 14925, 14970, 14925, 14492, 14662, 14836, 15060, 15106, 15060
append sys.path and import a module: 13889, 13927, 14084
do some stuff, then: 14662, 14368, 14577
restart python; 14836, 14881, 14836
restart python; 15337, 15337, 15337, 15773, 15873, 15674, 15479, 15432, 14792

Following the switch back to unpatched mentioned above:
18182, 18050, 18248, 18450, 18248, 18181
restart python; 14881, 14970, 14705, 14836, 14749, 14970
restart python; 16835, 16835, 16778, 17064, 17182, 16892 
restart python; append sys.path, import module; 20000, 20408, 20080, 20808, 19763, 20000
do some stuff; 19305, 19762, 19607, 19084, 19455, 19455
restart python; 17605, 17731, 17482, 17544, 17544, 17483

On these two sets:

Average unpatched: 17865 pystones
Average patched: 15136 pystones

A 15% reduction with the patch.

This *may* be significant, but requires a lot more data here and
investigation by others.  On this box the pystone rating jumps between
python restarts.

I haven't calculated any more advanced statistics; IMHO with small
data sets, especially jumpy, ragged ones like these, you can often get
a better feel by just eyeballing it.  The downside being, you tend to
see what you want to see....
-- 
KBK