[Patches] [ python-Patches-587076 ] Adaptive stable mergesort
noreply@sourceforge.net
noreply@sourceforge.net
Sun, 28 Jul 2002 11:14:00 -0700
Patches item #587076, was opened at 2002-07-26 11:51
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=587076&group_id=5470
Category: Core (C code)
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: Tim Peters (tim_one)
Assigned to: Nobody/Anonymous (nobody)
Summary: Adaptive stable mergesort
Initial Comment:
This adds method list.msort([compare]).
Lib/test/sortperf.py is already a sort performance
test. To run it on exactly the same data I used, run it
via
python -O sortperf.py 15 20 1
That will time the current samplesort (even after this
patch). After getting stable numbers for that, change
sortperf's doit() to say L.msort() instead of L.sort(),
and you'll time the mergesort instead.
CAUTION: To save time across many runs, sortperf
saves the random floats it generates, into temp files.
If those temp files already exist when sortperf starts,
it reads them up instead of generating new numbers.
As a result, it's important in the above to pass "1" as
the last argument the *first* time you run sortperf --
that forces the random # generator into the same
state it was when I used it.
This patch also gives lists a new list.hsort() method,
which is a weak heapsort I gave up on. Time it if you
want to see how bad an excellent sort can get <wink>.
----------------------------------------------------------------------
>Comment By: Tim Peters (tim_one)
Date: 2002-07-28 14:14
Message:
Logged In: YES
user_id=31435
Adding new patch, merge2.patch. Most of this is
semantically neutral compared to the last version -- more
asserts, better comments, minor code fiddling for clarity,
got rid of the weak heapsort.
There is one useful change, extracting more info out of the
pre-merge "find the endpoints" searches. This helps "in
theory" most of the time, but probably not enough to
measure. In some odd cases it can help a lot, though. See
Python-Dev for discussion. There's no strong reason to
time this stuff again, if you already did it once (and thanks
to those who did!).
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2002-07-28 14:09
Message:
Logged In: YES
user_id=31435
Adding new doc file.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2002-07-28 14:08
Message:
Logged In: YES
user_id=31435
Deleting old doc file.
----------------------------------------------------------------------
Comment By: Anthony Baxter (anthonybaxter)
Date: 2002-07-27 07:23
Message:
Logged In: YES
user_id=29957
PIII Mobile 1.2GHz, 512k cache, 256M, Redhat 7.2, gcc 2.96
(samplesort)
i 2**i *sort \sort /sort 3sort +sort ~sort =sort
!sort
15 32768 0.08 0.01 0.01 0.07 0.01 0.03 0.01
0.08
16 65536 0.18 0.02 0.02 0.17 0.02 0.06 0.01
0.19
17 131072 0.41 0.04 0.04 0.41 0.04 0.16 0.04
0.44
18 262144 0.93 0.09 0.08 0.90 0.10 0.33 0.08
0.97
19 524288 2.04 0.18 0.16 1.98 0.23 0.69 0.17
2.13
20 1048576 4.49 0.36 0.34 4.52 0.43 1.44 0.33
4.65
(timsort)
i 2**i *sort \sort /sort 3sort +sort ~sort =sort
!sort
15 32768 0.08 0.01 0.01 0.00 0.01 0.04 0.00
0.01
16 65536 0.18 0.02 0.02 0.02 0.01 0.07 0.02
0.04
17 131072 0.42 0.03 0.04 0.04 0.04 0.14 0.03
0.08
18 262144 0.95 0.08 0.08 0.09 0.08 0.30 0.07
0.17
19 524288 2.08 0.17 0.16 0.17 0.17 0.63 0.17
0.34
20 1048576 4.56 0.33 0.33 0.33 0.35 1.29 0.33
0.71
PIII Mobile 1.2GHz, 512k cache, 256M, Redhat 7.2, gcc 3.0.4
(samplesort)
i 2**i *sort \sort /sort 3sort +sort ~sort =sort
!sort
15 32768 0.08 0.01 0.01 0.08 0.00 0.02 0.01
0.08
16 65536 0.18 0.01 0.02 0.18 0.01 0.06 0.02
0.19
17 131072 0.41 0.04 0.04 0.39 0.04 0.16 0.04
0.44
18 262144 0.94 0.08 0.08 0.91 0.10 0.33 0.07
0.95
19 524288 2.05 0.17 0.16 2.07 0.20 0.70 0.16
2.11
20 1048576 4.50 0.34 0.32 4.30 0.42 1.41 0.32
4.61
(timsort)
i 2**i *sort \sort /sort 3sort +sort ~sort =sort
!sort
15 32768 0.09 0.01 0.00 0.01 0.01 0.04 0.01
0.01
16 65536 0.18 0.02 0.02 0.02 0.01 0.07 0.02
0.04
17 131072 0.41 0.04 0.04 0.04 0.03 0.14 0.03
0.08
18 262144 0.93 0.08 0.07 0.08 0.08 0.31 0.08
0.16
19 524288 2.07 0.15 0.15 0.16 0.16 0.63 0.16
0.34
20 1048576 4.54 0.33 0.31 0.32 0.33 1.28 0.32
0.67
----------------------------------------------------------------------
Comment By: Anthony Baxter (anthonybaxter)
Date: 2002-07-27 04:20
Message:
Logged In: YES
user_id=29957
Sun Ultra 5, gcc 2.95.2, 512M ram, sunos 5.7.
(sort)
imperial% ./python -O Lib/test/sortperf.py 15 20 1
i 2**i *sort \sort /sort 3sort +sort ~sort =sort
!sort
15 32768 0.29 0.03 0.02 0.29 0.03 0.09 0.02
0.31
16 65536 0.66 0.05 0.05 0.68 0.05 0.20 0.05
0.71
17 131072 1.50 0.11 0.11 1.51 0.12 0.47 0.11
1.60
18 262144 3.25 0.23 0.22 3.37 0.25 1.18 0.22
3.52
19 524288 6.88 0.45 0.43 7.30 0.51 1.91 0.43
7.43
20 1048576 14.90 0.92 0.88 15.49 1.05 3.89 0.90
16.04
(timsort)
imperial% ./python -O Lib/test/sortperf.py 15 20 1
i 2**i *sort \sort /sort 3sort +sort ~sort =sort
!sort
15 32768 0.28 0.02 0.02 0.03 0.02 0.13 0.02
0.05
16 65536 0.59 0.05 0.05 0.06 0.05 0.26 0.05
0.11
17 131072 1.33 0.10 0.09 0.11 0.11 0.54 0.10
0.21
18 262144 2.92 0.22 0.20 0.22 0.21 1.10 0.20
0.44
19 524288 6.33 0.44 0.42 0.43 0.43 2.21 0.41
0.90
20 1048576 13.56 0.89 0.85 0.84 0.87 4.51 0.87
1.82
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2002-07-26 21:24
Message:
Logged In: YES
user_id=31435
I attached timsort.txt, a plain-text detailed description of
the algorithm. After I dies, it's the only clue that will remain
<wink>.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2002-07-26 16:38
Message:
Logged In: YES
user_id=31435
Intrigued by a comment of McIlroy, I tried catenating all
the .c files in Objects and Modules, into one giant file, and
sorted that. msort got a 22% speedup there, suggesting
there's *some* kind of significant pre-existing lexicographic
order (and/or reverse order) in C source files that msort is
able to exploit.
Trying it again on about 1.33 million lines of Python-Dev
archive (including assorted uuencoded attachmets). msort
got a 32% speedup.
I'm not sure what to make of that, but we needed some real
life data here <wink>.
----------------------------------------------------------------------
Comment By: Skip Montanaro (montanaro)
Date: 2002-07-26 15:50
Message:
Logged In: YES
user_id=44345
Pentium III, 450MHz, 256KB L2 cache, Mandrake Linux 8.1, gcc 2.96
L.sort():
i 2**i *sort \sort /sort 3sort +sort ~sort =sort !sort
15 32768 0.32 0.02 0.03 0.30 0.03 0.09 0.03 0.32
16 65536 0.73 0.06 0.05 0.66 0.06 0.20 0.05 0.71
17 131072 1.53 0.11 0.12 1.42 0.13 0.44 0.11 1.51
18 262144 3.28 0.21 0.21 3.09 0.28 0.89 0.21 3.26
19 524288 7.05 0.44 0.42 6.60 0.59 1.81 0.42 7.03
20 1048576 15.30 0.90 0.86 14.10 1.13 3.62 0.86 14.96
L.msort():
i 2**i *sort \sort /sort 3sort +sort ~sort =sort !sort
15 32768 0.32 0.02 0.03 0.03 0.02 0.13 0.02 0.05
16 65536 0.70 0.05 0.06 0.05 0.06 0.27 0.07 0.10
17 131072 1.53 0.09 0.11 0.10 0.11 0.59 0.10 0.21
18 262144 3.27 0.22 0.21 0.23 0.21 1.13 0.21 0.43
19 524288 7.10 0.43 0.45 0.44 0.45 2.27 0.43 0.88
20 1048576 15.03 0.86 0.87 0.87 0.89 4.70 0.89 1.74
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2002-07-26 14:54
Message:
Logged In: YES
user_id=31435
Pentium III, 866 MHz, 16KB L1 D-cache, 16KB L1 I-
cache, 256KB L2 cache, Win98SE, MSVC 6
samplesort
i 2**i *sort \sort /sort 3sort +sort ~sort =sort !sort
15 32768 0.17 0.01 0.01 0.17 0.01 0.05
0.01 0.11
16 65536 0.24 0.02 0.02 0.25 0.02 0.08
0.02 0.24
17 131072 0.53 0.05 0.04 0.49 0.05 0.18
0.04 0.52
18 262144 1.16 0.09 0.09 1.06 0.12 0.37
0.09 1.14
19 524288 2.53 0.18 0.17 2.30 0.24 0.75
0.17 2.47
20 1048576 5.48 0.37 0.35 5.17 0.45 1.51
0.35 5.34
timsort
i 2**i *sort \sort /sort 3sort +sort ~sort =sort !sort
15 32768 0.15 0.03 0.02 0.02 0.01 0.04
0.01 0.02
16 65536 0.23 0.02 0.02 0.02 0.02 0.09
0.02 0.04
17 131072 0.53 0.04 0.04 0.05 0.04 0.19
0.04 0.09
18 262144 1.16 0.09 0.09 0.10 0.09 0.38
0.09 0.19
19 524288 2.54 0.18 0.17 0.18 0.18 0.78
0.17 0.36
20 1048576 5.50 0.36 0.35 0.36 0.37 1.60
0.35 0.73
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2002-07-26 13:52
Message:
Logged In: YES
user_id=31435
Numbers from Marc-Andre Lemburg, "AMD Athlon
1.2GHz/Linux/gcc".
samplesort
i 2**i *sort \sort /sort 3sort +sort ~sort =sort !sort
15 32768 0.07 0.00 0.01 0.09 0.01 0.03
0.01 0.08
16 65536 0.18 0.02 0.02 0.19 0.03 0.07
0.02 0.20
17 131072 0.43 0.05 0.04 0.46 0.05 0.18
0.05 0.48
18 262144 0.99 0.09 0.10 1.04 0.13 0.40
0.09 1.11
19 524288 2.23 0.19 0.21 2.32 0.24 0.83
0.20 2.46
20 1048576 4.96 0.40 0.40 5.41 0.47 1.72
0.40 5.46
samplesort again (run twice by mistake)
i 2**i *sort \sort /sort 3sort +sort ~sort =sort !sort
15 32768 0.08 0.01 0.01 0.09 0.01 0.03
0.00 0.09
16 65536 0.20 0.02 0.01 0.20 0.03 0.07
0.02 0.20
17 131072 0.46 0.06 0.02 0.45 0.05 0.20
0.04 0.49
18 262144 0.99 0.09 0.10 1.09 0.11 0.40
0.12 1.12
19 524288 2.33 0.20 0.20 2.30 0.24 0.83
0.19 2.47
20 1048576 4.89 0.40 0.41 5.37 0.48 1.71
0.38 6.22
timsort
i 2**i *sort \sort /sort 3sort +sort ~sort =sort !sort
15 32768 0.08 0.01 0.01 0.01 0.01 0.03
0.00 0.02
16 65536 0.17 0.02 0.02 0.02 0.02 0.07
0.02 0.06
17 131072 0.41 0.05 0.04 0.05 0.04 0.16
0.04 0.09
18 262144 0.95 0.10 0.10 0.10 0.10 0.33
0.10 0.20
19 524288 2.17 0.20 0.21 0.20 0.21 0.66
0.20 0.44
20 1048576 4.85 0.42 0.40 0.41 0.41 1.37
0.41 0.84
----------------------------------------------------------------------
Comment By: Kevin Jacobs (jacobs99)
Date: 2002-07-26 12:54
Message:
Logged In: YES
user_id=459565
Intel 1266 MHz Penguin III x2 (Dual processor)
512KB cache
Linux 2.4.19-pre1-ac2
gcc 3.1 20020205
samplesort:
i 2**i *sort \sort /sort 3sort +sort ~sort =sort !sort
15 32768 0.07 0.00 0.01 0.06 0.01 0.02 0.00 0.07
16 65536 0.16 0.02 0.01 0.15 0.01 0.06 0.02 0.17
17 131072 0.37 0.04 0.04 0.35 0.04 0.15 0.03 0.38
18 262144 0.84 0.07 0.08 0.80 0.09 0.31 0.07 0.86
19 524288 1.89 0.16 0.15 1.78 0.19 0.66 0.15 1.92
20 1048576 4.12 0.33 0.31 4.07 0.37 1.34 0.31
4.22
timsort:
i 2**i *sort \sort /sort 3sort +sort ~sort =sort !sort
15 32768 0.07 0.01 0.00 0.01 0.01 0.03 0.01 0.01
16 65536 0.17 0.01 0.02 0.01 0.02 0.06 0.02 0.04
17 131072 0.37 0.04 0.03 0.04 0.04 0.13 0.04 0.08
18 262144 0.84 0.07 0.07 0.08 0.08 0.27 0.07 0.16
19 524288 1.89 0.16 0.15 0.15 0.17 0.55 0.15 0.33
20 1048576 4.16 0.32 0.31 0.31 0.32 1.14 0.31
0.66
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2002-07-26 12:30
Message:
Logged In: YES
user_id=31435
Wow! Thanks, Neil! That's impressive, even if I say so
myself <wink>.
----------------------------------------------------------------------
Comment By: Neil Schemenauer (nascheme)
Date: 2002-07-26 12:23
Message:
Logged In: YES
user_id=35752
AMD 1.4 Ghz Athon CPU
L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
L2 Cache: 256K (64 bytes/line)
Linux 2.4.19-pre10-ac1
gcc 2.95.4
samplesort:
i 2**i *sort \sort /sort 3sort +sort ~sort =sort
!sort
15 32768 0.06 0.01 0.01 0.07 0.01 0.03 0.01
0.07
16 65536 0.16 0.02 0.02 0.15 0.02 0.07 0.02
0.17
17 131072 0.37 0.03 0.03 0.39 0.04 0.16 0.04
0.41
18 262144 0.84 0.07 0.08 0.87 0.10 0.34 0.07
0.93
19 524288 1.89 0.16 0.16 1.97 0.21 0.70 0.16
2.08
20 1048576 4.20 0.33 0.34 4.55 0.41 1.45 0.34
4.61
timsort:
i 2**i *sort \sort /sort 3sort +sort ~sort =sort
!sort
15 32768 0.06 0.00 0.01 0.01 0.01 0.03 0.00
0.01
16 65536 0.14 0.02 0.02 0.02 0.02 0.06 0.02
0.04
17 131072 0.35 0.04 0.04 0.04 0.04 0.12 0.04
0.08
18 262144 0.79 0.08 0.08 0.09 0.09 0.27 0.09
0.16
19 524288 1.79 0.17 0.17 0.18 0.17 0.54 0.17
0.33
20 1048576 3.96 0.35 0.34 0.34 0.36 1.12 0.34
0.70
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=587076&group_id=5470