Here is a statistic for most called PyObject_INIT or PyObject_INIT_VAR for types (collected during running Python tests on 32-bit Linux). type count % acc.% builtin_function_or_method 116012007 36.29% 36.29% method 52465386 16.41% 52.70% int 42828741 13.40% 66.09% str 37017098 11.58% 77.67% generator 14026583 4.39% 82.06% list_iterator 8731329 2.73% 84.79% bytes 7217934 2.26% 87.04% tuple_iterator 5042563 1.58% 88.62% float 4672980 1.46% 90.08% set 3319699 1.04% 91.12% _io.StringIO 3000369 0.94% 92.06% str_iterator 2126838 0.67% 92.73% list 2031059 0.64% 93.36% dict 1691993 0.53% 93.89% method-wrapper 1573139 0.49% 94.38% function 1472062 0.46% 94.84% traceback 1388278 0.43% 95.28% tuple 1132071 0.35% 95.63% memoryview 1092173 0.34% 95.97% cell 1049496 0.33% 96.30% managedbuffer 1036889 0.32% 96.63% bytearray 711969 0.22% 96.85% range_iterator 496924 0.16% 97.00% range 483971 0.15% 97.15% super 472447 0.15% 97.30% map 449567 0.14% 97.44% frame 427320 0.13% 97.58% set_iterator 423392 0.13% 97.71% Leaf 398705 0.12% 97.83% symtable 374412 0.12% 97.95% Types for which free lists already are used: builtin_function_or_method, method, float, tuple, list, dict, frame. Some free list implementations (e.g. for tuple) don't call PyObject_INIT/PyObject_INIT_VAR. That is why numbers are such low for tuples. Perhaps it is worth to add free lists for other types: int, str, bytes, generator, list and tuple iterators? Shortened tables for variable-sized objects (that calls PyObject_INIT_VAR): int 42828741 13.40% 0 425353 0.99% 0.99% 1 21399290 49.96% 50.96% 2 10496856 24.51% 75.47% 3 4873346 11.38% 86.85% 4 1021563 2.39% 89.23% 5 1246444 2.91% 92.14% 6 733676 1.71% 93.85% 7 123074 0.29% 94.14% 8 139203 0.33% 94.47% ... bytes 7217934 2.26% 0 842 0.01% 0.01% 1 179469 2.49% 2.50% 2 473306 6.56% 9.06% 3 254968 3.53% 12.59% 4 1169164 16.20% 28.79% 5 72806 1.01% 29.79% 6 128668 1.78% 31.58% 7 169694 2.35% 33.93% 8 155154 2.15% 36.08% 9 67320 0.93% 37.01% 10 51703 0.72% 37.73% 11 42574 0.59% 38.32% 12 108947 1.51% 39.83% 13 40812 0.57% 40.39% 14 126783 1.76% 42.15% 15 37873 0.52% 42.67% 16 447482 6.20% 48.87% 17 194320 2.69% 51.56% 18 251685 3.49% 55.05% 19 159435 2.21% 57.26% 20 212521 2.94% 60.20% ... 31 18751 0.26% 67.32% 32 159781 2.21% 69.54% 33 8332 0.12% 69.65% ... 63 19841 0.27% 79.21% 64 144982 2.01% 81.22% 65 5216 0.07% 81.29% ... 127 1354 0.02% 85.44% 128 376539 5.22% 90.66% 129 17468 0.24% 90.90% ... 255 178 0.00% 92.39% 256 11993 0.17% 92.55% 257 124 0.00% 92.56% ...
On May 9, 2015 5:07 PM, "Serhiy Storchaka" <storchaka@gmail.com> wrote:
https://mail.python.org/mailman/options/python-dev/graffatcolmingov%40gmail.... Can you share how you gathered them so someone could run them on a 64-bit build?
On 10.05.15 02:25, Ian Cordasco wrote:
Can you share how you gathered them so someone could run them on a 64-bit build?
This is quick and dirty patch. It generates 8 GB log file! patch --merge -p1 <PyObject_INIT_stat.diff make -s -j2 ./python -Wd -m test.regrtest -w -uall 2>PyObject_INIT.log python3 PyObject_INIT_stat.py <PyObject_INIT.log >PyObject_INIT.stat Perhaps compiling with COUNT_ALLOCS will produce similar statistic for types (but without statistics for sizes) and should be much faster.
On 05/09/2015 11:22 PM, Serhiy Storchaka wrote:
I ran it under 64-bit Linux. Actually it generated a 10GB log file. It was stalled at test_multiprocessing_fork for five hours so I killed it. What follows are the (apparently) partial results. I think it makes a good case for a one-element freelist for 64-bit builds. //arry/ -- type count % acc.% builtin_function_or_method 131028598 37.25% 37.25% method 52062496 14.80% 52.05% int 47600237 13.53% 65.59% str 43841584 12.46% 78.05% generator 14038624 3.99% 82.04% float 8617481 2.45% 84.49% list_iterator 8214121 2.34% 86.83% bytes 7884898 2.24% 89.07% tuple_iterator 5172174 1.47% 90.54% _io.StringIO 3482733 0.99% 91.53% set 3335168 0.95% 92.48% str_iterator 2856373 0.81% 93.29% list 2245981 0.64% 93.93% dict 1682253 0.48% 94.41% method-wrapper 1574412 0.45% 94.86% function 1475393 0.42% 95.28% traceback 1417094 0.40% 95.68% tuple 1181899 0.34% 96.01% memoryview 1103226 0.31% 96.33% cell 1047245 0.30% 96.63% managedbuffer 1044764 0.30% 96.92% bytearray 714337 0.20% 97.13% range_iterator 498240 0.14% 97.27% range 485325 0.14% 97.41% super 473542 0.13% 97.54% map 446608 0.13% 97.67% frame 426570 0.12% 97.79% set_iterator 424526 0.12% 97.91% Leaf 391824 0.11% 98.02% symtable 376815 0.11% 98.13% int 47600237 13.53% 0 294964 0.62% 0.62% 1 36135772 75.92% 76.53% 2 4504046 9.46% 86.00% 3 2109837 4.43% 90.43% 4 1277995 2.68% 93.11% 5 542775 1.14% 94.25% 6 485451 1.02% 95.27% ... bytes 7884898 2.24% 0 849 0.01% 0.01% 1 250357 3.18% 3.19% 2 450310 5.71% 8.90% 3 259659 3.29% 12.19% 4 1157554 14.68% 26.87% 5 77493 0.98% 27.85% 6 139816 1.77% 29.63% 7 165399 2.10% 31.72% 8 191821 2.43% 34.16% 9 63009 0.80% 34.96% 10 48751 0.62% 35.57% 11 50505 0.64% 36.22% 12 94186 1.19% 37.41% 13 33927 0.43% 37.84% 14 123546 1.57% 39.41% 15 36565 0.46% 39.87% 16 447183 5.67% 45.54% 17 186609 2.37% 47.91% 18 1301737 16.51% 64.42% ... tuple 1181899 0.34% 0 47 0.00% 0.00% 1 120156 10.17% 10.17% 2 340983 28.85% 39.02% 3 80924 6.85% 45.87% 4 78908 6.68% 52.54% 5 35502 3.00% 55.55% 6 171292 14.49% 70.04% 7 136474 11.55% 81.59% 8 48435 4.10% 85.69% ...
Here is comparable statistic collected from tests ran with an executable buil with COUNT_ALLOCS. type count % acc.% tuple 448855278 29.50% 29.50% frame 203515969 13.38% 42.88% str 182658237 12.01% 54.89% builtin_function_or_method 156724634 10.30% 65.19% int 106561963 7.00% 72.19% method 88269762 5.80% 78.00% list 50340630 3.31% 81.31% slice 36650028 2.41% 83.71% dict 34429310 2.26% 85.98% generator 33035375 2.17% 88.15% bytes 29230573 1.92% 90.07% function 24953392 1.64% 91.71% list_iterator 21236155 1.40% 93.11% tuple_iterator 16800947 1.10% 94.21% cell 16369317 1.08% 95.29% float 7079162 0.47% 95.75% _sre.SRE_Match 6342612 0.42% 96.17% set 5322829 0.35% 96.52% TokenInfo 5077251 0.33% 96.85% code 3643664 0.24% 97.09% traceback 3510709 0.23% 97.32% memoryview 2860799 0.19% 97.51% managedbuffer 2762975 0.18% 97.69% method-wrapper 2590642 0.17% 97.86% Name 1681233 0.11% 97.97% bytearray 1598429 0.11% 98.08% _io.StringIO 1439456 0.09% 98.17% weakref 1341485 0.09% 98.26% super 911811 0.06% 98.32% range 798254 0.05% 98.37%
Serhiy Storchaka schrieb am 09.05.2015 um 21:01:
Here is a statistic for most called PyObject_INIT or PyObject_INIT_VAR for types (collected during running Python tests on 32-bit Linux).
I'm aware that this includes lots of tests for the Python code in the stdlib, so these numbers are most likely not too far from what real-world code would give, but wouldn't it be even better to collect statistics from a (quick) benchmark suite run? Test suites tend to be very regular, flat and broad, unlike most production code. Stefan
On May 9, 2015 5:07 PM, "Serhiy Storchaka" <storchaka@gmail.com> wrote:
https://mail.python.org/mailman/options/python-dev/graffatcolmingov%40gmail.... Can you share how you gathered them so someone could run them on a 64-bit build?
On 10.05.15 02:25, Ian Cordasco wrote:
Can you share how you gathered them so someone could run them on a 64-bit build?
This is quick and dirty patch. It generates 8 GB log file! patch --merge -p1 <PyObject_INIT_stat.diff make -s -j2 ./python -Wd -m test.regrtest -w -uall 2>PyObject_INIT.log python3 PyObject_INIT_stat.py <PyObject_INIT.log >PyObject_INIT.stat Perhaps compiling with COUNT_ALLOCS will produce similar statistic for types (but without statistics for sizes) and should be much faster.
On 05/09/2015 11:22 PM, Serhiy Storchaka wrote:
I ran it under 64-bit Linux. Actually it generated a 10GB log file. It was stalled at test_multiprocessing_fork for five hours so I killed it. What follows are the (apparently) partial results. I think it makes a good case for a one-element freelist for 64-bit builds. //arry/ -- type count % acc.% builtin_function_or_method 131028598 37.25% 37.25% method 52062496 14.80% 52.05% int 47600237 13.53% 65.59% str 43841584 12.46% 78.05% generator 14038624 3.99% 82.04% float 8617481 2.45% 84.49% list_iterator 8214121 2.34% 86.83% bytes 7884898 2.24% 89.07% tuple_iterator 5172174 1.47% 90.54% _io.StringIO 3482733 0.99% 91.53% set 3335168 0.95% 92.48% str_iterator 2856373 0.81% 93.29% list 2245981 0.64% 93.93% dict 1682253 0.48% 94.41% method-wrapper 1574412 0.45% 94.86% function 1475393 0.42% 95.28% traceback 1417094 0.40% 95.68% tuple 1181899 0.34% 96.01% memoryview 1103226 0.31% 96.33% cell 1047245 0.30% 96.63% managedbuffer 1044764 0.30% 96.92% bytearray 714337 0.20% 97.13% range_iterator 498240 0.14% 97.27% range 485325 0.14% 97.41% super 473542 0.13% 97.54% map 446608 0.13% 97.67% frame 426570 0.12% 97.79% set_iterator 424526 0.12% 97.91% Leaf 391824 0.11% 98.02% symtable 376815 0.11% 98.13% int 47600237 13.53% 0 294964 0.62% 0.62% 1 36135772 75.92% 76.53% 2 4504046 9.46% 86.00% 3 2109837 4.43% 90.43% 4 1277995 2.68% 93.11% 5 542775 1.14% 94.25% 6 485451 1.02% 95.27% ... bytes 7884898 2.24% 0 849 0.01% 0.01% 1 250357 3.18% 3.19% 2 450310 5.71% 8.90% 3 259659 3.29% 12.19% 4 1157554 14.68% 26.87% 5 77493 0.98% 27.85% 6 139816 1.77% 29.63% 7 165399 2.10% 31.72% 8 191821 2.43% 34.16% 9 63009 0.80% 34.96% 10 48751 0.62% 35.57% 11 50505 0.64% 36.22% 12 94186 1.19% 37.41% 13 33927 0.43% 37.84% 14 123546 1.57% 39.41% 15 36565 0.46% 39.87% 16 447183 5.67% 45.54% 17 186609 2.37% 47.91% 18 1301737 16.51% 64.42% ... tuple 1181899 0.34% 0 47 0.00% 0.00% 1 120156 10.17% 10.17% 2 340983 28.85% 39.02% 3 80924 6.85% 45.87% 4 78908 6.68% 52.54% 5 35502 3.00% 55.55% 6 171292 14.49% 70.04% 7 136474 11.55% 81.59% 8 48435 4.10% 85.69% ...
Here is comparable statistic collected from tests ran with an executable buil with COUNT_ALLOCS. type count % acc.% tuple 448855278 29.50% 29.50% frame 203515969 13.38% 42.88% str 182658237 12.01% 54.89% builtin_function_or_method 156724634 10.30% 65.19% int 106561963 7.00% 72.19% method 88269762 5.80% 78.00% list 50340630 3.31% 81.31% slice 36650028 2.41% 83.71% dict 34429310 2.26% 85.98% generator 33035375 2.17% 88.15% bytes 29230573 1.92% 90.07% function 24953392 1.64% 91.71% list_iterator 21236155 1.40% 93.11% tuple_iterator 16800947 1.10% 94.21% cell 16369317 1.08% 95.29% float 7079162 0.47% 95.75% _sre.SRE_Match 6342612 0.42% 96.17% set 5322829 0.35% 96.52% TokenInfo 5077251 0.33% 96.85% code 3643664 0.24% 97.09% traceback 3510709 0.23% 97.32% memoryview 2860799 0.19% 97.51% managedbuffer 2762975 0.18% 97.69% method-wrapper 2590642 0.17% 97.86% Name 1681233 0.11% 97.97% bytearray 1598429 0.11% 98.08% _io.StringIO 1439456 0.09% 98.17% weakref 1341485 0.09% 98.26% super 911811 0.06% 98.32% range 798254 0.05% 98.37%
Serhiy Storchaka schrieb am 09.05.2015 um 21:01:
Here is a statistic for most called PyObject_INIT or PyObject_INIT_VAR for types (collected during running Python tests on 32-bit Linux).
I'm aware that this includes lots of tests for the Python code in the stdlib, so these numbers are most likely not too far from what real-world code would give, but wouldn't it be even better to collect statistics from a (quick) benchmark suite run? Test suites tend to be very regular, flat and broad, unlike most production code. Stefan
participants (4)
-
Ian Cordasco -
Larry Hastings -
Serhiy Storchaka -
Stefan Behnel