So I enhanced the PVM dynamic execution profiler to keep track of quads of instructions rather than just pairs. Running it on PyStone, I get the following results: Note: score is frequency * len(trace) http://www.bitfurnace.com/python/dxstats-traces.txt score trace 7204142 LOAD_FAST 3701796 LOAD_FAST, LOAD_FAST 3453243 COMPARE_OP, JUMP_IF_FALSE, POP_TOP 2701110 LOAD_GLOBAL 2603086 JUMP_IF_FALSE, POP_TOP 2402196 COMPARE_OP, JUMP_IF_FALSE 2302782 LOAD_CONST 2251111 STORE_FAST 2200676 LOAD_GLOBAL, COMPARE_OP, JUMP_IF_FALSE, POP_TOP 2101580 LOAD_FAST, LOAD_CONST 2100896 STORE_FAST, LOAD_FAST 2000188 COMPARE_OP, JUMP_IF_FALSE, POP_TOP, LOAD_GLOBAL 1800639 LOAD_FAST, LOAD_CONST, BINARY_ADD 1650507 LOAD_GLOBAL, COMPARE_OP, JUMP_IF_FALSE 1650177 JUMP_IF_FALSE, POP_TOP, LOAD_GLOBAL 1600986 LOAD_GLOBAL, LOAD_FAST 1600492 LOAD_FAST, LOAD_GLOBAL, COMPARE_OP, JUMP_IF_FALSE 1500536 LOAD_CONST, BINARY_ADD 1500339 STORE_FAST, LOAD_FAST, LOAD_FAST 1500282 LOAD_FAST, STORE_ATTR 1452043 POP_TOP 1400524 LOAD_FAST, LOAD_CONST, BINARY_ADD, STORE_FAST 1400056 LOAD_FAST, LOAD_FAST, BINARY_SUBSCR, LOAD_FAST 1351596 JUMP_IF_FALSE 1300164 POP_TOP, LOAD_GLOBAL 1251127 COMPARE_OP 1203536 COMPARE_OP, JUMP_IF_FALSE, POP_TOP, LOAD_FAST ... certainly, making JUMPs consume the top-of-stack still seems worthwhile other possibilities LOAD_FAST+FAST == LOAD_FAST, LOAD_FAST JUMP_IF_CMP == COMPARE_OP, JUMP_IF_FALSE -- consumes two items off the stack LOAD_FAST+CONST == LOAD_FAST, LOAD_CONST ADD_CONST == LOAD_CONST, BINARY_ADD FAST_SUBSCR == LOAD_FAST, LOAD_FAST, BINARY_SUBSCR If anyone has any suggestions of a better app than PyStone to profile, I would be grateful.
Damien Morton wrote:
So I enhanced the PVM dynamic execution profiler to keep track of quads of instructions rather than just pairs. Running it on PyStone, I get the following results:
pystone is *not* your typical Python application. If you want some interesting data, I'd suggest to use applications like Zope, PySol, Mailman, etc. as basis. The easiest way to do this is by instrumenting the Python interpreter used by all these applications and have it write its data to a pickle every now and then. Here's an example from the Python 1.5 days: Opcode frequencies: ------------------------------------------------------------------------ SET_LINENO(127) : 12036 ================================ LOAD_FAST(124) : 10366 =========================== LOAD_ATTR(105) : 4565 ============ LOAD_CONST(100) : 4549 ============ POP_TOP( 1) : 3756 ========= STORE_FAST(125) : 3580 ========= CALL_FUNCTION(131) : 2558 ====== JUMP_IF_FALSE(111) : 2142 ===== LOAD_GLOBAL(116) : 2056 ===== COMPARE_OP(106) : 2001 ===== RETURN_VALUE( 83) : 1593 ==== FOR_LOOP(114) : 1363 === BINARY_ADD( 23) : 1133 === STORE_NAME( 90) : 1062 == JUMP_ABSOLUTE(113) : 835 == MAKE_FUNCTION(132) : 753 == SETUP_LOOP(120) : 568 = POP_BLOCK( 87) : 536 = JUMP_FORWARD(110) : 503 = 32( 32) : 442 = JUMP_IF_TRUE(112) : 438 = LOAD_NAME(101) : 425 = DUP_TOP( 4) : 421 = UNARY_NEGATIVE( 11) : 411 = SETUP_EXCEPT(121) : 410 = 31( 31) : 409 = -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 27 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
Python UK 2003, Oxford: 33 days left EuroPython 2003, Charleroi, Belgium: 117 days left
Im not sure how I could properly excersise Zope, and Mailman doesn't run under windows. PySol is do-able. Any other suggestions as to apps worthy of profiling? Would the python standard library unit tests qualify?
-----Original Message----- From: M.-A. Lemburg [mailto:mal@lemburg.com] Sent: Thursday, 27 February 2003 03:41 To: Damien Morton Cc: python-dev@python.org Subject: Re: [Python-Dev] Dynamic bytecode analysis
Damien Morton wrote:
So I enhanced the PVM dynamic execution profiler to keep track of quads of instructions rather than just pairs. Running it on PyStone, I get the following results:
pystone is *not* your typical Python application. If you want some interesting data, I'd suggest to use applications like Zope, PySol, Mailman, etc. as basis.
The easiest way to do this is by instrumenting the Python interpreter used by all these applications and have it write its data to a pickle every now and then. Here's an example from the Python 1.5 days:
Opcode frequencies: -------------------------------------------------------------- ---------- SET_LINENO(127) : 12036 ================================ LOAD_FAST(124) : 10366 =========================== LOAD_ATTR(105) : 4565 ============ LOAD_CONST(100) : 4549 ============ POP_TOP( 1) : 3756 ========= STORE_FAST(125) : 3580 ========= CALL_FUNCTION(131) : 2558 ====== JUMP_IF_FALSE(111) : 2142 ===== LOAD_GLOBAL(116) : 2056 ===== COMPARE_OP(106) : 2001 ===== RETURN_VALUE( 83) : 1593 ==== FOR_LOOP(114) : 1363 === BINARY_ADD( 23) : 1133 === STORE_NAME( 90) : 1062 == JUMP_ABSOLUTE(113) : 835 == MAKE_FUNCTION(132) : 753 == SETUP_LOOP(120) : 568 = POP_BLOCK( 87) : 536 = JUMP_FORWARD(110) : 503 = 32( 32) : 442 = JUMP_IF_TRUE(112) : 438 = LOAD_NAME(101) : 425 = DUP_TOP( 4) : 421 = UNARY_NEGATIVE( 11) : 411 = SETUP_EXCEPT(121) : 410 = 31( 31) : 409 =
-- Marc-Andre Lemburg eGenix.com
Professional Python Software directly from the Source (#1, Feb 27 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
__________ Python UK 2003, Oxford: 33 days left EuroPython 2003, Charleroi, Belgium: 117 days left
damien morton wrote:
Im not sure how I could properly excersise Zope, and Mailman doesn't run under windows. PySol is do-able. Any other suggestions as to apps worthy of profiling?
Anything with more than 10kloc Python code should do, I guess.
Would the python standard library unit tests qualify?
Nope :-)
-----Original Message----- From: M.-A. Lemburg [mailto:mal@lemburg.com] Sent: Thursday, 27 February 2003 03:41 To: Damien Morton Cc: python-dev@python.org Subject: Re: [Python-Dev] Dynamic bytecode analysis
Damien Morton wrote:
So I enhanced the PVM dynamic execution profiler to keep track of quads of instructions rather than just pairs. Running it on
PyStone, I
get the following results:
pystone is *not* your typical Python application. If you want some interesting data, I'd suggest to use applications like Zope, PySol, Mailman, etc. as basis.
The easiest way to do this is by instrumenting the Python interpreter used by all these applications and have it write its data to a pickle every now and then. Here's an example from the Python 1.5 days:
Opcode frequencies: -------------------------------------------------------------- ---------- SET_LINENO(127) : 12036 ================================ LOAD_FAST(124) : 10366 =========================== LOAD_ATTR(105) : 4565 ============ LOAD_CONST(100) : 4549 ============ POP_TOP( 1) : 3756 ========= STORE_FAST(125) : 3580 ========= CALL_FUNCTION(131) : 2558 ====== JUMP_IF_FALSE(111) : 2142 ===== LOAD_GLOBAL(116) : 2056 ===== COMPARE_OP(106) : 2001 ===== RETURN_VALUE( 83) : 1593 ==== FOR_LOOP(114) : 1363 === BINARY_ADD( 23) : 1133 === STORE_NAME( 90) : 1062 == JUMP_ABSOLUTE(113) : 835 == MAKE_FUNCTION(132) : 753 == SETUP_LOOP(120) : 568 = POP_BLOCK( 87) : 536 = JUMP_FORWARD(110) : 503 = 32( 32) : 442 = JUMP_IF_TRUE(112) : 438 = LOAD_NAME(101) : 425 = DUP_TOP( 4) : 421 = UNARY_NEGATIVE( 11) : 411 = SETUP_EXCEPT(121) : 410 = 31( 31) : 409 =
-- Marc-Andre Lemburg eGenix.com
Professional Python Software directly from the Source (#1, Feb 27 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
__________ Python UK 2003, Oxford: 33 days left EuroPython 2003, Charleroi, Belgium: 117 days left
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev
-- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 27 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
Python UK 2003, Oxford: 33 days left EuroPython 2003, Charleroi, Belgium: 117 days left
Would the python standard library unit tests qualify?
Definitely not. It has highly atypical behavior. --Guido van Rossum (home page: http://www.python.org/~guido/)
damien> Im not sure how I could properly excersise Zope, and Mailman damien> doesn't run under windows. PySol is do-able. Any other damien> suggestions as to apps worthy of profiling? Zope can be exercised by simply creating a shell script which retrieves a given set of URLs from the server. Obviously, what you fetch will depend on what applications have been built using the server. Using sitecustomize, all apps can be instrumented without modification to dump their instruction counts at program termination. Here's my new sitecustomize.py, which dumps to my dxp server on manatee.mojam.com (please change EMAIL to suit if you decide to use this): import atexit import sys, os, xmlrpclib EMAIL = 'skip@pobox.com' if not hasattr(sys, 'getdxp'): print >> sys.stderr, "instruction count tracking not available" else: def rle(l): newl = [] lastel = None count = 0 for elt in l: if elt == lastel: count = count + 1 continue elif lastel is not None: if isinstance(lastel, list): lastel = rle(lastel) newl.append([lastel, count]) lastel = elt count = 1 if isinstance(lastel, list): lastel = rle(lastel) newl.append([lastel, count]) return newl def rld(l): newl = [] for elt, count in l: if isinstance(elt, list): elt = rld(elt) newl.extend([elt]*count) return newl def send_instruction_counts(email): print >> sys.stderr, "sending instruction counts to manatee" appname = os.path.basename(sys.argv[0]) or "interactive" dxpserver = xmlrpclib.Server('http://manatee.mojam.com:7304') dxpserver.add_dx_info(appname, email, sys.version_info[:3], rle(sys.getdxp())) atexit.register(send_instruction_counts, EMAIL) You can grab data from the server like so: from sitecustomize import rld x = dxpserver.get_dxp_info([2,3,0]) counts = rld(x['counts']) pairs = rld(x['pairs']) Skip
On Thu, 2003-02-27 at 04:16, damien morton wrote:
Im not sure how I could properly excersise Zope, and Mailman doesn't run under windows. PySol is do-able. Any other suggestions as to apps worthy of profiling?
Would the python standard library unit tests qualify?
Use the compiler package in the standard library to compile a non-trivial chunk of Python code -- like the compiler package. Use one of the standard XML or HTML parsers to parse a large file. Jeremy
mal> pystone is *not* your typical Python application. If you want some mal> interesting data, I'd suggest to use applications like Zope, PySol, mal> Mailman, etc. as basis. Let me mention once again that I have an XML-RPC server available to which applications can connect and upload their dynamic opcode frequencies. This can be done more-or-less transparently using the atexit module. Others can then fetch aggregate information for use like Damien's. It's running at dxp = xmlrpclib.ServerProxy("http://manatee.mojam.com:7304") To get a usage message, execute print dxp.usage() I've run it for the past couple months, since my last announcement here and in c.l.py but have yet to pick up any data. I'm going to take a dose of my own medicine and start running Python on my development machine with DYNAMIC_EXCUTION_PROFILE and DXPAIRS defined and modify my interactive setup and a couple applications I run frequently to dump their instruction counts. Skip
participants (6)
-
damien morton
-
Damien Morton
-
Guido van Rossum
-
Jeremy Hylton
-
M.-A. Lemburg
-
Skip Montanaro