Mailman 3 December 2011 - pypy-dev

PyPy Tool Chain
by Rinu Boney Dec. 10, 2011

Dec. 10, 2011

Can almost any language be implemented in RPython and be faster ? ( like can we implement a langugae like java ? )

4 3

Concerning bounded number of classes created
by Alexander Golec Dec. 10, 2011

Dec. 10, 2011

Hey all, Sorry to keep asking questions on the mailing list, but I've been doing my semester research project on PyPy, and my deadline is beginning to loom... I have a question concerning what I think is a conflict between the RPython coding guide and the 2005 EU paper. The paper states: Our approach goes further and analyses live programs in memory: the program is allowed to contain fully dynamic sections, as long as these sections are entered a bounded num- ber of times. For example, the source code of the PyPy interpreter, which is itself written in this bounded-dynamism style, makes extensive use of the fact that it is possible to build new classes at any point in time -- not just during an initialisation phase -- as long as the number of new classes is bounded. For example, interpreter/gateway.py builds a custom wrapper class corresponding to each function that a particular variable can reference. There is a finite num- ber of functions in total, so this can only create a finite number of extra wrapper classes. While the coding guide says: definitions run-time definition of classes or functions is not allowed. I'm I understood the paper correctly, any piece of code can create zero or one class, but not infinity. However, this doesn't seem to jive with what the coding guide says. Alex

3 6

In RPython, what is meant by 'live code objects'?
by Alexander Golec Dec. 10, 2011

Dec. 10, 2011

Hey all, When the RPython receives the code objects, are all the objects that the interpreter would push and pop on the stack already present? Or does the abstract interpreter handle creating those objects as well? In other words, is interpretation abstract from the start of the program to the end? Is the interpreter concrete until the entry point, at which point it switches to the abstract interpreter? Or is the RPython compiler not responsible for interpreting the initialization portion, just for the bit after the entry point? Alex

2 1

Could someone give me an idea of what is where with RPython?
by Alexander Golec Dec. 10, 2011

Dec. 10, 2011

Hey all, I have the following conception of the RPython toolchain: - RPython translator is written in RPython, meaning it can be interpreted by CPython, and then run as a standalone executable once translated - The translator runs the standard interpreter on the input program. - The input program has some marker to say 'alright, I'm initialized, begin translating,' at which point execution stops and RPython starts - RPython kicks in, performs flow object space analysis on the initialized code objects. - Type annotation occurs based on the initialized types - The annotated code is passed down to the backend-facing code generator Is this correct? Do I have a correct basic understanding of what happens where? Alex

3 9

Clang benchmarks
by Justin Noah Dec. 9, 2011

Dec. 9, 2011

Here are the llvm/clang build using the shadowstack gc. What do you think? Also, I will be downloading the prebuilt binary and running benchmarks and post them as well. Report on Linux infinity 3.1.1-gentoo #1 SMP PREEMPT Sun Nov 20 03:57:03 PST 2011 x86_64 AMD Turion(tm) II Dual-Core Mobile M520 Total CPU cores: 2 ### ai ### Min: 0.140570 -> 0.128204: 1.0965x faster Avg: 0.158339 -> 0.148346: 1.0674x faster Significant (t=2.351561, a=0.95) Stddev: 0.01762 -> 0.02434: 1.3809x larger ### bm_chameleon ### Min: 0.051691 -> 0.051436: 1.0050x faster Avg: 0.062364 -> 0.065282: 1.0468x slower Not significant Stddev: 0.01720 -> 0.01883: 1.0947x larger ### bm_mako ### Min: 0.152917 -> 0.153566: 1.0042x slower Avg: 0.169574 -> 0.169380: 1.0011x faster Not significant Stddev: 0.02681 -> 0.02674: 1.0025x smaller ### chaos ### Min: 0.024781 -> 0.024249: 1.0219x faster Avg: 0.037913 -> 0.037324: 1.0158x faster Not significant Stddev: 0.07050 -> 0.07000: 1.0072x smaller ### crypto_pyaes ### Min: 0.121740 -> 0.121273: 1.0038x faster Avg: 0.136873 -> 0.136358: 1.0038x faster Not significant Stddev: 0.05667 -> 0.05649: 1.0033x smaller ### django ### Min: 0.104028 -> 0.100156: 1.0387x faster Avg: 0.116562 -> 0.114053: 1.0220x faster Significant (t=2.133187, a=0.95) Stddev: 0.00590 -> 0.00586: 1.0077x smaller ### fannkuch ### Min: 0.514067 -> 0.536132: 1.0429x slower Avg: 0.520998 -> 0.542397: 1.0411x slower Significant (t=-6.626659, a=0.95) Stddev: 0.01654 -> 0.01575: 1.0501x smaller ### float ### Min: 0.085486 -> 0.084867: 1.0073x faster Avg: 0.106569 -> 0.104426: 1.0205x faster Not significant Stddev: 0.01745 -> 0.01596: 1.0931x smaller ### go ### Min: 0.271402 -> 0.269337: 1.0077x faster Avg: 0.483500 -> 0.483173: 1.0007x faster Not significant Stddev: 0.19943 -> 0.19894: 1.0025x smaller ### html5lib ### Min: 5.856357 -> 5.745086: 1.0194x faster Avg: 7.952101 -> 7.824551: 1.0163x faster Not significant Stddev: 2.80613 -> 2.76699: 1.0141x smaller ### json_bench ### Min: 3.525704 -> 3.533075: 1.0021x slower Avg: 3.563207 -> 3.577035: 1.0039x slower Not significant Stddev: 0.14274 -> 0.15135: 1.0604x larger ### meteor-contest ### Min: 0.335457 -> 0.336462: 1.0030x slower Avg: 0.343577 -> 0.344685: 1.0032x slower Not significant Stddev: 0.01572 -> 0.01579: 1.0045x larger ### nbody_modified ### Min: 0.085591 -> 0.085202: 1.0046x faster Avg: 0.087704 -> 0.087397: 1.0035x faster Not significant Stddev: 0.00601 -> 0.00633: 1.0538x larger ### pyflate-fast ### Min: 1.038416 -> 1.036536: 1.0018x faster Avg: 1.078068 -> 1.086279: 1.0076x slower Not significant Stddev: 0.02674 -> 0.02464: 1.0854x smaller ### raytrace-simple ### Min: 0.066854 -> 0.067021: 1.0025x slower Avg: 0.081450 -> 0.081594: 1.0018x slower Not significant Stddev: 0.02581 -> 0.02637: 1.0218x larger ### richards ### Min: 0.008039 -> 0.007999: 1.0050x faster Avg: 0.008915 -> 0.008894: 1.0024x faster Not significant Stddev: 0.00269 -> 0.00267: 1.0065x smaller ### rietveld ### Min: 0.243249 -> 0.245089: 1.0076x slower Avg: 0.614075 -> 0.609522: 1.0075x faster Not significant Stddev: 0.59714 -> 0.57957: 1.0303x smaller ### slowspitfire ### Min: 0.677112 -> 0.684585: 1.0110x slower Avg: 0.731244 -> 0.721026: 1.0142x faster Significant (t=2.087466, a=0.95) Stddev: 0.02165 -> 0.02700: 1.2469x larger ### spambayes ### Min: 0.151896 -> 0.151275: 1.0041x faster Avg: 0.292433 -> 0.293944: 1.0052x slower Not significant Stddev: 0.12329 -> 0.12527: 1.0161x larger ### spectral-norm ### Min: 0.028405 -> 0.028729: 1.0114x slower Avg: 0.032538 -> 0.033337: 1.0245x slower Not significant Stddev: 0.01354 -> 0.01373: 1.0144x larger ### spitfire ### Min: 9.910000 -> 10.030000: 1.0121x slower Avg: 10.039000 -> 10.147800: 1.0108x slower Significant (t=-5.090227, a=0.95) Stddev: 0.11163 -> 0.10189: 1.0957x smaller ### spitfire_cstringio ### Min: 4.920000 -> 4.920000: no change Avg: 4.955000 -> 4.956400: 1.0003x slower Not significant Stddev: 0.06225 -> 0.07056: 1.1336x larger ### sympy_expand ### Min: 1.990902 -> 2.037110: 1.0232x slower Avg: 2.497677 -> 2.532886: 1.0141x slower Not significant Stddev: 0.97914 -> 0.96325: 1.0165x smaller ### sympy_integrate ### Min: 4.294102 -> 4.395819: 1.0237x slower Avg: 6.417529 -> 6.457441: 1.0062x slower Not significant Stddev: 2.47638 -> 2.49616: 1.0080x larger ### sympy_str ### Min: 0.907993 -> 0.889144: 1.0212x faster Avg: 1.978583 -> 1.950795: 1.0142x faster Not significant Stddev: 1.11382 -> 1.09577: 1.0165x smaller ### sympy_sum ### Min: 1.349965 -> 1.286402: 1.0494x faster Avg: 1.730386 -> 1.727202: 1.0018x faster Not significant Stddev: 0.54941 -> 0.53929: 1.0188x smaller ### telco ### Min: 0.099984 -> 0.099985: 1.0000x slower Avg: 0.120782 -> 0.121002: 1.0018x slower Not significant Stddev: 0.05775 -> 0.05797: 1.0038x larger ### trans_annotate ### Raw results: [1368.2] None ### trans_rtype ### Raw results: [954.1] None ### trans_backendopt ### Raw results: [433.3] None ### trans_database ### Raw results: [586.7] None ### trans_source ### Raw results: [635.9] None ### twisted_iteration ### Min: 0.013585 -> 0.013649: 1.0047x slower Avg: 0.013755 -> 0.013809: 1.0039x slower Not significant Stddev: 0.00015 -> 0.00014: 1.0542x smaller ### twisted_names ### Min: 0.007508 -> 0.007530: 1.0030x slower Avg: 0.007914 -> 0.007958: 1.0056x slower Not significant Stddev: 0.00019 -> 0.00020: 1.0402x larger ### twisted_pb ### Min: 0.038095 -> 0.037879: 1.0057x faster Avg: 0.040128 -> 0.039598: 1.0134x faster Significant (t=2.789748, a=0.95) Stddev: 0.00101 -> 0.00089: 1.1368x smaller ### twisted_tcp ### Min: 1.128350 -> 1.122807: 1.0049x faster Avg: 1.167165 -> 1.153992: 1.0114x faster Significant (t=2.337046, a=0.95) Stddev: 0.02843 -> 0.02794: 1.0174x smaller Here is the output of "cat /proc/cpuinfo | grep cache," I have attached the full output of cpuinfo. -- - Justin Noah

2 2

{Positioning- URL - Review}
by Henry Mithchell Dec. 8, 2011

Dec. 8, 2011

pypy-dev(a)codespeak.net : Your web site is really good but you could be missing out on a lot of online business because of where your site shows up on the major search directories. A few simple changes could greatly increase your web traffic and your bottom line. Reply to us and we will give you a free analysis of your site and show you what will make the difference for your business. Include the best way to reach you with the results. Sincerely, Henry Mitchell

1 0

RPython
by Rinu Boney Dec. 8, 2011

Dec. 8, 2011

[ forgive me if it is a dumb question ] is there any article or a tutorial that shows how to use the rpython toolkit for a beginner ( just to hack around and learn stuff ) ? what is the relation between rpythonic ( http://code.google.com/p/rpythonic/ ) and rpython ? where can i get more information on rpython ( i have already seen what is written in the coding guide! ) ? - Rinu.

3 2

pypy/bin/checkmodule.py
by Armin Rigo Dec. 7, 2011

Dec. 7, 2011

Hi all, I fixed pypy/bin/checkmodules.py (I actually rewrote it from scratch). It may no longer accept the _clr module, I didn't check, but now it accepts a good number of other built-in modules. For example micronumpy. So I would recommend micronumpy developers to use it before they check in :-) It gives a "translates / does not translate" answer in 30 seconds. A bientôt, Armin.

2 4

Building lxml
by nitralime Dec. 7, 2011

Dec. 7, 2011

Hello! I'v tried to build lxml 2.3.2 with pypy 1.7 (linux64) and got the following error: ------------------------------------------------------------------------------------ $ pypy setup.py build_ext -i pypy: /usr/lib/libssl.so.0.9.8: no version information available (required by pypy) pypy: /usr/lib/libcrypto.so.0.9.8: no version information available (required by pypy) Building lxml version 2.3.2. Building without Cython. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib running build_ext building 'lxml.etree' extension creating build creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/src creating build/temp.linux-x86_64-2.7/src/lxml cc -fPIC -Wimplicit -I/usr/include/libxml2 -I/usr/local/pypy/include -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -w src/lxml/lxml.etree.c: In function ‘__Pyx_PyObject_Pop’: src/lxml/lxml.etree.c:3683:16: error: ‘PyListObject’ has no member named ‘allocated’ src/lxml/lxml.etree.c:3688:30: error: ‘PySet_Type’ undeclared (first use in this function) src/lxml/lxml.etree.c:3688:30: note: each undeclared identifier is reported only once for each function it appears in src/lxml/lxml.etree.c: In function ‘__pyx_f_4lxml_5etree_18_FileReaderContext__readDtd’: src/lxml/lxml.etree.c:72226:7: error: ‘_save’ undeclared (first use in this function) .... ... error: command 'cc' failed with exit status 1 ------------------------------------------------------------------------------------ Next I have tried the following (which is probabely a "nonsense"): ------------------------------------------------------------------------------------ $ pypy setup.py build_ext -i -I/usr/include/python2.7 pypy: /usr/lib/libssl.so.0.9.8: no version information available (required by pypy) pypy: /usr/lib/libcrypto.so.0.9.8: no version information available (required by pypy) Building lxml version 2.3.2. Building without Cython. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib running build_ext building 'lxml.etree' extension cc -fPIC -Wimplicit -I/usr/include/libxml2 -I/usr/include/python2.7 -I/usr/local/pypy/include -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -w cc -shared build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -L/usr/lib -lxslt -lexslt -lxml2 -lz -lm -o /home/nik/tmp/lxml-2.3.2/src/lxml/etree.pypy-17.so building 'lxml.objectify' extension cc -fPIC -Wimplicit -I/usr/include/libxml2 -I/usr/include/python2.7 -I/usr/local/pypy/include -c src/lxml/lxml.objectify.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.objectify.o -w cc -shared build/temp.linux-x86_64-2.7/src/lxml/lxml.objectify.o -L/usr/lib -lxslt -lexslt -lxml2 -lz -lm -o /home/nik/tmp/lxml-2.3.2/src/lxml/objectify.pypy-17.so ------------------------------------------------------------------------------------ $ PYTHONPATH=src pypy pypy: /usr/lib/libssl.so.0.9.8: no version information available (required by pypy) pypy: /usr/lib/libcrypto.so.0.9.8: no version information available (required by pypy) Python 2.7.1 (7773f8fc4223, Nov 18 2011, 18:47:11) [PyPy 1.7.0 with GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``happy new year'' >>>> from lxml import etree Traceback (most recent call last): File "<console>", line 1, in <module> ImportError: unable to load extension module '/home/nik/tmp/lxml-2.3.2/src/lxml/etree.pypy-17.so': /home/nik/tmp/lxml-2.3.2/src/lxml/etree.pypy-17.so: undefined symbol: PyClass_Type >>>> Has anyone ever tried to build lxml with pypy? What is the correct way to build an extension module? Any feedback is very much appreciated! Regards Nik PS: By the way, what do pypy: /usr/lib/libssl.so.0.9.8: no version information available (required by pypy) pypy: /usr/lib/libcrypto.so.0.9.8: no version information available (required by pypy) mean? Both libraries are available on my PC.

4 4

How are namespaces implemented in rpython?
by Alexander Golec Dec. 6, 2011

Dec. 6, 2011

Hey all, I'm preparing a presentation for Alfred Aho at Columbia, and I'd like to ask how are namespaces translated during the translation phase. Are they implemented dynamically, or are they actually compiled down to C? Alex

5 6

PyPy 1.6 not working on Windows XP
by Ram Rachum Dec. 5, 2011

Dec. 5, 2011

Hey guys, I upgraded to PyPy 1.6 on my 2 Windows XP 32 bit machines. It crashes on both system when running the GarlicSim test suite. It shows a Windows error dialog saying "pypy.exe has encountered a problem and needs to close. We are sorry for the inconvenience." and giving this data: AppName: pypy.exe AppVer: 0.0.0.0 ModName: kernel32.dll ModVer: 5.1.2600.5512 Offset: 00040b0d I can also open a dialog with a lot of data on the error (don't know how useful it is) but Windows won't let me Ctrl-C it. What can I do? Thanks, Ram.

8 32

Product Recommended by Ashley
by Ashley Dec. 4, 2011

Dec. 4, 2011

Dear pypy-dev, I hope you've got a good attorney. . . Because once you flick this switch on this NEW lethal 'business in a box' machine. . . http://wealthgoodincome.co.cc/rpa1.php?e=pypy-dev@codespeak.net The money starts rushing in so fast. . .you might think you broke some State or Federal laws. . . But not to worry. It's actually COMPLETELY legal.. And yet it's probably the most powerful 'Business in a Box' system that's ever come across my desk. I'm not so sure you're ready for something QUITE this powerful? Or are you. . . Let me know what you think. http://wealthgoodincome.co.cc/rpa1.php?e=pypy-dev@codespeak.net To your success, support supremeSoft LLC 302 Park Avenue, New York New YorkY 10022 USA P.S. If that video doesn't get your heart racing. . .you might have to check your pulse for signs of life. . . ~~~~~~~~~~~~~~~~~~~~~~~~~~ To view this product please follow the link below: http://www.greentoysandgames.co.uk/index.php?_a=viewProd&productId=290 ~~~~~~~~~~~~~~~~~~~~~~~~~~ This email was sent from http://www.greentoysandgames.co.uk Sender's IP Address: 180.190.136.128

1 0

Question about pypy
by Alexander Golec Dec. 4, 2011

Dec. 4, 2011

Hi all, I'm a student at Columbia University, and I'm taking a graduate course with Alfred Aho, the author of the dragon book, on advanced compilers techniques. I've been researching the pypy project in general, and rpython in particular, and I'd like to ask you guys for some feedback on the current sketch of my presentation. Aho has mentioned on several occasions that he is very excited to receive my talk, and I'd like to get some feedback from you guys about it before I put it forward to him. So then, my talk will discuss rpython's approach to translation, and here is the current outline: - Compiling python to C is easy: just inline the implementation of every opcode handler durr hurr hurr - Ok, seriously, can you do it in a performant manner? - Python has some semantics that make this difficult, in particular: - opcodes are type-agnostic - opcodes are high-level, they do high-level things with high-level arguments. eg. the BUILD_CLASS opcode - opcodes include namespace operations - This type-agnostic bit is the real tricky part because C requires all expressions to have a type, while python does not - Vanilla cartesian product type inference doesn't really work because the number of types is undecidable - rpython gets around this by imposing a restriction on dynamic type creation. - The details of the annotator are omitted due to time constraints So then, the crux of my talk is the type annotator. I won't be going deeply into the flow and object spaces in the talk, deferring to my paper. Is there anything major I'm missing? Did I get anything horribly wrong? Alex

4 6

Metaprogramming with syntax extension
by Valentin Perrelle Dec. 4, 2011

Dec. 4, 2011

Hi, Last months i played a bit with syntax extension in order to develop a network programming paradigm inspired by the Unreal Script language. I did that with LUA and the help of metalua which use a lua lua compiler. I was wondering if i could do the same with Python. Is it possible with Pypy to dynamically (or even statically) change the parser in order to accept new syntax constructions ? All i could need is to perform program transformation : any program with the new syntax could be translated to a valid python program without the syntax extension. Basically, i need to change the way some class methods are called and some properties are accessed. Of course, it could be done with the dedicated metatmethods. But the amount of code needed to perform this would be big, repetitive, uninformative and hardly factorizable. Valentin Perrelle.

2 1

1.6 -> 1.7
by Russel Winder Dec. 2, 2011

Dec. 2, 2011

http://doc.pypy.org/en/latest/index.html#getting-into-pypy still claims 1.6 is the latest official release! What is the right way of reporting these sorts of thing? -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder@ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel(a)russel.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder

3 3

Win32 build of the release 1.7
by Armin Rigo Dec. 1, 2011

Dec. 1, 2011

Hi all, We have a tentative Win32 build of PyPy 1.7 including both the JIT and Stackless features (the '_continuation' primitive module and the app-level 'stackless' and 'greenlet' modules). It built and passed a good proportion of tests. If you are interested in helping, could a few people try to download it and tell us if it works, and if something is obviously missing? Thank you! http://buildbot.pypy.org/nightly/release-1.7.x/pypy-c-jit-49856-930f0bc4125… A bientôt, Armin.

7 8