Mailman 3 December 2011 - pypy-dev

.py -> .pyc -> (.pyc2 + .so) ?
by Bengt Richter 13 Dec '11

13 Dec '11

Just musing -- if pypy can discover pure functions (or functions with simple static global side effects) and the jit transforms them into machine code in a runtime memory-resident image somewhere, could this image be transformed with some .so boilerplate and metadata into a loadable module which could be discovered for use more or less like a .pyc that corresponds to .py is used? Could the jit perhaps write into a memory-mapped file that could be closed on exit to preserve the image for possible final post-processing into a standard format .so full of ready-made functions and supporting data? If suitable metadata were included, perhaps the .so could be loaded and used by a C or C++ program linking it? Meaning there would be a way to write a C-compatible "library" using pypy and python module source syntax. If you 'can never be sure' some function is suitable, could you pass pypy a support file (e.g. -jit purefuncfile=thesupportfile) with a line per function specifying modulefile and access path (maybe start with global functions only), asserting suitability? Regards, Bengt Richter

6 6

GZip benchmark
by Maciej Fijalkowski 13 Dec '11

13 Dec '11

Hello There is an open pull request with gzip benchmark. Do we want a gzip benchmark in our nightly run? https://bitbucket.org/pypy/benchmarks/pull-request/1/added-gzip-benchmark How much of stdlib we really want to measure? Cheers, fijal

2 4

OS
by Rinu Boney 11 Dec '11

11 Dec '11

can RPython be converted to low level code as to create something like a kernel or something low level as that ? what about an OS with assembly lang , little C/C++ and RPython ? can an OS like android be developed using RPython ? ( by RPython i also mean using the PyPy tool chain )

5 7

Re: [pypy-dev] OS
by Laura Creighton 11 Dec '11

11 Dec '11

I worked on the Tunis OS at the University of Toronto, which was an OS written in Concurrent Euclid. I think -- especially if Software Transactional Memory works out, there will be interest in writing an OS, not in RPython, but in Python. So then, well, maybe you will want a JIT ... I think we will only be able to finally ditch unix/linux for _something significantly better_ when we get to write the something better in a higher level language than C. Or Java. Laura

1 0

python 2 and python 3 sharing an interpreter?
by Martijn Faassen 10 Dec '11

10 Dec '11

Hi PyPy folks, I've said so before in comments on your blog, but much kudos to the project, you guys have been doing a great job! Some time ago I tried PyPy against an artificial life experiment of mine which implemented a simple stack-based language in Python and was pleasantly surprised to see that PyPy actually sped things up quite a bit - I'd heard about the JIT not being too good with interpreters. Good enough to make it a lot faster, though. :) Anyway, on to my question. I already asked it once in 2007, and didn't get a very encouraging response, but we're years down the road now and crazy difficulties is something the PyPy folks are good at, so I'll ask it again. You all are, as I understand, exploring supporting Python 3 with PyPy. Cool. Would it be possible to have an interpreter that could support both Python 2 and Python 3 modules in the same runtime? I.e, an interpreter that supports importing Python 3 code from Python 2 modules, and vice versa? It would tremendously help the Python ecosystem for such a solution to be out there, because in such a situation people can start using Python 3 codebases in Python 2, encouraging more people to port their libraries to Python 3, and people can use Python 2 codebases in Python 3, encouraging more people to start projects in Python 3. I can't stress enough how much I believe that would help the Python ecosystem! I understand that the problems involved would be decidedly non-trivial, but that has never stopped you guys before. Modules would need to declare somehow that they were Python 2 or Python 3. There'd need to be, somehow, two separate import "spaces", where Python 2 code would import from Python 2 modules by default and Python 3 code would import from Python 3 modules by default. You'd also need something explicit to allow cross imports. Python 2 objects represented in Python 3 (and vice versa) would either need to be transformed for immutable built-in objects (a Python 2 string would become a Python 3 bytes), or proxied for custom objects. I remember when I brought this up at the time that Armin Rigo suggested a few conundrums that were hard to solve. What these were exactly I don't remember anymore, so I hope Armin will chime in and remind me. :) Regards, Martijn

8 20

PyPy Tool Chain
by Rinu Boney 10 Dec '11

10 Dec '11

Can almost any language be implemented in RPython and be faster ? ( like can we implement a langugae like java ? )

4 3

Concerning bounded number of classes created
by Alexander Golec 10 Dec '11

10 Dec '11

Hey all, Sorry to keep asking questions on the mailing list, but I've been doing my semester research project on PyPy, and my deadline is beginning to loom... I have a question concerning what I think is a conflict between the RPython coding guide and the 2005 EU paper. The paper states: Our approach goes further and analyses live programs in memory: the program is allowed to contain fully dynamic sections, as long as these sections are entered a bounded num- ber of times. For example, the source code of the PyPy interpreter, which is itself written in this bounded-dynamism style, makes extensive use of the fact that it is possible to build new classes at any point in time -- not just during an initialisation phase -- as long as the number of new classes is bounded. For example, interpreter/gateway.py builds a custom wrapper class corresponding to each function that a particular variable can reference. There is a finite num- ber of functions in total, so this can only create a finite number of extra wrapper classes. While the coding guide says: definitions run-time definition of classes or functions is not allowed. I'm I understood the paper correctly, any piece of code can create zero or one class, but not infinity. However, this doesn't seem to jive with what the coding guide says. Alex

3 6

In RPython, what is meant by 'live code objects'?
by Alexander Golec 09 Dec '11

09 Dec '11

Hey all, When the RPython receives the code objects, are all the objects that the interpreter would push and pop on the stack already present? Or does the abstract interpreter handle creating those objects as well? In other words, is interpretation abstract from the start of the program to the end? Is the interpreter concrete until the entry point, at which point it switches to the abstract interpreter? Or is the RPython compiler not responsible for interpreting the initialization portion, just for the bit after the entry point? Alex

2 1

Could someone give me an idea of what is where with RPython?
by Alexander Golec 09 Dec '11

09 Dec '11

Hey all, I have the following conception of the RPython toolchain: - RPython translator is written in RPython, meaning it can be interpreted by CPython, and then run as a standalone executable once translated - The translator runs the standard interpreter on the input program. - The input program has some marker to say 'alright, I'm initialized, begin translating,' at which point execution stops and RPython starts - RPython kicks in, performs flow object space analysis on the initialized code objects. - Type annotation occurs based on the initialized types - The annotated code is passed down to the backend-facing code generator Is this correct? Do I have a correct basic understanding of what happens where? Alex

3 9

Clang benchmarks
by Justin Noah 08 Dec '11

08 Dec '11

Here are the llvm/clang build using the shadowstack gc. What do you think? Also, I will be downloading the prebuilt binary and running benchmarks and post them as well. Report on Linux infinity 3.1.1-gentoo #1 SMP PREEMPT Sun Nov 20 03:57:03 PST 2011 x86_64 AMD Turion(tm) II Dual-Core Mobile M520 Total CPU cores: 2 ### ai ### Min: 0.140570 -> 0.128204: 1.0965x faster Avg: 0.158339 -> 0.148346: 1.0674x faster Significant (t=2.351561, a=0.95) Stddev: 0.01762 -> 0.02434: 1.3809x larger ### bm_chameleon ### Min: 0.051691 -> 0.051436: 1.0050x faster Avg: 0.062364 -> 0.065282: 1.0468x slower Not significant Stddev: 0.01720 -> 0.01883: 1.0947x larger ### bm_mako ### Min: 0.152917 -> 0.153566: 1.0042x slower Avg: 0.169574 -> 0.169380: 1.0011x faster Not significant Stddev: 0.02681 -> 0.02674: 1.0025x smaller ### chaos ### Min: 0.024781 -> 0.024249: 1.0219x faster Avg: 0.037913 -> 0.037324: 1.0158x faster Not significant Stddev: 0.07050 -> 0.07000: 1.0072x smaller ### crypto_pyaes ### Min: 0.121740 -> 0.121273: 1.0038x faster Avg: 0.136873 -> 0.136358: 1.0038x faster Not significant Stddev: 0.05667 -> 0.05649: 1.0033x smaller ### django ### Min: 0.104028 -> 0.100156: 1.0387x faster Avg: 0.116562 -> 0.114053: 1.0220x faster Significant (t=2.133187, a=0.95) Stddev: 0.00590 -> 0.00586: 1.0077x smaller ### fannkuch ### Min: 0.514067 -> 0.536132: 1.0429x slower Avg: 0.520998 -> 0.542397: 1.0411x slower Significant (t=-6.626659, a=0.95) Stddev: 0.01654 -> 0.01575: 1.0501x smaller ### float ### Min: 0.085486 -> 0.084867: 1.0073x faster Avg: 0.106569 -> 0.104426: 1.0205x faster Not significant Stddev: 0.01745 -> 0.01596: 1.0931x smaller ### go ### Min: 0.271402 -> 0.269337: 1.0077x faster Avg: 0.483500 -> 0.483173: 1.0007x faster Not significant Stddev: 0.19943 -> 0.19894: 1.0025x smaller ### html5lib ### Min: 5.856357 -> 5.745086: 1.0194x faster Avg: 7.952101 -> 7.824551: 1.0163x faster Not significant Stddev: 2.80613 -> 2.76699: 1.0141x smaller ### json_bench ### Min: 3.525704 -> 3.533075: 1.0021x slower Avg: 3.563207 -> 3.577035: 1.0039x slower Not significant Stddev: 0.14274 -> 0.15135: 1.0604x larger ### meteor-contest ### Min: 0.335457 -> 0.336462: 1.0030x slower Avg: 0.343577 -> 0.344685: 1.0032x slower Not significant Stddev: 0.01572 -> 0.01579: 1.0045x larger ### nbody_modified ### Min: 0.085591 -> 0.085202: 1.0046x faster Avg: 0.087704 -> 0.087397: 1.0035x faster Not significant Stddev: 0.00601 -> 0.00633: 1.0538x larger ### pyflate-fast ### Min: 1.038416 -> 1.036536: 1.0018x faster Avg: 1.078068 -> 1.086279: 1.0076x slower Not significant Stddev: 0.02674 -> 0.02464: 1.0854x smaller ### raytrace-simple ### Min: 0.066854 -> 0.067021: 1.0025x slower Avg: 0.081450 -> 0.081594: 1.0018x slower Not significant Stddev: 0.02581 -> 0.02637: 1.0218x larger ### richards ### Min: 0.008039 -> 0.007999: 1.0050x faster Avg: 0.008915 -> 0.008894: 1.0024x faster Not significant Stddev: 0.00269 -> 0.00267: 1.0065x smaller ### rietveld ### Min: 0.243249 -> 0.245089: 1.0076x slower Avg: 0.614075 -> 0.609522: 1.0075x faster Not significant Stddev: 0.59714 -> 0.57957: 1.0303x smaller ### slowspitfire ### Min: 0.677112 -> 0.684585: 1.0110x slower Avg: 0.731244 -> 0.721026: 1.0142x faster Significant (t=2.087466, a=0.95) Stddev: 0.02165 -> 0.02700: 1.2469x larger ### spambayes ### Min: 0.151896 -> 0.151275: 1.0041x faster Avg: 0.292433 -> 0.293944: 1.0052x slower Not significant Stddev: 0.12329 -> 0.12527: 1.0161x larger ### spectral-norm ### Min: 0.028405 -> 0.028729: 1.0114x slower Avg: 0.032538 -> 0.033337: 1.0245x slower Not significant Stddev: 0.01354 -> 0.01373: 1.0144x larger ### spitfire ### Min: 9.910000 -> 10.030000: 1.0121x slower Avg: 10.039000 -> 10.147800: 1.0108x slower Significant (t=-5.090227, a=0.95) Stddev: 0.11163 -> 0.10189: 1.0957x smaller ### spitfire_cstringio ### Min: 4.920000 -> 4.920000: no change Avg: 4.955000 -> 4.956400: 1.0003x slower Not significant Stddev: 0.06225 -> 0.07056: 1.1336x larger ### sympy_expand ### Min: 1.990902 -> 2.037110: 1.0232x slower Avg: 2.497677 -> 2.532886: 1.0141x slower Not significant Stddev: 0.97914 -> 0.96325: 1.0165x smaller ### sympy_integrate ### Min: 4.294102 -> 4.395819: 1.0237x slower Avg: 6.417529 -> 6.457441: 1.0062x slower Not significant Stddev: 2.47638 -> 2.49616: 1.0080x larger ### sympy_str ### Min: 0.907993 -> 0.889144: 1.0212x faster Avg: 1.978583 -> 1.950795: 1.0142x faster Not significant Stddev: 1.11382 -> 1.09577: 1.0165x smaller ### sympy_sum ### Min: 1.349965 -> 1.286402: 1.0494x faster Avg: 1.730386 -> 1.727202: 1.0018x faster Not significant Stddev: 0.54941 -> 0.53929: 1.0188x smaller ### telco ### Min: 0.099984 -> 0.099985: 1.0000x slower Avg: 0.120782 -> 0.121002: 1.0018x slower Not significant Stddev: 0.05775 -> 0.05797: 1.0038x larger ### trans_annotate ### Raw results: [1368.2] None ### trans_rtype ### Raw results: [954.1] None ### trans_backendopt ### Raw results: [433.3] None ### trans_database ### Raw results: [586.7] None ### trans_source ### Raw results: [635.9] None ### twisted_iteration ### Min: 0.013585 -> 0.013649: 1.0047x slower Avg: 0.013755 -> 0.013809: 1.0039x slower Not significant Stddev: 0.00015 -> 0.00014: 1.0542x smaller ### twisted_names ### Min: 0.007508 -> 0.007530: 1.0030x slower Avg: 0.007914 -> 0.007958: 1.0056x slower Not significant Stddev: 0.00019 -> 0.00020: 1.0402x larger ### twisted_pb ### Min: 0.038095 -> 0.037879: 1.0057x faster Avg: 0.040128 -> 0.039598: 1.0134x faster Significant (t=2.789748, a=0.95) Stddev: 0.00101 -> 0.00089: 1.1368x smaller ### twisted_tcp ### Min: 1.128350 -> 1.122807: 1.0049x faster Avg: 1.167165 -> 1.153992: 1.0114x faster Significant (t=2.337046, a=0.95) Stddev: 0.02843 -> 0.02794: 1.0174x smaller Here is the output of "cat /proc/cpuinfo | grep cache," I have attached the full output of cpuinfo. -- - Justin Noah

2 2