Mailman 3 September 2012 - pypy-dev

PyPy at FOSDEM
by Maciej Fijalkowski 19 Sep '12

19 Sep '12

Hi Just a quick question - is anyone from the team planning on attending FOSDEM in Brussels on 2-3rd of Feb 2013? Cheers, fijal

1 0

Flow graphs, backends and JIT
by haael 17 Sep '12

17 Sep '12

OK, I read almost all the documentation I found on the web page. But I still don't understand few things. There are 3 layers in the whole picture. The user application written in Python, the Python interpreter written in RPython and the RPython interpreter itself. 1. Where do flow graphs are generated from? Is it the representation of the user application, or the interpreter? 2. Where does the JIT fit here? I read it traces the execution of the iterpreter and indirectly the user application. Does it operate on the flow graphs or something else? 3. Which component actually does the JIT? Is it just a tweak on the code generator or are the flow graphs generated differently? 4. Is there some documentation how to write a backend (code generator)? The source code is poorly documented and the topic is not mentioned on the web page. What exactly do I need to implement to have a backend? haael

2 1

rlib/runicode, python 3 and pypy
by Maciej Fijalkowski 13 Sep '12

13 Sep '12

Hi This is mostly a mail to Antonio, but I'm interested in everyone's opinion. Al rlib/runicode.py differs on py3k branch from the default. IMO this is a very very bad idea. This is a push towards more convoluted interpreter/translator interaction rather than less. Also it makes RPython a less defined language rather than more. I see the following options: * decide utf-8 encoding in RPython is a bad idea alltogether * move codec somewhere else or patch unicodeobject in python 3 and decide to keep the default version on py3k * declare it a temporary hack, but then it confuses people and I'm generally skeptical about the amount of temporary hacks on py3k branch. The one thing I *don't* want to do is to declare "meh, too bad, RPython is a different language on py3k and on default" as long as py3k branch stays as a part of PyPy project. Cheers, fijal

1 0

build error with --shared flag
by Max Lavrenov 11 Sep '12

11 Sep '12

Hello When i tried build master pypy with --shared flag i got error: [translation:ERROR] /usr/bin/ld: /usr/lib/libffi.a(ffi64.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC [translation:ERROR] /usr/lib/libffi.a: could not read symbols: Bad value [translation:ERROR] collect2: ошибка: выполнение ld завершилось с кодом возврата 1 [translation:ERROR] make: *** [libpypy-c.so] Ошибка 1 [translation:ERROR] """) What that means? Should i rebuild my libffi package with -fPIC? Right now i've temporary fixed this probleb with patch def find_libffi_a(): function so it always use dynamic linking. Will it cause "endless troubles for installing" ( according comment from clibffi.py ) ? Thanks. Best regards, Max

2 1

Windows 7 64-bit Problems installing pip or virutalenv
by Gelonida N 09 Sep '12

09 Sep '12

Hi, Just doing some experiments with PyPy I'd like to install pip / virutalenv. With pypy and Linux no problem With CPython (32 bit) and windows 7 64-bit no problem With pypy and Windows 7 64-bit no success. Any hints ???? I even tried it with disabling the antivirus. In fact so far I couldn't download and install any package successfully. Perhaps somebody could recommend a very simple package as a starting poimt. Experimenting without virtualenv is a serious pain. Installing pypy and easy_install (setuptools) seems to work Installing virtualenv / pip always fails. I'm running out of ideas. SO any help is welcome. Below the details. Step 1: Download and unpack pypy ------------------------------------- I open a cmd window I downloaded pypy1.9.0 and extracted it to a directory (%PYPY_PATH%) Step 2: Installing easy_install (setup_tools) ----------------------------------------------- I download http://python-distribute.org/distribute_setup.py and install it with cd %PYPY_PATH% pypy.exe distribute_setup.py This seems to work basically. However a really strange message is: Don't have permissions to write C:\test\pypy-1.9\site-packages\setuptools-0.6c11-py2.7.egg-info, skipping Not sure, where this is coming from. I assume, that another thread / process didn't close the file properly Just for fun I tried this also with a privileged window. Same problem: Step 3: Try to install virtualenv ------------------------------------- I tried following steps in a normal cmd window awith a cmd window with admin privileges. When starting virtualenv from a admin-cmd window. Then I get following output: C:\test>pypy-1.9.2\bin\easy_install.exe virtualenv Searching for virtualenv Reading http://pypi.python.org/simple/virtualenv/ Reading http://www.virtualenv.org Reading http://virtualenv.openplans.org Best match: virtualenv 1.8.2 Downloading http://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.8.2.tar.gz… Processing virtualenv-1.8.2.tar.gz Writing c:\users\XXXXX\appdata\local\temp\easy_install-ubcvxv\virtualenv-1.8.2\setup.cfg Running virtualenv-1.8.2\setup.py -q bdist_egg --dist-dir c:\users\XXXXX\appdata\local\temp\easy_install-ubcvxv\virtualenv-1.8.2\egg-dist-tmp-cd7mao warning: no previously-included files matching '*' found under directory 'docs\_templates' warning: no previously-included files matching '*' found under directory 'docs\_build' No eggs found in c:\users\XXXX\appdata\local\temp\easy_install-ubcvxv\virtualenv-1.8.2\egg-dist-tmp-cd7mao (setup script problem?) C:\test> Step 4) Trying to Install pip ------------------------------- Installing pip fails as well. C:\test>pypy-1.9.2\bin\easy_install.exe pip Searching for pip Reading http://pypi.python.org/simple/pip/ Reading http://www.pip-installer.org Reading http://pip.openplans.org Best match: pip 1.2.1 Downloading http://pypi.python.org/packages/source/p/pip/pip-1.2.1.tar.gz#md5=db8a6d8a4… Processing pip-1.2.1.tar.gz Writing c:\users\XXXX\appdata\local\temp\easy_install-rxf5c5\pip-1.2.1\setup.cfg Running pip-1.2.1\setup.py -q bdist_egg --dist-dir c:\users\XXXX\appdata\local\temp\easy_install-rxf5c5\pip-1.2.1\egg-dist-tmp-an2nbn warning: no files found matching '*.html' under directory 'docs' warning: no previously-included files matching '*.txt' found under directory 'docs\_build' no previously-included directories found matching 'docs\_build\_sources' No eggs found in c:\users\XXXX\appdata\local\temp\easy_install-rxf5c5\pip-1.2.1\egg-dist-tmp-an2nbn (setup script problem?) error: c:\users\XXXX\appdata\local\temp\easy_install-rxf5c5\pip-1.2.1\docs\index.txt: The process cannot access the file because it is being used by another p rocess. C:\test> If I start, easy_install as normal user, then I get first a popu asking me if I want to allow easy_install to make changes to my computer. then a new cmd window opens with some messages (dowmloading, . . . ) then the window disappears and then I get a popup, that windows is not sure my program installed correctly

2 2

Locals clearing in RPython
by Timothy Baldridge 07 Sep '12

07 Sep '12

Let's imagine that I have some code like the following in RPython: def wrapper_func(arg1, arg2): return inner_func(arg2) def inner_func(x): for y in range(x): # do something here pass return -1 bigint = 1000000 wrapper_func(list(range(bigint)), bigint) The problem here is that arg1 is going to be held onto in (in CPython at least) until inner_func returns. This means that the list created on the invocation of wrapper_func is going to stick around during the entire execution time of inner_func. This would not make much of a difference normally, but in languages with extensive use of lazy evaluation, holding onto the head of a sequence could cause out-of-memory errors. In Clojure this is fixed-up by the compiler via "locals clearing". Basically the compiler inserts "arg1 = None" before the invocation of inner_func. What's the story here in RPython? Since RPython basically compiles down to single-assignment code I'm guessing the Clojure fix won't help me. When is the GC able to go and free data held by arguments? Timothy

3 4

STM and concurrent GC
by Wiktor Mizdal 06 Sep '12

06 Sep '12

Hi, will STM can help implement concurrent garbage collector? Wiktor Mizdal

2 1

PyPy got an Editors' Choice award !
by Vera Zubor - DownloadAtlas.com 06 Sep '12

06 Sep '12

Hello PyPy Development Team, we would like to remind you and announce an Editor's Choice award to PyPy Current method of awarding is chosen by us to reflect the high quality of your software product. More informations at http://www.downloadatlas.com/open-source-eac5cece.html#awards Important Note: If you have any questions, please don't hesitate to contact us. Best wishes, Vera Zubor - editor e-mail: editor(a)downloadatlas.com ----------------------------------------------------------------------------------------------------------------------- This message and any attached files are confidential and intended solely for the addressee(s). Any publication, transmission or other use of the information by a person or entity other than the intended addressee is prohibited. If you receive this in error please contact the sender and delete the material. The sender does not accept liability for any errors or omissions as a result of the transmission. To unsubscribe or change options please login to your account using email address http://user.downloadatlas.com/?account=pypy-dev@python.org

1 0

STM "version 2"
by Armin Rigo 06 Sep '12

06 Sep '12

Hi all, To keep you informed of the progress on STM: In the middle of August I found a potentially better approach to STM, which uses copies of objects more extensively (something which is neither natural nor easy to do in C/C++, which is probably why it was not researched before). The main change is that all globally visible objects are and stay read-only. A write that occurs in theory on one such object is --- like before --- done inside a local copy of the object. The difference with before is what occurs on commit. Before, the local copy would have its content copied back over the global object. But now, the local copy "becomes" the next version of the global object. It makes the reads simpler and cheaper, because we don't have to worry about other threads committing changes to the object in parallel. We end up with the old version of the global object that says, in its header, "I'm outdated, here's a pointer to some newer version". Obviously, this approach relies on good GC support to eventually collect the old copies. I ended up documenting it extensively there, in a very terse form so far: * https://bitbucket.org/pypy/extradoc/raw/extradoc/talk/stm2012/stmimpl.rst I adapted the high-level testing framework I wrote last year to check that it is, or appears to be, correct (and found of course a couple of subtle bugs): * https://bitbucket.org/arigo/arigo/raw/default/hack/stm/python/c2.py * https://bitbucket.org/arigo/arigo/raw/default/hack/stm/python/test_c2.py And I rewrote and tested it in C: * https://bitbucket.org/arigo/arigo/raw/default/hack/stm/c2 In the next days I will start to adapt PyPy to it. This approach has the advantage over the previous one to not require complex support for the reads, which should also make the JIT backend work more directly. Also, unlike the previous one, this approach works on non-Intel CPUs (the C code has been tested on POWER64). A bientôt, Armin.

2 1

pypy arm progress
by Wiktor Mizdal 05 Sep '12

05 Sep '12

Hi, in "Almost There - PyPy's ARM Backend" article is "The incomplete list of open topics": -We are looking for a better way to translate PyPy for ARM, than the one describe above. I am not sure if there currently is hardware with enough memory to directly translate PyPy on an ARM based system, this would require between 1.5 or 2 Gig of memory. A fully QEMU based approach could also work, instead of Scratchbox2 that uses QEMU under the hood. -Test the JIT on different hardware. -Experiment with the JIT settings to find the optimal thresholds for ARM. -Continuous integration: We are looking for a way to run the PyPy test suite to make sure everything works as expected on ARM, here QEMU also might provide an alternative. -A long term plan would be to port the backend to ARMv5 ISA and improve the support for systems without a floating point unit. This would require to implement the ISA and create different code paths and improve the instruction selection depending on the target architecture. -Review of the generated machine code the JIT generates on ARM to see if the instruction selection makes sense for ARM. -Build a version that runs on Android. -Improve the tools, i.e. integrate with jitviewer. Which points is already done? Wiktor Mizdal

2 1