Mailman 3 September 2012 - pypy-dev

Python vs pypy: interesting performance difference [dict.setdefault]
by David Naylor March 10, 2014

March 10, 2014

Hi, I needed to create a cache of date and time objects and I wondered what was the best way to handle the cache. For comparison I put together the following test: <file="iforkey.py"> import datetime import random import timeit ranges = [datetime.datetime(2011,01, random.randint(1, 31)) for i in xrange(1000)] def ifdict(): cache = {} b = [] for i in ranges: key = i.day if key in cache: b.append(cache[key]) else: date = i.date() cache[key] = date b.append(date) def keydict(): cache = {} b = [] for i in ranges: key = i.day try: b.append(cache[key]) except KeyError: date = i.date() cache[key] = date b.append(date) def defaultdict(): cache = {} b= [] for i in ranges: b.append(cache.setdefault(i, i.date())) print "ifdict:", timeit.repeat("ifdict()", "from __main__ import ifdict", number=10000) print "keydict:", timeit.repeat("keydict()", "from __main__ import keydict", number=10000) print "defaultdict:", timeit.repeat("defaultdict()", "from __main__ import defaultdict", number=10000) </file> # python iforfile.py ifdict: [2.432887077331543, 2.4002890586853027, 2.397233009338379] keydict: [2.3483030796051025, 2.358638048171997, 2.314802885055542] defaultdict: [3.5384328365325928, 3.5329859256744385, 3.5728111267089844] # pypy iforfile.py (pypy 1.5) ifdict: [0.8129069805145264, 0.74648118019104, 0.7432689666748047] keydict: [0.5187451839447021, 0.4662129878997803, 0.4504108428955078] defaultdict: [37.98510789871216, 37.859113931655884, 37.92770600318909] Pypy displays significant slowdown in the defaultdict function, otherwise displays its usual speedup. To check what is the cause I replaced i.date() with i.day and found no major difference in times. It appears dict.setdefault (or it's interaction with jit) is causing a slow down. Regards

4 14

numpy fails trigonometry with complex numbers, what to do?
by Matti Picus Dec. 22, 2013

Dec. 22, 2013

I am trying to complete complex numbers in numpypy. Progress is good, I picked up from previous work on the numpypy-complex2 branch. Complex numbers come with extensive tests, it seems all the corner cases are covered. In porting the tests to numpypy, I came across a problem: numpy returns different results than cmath. Some of the differences are due to the fact that numpy does not raise a ValueError for dividing by 0 or other silly input values, but other differences are inexplicable (note the sign of the imaginary part): >>> numpy.arccos(complex(0.,-0.)) (1.5707963267948966-0j) >>> cmath.acos(complex(0.,-0.)) (1.5707963267948966+0j) >>> or this one: >>> cmath.acos(complex(float('inf'),2.3)) -infj >>> numpy.arccos(complex(float('inf'),2.3)) (0.78539816339744828-inf*j) Should I ignore the inconsistencies, or fix the 700 out of 2300 test instance failures? What should pypy's numpypy do - be consistent with numpy or with cmath? cmath is easier and probably faster (no need to mangle results or input args), so I would prefer cmath to trying to understand the logic behind numpy. Matti 2280c2365844

4 4

Any publications regarding PyPy / trace-based JIT compiler?
by Michael Sioutis Nov. 11, 2012

Nov. 11, 2012

Dear PyPy team, I am in the process of writing a paper that will target some AI conference, and I would like to ask if there are any relevant publications of yours or in general that showcase the possible advantages of trace-based JIT compilation over method-based JIT compilation or static compilation. I just want to use 2 or 3 of them as references when I explain why my implementation is more scalable and robust when compared to some C/C++/Java implementations (appart from different data structures and algorithms). Thank you :) Mike

7 12

PyPy STM
by Russel Winder Oct. 5, 2012

Oct. 5, 2012

Armin, Sarah and I are restarting our work on CSP, and extending to creating actors and a dataflow library. It would be good to make this work on Jython, IronPython and PyPy as well as CPython. However we want to get away from a reliance on multiprocessing since it is rather heavyweight for the sort of parallelism we are after. STM as an infrastructure layer in PyPy and CPython would get us away from the GIL and allow for using Python threads bound to kernel threads to allow a single Python process to allow us to create lighweight processes (no application level shared memory). Is the STM variant of PyPy and/or CPython in any form of usable state? If so then we can investigate building from source for ourselves so as to use it as a foundation for building the higher abstraction parallelism for applications programmers. Thanks. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder@ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel(a)winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder

6 12

MalGen as a benchmark?
by Chris Leary Sept. 30, 2012

Sept. 30, 2012

Found a red-hot, branchy-looking Python kernel in the wild and naturally I thought of you trace compiler folks! ;-) Hope that it might be useful: I think it could make a nice addition to the speed center, seeing as how it's a CPU bound workload on all the machines I have access to (though I haven't profiled it at all so it could potentially be leaning heavily on paths in some unoptimized builtins). MalGen is a set of scripts which generate large, distributed data sets suitable for testing and benchmarking software designed to perform parallel processing on large data sets. The data sets can be thought of as site-entity log files. After an initial seeding, the scripts allow for the data generation to be initiated from a single central node to run the generation concurrently on multiple remote nodes of the cluster. -- http://code.google.com/p/malgen/ Specifically, http://code.google.com/p/malgen/source/browse/trunk/bin/cloud/malgen/malgen… which gets run thusly: :: pypy malgen.py -O /tmp/ -o INITIAL.txt 0 50000000 10000000 21 (Where 5e7 is the "initial block size" and 1e7 is the other-than-inital block size.) This generates the initial seeding they were talking about, followed by a run for each of N blocks on each node (in this hypothetical setup, for 5 blocks on each of four nodes the following is run): :: pypy malgen.py -O /tmp [start_value] The metadata is read out of the INITIAL.txt file and used to determine the size of the block, and the parameter [start_value] is used to bump to the appropriate start id count for the current block. Inner loop: http://code.google.com/p/malgen/source/browse/trunk/bin/cloud/malgen/malgen… Thoughts? - Leary

3 6

nightly builds require OpenSSL 1.0 ?
by Stefan Behnel Sept. 30, 2012

Sept. 30, 2012

Hi, my nightly fetch-PyPy build job stopped working two days ago because the nightly builds now require libssl 1.0.0. It's not currently available on our integration test server. Is that dependency really necessary or was it just an accidental side effect of a system update? Stefan

5 9

Bachelor Thesis - STM / ATM
by Andreas Pedersen Sept. 30, 2012

Sept. 30, 2012

Hello PyPy team. I would like to write my Bachelor Thesis on Python Parallel programming and your recent blog post Multicore Programming in PyPy and CPython<http://morepypy.blogspot.dk/2012/08/multicore-programming-in-pypy-and.html> sounds very intriguing. I would like to assist you with implementing STM / AME in PyPy and I can provide you with the equivalent of about 8 weeks of full time work over the next 3 months for this purpose. The Guidelines require an overall theme for the thesis, so I was thinking of focusing on a subset of the STM / ATM problem as the full problem seems too big. I don't currently have an overview of the code, so I can't see how far you are and what subproblems you are tackling. Any help on finding a good subject would be appreciated. The problem for me, is that I had the misconception my supervisor was more up-to-date on parallel programming in Python than he was. So now I'm under a very tight schedule to get my synopsis written as quickly as possible. Kind Regards - Andreas

2 1

source code documentation in PyPy
by Maciej Fijalkowski Sept. 26, 2012

Sept. 26, 2012

Hi I would like to suggest we add a requirement to document PyPy source code slightly better. Step one would be to have few-sentences "where are you now" info at the top of each file. How about we try to stick to a policy where each time anyone does a major work on a file, he adds documentation to the top or checks if what's there is already correct? Opinions? Cheers, fijal

2 1

MinGW32 support PyPy with mscr90.dll
by bookaa Sept. 19, 2012

Sept. 19, 2012

Windows system, after set cc=mingw32-gcc PyPy can not pass many tests: pytest.py pypy/translator/test/test_unsimplify.py pytest.py pypy/translator/c/test/test_extfunc.py::test_dictlike_environ_getitem ... I find this is because MinGW32 gcc compile default use msvcrt.dll, but we need msvcr90.dll, as Python 2.7 use msvcr90.dll. >>> import ctypes >>> ctypes.util.find_msvcrt() 'msvcr90.dll'And even python.exe dependence on msvcr90.dllref to http://trac.seqan.de/wiki/HowTo/SelectASpecificMsvcrtVersionWithMinGw this is how I make MinGW32 gcc use msvcr90.dll instead: save these to you MinGW directory as file D:\MinGW\lib\gcc\mingw32\4.6.2\specs *cpp: %{posix:-D_POSIX_SOURCE} %{mthreads:-D_MT} -D__MSVCRT_VERSION__=0x0900 *libgcc: %{mthreads:-lmingwthrd} -lmingw32 %{shared-libgcc:-lgcc_s} %{!shared-libgcc:-lgcc_eh} -lgcc -lmoldname90 -lmingwex -lmsvcr90 I suggest this instruction should be add to PyPy doc: MinGW32 support

2 1

Re: [pypy-dev] Flow graphs, backends and JIT
by haael Sept. 19, 2012

Sept. 19, 2012

>>>> 3. Which component actually does the JIT? Is it just a tweak on the code >>>> generator or are the flow graphs generated differently? >>> >>> >>> The flow graphs are taken from the translator and modified by the JIT >>> generator. >> >> >> My question is: >> >> Does JIT involve another "transformation" of the flow graphs? In normal >> (non-JIT) code generation some flow graphs are fed to the backend generator. >> Wich step is different in the JIT case? Does the backend generator get >> different flow graphs or are the same flow graphs compiled differently by a >> tweaked code generator? > > They get the same flowgraphs. So, if I understand well, there is no common JIT code among different backends? The JIT we have is the C-backend specific? Different backends would need a new JIT approach? >>>> 4. Is there some documentation how to write a backend (code generator)? >>>> The >>>> source code is poorly documented and the topic is not mentioned on the >>>> web >>>> page. What exactly do I need to implement to have a backend? >>> >>> >>> You mean a JIT backend or a RPython backend? >> >> >> >> A RPython backend first. Is there any documentation, tutorial, simple toy >> backend or anything I could start with? > > No. In fact, the only RPython backend that is well-maintained is the C one. OK, so where could I start from? Is there for example some list of flow graphs opcodes? >>> You might find this useful: http://www.aosabook.org/en/pypy.html >>> >> >> OK, that was useful. It seems that the JIT generator is some assembler >> embedded into the final binary. Does JIT generator share some code with the >> backend generator? > > No. > >> >> Would it be possible to get rid of the normal code generator (leaving only >> some glue code) and relaying only on the JIT generator, that would produce >> the whole code? > > No. The JIT generator is specialized for dynamic languages not ones > like RPython, which can be translated to C. > >> >> This would reduce the size of the binary and would not hit performance much, >> since loops would be generated as usual, only the non-looping execution >> would be different. > > Why would it reduce the size of the binary? That is my poor understanding, I might be wrong. In the current approach in a binary there is a compiled machine code, the flow graph representation and the JIT compiler. I think we could get rid of (most) compiled machine code, leaving only some startup code to spawn the JIT compiler. Then, each code path would be compiled by JIT and executed. Loops would run as fast as usual. Non-loop code would run slower, but I think this would be a minor slowdown. Most importantly, as I understand, the binary contains many versions of the same code paths specialized for different types. If we throw it out, the binary would be smaller. This is not a proposal. It is just a try at understanding things. haael

5 9