Mailman 3 June 2010 - pypy-dev

Building pypy on freebsd 64bit
by Shadrin Eugene 18 Nov '11

18 Nov '11

Hi! I tryed to build pypy on the freebsd, 64bit platform, but failed. First of all I downloaded a 32bit-Python to run pypy. And then: [user@server_name ~/pypy-trunk/pypy/translator/goal]$ ~/py32/bin/python translate.py -Ojit [translation:info] Translating target as defined by targetpypystandalone [platform:execute] gcc -m32 -c -O3 -pthread -fomit-frame-pointer /var/tmp/usession-trunk-22/gcctest.c -o /var/tmp/usession-trunk-22/gcctest.o [platform:WARNING] /var/tmp/usession-trunk-22/gcctest.c:28:2: warning: no newline at end of file [platform:execute] gcc -m32 /var/tmp/usession-trunk-22/gcctest.o -L/lib32 -L/usr/lib32 -L`pwd`/lib32 -Wl,-rpath,/lib32 -Wl,-rpath,/usr/lib32 -pthread -o /var/tmp/usession-trunk-22/gcctest printf("sizeof short=%ld\n", (long)sizeof(short)); printf("sizeof unsigned short=%ld\n", (long)sizeof(unsigned short)); printf("sizeof int=%ld\n", (long)sizeof(int)); printf("sizeof unsigned int=%ld\n", (long)sizeof(unsigned int)); printf("sizeof long=%ld\n", (long)sizeof(long)); printf("sizeof unsigned long=%ld\n", (long)sizeof(unsigned long)); printf("sizeof signed char=%ld\n", (long)sizeof(signed char)); printf("sizeof unsigned char=%ld\n", (long)sizeof(unsigned char)); printf("sizeof long long=%ld\n", (long)sizeof(long long)); printf("sizeof unsigned long long=%ld\n", (long)sizeof(unsigned long long)); printf("sizeof size_t=%ld\n", (long)sizeof(size_t)); printf("sizeof time_t=%ld\n", (long)sizeof(time_t)); printf("sizeof wchar_t=%ld\n", (long)sizeof(wchar_t)); printf("sizeof mode_t=%ld\n", (long)sizeof(mode_t)); printf("sizeof pid_t=%ld\n", (long)sizeof(pid_t)); sizeof short=2 sizeof unsigned short=2 sizeof int=4 sizeof unsigned int=4 sizeof long=8 sizeof unsigned long=8 sizeof signed char=1 sizeof unsigned char=1 sizeof long long=8 sizeof unsigned long long=8 sizeof size_t=8 sizeof time_t=8 sizeof wchar_t=4 sizeof mode_t=2 sizeof pid_t=4 Traceback (most recent call last): File "translate.py", line 300, in <module> main() File "translate.py", line 205, in main targetspec_dic, translateconfig, config, args = parse_options_and_load_target() File "translate.py", line 153, in parse_options_and_load_target targetspec_dic = load_target(targetspec) File "translate.py", line 105, in load_target mod = __import__(specname) File "targetpypystandalone.py", line 5, in <module> from pypy.objspace.std.objspace import StdObjSpace File "/place/home/hitrobot/pypy-trunk/pypy/objspace/std/__init__.py", line 1, in <module> from pypy.objspace.std.objspace import StdObjSpace File "/place/home/hitrobot/pypy-trunk/pypy/objspace/std/objspace.py", line 3, in <module> from pypy.interpreter import pyframe, function, special File "/place/home/hitrobot/pypy-trunk/pypy/interpreter/pyframe.py", line 13, in <module> from pypy.rlib import jit, rstack File "/place/home/hitrobot/pypy-trunk/pypy/rlib/rstack.py", line 10, in <module> from pypy.rpython.lltypesystem import rffi, lltype File "/place/home/hitrobot/pypy-trunk/pypy/rpython/lltypesystem/rffi.py", line 824, in <module> sys.maxint, sizeof(lltype.Signed))) AssertionError: Mixed configuration of the word size of the machine: the underlying Python was compiled with maxint=2147483647, but the C compiler says that 'long' is 8 bytes I got this error, then I added "-m32" flag to gcc argument list (in the posix.py source file) , but error occured again :( Does anybody tryed to build pypy-c on the 64bit FreeBsd? Thank you for your answer! -- Shadrin Eugene

5 7

Re: [pypy-dev] New speed.pypy.org version
by Paolo Giarrusso 02 Jul '10

02 Jul '10

Hi Miquel, I'm quite busy (because of a paper deadline next Tuesday), sorry for not answering earlier. I was just struck by an idea: there is a stacked bar plot where the total bar is related to the geometric mean, such that it is normalization-invariant. But this graph _is_ complicated. It is a stacked plot of _logarithms_ of performance ratios? This way, the complete stacked bar shows the logarithm of the product, rather than their sum, i.e. the log of the (geometric mean)^N rather than their arithmetic mean. log of the (geometric mean)^N = N*log of the (geometric mean). Some simple maths (I didn't write it out, so please recheck!) seems to show that showing (a+b*log (ratio)), instead of log(ratio), gives still a fair comparison, obtaining N*a+b*N*log(geomean) = \Theta(log(geomean)). You need to put a and b because showing if the ratio is 1, log(1) is zero (b is the representation scale which is always there). About your workaround: I would like a table with the geometric mean of the ratios, where we get the real global performance ratio among the interpreters. As far as the results of your solution do not contradict that _real_ table, it should be a reasonable workaround (but I would embed the check in the code - otherwise other projects _will be_ bitten by that). Probably, I would like the website to offer such a table to users, and I would like a graph of the overall performance ratio over time (actually revisions). Finally, the docs of your web application should at the very least reference the paper and this conversation (if there's a public archive of the ML, as I think), and ideally explain the issue. Sorry for being too dense, maybe - if I was unclear, please tell me and I'll answer next week. Best regards, Paolo On Mon, Jun 28, 2010 at 11:21, Miquel Torres <tobami(a)googlemail.com> wrote: > Hi Paolo, > > I read the paper, very interesting. It is perfectly clear that to > calculate a normalized total only the geometric mean makes sense. > > However, a stacked bars plot shows the individual benchmarks so it > implicitly is an arithmetic mean. The only solution (apart from > removing the stacked charts and only offering total bars) is the > weighted approach. > > External weights are not very practical though. Codespeed is used by > other projects so an extra option would need to be added to the > settings to allow the introducing of arbitrary weights to benchmarks. > A bit cumbersome. I have an idea that may work. Take the weights from > a defined baseline so that the run times are equal, which is the same > as normalizing to a baseline. It would be the same as now, only that > you can't choose the normalization, it will be weighted (normalized) > according the default baseline (which you already can already > configure in the settings). > > You may say that it is still an arithmetic mean, but there won't be > conflicting results because there is only a single normalization. For > PyPy that would be cpython, and everything would make sense. > I know it is a work around, not a solution. If you think it is a bad > idea, the only other possibility is not to have stacked bars (as in > "showing individual benchmarks"). But I find them useful. Yes you can > see the individual benchmark results better in the normal bars chart, > but there you don't see visually which benchmarks take the biggest > part of the pie, which helps visualize what parts of your program need > most improving. > > What do you think? > > Regards, > Miquel > > > 2010/6/25 Paolo Giarrusso <p.giarrusso(a)gmail.com>: >> On Fri, Jun 25, 2010 at 19:08, Miquel Torres <tobami(a)googlemail.com> wrote: >>> Hi Paolo, >>> >>> I am aware of the problem with calculating benchmark means, but let me >>> explain my point of view. >>> >>> You are correct in that it would be preferable to have absolute times. Well, >>> you actually can, but see what it happens: >>> http://speed.pypy.org/comparison/?hor=true&bas=none&chart=stacked+bars >> >> Ahah! I didn't notice that I could skip normalization! This does not >> fully invalidate my point, however. >> >>> Absolute values would only work if we had carefully chosen benchmaks >>> runtimes to be very similar (for our cpython baseline). As it is, html5lib, >>> spitfire and spitfire_cstringio completely dominate the cummulative time. >> >> I acknowledge that (btw, it should be cumulative time, with one 'm', >> both here and in the website). >> >>> And not because the interpreter is faster or slower but because the >>> benchmark was arbitrarily designed to run that long. Any improvement in the >>> long running benchmarks will carry much more weight than in the short >>> running. >> >>> What is more useful is to have comparable slices of time so that the >>> improvements can be seen relatively over time. >> >> If you want to sum up times (but at this point, I see no reason for >> it), you should rather have externally derived weights, as suggested >> by the paper (in Rule 3). >> As soon as you take weights from the data, lots of maths that you need >> is not going to work any more - that's generally true in many cases in >> statistics. >> And the only way making sense to have external weights is to gather >> them from real world programs. Since that's not going to happen >> easily, just stick with the geometric mean. Or set an arbitrarily low >> weight, manually, without any math, so that the long-running >> benchmarks stop dominating the res. It's no fraud, since the current >> graph is less valid anyway. >> >>> Normalizing does that i >>> think. >> Not really. >> >>> It just says: we have 21 tasks which take 1 second to run each on >>> interpreter X (cpython in the default case). Then we see how other >>> executables compare to that. What would the geometric mean achieve here, >>> exactly, for the end user? >> >> You actually need the geomean to do that. Don't forget that the >> geomean is still a mean: it's a mean performance ratio which averages >> individual performance ratios. >> If PyPy's geomean is 0.5, it means that PyPy is going to run that task >> in 11.5 seconds instead of 21. To me, this sounds exactly like what >> you want to achieve. Moreover, it actually works, unlike what you use. >> >> For instance, ignore PyPy-JIT, and look only CPython and pypy-c (no >> JIT). Then, change the normalization among the two: >> http://speed.pypy.org/comparison/?exe=2%2B35,3%2BL&ben=1,2,3,4,5,6,7,8,9,10… >> http://speed.pypy.org/comparison/?exe=2%2B35,3%2BL&ben=1,2,3,4,5,6,7,8,9,10… >> with the current data, you get that in one case cpython is faster, in >> the other pypy-c is faster. >> It can't happen with the geomean. This is the point of the paper. >> >> I could even construct a normalization baseline $base such that >> CPython seems faster than PyPy-JIT. Such a base should be very fast >> on, say, ai (where CPython is slower), so that $cpython.ai/$base.ai >> becomes 100 and $pypyjit.ai/$base.ai becomes 200, and be very slow on >> other benchmarks (so that they disappear in the sum). >> >> So, the only difference I see is that geomean works, arithm. mean >> doesn't. That's why Real Benchmarkers use geomean. >> >> Moreover, you are making a mistake quite common among non-physicists. >> What you say makes sense under the implicit assumption that dividing >> two times gives something you can use as a time. When you say "Pypy's >> runtime for a 1 second task", you actually want to talk about a >> performance ratio, not about the time. In the same way as when you say >> "this bird runs 3 meters long in one second", a physicist would sum >> that up as "3 m/s" rather than "3 m". >> >>> I am not really calculating any mean. You can see that I carefully avoided >>> to display any kind of total bar which would indeed incur in the problem you >>> mention. That a stacked chart implicitly displays a total is something you >>> can not avoid, and for that kind of chart I still think normalized results >>> is visually the best option. >> >> But on a stacked bars graph, I'm not going to look at individual bars >> at all, just at the total: it's actually less convenient than in >> "normal bars" to look at the result of a particular benchmark. >> >> I hope I can find guidelines against stacked plots, I have a PhD >> colleague reading on how to make graphs. >> >> Best regards >> -- >> Paolo Giarrusso - Ph.D. Student >> http://www.informatik.uni-marburg.de/~pgiarrusso/ >> > -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/

2 2

Improving Stackless/Coroutines implementation
by Gabriel Lavoie 01 Jul '10

01 Jul '10

Hello everyone, as a few knows here, I'm working heavily with PyPy's "stackless" module for my Master degree project to make it more distributed. Since I started to work full time on this project I've encountered a few bugs (mostly related to pickling of tasklets) and missing implementation details in the module. The latest problem I've encountered is to be able to detect when tasklet.kill() is called, within the tasklet being killed. With Stackless CPython, TaskletExit is raised and can be caught but this part wasn't really implemented in PyPy's stackless module. Since the module is implemented on top of coroutines and since coroutine.kill() is called within tasklet.kill(), the exception thrown by the coroutine implementation needs to be caught. Here's the problem: http://codespeak.net/pypy/dist/pypy/doc/stackless.html#coroutines - coro.kill() Kill coro by sending an exception to it. (At the moment, the exception is not visible to app-level, which means that you cannot catch it, and that try: finally: clauses are not honored. This will be fixed in the future.) The exception is not thrown at app level and a coroutine dies silently. Took a look at the code and I've been able to expose a CoroutineExit exception to app level on which I intend implementing TaskletExit correctly. I'm also able to catch the exception as expected but the code is not yet complete. Right now, I have a question on how to expose correctly the CoroutineExit and TaskletExit exceptions to app level. Here's what I did: W_CoroutineExit = _new_exception('CoroutineExit', W_Exception, 'Exit requested...') class AppCoroutine(Coroutine): # XXX, StacklessFlags): def __init__(self, space, state=None): # Some other code here # Exporting new exception to __builtins__ and "exceptions" modules self.w_CoroutineExit = space.gettypefor(W_CoroutineExit) space.setitem( space.exceptions_module.w_dict, space.new_interned_str('CoroutineExit'), self.w_CoroutineExit) space.setitem(space.builtin.w_dict, space.new_interned_str('CoroutineExit'), self.w_CoroutineExit) I talked about this on #pypy (IRC) but people weren't sure about exporting new names to __builtins__. On my side I wanted to make it look as most as possible as how Stackless CPython did it with TaskletExit, which is directly available in __builtins__. This would make code compatible with both Stackless Python and PyPy's stackless module. Also, exporting names this way would only make them appear in __builtins__ when the "_stackless" module is enabled (pypy-c built with --stackless). What are your opinions about it? (Maciej, I already know about yours! ;) Thank you very much, Gabriel (WildChild) -- Gabriel Lavoie glavoie(a)gmail.com

1 1

PyPy 1.3 released
by Maciej Fijalkowski 01 Jul '10

01 Jul '10

======================= PyPy 1.3: Stabilization ======================= Hello. We're please to announce release of PyPy 1.3. This release has two major improvements. First of all, we stabilized the JIT compiler since 1.2 release, answered user issues, fixed bugs, and generally improved speed. We're also pleased to announce alpha support for loading CPython extension modules written in C. While the main purpose of this release is increased stability, this feature is in alpha stage and it is not yet suited for production environments. Highlights of this release ========================== * We introduced support for CPython extension modules written in C. As of now, this support is in alpha, and it's very unlikely unaltered C extensions will work out of the box, due to missing functions or refcounting details. The support is disable by default, so you have to do:: import cpyext before trying to import any .so file. Also, libraries are source-compatible and not binary-compatible. That means you need to recompile binaries, using for example:: python setup.py build Details may vary, depending on your build system. Make sure you include the above line at the beginning of setup.py or put it in your PYTHONSTARTUP. This is alpha feature. It'll likely segfault. You have been warned! * JIT bugfixes. A lot of bugs reported for the JIT have been fixed, and its stability greatly improved since 1.2 release. * Various small improvements have been added to the JIT code, as well as a great speedup of compiling time. Cheers, Maciej Fijalkowski, Armin Rigo, Alex Gaynor, Amaury Forgeot d'Arc and the PyPy team

3 3

Is MySQLdb Working?
by Phyo Arkar 29 Jun '10

29 Jun '10

Hello Pypy team. Thank you for the Greatest project every happen to Programming World. I am trying to run web2py - http://www.web2py.com on pypy , everything working fine except MySQLdb , which i installed over easy_install . >>>> import MySQLdb Traceback (most recent call last): File "<console>", line 1, in <module> ZipImportError: No module named pkg_resources I saw the patch for mysqldb in the trunk , so i have to apply it on mysqldb ? Which version for it ?

1 0

PyPy Master thesis sandboxing
by Carl Friedrich Bolz 28 Jun '10

28 Jun '10

Hi all, just wanted to point this Master thesis using PyPy out: http://www.diku.dk/english/Calendar/masters_defence_soeren/ Didn't know about this, but interesting anyway. Cheers, Carl Friedrich

3 9

Unsubscribe
by Michael Maloney 28 Jun '10

28 Jun '10

On Mon, Jun 28, 2010 at 12:51 PM, <pypy-dev-request(a)codespeak.net> wrote: > Send pypy-dev mailing list submissions to > pypy-dev(a)codespeak.net > > To subscribe or unsubscribe via the World Wide Web, visit > http://codespeak.net/mailman/listinfo/pypy-dev > or, via email, send a message with subject or body 'help' to > pypy-dev-request(a)codespeak.net > > You can reach the person managing the list at > pypy-dev-owner(a)codespeak.net > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of pypy-dev digest..." > > > Today's Topics: > > 1. PyPy 1.3 released (Maciej Fijalkowski) > 2. Re: New speed.pypy.org version (Miquel Torres) > 3. Re: PyPy 1.3 released (Armin Rigo) > 4. PyPy Master thesis sandboxing (Carl Friedrich Bolz) > 5. Re: PyPy Master thesis sandboxing (Maciej Fijalkowski) > 6. Re: PyPy Master thesis sandboxing (Carl Friedrich Bolz) > 7. Re: PyPy Master thesis sandboxing (Maciej Fijalkowski) > 8. Re: PyPy Master thesis sandboxing (S?ren Laursen) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 25 Jun 2010 17:27:52 -0600 > From: Maciej Fijalkowski <fijall(a)gmail.com> > Subject: [pypy-dev] PyPy 1.3 released > To: PyPy Dev <pypy-dev(a)codespeak.net>, "<python-dev(a)python.org>" > <python-dev(a)python.org>, python-announce(a)python.org > Message-ID: > <AANLkTikaN3p6BNFUfXL4RlWB28ZwjrajeUb34r8SGvdy(a)mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > ======================= > PyPy 1.3: Stabilization > ======================= > > Hello. > > We're please to announce release of PyPy 1.3. This release has two major > improvements. First of all, we stabilized the JIT compiler since 1.2 release, > answered user issues, fixed bugs, and generally improved speed. > > We're also pleased to announce alpha support for loading CPython extension > modules written in C. While the main purpose of this release is increased > stability, this feature is in alpha stage and it is not yet suited for > production environments. > > Highlights of this release > ========================== > > * We introduced support for CPython extension modules written in C. As of now, > this support is in alpha, and it's very unlikely unaltered C extensions will > work out of the box, due to missing functions or refcounting details. The > support is disable by default, so you have to do:: > > import cpyext > > before trying to import any .so file. Also, libraries are source-compatible > and not binary-compatible. That means you need to recompile binaries, using > for example:: > > python setup.py build > > Details may vary, depending on your build system. Make sure you include > the above line at the beginning of setup.py or put it in your PYTHONSTARTUP. > > This is alpha feature. It'll likely segfault. You have been warned! > > * JIT bugfixes. A lot of bugs reported for the JIT have been fixed, and its > stability greatly improved since 1.2 release. > > * Various small improvements have been added to the JIT code, as well as a great > speedup of compiling time. > > Cheers, > Maciej Fijalkowski, Armin Rigo, Alex Gaynor, Amaury Forgeot d'Arc and > the PyPy team > > > ------------------------------ > > Message: 2 > Date: Sat, 26 Jun 2010 09:16:52 +0200 > From: Miquel Torres <tobami(a)googlemail.com> > Subject: Re: [pypy-dev] New speed.pypy.org version > To: Paolo Giarrusso <p.giarrusso(a)gmail.com> > Cc: pypy-dev <pypy-dev(a)codespeak.net> > Message-ID: > <AANLkTilgNm7a3o9pSUDgbHwxJ2bNqsYNajUK21JnFBT7(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Hi Paolo, > > well, you are right of course. I had forgotten about the real problem, which > you actually demonstrate quite well with your CPython and pypy-c case: > depending on the normalization you can make any stacked series look faster > than the others. > > I will have a look at the literature and modify normalized stacked plots > accordingly. > > Thanks for taking the time to explain things in such detail. > > Regards, > Miquel > > > 2010/6/25 Paolo Giarrusso <p.giarrusso(a)gmail.com> > >> On Fri, Jun 25, 2010 at 19:08, Miquel Torres <tobami(a)googlemail.com> >> wrote: >> > Hi Paolo, >> > >> > I am aware of the problem with calculating benchmark means, but let me >> > explain my point of view. >> > >> > You are correct in that it would be preferable to have absolute times. >> Well, >> > you actually can, but see what it happens: >> > http://speed.pypy.org/comparison/?hor=true&bas=none&chart=stacked+bars >> >> Ahah! I didn't notice that I could skip normalization! This does not >> fully invalidate my point, however. >> >> > Absolute values would only work if we had carefully chosen benchmaks >> > runtimes to be very similar (for our cpython baseline). As it is, >> html5lib, >> > spitfire and spitfire_cstringio completely dominate the cummulative time. >> >> I acknowledge that (btw, it should be cumulative time, with one 'm', >> both here and in the website). >> >> > And not because the interpreter is faster or slower but because the >> > benchmark was arbitrarily designed to run that long. Any improvement in >> the >> > long running benchmarks will carry much more weight than in the short >> > running. >> >> > What is more useful is to have comparable slices of time so that the >> > improvements can be seen relatively over time. >> >> If you want to sum up times (but at this point, I see no reason for >> it), you should rather have externally derived weights, as suggested >> by the paper (in Rule 3). >> As soon as you take weights from the data, lots of maths that you need >> is not going to work any more - that's generally true in many cases in >> statistics. >> And the only way making sense to have external weights is to gather >> them from real world programs. Since that's not going to happen >> easily, just stick with the geometric mean. Or set an arbitrarily low >> weight, manually, without any math, so that the long-running >> benchmarks stop dominating the res. It's no fraud, since the current >> graph is less valid anyway. >> >> > Normalizing does that i >> > think. >> Not really. >> >> > It just says: we have 21 tasks which take 1 second to run each on >> > interpreter X (cpython in the default case). Then we see how other >> > executables compare to that. What would the geometric mean achieve here, >> > exactly, for the end user? >> >> You actually need the geomean to do that. Don't forget that the >> geomean is still a mean: it's a mean performance ratio which averages >> individual performance ratios. >> If PyPy's geomean is 0.5, it means that PyPy is going to run that task >> in 11.5 seconds instead of 21. To me, this sounds exactly like what >> you want to achieve. Moreover, it actually works, unlike what you use. >> >> For instance, ignore PyPy-JIT, and look only CPython and pypy-c (no >> JIT). Then, change the normalization among the two: >> >> http://speed.pypy.org/comparison/?exe=2%2B35,3%2BL&ben=1,2,3,4,5,6,7,8,9,10… >> >> http://speed.pypy.org/comparison/?exe=2%2B35,3%2BL&ben=1,2,3,4,5,6,7,8,9,10… >> with the current data, you get that in one case cpython is faster, in >> the other pypy-c is faster. >> It can't happen with the geomean. This is the point of the paper. >> >> I could even construct a normalization baseline $base such that >> CPython seems faster than PyPy-JIT. Such a base should be very fast >> on, say, ai (where CPython is slower), so that $cpython.ai/$base.ai >> becomes 100 and $pypyjit.ai/$base.ai becomes 200, and be very slow on >> other benchmarks (so that they disappear in the sum). >> >> So, the only difference I see is that geomean works, arithm. mean >> doesn't. That's why Real Benchmarkers use geomean. >> >> Moreover, you are making a mistake quite common among non-physicists. >> What you say makes sense under the implicit assumption that dividing >> two times gives something you can use as a time. When you say "Pypy's >> runtime for a 1 second task", you actually want to talk about a >> performance ratio, not about the time. In the same way as when you say >> "this bird runs 3 meters long in one second", a physicist would sum >> that up as "3 m/s" rather than "3 m". >> >> > I am not really calculating any mean. You can see that I carefully >> avoided >> > to display any kind of total bar which would indeed incur in the problem >> you >> > mention. That a stacked chart implicitly displays a total is something >> you >> > can not avoid, and for that kind of chart I still think normalized >> results >> > is visually the best option. >> >> But on a stacked bars graph, I'm not going to look at individual bars >> at all, just at the total: it's actually less convenient than in >> "normal bars" to look at the result of a particular benchmark. >> >> I hope I can find guidelines against stacked plots, I have a PhD >> colleague reading on how to make graphs. >> >> Best regards >> -- >> Paolo Giarrusso - Ph.D. Student >> http://www.informatik.uni-marburg.de/~pgiarrusso/<http://www.informatik.uni-marburg.de/%7Epgiarrusso/> >> >

1 0

New speed.pypy.org version
by Miquel Torres 26 Jun '10

26 Jun '10

Hi all!, I want to announce a new version of the benchmarks site speed.pypy.org. After about 6 months, it finally shows the vision I had for such a website: usefull for pypy developers but also for the general public following pypy's or even other python implementation's development. On to the changes. There are now three views: "Changes", "Timeline" and "Comparison": The Overview was renamed to Changes, and its inline plot bars got removed because you can get the exact same plot in the Comparison view now (and then some). The Timeline got selectable baseline and "humanized" date labels for the x axis. The new Comparison view allows, well, comparing of "competing" interpreters, which will also be of interest to the wider Python community (specially if we can add unladen, ironpython and JPython results). Two examples of interesting comparisons are: - relative bars ( http://speed.pypy.org/comparison/?bas=2%2B35&chart=relative+bars): here we see that the jit is faster than psyco in all cases except spambayes and slowspitfire, were the jit cannot make up for pypy-c's abismal performance. Interestingly, in the only other case where the jit is slower than cpython, the ai benchmark, psyco performs even worse. - stacked bars horizontal( http://speed.pypy.org/comparison/?hor=true&bas=2%2B35&chart=stacked+bars): This is not meant to "demonstrate" that overall the jit is over two times faster than cpython. It is just another way for a developer to picture how long a programme would take to complete if it were composed of 21 such tasks. You can see that cpython's (the normalization chosen) benchmarks all take 1"relative" second. pypy-c needs more or less the same time, some "tasks" being slower and some faster. Psyco shows an interesting picture: >From meteor-contest downwards (fortuitously) , all benchmarks are extremely "compressed", which means they are speeded up by psyco quite a lot. But any further speed up wouldn't make overall time much shorter because the first group of benchmarks now takes most of the time to complete. pypy-c-jit is a more extreme case of this: If the jit accelerated all "fast" benchmarks to 0 seconds (infinitely fast), it would only get about twice as fast as now because ai, slowspitfire, spambayes and twisted_tcp now need half the entire execution time. An good demonstration of "you are only as fast as your slowest part". Of course the aggregate of all benchmarks is not a real app, but it is still fun. I hope you find the new version useful, and as always any feedback is welcome. Cheers! Miquel

4 12

virtualenv support and directory hierarchy
by Antonio Cuni 19 Jun '10

19 Jun '10

Hi all, I am investigating how to make virtualenv working on pypy and I'm running into a couple of issues: the most important one is that virtualenv relies on sys.prefix (which does not exists in pypy) to find the standard library, and the other is that the standard library of pypy is supposed to be put in /usr/share instead of /usr/lib (or /usr/local/*). Currently, a pypy installation is supposed to have this structure: /usr/bin/pypy-c /usr/share/pypy-1.2/pypy/lib/ /usr/share/pypy-1.2/lib-python/modified-2.5.2 /usr/share/pypy-1.2/lib-python/2.5.2 In such a situation, sys.pypy_prefix is set to '/usr/share/pypy-1.2'. I propose to change it in this way: /usr/bin/pypy-c /usr/lib/pypy1.2/lib-pypy/ /usr/lib/pypy1.2/lib-python/modified-2.5.2 /usr/lib/pypy1.2/lib-python/2.5.2 where lib-pypy contains what it's now in pypy/lib. In such a situation, sys.prefix would be set to '/usr', in a similar way as cpython. Also, we should also add a sys.exec_prefix which is meant to be always equal to sys.prefix (at least for now). (I removed the dash in pypy-1.2 for consistency with cpython, which uses something like lib/python2.6). Moreover, I would also like virtualenv to work from an svn checkout/source tarball of pypy, without any needing of installing it system-wide. To do so, we need to find a sensible value for sys.prefix, considering that tools like virtualenv suppose to find the stdlib under sys.prefix+'lib/'+something_else. So, the proposed new structure is this: /path/to/pypy-trunk/ /path/to/pypy-trunk/lib/pypy1.2/{lib-pypy,modified-2.5.2,2.5.2} sys.prefix == '/path/to/pypy-trunk' The drawback is that before getting to the real files you have to walk a lot of empty directories, and that we should manually change the name of pypy1.2 each time we increase the version number (not that it happens very often :-)). One side-advantage is that in this way we would move pypy/lib outside the main pypy package, which is good because it's not really a part of it. Finally, we should probably think of where to put the include/ directory (plus other that might be needed to build extension), but I'll let cpyext experts to say what it's better :-) What do you think? Any comment/suggestion/problem that I overlooked? ciao, Anto

4 6

lltype Questions
by Dan Roberts 18 Jun '10

18 Jun '10

Hey Everybody, It seems no one's in #pypy at the moment, so I figured I'd post to the mailing list for non-realtime questioning. I wrote my implementation of PyUnicode_DecodeUTF16() which is correct as best I can tell (and it passes its test, but I wrote the test too :-) ). The code is here: http://paste.pocoo.org/show/226753/ and its test is here: http://paste.pocoo.org/show/226752/ . I have a few questions, like I said, the test passes, but I'm unsure of some of the code I've written. There are XXX and FIXME and other comments throughout the code, those represent places where I'm unsure. In the test code, line 6 I'm not sure if that malloc() is correct, and even if it is, would it have been better (and also valid) to lltype.malloc(lltype.Signed, ...) ? Then from line 7 to 12 I assign "native" integers to the malloc-ed memory, is this done correctly? ex. pendian[0] = -1 or should it be pendian[0] = rffi.cast(lltype.Signed, -1) ? In the implementation, I'm fairly certain I've marked everything with FIXME or XXX, so there's not much more to say about that... I'd really appreciate anyone who's willing to take a look. Thanks, Dan

2 1