Mailman 3 September 2010 - pypy-dev

Building pypy on freebsd 64bit
by Shadrin Eugene 18 Nov '11

18 Nov '11

Hi! I tryed to build pypy on the freebsd, 64bit platform, but failed. First of all I downloaded a 32bit-Python to run pypy. And then: [user@server_name ~/pypy-trunk/pypy/translator/goal]$ ~/py32/bin/python translate.py -Ojit [translation:info] Translating target as defined by targetpypystandalone [platform:execute] gcc -m32 -c -O3 -pthread -fomit-frame-pointer /var/tmp/usession-trunk-22/gcctest.c -o /var/tmp/usession-trunk-22/gcctest.o [platform:WARNING] /var/tmp/usession-trunk-22/gcctest.c:28:2: warning: no newline at end of file [platform:execute] gcc -m32 /var/tmp/usession-trunk-22/gcctest.o -L/lib32 -L/usr/lib32 -L`pwd`/lib32 -Wl,-rpath,/lib32 -Wl,-rpath,/usr/lib32 -pthread -o /var/tmp/usession-trunk-22/gcctest printf("sizeof short=%ld\n", (long)sizeof(short)); printf("sizeof unsigned short=%ld\n", (long)sizeof(unsigned short)); printf("sizeof int=%ld\n", (long)sizeof(int)); printf("sizeof unsigned int=%ld\n", (long)sizeof(unsigned int)); printf("sizeof long=%ld\n", (long)sizeof(long)); printf("sizeof unsigned long=%ld\n", (long)sizeof(unsigned long)); printf("sizeof signed char=%ld\n", (long)sizeof(signed char)); printf("sizeof unsigned char=%ld\n", (long)sizeof(unsigned char)); printf("sizeof long long=%ld\n", (long)sizeof(long long)); printf("sizeof unsigned long long=%ld\n", (long)sizeof(unsigned long long)); printf("sizeof size_t=%ld\n", (long)sizeof(size_t)); printf("sizeof time_t=%ld\n", (long)sizeof(time_t)); printf("sizeof wchar_t=%ld\n", (long)sizeof(wchar_t)); printf("sizeof mode_t=%ld\n", (long)sizeof(mode_t)); printf("sizeof pid_t=%ld\n", (long)sizeof(pid_t)); sizeof short=2 sizeof unsigned short=2 sizeof int=4 sizeof unsigned int=4 sizeof long=8 sizeof unsigned long=8 sizeof signed char=1 sizeof unsigned char=1 sizeof long long=8 sizeof unsigned long long=8 sizeof size_t=8 sizeof time_t=8 sizeof wchar_t=4 sizeof mode_t=2 sizeof pid_t=4 Traceback (most recent call last): File "translate.py", line 300, in <module> main() File "translate.py", line 205, in main targetspec_dic, translateconfig, config, args = parse_options_and_load_target() File "translate.py", line 153, in parse_options_and_load_target targetspec_dic = load_target(targetspec) File "translate.py", line 105, in load_target mod = __import__(specname) File "targetpypystandalone.py", line 5, in <module> from pypy.objspace.std.objspace import StdObjSpace File "/place/home/hitrobot/pypy-trunk/pypy/objspace/std/__init__.py", line 1, in <module> from pypy.objspace.std.objspace import StdObjSpace File "/place/home/hitrobot/pypy-trunk/pypy/objspace/std/objspace.py", line 3, in <module> from pypy.interpreter import pyframe, function, special File "/place/home/hitrobot/pypy-trunk/pypy/interpreter/pyframe.py", line 13, in <module> from pypy.rlib import jit, rstack File "/place/home/hitrobot/pypy-trunk/pypy/rlib/rstack.py", line 10, in <module> from pypy.rpython.lltypesystem import rffi, lltype File "/place/home/hitrobot/pypy-trunk/pypy/rpython/lltypesystem/rffi.py", line 824, in <module> sys.maxint, sizeof(lltype.Signed))) AssertionError: Mixed configuration of the word size of the machine: the underlying Python was compiled with maxint=2147483647, but the C compiler says that 'long' is 8 bytes I got this error, then I added "-m32" flag to gcc argument list (in the posix.py source file) , but error occured again :( Does anybody tryed to build pypy-c on the 64bit FreeBsd? Thank you for your answer! -- Shadrin Eugene

5 7

Re: [pypy-dev] Question on the future of RPython
by Paolo Giarrusso 01 Oct '10

01 Oct '10

On Thu, Sep 30, 2010 at 03:21, Terrence Cole <list-sink(a)trainedmonkeystudios.org> wrote: > On Wed, 2010-09-29 at 23:50 +0200, Paolo Giarrusso wrote: > I think it makes sense to treat the sorts of optimizations you can do > with static analysis of a dynamic language with the same sort of guarded > optimism that is used in pypy's jit compiler: run with the assumption > that it will all work out, but watch for failure and fallback to a safe > slow-path nicely when things go off the rails. Agreed, but "watch out for failure" is done by guards, and one of the further advantages of type inference, on top of "type feedback", described there, is being able to remove some of those guards, because that might help in some inner loops. In some cases, that's still possible, by "watching out" not during code execution but during execution of infrequent events. In Java loading a class might invalidate some optimization assumptions (like "this class has no subclass", useful for inlining without guards), but that's checked by the classloading fast-path. Same things could apply to allow Python cross-module type-inferencing. I don't know exactly how modules at JIT compile-time and runtime can be different, but I guess that invalidation at module loading should catch that. And invalidate lots of compiled code, which is usually fine. The interaction of this with tracing is actually interesting: in a Python tracing JIT, you could keep the traces and restore omitted guards, but when your JIT traces the Python interpreter, I wonder how do you express any of this. One can insert a guard only if needed, but telling "hey, this is invalid" requires some special API. My proposal, here, would be a "virtual guard", which is recorded in the guard but omitted from the output. The "omission" is what can be invalidated, but the trace itself (not its compiled version) is kept (because it can still be executed). If some form of this makes actually sense (I've just thought on it 5 minutes), this is something worth publishing, which would allow me to take some time of my PhD to work on it - if it's not already done by papers on tracing JITs. -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/

3 4

Fwd: Question on the future of RPython
by Paolo Giarrusso 30 Sep '10

30 Sep '10

I'm forwarding this mail that Terrence Cole sent me privately for no reason apparent to me, because without it my mail makes less sense. ---------- Forwarded message ---------- From: Terrence Cole <list-sink(a)trainedmonkeystudios.org> Date: Thu, Sep 30, 2010 at 03:21 Subject: Re: [pypy-dev] Question on the future of RPython To: Paolo Giarrusso <p.giarrusso(a)gmail.com> On Wed, 2010-09-29 at 23:50 +0200, Paolo Giarrusso wrote: > The uselessness of static type analysis in dynamic languages was shown > concretely in 1991 on a SmallTalk derivative, Self, the father of > JavaScript, but I've seen the result only in the PhD thesis* of U. > Hölzle, one of Self authors - and I'm pointing it out because I'm not > sure how well known it is. I've read parts of that at some point in the past. > That might explain why Brett Canon's PhD thesis was not that > successful. Of course, after reading his thesis you might be able in > theory to address his concerns, and he acknowledges that type analysis > might help with a few specific cases. I'm not sure about Brendan > Eich's points, but I don't have the time, unfortunately, to > investigate them. Dr. Canon achieved an ~1% performance improvement. From his conclusion, the most significant problem he found was that most program data is stored in object attributes and object attributes were not type-inferable with the methods he used. > * Title: "Adaptive optimization for Self: Reconciling High Performance > with Exploratory Programming". See, among others, sec. 7.2, "Type > analysis does not improve performance of object-oriented programs", > sec. 7.4.1, "type analysis exhibits unstable performance". The > conclusion is that in most cases it helps more to use "type feedback" > , indicates profile-guided optimization applied Scanning through the 7.2, they say: "In general, type analysis cannot infer the result types of non-inlined message sends, of arguments of non-inlined sends, or of assignable slots (i.e., instance variables). Since sends to such values are fairly frequent, a large fraction of message sends does not benefit from type analysis." So they seem to have run into similar problems: the type inferencing algorithm doesn't get to sink its teeth into any of the bits that most need it. Chapter3 of Cannon05 details what specific parts of python can't be type-inferred. It turns out this is most of python. For example, type inferencing can't cross module boundaries because the module used at runtime might be different from the one present at compile time. While perfectly true, this is probably going to be quite rare in practice. I think it makes sense to treat the sorts of optimizations you can do with static analysis of a dynamic language with the same sort of guarded optimism that is used in pypy's jit compiler: run with the assumption that it will all work out, but watch for failure and fallback to a safe slow-path nicely when things go off the rails. > See below further inline replies. > > On Tue, Sep 28, 2010 at 21:49, Terrence Cole > <list-sink(a)trainedmonkeystudios.org> wrote: > > I think this is a disconnect. Applying a jit to a non-interpretted > > language -- Jacob here seems to think I was talking about a static, > > compiled subset of python -- makes little sense. > > JIT just makes much _more_ sense for dynamic languages. Profile-guided > optimization (PGO) give impressive performance improvements for C, but > can also worsen performance when the execution profile changes > (because of different inputs), and JIT would solve this problem. Since > we are talking about optimized C, "impressive performance > improvements" is more like 20% more (so to say) than like 10x-100x, > and it's mostly about things like static branch prediction. > For RPython, it makes more sense to reuse the support from the > translation backend (JVM and .NET should do this by default, in C you > can use PGO on the translation output. > > >> There are some undesirable things about static analysis, but it can > >> sure be useful from optimisation, security and reliability > >> perspectives. > > > > Brendan Eich agrees [2]. This is heartening, because javascript has > > much in common with python. > > > [1] http://www.ocf.berkeley.edu/~bac/thesis.pdf > > [2] http://brendaneich.com/2010/08/static-analysis-ftw/ > -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/

3 2

Re: [pypy-dev] PyPy JIT & C extensions, greenlet
by Andrew Francis 30 Sep '10

30 Sep '10

Hi Maciej: Message: 4 Date: Mon, 27 Sep 2010 14:23:14 +0200 From: Maciej Fijalkowski <fijall(a)gmail.com> Subject: Re: [pypy-dev] PyPy JIT & C extensions, greenlet To: "Ian P. Cooke" <ipc(a)srand.net> Cc: pypy-dev(a)codespeak.net Message-ID: <AANLkTinYC+Orf9RXfyST3ZJPX0ZM0nAXYsh=tH8PtAxK(a)mail.gmail.com> Content-Type: text/plain; charset=UTF-8 >greenlet C module is quite incompatible with pypy and won't work. >However making pypy work with jit and stackless is something that >requires a bit of work only (teaching jit how to unroll the stack >mostly) and I plan to look into it in the very near future. I talked briefly with Armin at EuroPython about Stackless and JIT. I poke around pypy-dev and use stackless.py. However I am very interested in learning about how the stackless transform works and how pypy works so I could help. I have been at Stackless for about five years now and I wouldn't mind spending a year learning pypy. Cheers, Andrew

2 2

Question on the future of RPython
by Saravanan Shanmugham 30 Sep '10

30 Sep '10

I understand from various threads here, that RPython is not for general purpose use. Why this lack of Focus on general use. I am looking at this and I am thinking and comparing this to a corporation that is working on this awesome product. They are so focused on this awesome final product vision that they fail to realize the awesome potential of some if its intermediate side deliverables. PyPy is definitely gaining momentum. But as a strategy to build that momentum, and gain new converts it should put some focus on some of its niche strengths. Things other python implementions cannot do. One such niche is its RPython and RPython Compiler. No other python implementation can convert python programs to executables. I am seeing growing interest in writing Rpython code for performance critical code and even potentially compiling it to binaries. http://olliwang.com/2009/12/20/aes-implementation-in-rpython/ http://alexgaynor.net/2010/may/15/pypy-future-python/ Is it possible the PyPy team may be understating the significance of RPython? Am I crazy to think this way? :-) Sarvi

14 57

PyPy JIT & C extensions, greenlet
by Ian P. Cooke 28 Sep '10

28 Sep '10

There was a recent thread with the same subject and I would like to look into this a bit more. I knew pypy-stackless wouldn't work after I built a working 64-bit pypy w/ JIT, well, now I'm intrigued. I will look at the code more closely soon. Armin, Carl Friedrich, would you answer a couple of questions in the mean-time? What is the largest roadblock to making pypy-stackless work on pypy w/ JIT? Would it be possible/easier to port the greenlet module? Having built-in support for co-routines would be very nice but my own goal is to get greenlet working in any manner. If I could build a 64-bit pypy w/ JIT and then easy_install greenlet, that would work for me. Thanks, Ian P.S. congratulations on all your recent progress! I always look forward for the next pypy blog update :)

4 4

Readiness of asmgcc for x86_64 linux?
by Gary Robinson 24 Sep '10

24 Sep '10

Hi, I saw the PyPy Status Blog post mentioning that there is a working asmgcc for x86_64 linux. I wonder if you could clarify the status of it a bit further. The last thing Jason Creighton wrote on the subject that I can find, from Aug 13, was: "...the bottom line is that the main goal of my GSoC was accomplished: A working 64-bit PyPy JIT. Hopefully I'll be able to complete asmgcc-64, and make the JIT even faster..." But the new Status Blog post says " It not only includes working 64bit JIT (merged into PyPy trunk), but also a working asmgcc for x86_64 linux platform, that makes it possible to run the JIT on this architecture with our advanced garbage collectors" So it sounds like he (or someone) DID get the Linux version of it working. Has it been merged into the trunk? Does it seem stable? You say: "Expect this to be a major selling point for the next PyPy release :-)" Do you have an estimate of when that'll come out? I'm looking forward to testing PyPy for some of our music recommendation code. The main thing holding me back so far is the lack of 64-bit support. The other thing in the way is that I need to use multiple cores. I can home-grow a solution for my needs, but it would be great if the python multiprocessing library were to be supported. I see "r77223 - in pypy/branch/fast-forward/pypy/module/_multiprocessing: . test" in the svn commit log, dated Tuesday of this week (http://permalink.gmane.org/gmane.comp.python.pypy.cvs/29865)... I'm hoping that means it's going to be supported soon? That would be really great. Thanks, Gary -- Gary Robinson CTO Emergent Discovery, LLC personal email: garyrob(a)me.com work email: grobinson(a)emergentdiscovery.com Company: http://www.emergentdiscovery.com Blog: http://www.garyrobinson.net

5 9

PyCon Call for Proposals -- PyCon 2011
by Alex Gaynor 23 Sep '10

23 Sep '10

Call for proposals -- PyCon 2011 -- <http://us.pycon.org/2011/> =============================================================== Proposal Due date: November 1st, 2010 PyCon is back! With a rocking new website, a great location and more Python hackers and luminaries under one roof than you could possibly shake a stick at. We've also added an "Extreme" talk track this year - no introduction, no fluff - only the pure technical meat! PyCon 2011 will be held March 9th through the 17th, 2011 in Atlanta, Georgia. (Home of some of the best southern food you can possibly find on Earth!) The PyCon conference days will be March 11-13, preceded by two tutorial days (March 9-10), and followed by four days of development sprints (March 14-17). PyCon 2011 is looking for proposals for the formal presentation tracks (this includes "extreme talks"). A request for proposals for poster sessions and tutorials will come separately. Want to showcase your skills as a Python Hacker? Want to have hundreds of people see your talk on the subject of your choice? Have some hot button issue you think the community needs to address, or have some package, code or project you simply love talking about? Want to launch your master plan to take over the world with Python? PyCon is your platform for getting the word out and teaching something new to hundreds of people, face to face. In the past, PyCon has had a broad range of presentations, from reports on academic and commercial projects, tutorials on a broad range of subjects, and case studies. All conference speakers are volunteers and come from a myriad of backgrounds: some are new speakers, some have been speaking for years. Everyone is welcome, so bring your passion and your code! We've had some incredible past PyCons, and we're looking to you to help us top them! Online proposal submission is open now! Proposals will be accepted through November 10th, with acceptance notifications coming out by January 20th. To get started, please see: <http://us.pycon.org/2011/speaker/> For videos of talks from previous years - check out: <http://python.mirocommunity.org/category/conferences> For more information on "Extreme Talks" see: <http://us.pycon.org/2011/speaker/extreme/> We look forward to seeing you in Atlanta! Please also note - registration for PyCon 2011 will also be capped at a maximum of 1,500 delegates, including speakers. When registration opens (soon), you're going to want to make sure you register early! Speakers with accepted talks will have a guaranteed slot. Important Dates: * November 1st, 2010: Talk proposals due. * December 15th, 2010: Acceptance emails sent. * January 19th, 2010: Early bird registration closes. * March 9-10th, 2011: Tutorial days at PyCon. * March 11-13th, 2011: PyCon main conference. * March 14-17th, 2011: PyCon sprints days. Contact Emails: Van Lindberg (Conference Chair) - van(a)python.org Jesse Noller (Co-Chair) - jnoller(a)python.org PyCon Organizers list: pycon-organizers(a)python.org -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Voltaire "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me

1 0

Re: [pypy-dev] [pypy-svn] r77101 - in pypy/trunk/pypy: jit/tl module/__builtin__ module/__builtin__/test module/pypyjit/test
by Antonio Cuni 16 Sep '10

16 Sep '10

Hi, On 16/09/10 07:27, hakanardo(a)codespeak.net wrote: > Log: > Allow jit to unroll calls to max() and min() with more than one argument. > [cut] > +@unroll_safe > @specialize.arg(2) > def min_max(space, args, implementation_of): > if implementation_of == "max": > compare = space.gt > else: > compare = space.lt > + > + args_w = args.arguments_w > + if len(args_w)> 1 and not args.keywords: # Unrollable case > + w_max_item = None > + for w_item in args_w: > + if w_max_item is None or \ > + space.is_true(compare(w_item, w_max_item)): > + w_max_item = w_item > + return w_max_item > + else: > + return min_max_loop(space, args, implementation_of) I don't think it's a good idea. What happens if I call max() over a list of 1 million of elements? We obviously don't want the jit to unroll 1 million of iterations. Or am I missing something? ciao, Anto

3 2

Re: [pypy-dev] [pypy-svn] r77083 - pypy/branch/jitffi
by Maciej Fijalkowski 16 Sep '10

16 Sep '10

Hey anto. There was a SoC about that, I guess it would be good to chat about it at least (personally I think jitting rlib/libffi is exactly bad layer to be jitted and some experiments were done). Cheers, fijal On Wed, Sep 15, 2010 at 1:07 PM, <antocuni(a)codespeak.net> wrote: > Author: antocuni > Date: Wed Sep 15 13:07:20 2010 > New Revision: 77083 > > Added: > pypy/branch/jitffi/ (props changed) > - copied from r77082, pypy/trunk/ > Log: > a branch in which to try to jit rlib/libffi.py > > > _______________________________________________ > pypy-svn mailing list > pypy-svn(a)codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn >

2 1

pypy-dev September 2010