Mailman 3 April 2011 - pypy-dev

[GSoC] Developing a benchmark suite (for Python 3.x)
by DasIch April 8, 2011

April 8, 2011

Hello Guys, I would like to present my proposal for the Google Summer of Code, concerning the idea of porting the benchmarks to Python 3.x for speed.pypy.org. I think I have successfully integrated the feedback I got from prior discussions on the topic and I would like to hear your opinion. Abstract ======= As of now there are several benchmark suites used by Python implementations, PyPy[1] uses the benchmarks developed for the Unladen Swallow[2] project as well as several other benchmarks they implemented on their own, CPython[3] uses the Unladen Swallow benchmarks and several "crap benchmarks used for historical reasons"[4]. This makes comparisons unnecessarily hard and causes confusion. As a solution to this problem I propose merging the existing benchmarks - at least those considered worth having - into a single benchmark suite which can be shared by all implementations and ported to Python 3.x. Milestones The project can be divided into several milestones: 1. Definition of the benchmark suite. This will entail contacting developers of Python implementations (CPython, PyPy, IronPython and Jython), via discussion on the appropriate mailing lists. This might be achievable as part of this proposal. 2. Implementing the benchmark suite. Based on the prior agreed upon definition, the suite will be implemented, which means that the benchmarks will be merged into a single mercurial repository on Bitbucket[5]. 3. Porting the suite to Python 3.x. The suite will be ported to 3.x using 2to3[6], as far as possible. The usage of 2to3 will make it easier make changes to the repository especially for those still focusing on 2.x. It is to be expected that some benchmarks cannot be ported due to dependencies which are not available on Python 3.x. Those will be ignored by this project to be ported at a later time, when the necessary requirements are met. Start of Program (May 24) ====================== Before the coding, milestones 2 and 3, can begin it is necessary to agree upon a set of benchmarks, everyone is happy with, as described. Midterm Evaluation (July 12) ======================= During the midterm I want to finish the second milestone and before the evaluation I want to start in the third milestone. Final Evaluation (Aug 16) ===================== In this period the benchmark suite will be ported. If everything works out perfectly I will even have some time left, if there are problems I have a buffer here. Probably Asked Questions ====================== Why not use one of the existing benchmark suites for porting? The effort will be wasted if there is no good base to build upon, creating a new benchmark suite based upon the existing ones ensures that. Why not use Git/Bazaar/...? Mercurial is used by CPython, PyPy and is fairly well known and used in the Python community. This ensures easy accessibility for everyone. What will happen with the Repository after GSoC/How will access to the repository be handled? I propose to give administrative rights to one or two representatives of each project. Those will provide other developers with write access. Communication ============= Communication of the progress will be done via Twitter[7] and my blog[8], if desired I can also send an email with the contents of the blog post to the mailing lists of the implementations. Furthermore I am usually quick to answer via IRC (DasIch on freenode), Twitter or E-Mail(dasdasich(a)gmail.com) if anyone has any questions. Contact to the mentor can be established via the means mentioned above or via Skype. About Me ======== My name is Daniel Neuhäuser, I am 19 years old and currently a student at the Bergstadt-Gymnasium Lüdenscheid[9]. I started programming (with Python) about 4 years ago and became a member of the Pocoo Team[10] after successfully participating in the Google Summer of Code last year, during which I ported Sphinx[11] to Python 3.x and implemented an algorithm to diff abstract syntax trees to preserve comments and translated strings which has been used by the other GSoC projects targeting Sphinx. .. [1]: https://bitbucket.org/pypy/benchmarks/src .. [2]: http://code.google.com/p/unladen-swallow/ .. [3]: http://hg.python.org/benchmarks/file/tip/performance .. [4]: http://hg.python.org/benchmarks/file/62e754c57a7f/performance/README .. [5]: http://bitbucket.org/ .. [6]: http://docs.python.org/library/2to3.html .. [7]: http://twitter.com/#!/DasIch .. [8]: http://dasdasich.blogspot.com/ .. [9]: http://bergstadt-gymnasium.de/ .. [10]: http://www.pocoo.org/team/#daniel-neuhauser .. [11]: http://sphinx.pocoo.org/ P.S.: I would like to get in touch with the IronPython developers as well, unfortunately I was not able to find a mailing list or IRC channel is there anybody how can send me in the right direction?

1 1

Pypy custom interpreter JIT question
by Andrew Brown April 6, 2011

April 6, 2011

Hello PyPy team, So I became interested in the translation toolchain plus JIT compiler generator, and in attempt to learn more about it, I set out to write a very simple interpreter for the language BF (brainf***). Simple enough, 8 opcodes, each with no arguments, and most have a one line implementation. Also plenty of examples out there to run, like a mandelbrot generator =) So I wrote up an interpreter for it in RPython. This worked great until I tried to enable the JIT option, at which point it would produce incorrect results. Strange, I thought I may have been using the hints incorrectly, I couldn't find too many details on exactly what the red and green vars should be. Here's the strange part though: I finally fixed it by changing an implementation detail that shouldn't have changed semantics at all. My implementation creates an instance of a Tape object which has two attributes: a list of integers representing the state of the machine, and a single integer identifying the current active cell on the tape. The implementation of each opcode was a method of this class, and the state of the program (what I passed as the "red" variable) was the instance of this class. After I manually factored the functionality of the class directly into the main dispatch loop and got rid of the class entirely, the JIT compiler started producing correct results. Can anyone help me figure out why my first attempt didn't work? Do red variables that are in class instances need to be handled different somehow? Here's the initial version that runs incorrectly when translated with JIT: https://bitbucket.org/brownan/bf-interpreter/src/c4679b354313/targetbf.py Here's the modified version that seems to work just fine: https://bitbucket.org/brownan/bf-interpreter/src/8095853278e9/targetbf.py In particular, note the elimination of the Tape object in the second version, and the differences in the mainloop function as well as the differences in the "red" variables. I've also included a few example BF programs if someone wants to try it out. The hanoi example crashes almost immediately with the first version translated with JIT. By the way, I've been translating it with the latest version of PyPy off of bitbucket. (latest as of a few weeks ago, that is) Thanks and great work on this project! -Andrew Brown

9 37

Re: [pypy-dev] A Short Introduction
by Ryan Baker April 5, 2011

April 5, 2011

I apologize for the additional message, but it seems I forgot to put mention IRC nick (thanks Laura!). I'll be using the nick barren_yak. See you all there. Best Regards, Ryan

1 0

A Short Introduction
by Ryan Baker April 5, 2011

April 5, 2011

Greetings PyPy-dev! I'm new here on the mailing list (and to PyPy) and thought I would give a short introduction. My name is Ryan and I'm a student at Oregon State University in Computer Science. I'm currently interning at Intel until September. A little technical background: I love Python (surprise!), and have been programming in various languages since I was about 12. I have a large C and Java background while dabbling in others such as LISP, Ruby, Obj-C, and Lua. Most of my coding has been small personal projects or academic work and decided to finally get involved in a community. I was drawn to the PyPy project after I played with if over a weekend and found it to be extremely relevant to my personal interests. I look forward to meeting all of you both on the mailing list and on IRC and I hope that I can put some of my time to good use and help PyPy! On a side note, I usually am very quick to respond to emails so if you ever have a question, idea, or anything you want to bounce off of me, please feel free to shoot me a message. Best Regards, Ryan

1 0

Re: [pypy-dev] minimum system requirements, build configuration
by Liam Staskawicz April 5, 2011

April 5, 2011

Hi David, thanks for the quick response. On Tuesday, April 5, 2011 at 7:18 AM, David Schneider wrote: > Hi Liam, > > On Tue, Apr 5, 2011 at 07:25, Liam Staskawicz <lstask(a)gmail.com> wrote: > > Hi - I've been looking at pypy as a possible application runtime in an ARM9 > > Linux based system. I understand that the in-progress ARM jit backend > > targets only ARMv7, but I'm still interested in characterizing a translated > > pypy-c interpreter on this system with regard to CPU and memory usage. Of > > course I look forward to possible ARMv5 support for the jit in the future, > > but if I've understood correctly, fully interpreted mode should be supported > > - please let me know if this is not correct! > > This is correct, the non-jitted version can be cross-translated to > ARM. Although it still only supports the Boehm GC so running PyPy > without the JIT will be slower than CPython. The JIT currently targets > the ARMv7 architecture, we have not tested on an ARMv5 processor yet, > but it would be interesting to see what is missing for compatibility > with ARMv5. I see - non-jitted is perhaps not such an interesting option at the moment then, I suppose. Are the other GC options not available because they rely on the jit, or is it another dependency? I'll try to find some time to look into what's required for ARMv5 support. If you have any thoughts on particularly ARMv7-centric functionality in the current port that would need to be addressed, that would be interesting to know. > > > To that end, are there minimum recommended system requirements for pypy, > > specifically in terms of memory? As a reference, something like the Mono > > 'Small Footprint' wiki page applied to pypy would be what I'm looking > > for: http://www.mono-project.com/Small_footprint > At this point in time we have no official minimum requirements to run > pypy on ARM, but (quoting Armin) probably anything starting from 64MB > should be fine, depending on the application. The BeagleBoard which is > used to develop the ARM backend has 512 MB and pypy works fine on it > so far. OK, good to know - thanks! > > > On a related note, most of the build config info I've come across seems to > > revolve around a scratchbox2 build environment targeting maemo - indeed it > > seems to be the main other 'platform' option in the pypy build script, other > > than 'host'. Is there any particularly tight coupling between maemo and > > pypy, or should I hope to be able to set up a generic sbox2 environment for > > my target and come away with a working build? Are there any other docs or > > config snippets that detail how to set up a generic cross compile > > environment for pypy? > > We used the scratchbox2 because it allows to create an environment > which can cross-compile programs, but also run small programs in a > context similar to the target environment. This is used to gather > information about the target, required for the translation. > For the ARM backend we are using an Ubuntu rootfs with the build-tools > installed and the gcc-arm cross compiler installed on the host. The > translation toolchain only needs to now the name of the sb2 > environment in which to compile and execute the programs. One thing to > note is that the Python interpreter on the host must match the target > system in the sense of 32/64 bit. Besides these points any sb2 > environment could be used to try the translation for a given target. Sounds good - I'll give it a shot. Thanks again.

3 7

minimum system requirements, build configuration
by Liam Staskawicz April 5, 2011

April 5, 2011

Hi - I've been looking at pypy as a possible application runtime in an ARM9 Linux based system. I understand that the in-progress ARM jit backend targets only ARMv7, but I'm still interested in characterizing a translated pypy-c interpreter on this system with regard to CPU and memory usage. Of course I look forward to possible ARMv5 support for the jit in the future, but if I've understood correctly, fully interpreted mode should be supported - please let me know if this is not correct! To that end, are there minimum recommended system requirements for pypy, specifically in terms of memory? As a reference, something like the Mono 'Small Footprint' wiki page applied to pypy would be what I'm looking for: http://www.mono-project.com/Small_footprint On a related note, most of the build config info I've come across seems to revolve around a scratchbox2 build environment targeting maemo - indeed it seems to be the main other 'platform' option in the pypy build script, other than 'host'. Is there any particularly tight coupling between maemo and pypy, or should I hope to be able to set up a generic sbox2 environment for my target and come away with a working build? Are there any other docs or config snippets that detail how to set up a generic cross compile environment for pypy? Thanks for any tips or relevant info! Liam

1 0

A Little Bit of Python Episode 17
by Michael Foord April 5, 2011

April 5, 2011

An interview with Armin, recorded at the Leysin sprint: http://advocacy.python.org/podcasts/littlebit/2011-04-04.mp3 It turned out pretty well I think. All the best, Michael Foord -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

1 0

Post Easter PyPy Sprint, G�teborg Sweden April 25 -- May 1.
by Laura Creighton April 4, 2011

April 4, 2011

he next PyPy sprint will be in Gothenburg, Sweden. It is a public sprint, very suitable for newcomers. We'll focus on making the 1.5 release (if it hasn't already happened) and whatever interests the Sprint attendees. Topics and goals ---------------- The main goal is to polish and release PyPy 1.5, supporting Python 2.7 as well as the last few months' improvements in the JIT (provided that it hasn't already happened). Other topics: - Going over our documentation, and classifying our docs in terms of mouldiness. Deciding what needs writing, and maybe writing it. - Helping people get their code running with PyPy - maybe work on EuroPython Training, and talks - Summer of Code preparation - speed.pypy.org - any other programming task is welcome too -- e.g. tweaking the Python or JavaScript interpreter, Stackless support, and so on. Location -------- The sprint will be held in the apartment of Laura Creighton and Jacob Hallén which is at Götabergsgatan 22 in Gothenburg, Sweden. Here is a map_. This is in central Gothenburg. It is between the tram_ stops of Vasaplatsen and Valand, (a distance of 4 blocks) where many lines call -- the 2, 3, 4, 5, 7, 10 and 13. .. _tram: http://www.vasttrafik.se/en/ .. _map: http://bit.ly/grRuQe Probably cheapest and not too far away is to book accomodation at `SGS Veckobostader`_. The `Elite Park Avenyn Hotel`_ is a luxury hotel just a few blocks away. There are scores of hotels a short walk away from the sprint location, suitable for every budget, desire for luxury, and desire for the unusual. You could, for instance, stay on a `boat`_. Options are oo numerous to go into here. Just ask in the mailing list or on the blog. .. _`SGS Veckobostader`: http://www.sgsveckobostader.se/en .. _`Elite Park Avenyn Hotel`: http://www.elite.se/hotell/goteborg/park/ .. _`boat`: http://www.liseberg.se/en/home/Accommodation/Hotel/Hotel-Barken-Viki ng/ Hours will be from 10:00 until people have had enough. It's a good idea to arrive a day before the sprint starts and leave a day later. In the middle of the sprint there usually is a break day and it's usually ok to take half-days off if you feel like it. Good to Know ------------ Sweden is not part of the Euro zone. One SEK (krona in singular, kronor in plural) is roughly 1/10th of a Euro (9.36 SEK to 1 Euro). The venue is central in Gothenburg. There is a large selection of places to get food nearby, from edible-and-cheap to outstanding. We often cook meals together, so let us know if you have any food allergies, dislikes, or special requirements. Sweden uses the same kind of plugs as Germany. 230V AC. The Sprint will be held the week following Easter. This means, as always, that Gothcon_ will be taking place the weekend before (Easter weekend). Gothcon, now in its 35 year, is the largest European game players conference. Some of you may be interested in arriving early for the board games. The conference site is only in Swedish, alas. You don't need to register in advance unless you are planning to host a tournament, (and it's too late for that anyway). .. _Gothcon: http://www.gothcon.se/ Getting Here ------------ If are coming train, you will arrive at the `Central Station`_. It is about 12 blocks to the site from there, or you can take a tram_. There are two airports which are local to Göteborg, `Landvetter`_ (the main one) and `Gothenburg City Airport`_ (where some budget airlines fly). If you arrive at `Landvetter`_ the airport bus stops right downtown at `Elite Park Avenyn Hotel`_ which is the second stop, 4 blocks from the Sprint site, as well as the end of the line, which is the `Central Station`_. If you arrive at `Gothenburg City Airport`_ take the bus to the end of the line. You will be at the `Central Station`_. You can also arrive by ferry_, from either Kiel in Germany or Frederikshavn in Denmark. .. _`Central Station`: http://bit.ly/fON43p .. _`Landvetter`: http://swedavia.se/en/Goteborg/Traveller-information/Traffic-i nformation/ .. _`Gothenburg City Airport`: http://www.goteborgairport.se/eng.asp .. _ferry: http://www.stenaline.nl/en/ferry/ Who's Coming? -------------- If you'd like to come, please let us know when you will be arriving and leaving, as well as letting us know your interests We'll keep a list of `people`_ which we'll update (which you can do so yourself if you have bitbucket pypy commit rights). .. _`people`: https://bitbucket.org/pypy/extradoc/src/tip/sprintinfo/gothenburg- 2011/people.txt

1 0

Europython reservations
by Laura Creighton April 2, 2011

April 2, 2011

Jacob and I just arranged to be staying at Residence Michelangiolo June 19-26 . see: http://www.tripadvisor.com/Hotel_Review-g187895-d581477-Reviews-Residence_M… among others for review. At 980 Euros for the whole time, it seems to be substantially cheaper that the conference hotel, and not that far a walk. Other differences -- no included breakfast, but pantry -- refrigerator, counter, and clean up space -- in your room. For people who wanted to make their own breakfast anyway, this is a win. There may be a cook space too. There is internet by wire in each room but no wifi. On the 5th, we should be on a ferry to get us to kayaking vacation in Corsica. see: http://ucpa.se/Aktivitetsprogram-havskajaksafari-Korsika (in Swedish, but maybe it will help you to find the same program from ucpa described in whatever language you prefer.) This leaves the 26-5th to figure out. Vacation in Tuscany is one idea. Or to extend the sprint because 2 days is not enough is another. If so, maybe we could hold it at this place? Seems spacious enough. I thought it would be fun to fill Residence Michelangiolo with friends, at least for the 19-26 so posted this here. Laura

1 0

[PATCH] fix executing a file under sandbox by implementing fstat
by Seth de l'Isle April 1, 2011

April 1, 2011

I ran into a problem trying to follow the instructions to run a python file under the sandbox version of pypy-c. See the console log: http://paste.pocoo.org/show/363348 I was following the instructions here: http://codespeak.net/pypy/dist/pypy/doc/sandbox.html I got some help from arigato, ronny and antocuni on IRC (see log below) and they gave me the courage to hack on sandlib.py a little to see if I could get things working. http://www.tismer.com/pypy/irc-logs/pypy/%23pypy.log.20110331 The following patch changes the code so that it tracks the virtual file system nodes that correspond to each virtual file descriptor so that the node.stat() function can be used for fstat the same way it is used for stat. Thanks! diff -r 601862ed288e pypy/translator/sandbox/sandlib.py --- a/pypy/translator/sandbox/sandlib.py Mon Mar 28 13:12:49 2011 +0200 +++ b/pypy/translator/sandbox/sandlib.py Thu Mar 31 16:13:41 2011 -0800 @@ -391,6 +391,7 @@ super(VirtualizedSandboxedProc, self).__init__(*args, **kwds) self.virtual_root = self.build_virtual_root() self.open_fds = {} # {virtual_fd: real_file_object} + self.fd_to_node = {} def build_virtual_root(self): raise NotImplementedError("must be overridden") @@ -425,10 +426,17 @@ def do_ll_os__ll_os_stat(self, vpathname): node = self.get_node(vpathname) return node.stat() + do_ll_os__ll_os_stat.resulttype = s_StatResult do_ll_os__ll_os_lstat = do_ll_os__ll_os_stat + def do_ll_os__ll_os_fstat(self, fd): + node = self.fd_to_node[fd] + return node.stat() + + do_ll_os__ll_os_fstat.resulttype = s_StatResult + def do_ll_os__ll_os_isatty(self, fd): return self.virtual_console_isatty and fd in (0, 1, 2) @@ -452,11 +460,14 @@ raise OSError(errno.EPERM, "write access denied") # all other flags are ignored f = node.open() - return self.allocate_fd(f) + fd = self.allocate_fd(f) + self.fd_to_node[fd] = node + return fd def do_ll_os__ll_os_close(self, fd): f = self.get_file(fd) del self.open_fds[fd] + del self.fd_to_node[fd] f.close() def do_ll_os__ll_os_read(self, fd, size):

2 1