Mailman 3 October 2009 - pypy-dev

Re: [pypy-dev] Nightly Benchmarks
by Miquel Torres 06 Oct '09

06 Oct '09

> as I'm not doing any more > work on the benchmarking framework. That is a pity. But well, I'll have to do with asking. There are a couple of things I want to discuss: - An extra table (class) may be needed to store the hardware the benchmark was run on. pypy will probably need to be tested on other architectures. - The interpreter table needs extra fields: - revision: absolutely necessary - tag: this is for 1.0, 1.1, beta, trunk, snapshot, or what ever. This is needed to know what pypy revisions to list in the web page that allows comparison to other python implementations (most importantly cpython). - Furthermore: You state that compile options are not needed, that they would be different interpreters. I agree that we can have pypy-c, pypy-crl and pypy-c-jit (for example) as different interpreters. But things can get messy if we add garbage collectors and other options: pypy-c-68150-gc=hybrid--opt=3--no-allworkingmodules , etc... What are your thoughts on this? 2009/10/5 Anders Hammarquist <iko(a)openend.se> > Hi, > > In a message of Thu, 01 Oct 2009 23:07:38 +0200, Miquel Torres writes: > >SQLAlchemy is problematic. there is a django-sqlalchemy project, but it is > >still work in progress: > >http://code.google.com/p/django-sqlalchemy/wiki/Roadmap > > And that seems rather dead, no checkins since april... > > >Otherwise, your DB design is valid. The simpler, the better. > >One question: did you intend to keep a DB locally and then copy to the web > >server backend or to store the data directly on the web server ( > >speed.pypy.org)? > > Well, my thoughts hadn't gone much past getting the runner up and storing > the results. My intention was to have whatever web presentation running > on that same host, but I had no plans for more complicated presentation. > > >About the DB design, while sufficient, I would probably add a couple of > >things like version tags (for betas and final releases, for example), > maybe > >compile options (??). Why? because we want to be able to filter trunk > >versions in every possible way. > > I think remembering the svn revision of the run makes sense. Compile > options, and probably branch tags, are probably better implemented as > separate interpreters in my DB design. > > >In any case I want to discuss this a little bit more, because the DB needs > >to be as complete as possible from the beginning. While the first version > of > >the visualization interface will be quick and dirty so that developers > don't > >have to wait months to have something useful (And only then will I design > a > >more powerful interface), I don't want to make big changes to the DB once > we > >start to regularly run benchmarks. That way the data can be safely stored > >from the start. > > This makes perfect sense. I will be happy to discuss with you, but I will > leave the implementation in your capable hands as I'm not doing any more > work on the benchmarking framework. > > /Anders >

1 0

Re: [pypy-dev] Extension Modules Re: Question on Hard Switching and Soft Switching
by Andrew Francis 05 Oct '09

05 Oct '09

Hello Richard, Armin et al: --- On Sun, 10/4/09, Richard Tew <richard.m.tew(a)gmail.com> wrote: > In any case, select would not be the right place. It > is where someone goes to work with asynchronous IO. I would rather > suggest that the uber-module would be the socket module, and that > behind the scenes it might use IO completion ports on Windows and > poll or something similar on other platforms. But the goal in doing > it, would be to make IO just work with a microthread scheduling > framework for the user with minimal difficulty or effort. > In order for this to be so, how it works with the scheduler >would have to be equally simple for the user. Yes using IO completion ports or epoll, if done right, is a quick way to give better performance. However I think the interesting area is analysing how the scheduler interacts with the reactor (assuming we are using a reactor). I think a strategy will start with looking at stuff like many requests come in and how fast they can be processed (trying to determine what is a steady state). I will also assume that a good solution, given the aforementioned, is trying minimize stuff like context switches (even though they are fast) and calls to the OS (where one takes a hit). Then there is the issue of how much overhead is incurred from laying, that is locating "networking smarts" in a tasklet, as opposed to the interpreter. Again, as I stated in a previous post, I don't have a good conceptual framework. I have read some papers on high performance server design. Would have to read more. I also suspect a lot of experimentation would have to be done to see what works. Cheers, Andrew

1 0

Extension Modules Re: Question on Hard Switching and Soft Switching
by Andrew Francis 05 Oct '09

05 Oct '09

Hello Armin: --- On Wed, 9/30/09, Armin Rigo <arigo(a)tunes.org> wrote: > This is a question for Stackless Python only, not PyPy, so > you should post in that mailing list, not here :-) To answer it, > it's not visible to the user: it's just two different implementations of > switching. "Hard switching" is by actually moving part of the C stack > away; "soft switching" is using a purely standard C approach. > PyPy only has soft switching (but with the same Stackless interface). But is soft switching still about roughly 10 times faster than hard switching (value quoted in the Nuts and Bolts presentation). Because of only support for soft switching, I will assume I will have to write my own PyPy extension for say, networking I/O like select()/asyncore. Or does that already exist? Cheers, Andrew

4 3

how can i pickle an application-level object from interpreter code?
by Philip Guo 03 Oct '09

03 Oct '09

Hi all, I'm brand-new to PyPy, and here is my question: I want to selectively pickle Python objects that appear in application-level code, from *within* the interpreter. e.g., let's say i'm using the PyPy Python interpreter to run this program: import foo x = foo.ComplicatedObject() right after the interpreter creates an instance of foo.ComplicatedObject(), I want to serialize that instance to disk, in effect simulating this statement: pickle.dump(x, open('data.pickle')) What is the easiest way for me to do this from within the interpreter code? It seems like an instance object is represented within the interpreter as of type "class W_InstanceObject(Wrappable)", and I can't directly pickle that object. If I could just call the Python standard library's pickle on 'x' from within the interpreter, that would be great. Thanks in advance, Philip

3 3

benchmarking input
by Anders Hammarquist 03 Oct '09

03 Oct '09

Hi all, I need some input for the benchmarking infrastructure. I'm nearly at the point where I need to have some place to run it before continuing (i.e. I need to try and use it, not just speculate). Anyway, what I was thinking about, and need input on, is how to get at the interpreters to run the benchmark. When we were talking just benchmarks, and not profiling, my thought was to just use whatever python the machine has, and fetch the pypy from the last buildbot run, but for profiling that will not work (and anyway, running the profiling on the standard python is quite pointless). So benchmarks will obviously have to specify what interpreter(s) they should be run by somehow. The bigger question is how to get those interpreters. Should running the benchmarks also trigger building one (or more) pypy interpreters according to specs in the benchmarking framework? (but then if you only want it to run one benchmark, you may have to wait for all the interpreters to build) Perhaps each benchmark should build its own interpreter (though this seems slow, given that most benchmarks can probably run on an identically built interpreter). Or maybe the installed infrastructure should only care about history, and if you want to run a single benchmark, you do that on your own. Thoughts please! /Anders

8 7

Re: [pypy-dev] Nightly Benchmarks
by Anders Hammarquist 02 Oct '09

02 Oct '09

Hello, In a message of Tue, 29 Sep 2009 11:31:21 +0200, Miquel Torres writes: >We agreed on a Django DB backend, and plot rendering by the browser, so we >need to talk about the way to store the nightly benchmark data into the DB. Hmm, I've already started to implement a DB store using SQLalchemy. Can Django use that? (it is in the repository at http://codespeak.net/svn/pypy/build/benchmark) >The end goal is to build a speed.pypy.org where you can compare all kind of >pypy versions to cpython and other python interpreters and other languages[...] Could prove interesting, and the DB structure I've got should be able to deal with that as well (it is quite minimal). The runner that is there is mostly useless for comparing different lanugagues though. >Can you please explain to me what you are doing in that area now? The intended next step is to interface a benchmark runner (of pypy trunk) with the database so the results will be persistent. I've been caught up in other (non-PyPy-related) work though. and if the database interface I've got is not useful, maybe I shouldn't do that. /Anders

2 1

Re: [pypy-dev] [Stackless] Question on Hard Switching and Soft Switching
by Andrew Francis 01 Oct '09

01 Oct '09

Hi Kristján: --- On Tue, 9/29/09, Kristján Valur Jónsson <kristjan(a)ccpgames.com> wrote: > I've been exporing the bowels of Stackless a bit in the > last weeks and am somewhat familiar with the topic. Thanks a lot for the detailed description! As per Armin's suggestion, I will move this thread to the stackless mailing list. > I have not done any performance measurements to evaluate > the merits of soft switching. But I must confess that > the presence of soft-switching makes stackless much more > difficult to understand for me. And a lot of the diff > between trunk python and stackless python is because of the > boilerplate code required to get soft switching to work. Well I came across two slides in the "Stackless Python and PyPy Nults and Bolts" talk. It states that soft switching is 100 x faster than a thread. Hard switching is 10 x. I don't know if this fact is helpful. > I hope this clears things up. It helps. I need to learn a lot more to better use the information you have given me. Cheers, Andrew

1 0

support for 64-bit processors and eliminating global state
by Jeff Anderson-Lee 01 Oct '09

01 Oct '09

I'm new to pypy but would encourage the development folks to apply some focus towards two things: support for both 32 and 64-bit processors and eliminating global state including the GIL. The near future of mainstream processors is multi-core x86_64. For the short-term both 32-bit and 64-bit platforms will be around. Code that makes "naked" assumptions about word size will break and needs to be re-factored to hide the word-size dependencies. Similarly code that assumes a single thread of execution or uses a GIL to protect global state will make efficient use of modern processors. Any language or system that cannot make the transition to 64-bit multi-core will start to loose ground to those that do. At the Parallel Computing Laboratory (UC Berkeley) one of the projects we are working on is called SEJITS which stands for Selective Embedded Just in Time Specialization. The idea is that one can extend a self-introspecting modern scripting language for calling native coded modules (e.g. C) at selected points for handling specialized operations (e.g. vector/matrix operations using tuned SIMD or CUDA code). You can see the abstract of a recent SEJITS paper at http://pmea.ac.upc.edu/program.html (session 1a) but unfortunately the paper is not online yet. Both Python and Ruby are being looked at as potential target languages for SEJITS work. Both have sufficient introspection facilities to support selective JIT operations. Python has an advantage in having been used by the scientific community for longer than Ruby with more established users. I'd love to see this work integrate with pypy. At the moment the folks involved are targeting CPython. In any case, I think the transition to multi-core/multi-threaded 64-bit machines is a potential watershed of major importance which it would behoove pypy-dev folks to keep in mind. Respectfully. Jeff Anderson-Lee

5 9

Re: [pypy-dev] Stackless Python and PyPy Stackless.py
by Gabriel Lavoie 01 Oct '09

01 Oct '09

2009/9/28 Leonardo Santagada <santagada(a)gmail.com>: > I am very interested, I thought to do something like it. Where could I see > your code? > > On Sep 28, 2009, at 10:04 AM, Gabriel Lavoie wrote: > >> Hello Andrew, >> I'm currently experimenting with PyPy's implementation of >> Stackless to add new features for a university master degree project. >> I chose PyPy's implementation because it's easier to play with Python >> code than with C code. Also, since PyPy is "still experimental", it >> was the best implementation to choose to hack with and I don't regret >> my choice. What I'm trying to achieve is to add distributed features >> to Stackless: >> >> - Local and networked channels with automatic switch between both >> - Easy tasklet migration to a remote host, keeping the channel >> connections between tasklets. >> - Transparent/automatic dependencies migration when a tasklet is sent >> to a remote host. >> >> Most of the features are done and I'm currently working on the >> dependencies migration. The only bad part is that I'm doing this >> project part time since I have a full time job but I have to complete >> the programming part in the next two months (I've been working for too >> long on this). >> >> If you're interested to see what I've done, just ask! :) >> >> See ya, >> >> Gabriel >> >> 2009/9/25 Andrew Francis <andrewfr_ice(a)yahoo.com>: >>> >>> Hi Folks: >>> >>> Again as a part of my Stackless Python talk, I wanted to include a >>> section on the "Future." I assume a part of Stackless Python's future is >>> PyPy? Or am I being presumptuous? >>> >>> Regardless I would like to end the talk with a brief section on PyPy. I >>> noticed the Stackless.py module in lib that contains the Stackless >>> implementation in Python. >>> >>> What I plan to do in my talk is show how a rough approximation of Limbo's >>> alt (selecting the first ready channel from a list) could be implemented. >>> >>> I am a newbie in regards to PyPy. However I have been reading the >>> Stackless documentation. I thought it would be neat if I ended the talk with >>> redoing this, but in PyPy as a part of how one could quickly prototype new >>> Stackless Python features. Any thoughts? Is there anything gotchas? >>> >>> Cheers, >>> Andrew >>> >>> >>> >>> _______________________________________________ >>> pypy-dev(a)codespeak.net >>> http://codespeak.net/mailman/listinfo/pypy-dev >>> >> >> >> >> -- >> Gabriel Lavoie >> glavoie(a)gmail.com >> _______________________________________________ >> pypy-dev(a)codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > > -- > Leonardo Santagada > santagada at gmail.com > > > > I still haven't shown publicly my work. I'll try to quickly prepare something this week with my current test code samples and a quick description of the API. I think the code quality is pretty bad as this is my first real Python project. My priority is to have something that works before doing a big cleanup. Gabriel -- Gabriel Lavoie glavoie(a)gmail.com

2 3