Mailman 3 November 2007 - pypy-dev

thunk object space : become() quirks
by Boris Borcic 10 Dec '07

10 Dec '07

Hello, Looking for a way to use pypy in any manner, I came up with an idea for a deterministic sudoku solver that would use the 'become()' feature of the thunk object space. However, this doesn't work as I expected. The following shell trace illustrates my problem. Is this a bug or otherwise ? Can anybody explain ? BTW, I am using the pypy 1.0.0 build 41438 windows binary distribution. [subquestion : is pypy-dev appropriate for such a question ?] TIA, Boris Borcic >>>> s = lambda *x : set(x) >>>> s1 = s(1) >>>> s2 = s(2) >>>> s3 = s(3) >>>> s4 = s(4) >>>> s1,s2,s3,s4 (set([1]), set([2]), set([3]), set([4])) >>>> s1.update(s2) >>>> become(s2,s1) >>>> s3.update(s4) >>>> become(s4,s3) >>>> s1,s2,s3,s4 (set([1, 2]), set([1, 2]), set([3, 4]), set([3, 4])) >>>> s4.update(s2) >>>> s4 set([3, 4]) ------------------> expected set([1,2,3,4]) >>>> s2 set([1, 2]) >>>> s4 is s3 True >>>> s3.update(s2) >>>> s4 set([3, 4, 2]) ------------------> expected set([1,2,3,4]) >>>> s3 set([3, 4, 2]) >>>> s1 set([1, 2]) >>>> s1 is s2 True >>>> s3.update(s1) >>>> s4 set([3, 4, 2, 1]) >>>>

2 4

PyPy and GHOP
by VanL 30 Nov '07

30 Nov '07

For those who are not aware, Google just launched the Google Highly Open Participation (GHOP) project. GHOP introduces high-school-age kids to Open Source by giving them small but meaningful tasks to do. I have noticed at least one PyPy-related task proposal for GHOP, but I think that there is more opportunity here. Specifically, some of the "production-ready" threads noted that it is difficult to get a PyPy environment set up. Could that be a task? How about an example of translating a function or module to rpython? How about actual translations of simple parts of the standard library - maybe a function at a time? PyPy is well suited to providing GHOP tasks, because there are lots of interesting bits of code that could be translated into rpython. That is a little bit more substantial than simple documentation tasks, but still easy enough to do (if the translation is of single functions or a small group of related functions). What do you all think?

2 1

svn statistics
by Carl Friedrich Bolz 30 Nov '07

30 Nov '07

Hi all! I found a reasonably nice analysis tool for SVN repositories and set it loose on pypy-dist. The result can be found here: http://wyvern.cs.uni-duesseldorf.de/~cfbolz/pypystats/ It's not yet perfect, I should tell it to exclude unicodedata. However, it seems to be able to deal with our strange merging patterns and is much better than my hacked up scripts, so I tend to keep using this after some tweaking. Any suggestions on improvements? The program used is this: http://www.statsvn.org/ Cheers, Carl Friedrich

2 1

is anyone actively hacking on the JVM backend?
by Niko Matsakis 30 Nov '07

30 Nov '07

I have been making small improvements to the JVM backend: I am aiming towards eliminating any remaining conformance problems, and also towards fixing a few warts in the implementation. Right now I am wringing out the last few bugs from a check-in that would allow multiple PyPy-generated interpreters to be used simultaneously. Currently, the code uses a few static fields that would conflict if, for example, someone tried to load both a Python and a Smalltalk interpreter into the same JVM. These static fields were used to allow the helper code (src/PyPy.java and friends) to create instances of exception classes and other RPython-generated code. The new code creates a single instance of the PyPy helper class per generated interpreter and use non-static-methods instead, which should be better. Anyway, I just want to be sure I am not stepping on anyone's toes if I check this in, since it makes small changes all over the place to ensure that the proper PyPy object is pushed onto the stack, etc. Niko

7 10

naive VM question
by Adam Olsen 29 Nov '07

29 Nov '07

> From: Armin Rigo <arigo(a)tunes.org> > > Hi Peter, > > On Fri, Nov 16, 2007 at 10:53:39PM +1300, Peter Fraser wrote: > > Is "multiple isolated (but bridged) VM's in a single process" a > > possible general technique? > > > > If multiple cooperating processes is a reasonable response to the > > "single threaded VM meets multi-core architecture" problem -then is > > "multiple VM's in a single process" a better one? > > This is probably yet another place where PyPy could go much more easily > than CPython: multiple VMs in the same process which would by default > not have any share state, but expose primitives to the Python programmer > to explicitly share some objects. This is not trivial: there is some > research and experimentation to be done here. This sounds similar to what I'm doing with CPython: normal objects are contained within a single Monitor, whereas shareable objects (explicitly given a thread-safe API) can be passed between. http://code.google.com/p/python-safethread/ -- Adam Olsen, aka Rhamphoryncus

1 0

rctypes RIP
by Simon Burton 26 Nov '07

26 Nov '07

Just for the record can someone say what was the revision number where rctypes was removed ? I still have code that uses this. thanks, Simon.

2 1

Re: [pypy-dev] pypy on Tim Bray's blog
by Laura Creighton 23 Nov '07

23 Nov '07

In a message of Wed, 14 Nov 2007 03:53:18 +0100, Jacob Hallén writes: <snip> >> So I'll definitely complain if you spend a lot of time on Ruby (or >> Smalltalk for that matter) before Python's all the way there. I think >> that'd be a bad idea for a whole range of reasons - get the last 10% >> done for Python first before you take even more on your plates. We all >> know that last 10% tends to be the hardest part. Focus is important. If >> you can't make that work for Python, you'd have a hard time making it >> work for any other language too (or convincing people that you can). Of >> course I realize I have no real voice in this project, but that's my in >put. I don't think you understand the state of our project. There is practically nothing left that is 'python-centric' left to work on, unless you have a burning desire to have the 2.5 features we don't have yet, which is scheduled to get finished next week anyway. This is hardly any work. Where we need work is to make the thing run a lot faster. So, better gc, more work on the JIT -- and play better with c-extensions, and enable us to use the stackless transform on CLI and JVM, and the generated JIT on that as well. All this stuff is translation level stuff. Thus it is relevant to all the app-level input languages we support, not any particular one. Speed up one, you speed up them all. It took a 10 day sprint for some of the pypyers and the squeak experts at the SCG to get Smalltalk up and running. http://pypysqueak.blogspot.com/ Assuming we could find a small group of similarly competant ruby language experts, we ought to be able to do the same thing in the same sort of time range. If we cannot find them, and have to do it all ourselves, it will take longer, since we aren't experts on the arcane and obscure details about the ruby language that only ruby wizards know. And then we can take Tim Bray's money and give him the fast ruby he wants by working on precisely the same things that will give you the fast python you want. So I don't see why you aren't up in the front row cheering these developments as loudly as you can. Right now the most promising path to get you what you want, which I see as a fast python that is production-quality and suitable for deploying Grok web apps, runs hand in hand with Tim Bray's desire for a fast ruby that is production-quality and suitable to deploy rails apps. You guys want the same thing! You don't even have to squint! Why aren't you cheering? Laura

9 9

Re: [pypy-dev] [pypy-svn] r48962 - pypy/dist/pypy/translator/microbench
by Carl Friedrich Bolz 22 Nov '07

22 Nov '07

antocuni(a)codespeak.net wrote: > Author: antocuni > Date: Fri Nov 23 00:05:30 2007 > New Revision: 48962 > > Modified: > pypy/dist/pypy/translator/microbench/microbench.py > Log: > print the unexpected output in case of error Puh, thank you. This has been annoying me since forever. Cheers, Carl Friedrich

1 0

naive VM question
by Peter Fraser 22 Nov '07

22 Nov '07

I have to assume the following is either not useful, not practical, or both -but I have to ask... Version I. It seems to me that PyPy might be useful as a kind of "Co-Python" -viz. the PyPy VM running as a CPython extension, as a way to utilise a second core from a single CPython process. I'm realistic about the level of integration that could be achieved, -but even something that amounted to (fast) RPC inside a process might be useful given the second vm was arguably "free for the using" on a multicore system. If this was in fact useful, it might be an on-ramp into the Python mainstream for PyPy. PyPy needs on-ramps imho. Version II. Is "multiple isolated (but bridged) VM's in a single process" a possible general technique? If multiple cooperating processes is a reasonable response to the "single threaded VM meets multi-core architecture" problem -then is "multiple VM's in a single process" a better one? I'm talking about an ugly, brute force alternative to a multi threaded VM. Viz. simply duplicating / munging / mangling / tagging as much of the interpreter code as was necessary to allow say two (but not n) threads to safely run at once. 2 heaps, 2 GILS and any given thread only ever running against one of these "sub vm's". Then perhaps add explicitly managed opportunities for memory sharing and other integration strategies betwen the two. It strikes me as something that you wouldn't want to attempt in C, but might be possible in the PyPy translation. "Dual-core Python" .."Hydra" -or is this just crazy talk? Pete F

5 4

Very confused? -can someone at least tell me..? (was naive vm question)
by Peter Fraser 22 Nov '07

22 Nov '07

Ok -let me try again. If someone could answer 1. below, then I would be very grateful. Better yet, tell me where I leave the road thereafter. If I still sound hopelessly confused then I won't bother the list further, but I will be looking for someone to pay money to, for some skype-consulting to explain where I am going wrong. 1. Suppose you ran the PyPy-C interpreter as a CPython extension. Could the extension release the CPython GIL? 2. My naive bet is that it could -because the PyPy interpreter is not using CPython "in any essential way". 3. Leaving aside how useful this would be -can we agree that if indeed you could release the CPython GIL, then you could have two threads interpreting Python code, in one process, simultaneously. 4. This is possible because you have two of everything. Two code bases, two memory managers, two GILS etc etc etc. 5. The separation is so extreme that opportunities for cooperation between the two interpreters is severly curtailed. The type implementations aren't interchangeable for example. 6. Would this arrangement even be useful? This point is tangential. Personally I'm interested in any opportunity to utilize a second core, from one CPython process, simply using an extension -even with the severe constraints of messaging between two totally separate, incompatible VMs. But that is tangential. If this proved popular then that would be great for PyPy i think. But it is tangential. 7. The question I'm really getting at is, are there opportunities for a middle ground? Ok, so two threads running in one interpreter with no GIL is too intimate. Two distinct interpreters (a CPython and a PyPyC) in one process are too isolated (as are two threads in two processes). What is in the middle (other than locking)? 8. Yes, I am confusing designs and instances Boris. I'm trying to understand exactly what are the design differences, that create the implementation diferences, that conspire to make the CPython with PyPy-C extension mix, two separate instances. I'm speculating about whether you could contrive to create *just enough difference* to sustain two "instances" of essentialy the same interpreter, in one process. 9. I'm just talking about sufficient partitioning to achieve 2 instances in one process -where "instance" means that a thread running in either will never touch the data of the other. 10. A further wild hunch is that this partitioning might in part be achieved by wholesale code duplication, and the munging of names -as opposed to smarter code (viz structurally more sophisticated code to support the partition). I wouldn't be surprised to be totally wrong on this one and the concept doesn't hinge on it. It does scream "translation aspect" to me however. 11. I assume that if its possible, this partitioned "dual core" (as it were) interpreter would be far more easily achieved as a PyPy translation aspect, than by hand. This, the PyPyC extension GIL question, and the aim of PyPy to support different features is why I'm asking the question here. Also, much as I don't like fandom as a phenomena -I allow myself to be a PyPy fan. 12. But how useful would this be? Why do I think this would be better than simply running two processes with some kind of inter-process messaging? Simply speed. Suffice to say that I have a "cross process bad, in process good" mindset. 13. I'm betting on the identical types to give you far better cooperative efficiency than the CPython + PyPy-C approach, even if we don't try to actually share objects (but hey, that might be possible, a la posh, but in one process and on windows). 14. Finally -the key idea that niggles at me is that while the expensive, dense locking required for free threading presumably gives you an unbounded(ish) number of threads -the number of cores on modern machines is very bounded. Hence my grasping for fast, in process, messaging based concurrency that takes advantage of only having to support a bounded number of threads (2 for now) with their own instances -ie "dual core Python" (a good metaphor that) as a PyPy translation aspect. >>Putting aside the relevance to pypy<< Well I thought it was, but if someone could just answer 1. above, I concede that I could go away. If there is a more appropriate PyPy list for this kind of question -please tell me. Twins? I'm after viable Siamese twins. Co-joined but not too co-joined. PyPy really is aboue genetic engineering isn't it? Tweak the DNA and grow a new interpreter. regards Pete F

2 1