Mailman 3 July 2010 - pypy-dev

Building pypy on freebsd 64bit
by Shadrin Eugene 18 Nov '11

18 Nov '11

Hi! I tryed to build pypy on the freebsd, 64bit platform, but failed. First of all I downloaded a 32bit-Python to run pypy. And then: [user@server_name ~/pypy-trunk/pypy/translator/goal]$ ~/py32/bin/python translate.py -Ojit [translation:info] Translating target as defined by targetpypystandalone [platform:execute] gcc -m32 -c -O3 -pthread -fomit-frame-pointer /var/tmp/usession-trunk-22/gcctest.c -o /var/tmp/usession-trunk-22/gcctest.o [platform:WARNING] /var/tmp/usession-trunk-22/gcctest.c:28:2: warning: no newline at end of file [platform:execute] gcc -m32 /var/tmp/usession-trunk-22/gcctest.o -L/lib32 -L/usr/lib32 -L`pwd`/lib32 -Wl,-rpath,/lib32 -Wl,-rpath,/usr/lib32 -pthread -o /var/tmp/usession-trunk-22/gcctest printf("sizeof short=%ld\n", (long)sizeof(short)); printf("sizeof unsigned short=%ld\n", (long)sizeof(unsigned short)); printf("sizeof int=%ld\n", (long)sizeof(int)); printf("sizeof unsigned int=%ld\n", (long)sizeof(unsigned int)); printf("sizeof long=%ld\n", (long)sizeof(long)); printf("sizeof unsigned long=%ld\n", (long)sizeof(unsigned long)); printf("sizeof signed char=%ld\n", (long)sizeof(signed char)); printf("sizeof unsigned char=%ld\n", (long)sizeof(unsigned char)); printf("sizeof long long=%ld\n", (long)sizeof(long long)); printf("sizeof unsigned long long=%ld\n", (long)sizeof(unsigned long long)); printf("sizeof size_t=%ld\n", (long)sizeof(size_t)); printf("sizeof time_t=%ld\n", (long)sizeof(time_t)); printf("sizeof wchar_t=%ld\n", (long)sizeof(wchar_t)); printf("sizeof mode_t=%ld\n", (long)sizeof(mode_t)); printf("sizeof pid_t=%ld\n", (long)sizeof(pid_t)); sizeof short=2 sizeof unsigned short=2 sizeof int=4 sizeof unsigned int=4 sizeof long=8 sizeof unsigned long=8 sizeof signed char=1 sizeof unsigned char=1 sizeof long long=8 sizeof unsigned long long=8 sizeof size_t=8 sizeof time_t=8 sizeof wchar_t=4 sizeof mode_t=2 sizeof pid_t=4 Traceback (most recent call last): File "translate.py", line 300, in <module> main() File "translate.py", line 205, in main targetspec_dic, translateconfig, config, args = parse_options_and_load_target() File "translate.py", line 153, in parse_options_and_load_target targetspec_dic = load_target(targetspec) File "translate.py", line 105, in load_target mod = __import__(specname) File "targetpypystandalone.py", line 5, in <module> from pypy.objspace.std.objspace import StdObjSpace File "/place/home/hitrobot/pypy-trunk/pypy/objspace/std/__init__.py", line 1, in <module> from pypy.objspace.std.objspace import StdObjSpace File "/place/home/hitrobot/pypy-trunk/pypy/objspace/std/objspace.py", line 3, in <module> from pypy.interpreter import pyframe, function, special File "/place/home/hitrobot/pypy-trunk/pypy/interpreter/pyframe.py", line 13, in <module> from pypy.rlib import jit, rstack File "/place/home/hitrobot/pypy-trunk/pypy/rlib/rstack.py", line 10, in <module> from pypy.rpython.lltypesystem import rffi, lltype File "/place/home/hitrobot/pypy-trunk/pypy/rpython/lltypesystem/rffi.py", line 824, in <module> sys.maxint, sizeof(lltype.Signed))) AssertionError: Mixed configuration of the word size of the machine: the underlying Python was compiled with maxint=2147483647, but the C compiler says that 'long' is 8 bytes I got this error, then I added "-m32" flag to gcc argument list (in the posix.py source file) , but error occured again :( Does anybody tryed to build pypy-c on the 64bit FreeBsd? Thank you for your answer! -- Shadrin Eugene

5 7

pre-emptive micro-threads utilizing shared memory message passing?
by Kevin Ar18 12 Aug '10

12 Aug '10

Might as well warn you: This is going to be a rather long post. I'm not sure if this is appropriate to post here or if would fit right in with the mailing list. Sorry, if it is the wrong place to post about this. I've looked through the documenation (http://codespeak.net/pypy/dist/pypy/doc/stackless.html) and didn't really see what I was looking for. I've also investigated several options in the default CPython. What I'm trying to accomplish: I am trying to write a particular threading scenario that follows these rules. It is partly an experiment and partly for actual production code. 1. Hundreds or thousands of micro-threads that are essentially small self-contained programs (not really, but you can think of them that way). 2. No shared state - data is passed around from one micro-thread to another; only one micro-thread has access to the data at a time. (although the programmer gets the impression there is no shared state, in reality, the underlying implementation uses shared memory / shared state for speed; the data does not move; you just pass around a reference/pointer to some shared memory) 3. The micro-threads can run in parallel on different cpu cores, get moved to a different core, etc.... 4. The micro-threads are truly pre-emptive (uses hardware interrupt pre-emption). 5. It is my intention to write my own scheduler that will suspend the micro-threads, start them, control the sharing of data, assign them to different CPU cores etc.... In fact, for my purposes, I MUST write my own scheduler as I have very specific requirements on when they should and should not run. Now, I have spent some time trying to find a way to achieve this ... and I can implement a rather poor version using default Python. However, I don't see any way to implement my ideal version. Maybe someone here might have some pointers for me. Shared Memory between parallel processes ---------------------------------------- Quick Question: Do queues from the multiprocessing module use shared memory? If the answer is YES, you can just skip this section, because that would solve this particular problem. (For simplicity, let's assume a quad core CPU) It is my intent to create 4 threads/processs (one per core) and use the scheduler to assign a micro-thread (of which there may be hundreds) to one of the 4 threads/processes. However, the micro-threads need to exchange data quickly; to do that I need shared memory -- and that is where I'm having some trouble. Normally, 4 threads would be the ideal solution -- as they can run in parallel and use shared memory. However, because of the Python GIL, I can't use threads in this way; thus, I have to use 4 processes, which are not setup to share memory. Question: How can I share Python Objects between processes USING SHARED MEMORY? I do not want to have to copy or "pass" data back and forth between processes or have to use a proxy "server" process. These are both too much of a performance hit for my needs; shared memory is what I need. The multiprocessing module offers me 4 options: "queues", "pipes", "shared memory map", and a "server process". "Shared memory map" won't work as it only handles C values and arrays (not Python objects or variables). "Server Process" sounds like a bad idea. Am I correct in that this option requires extra processing power and does not even use shared memory? If so, that would be a very bad choice for me. The big question then... do "queues" and "pipes" used shared memory or do they pass data back and forth between processes? (if they used shared memory, then that would be perfect) Does PyPy have any other options for me? True Pre-emptive scheduling? ---------------------------- Any way to get pre-emptive micro-threads? Stackless (the real Stackless, not the one in PyPy) has the ability to suspend them after a certain number of interpreter instructions; however, this is prone to problems because it can run much longer than expected. Ideally, I would like to have true pre-emptive scheduling using hardware interrupts based on timing or CPU cycles (like the OS does for real threads). I am currently not aware of any way to achieve this in CPython, PyPy, Unladen Swallow, Stackless, etc.... Are there detailed docs on why the Python GIL exists? ----------------------------------------------------- I don't mean trivial statements like "because of C extensions" or "because the interpreter can't handle it". It may be possible that my particular usage would not require the GIL. However, I won't know this until I can understand what threading problems the Python interpreter has that the GIL was meant to protect against. Is there detailed documentation about this anywhere that covers all the threading issues that the GIL was meant to solve? Thanks, Kevin _________________________________________________________________ Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox. http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:…

10 39

Re: [pypy-dev] FW: Would the following shared memory model be possible?
by Michael Sparks 01 Aug '10

01 Aug '10

[ cc'ing the list in case anyone else took my words the same way as Kevin :-( ] On Sat, Jul 31, 2010 at 5:26 PM, Kevin Ar18 <kevinar18(a)hotmail.com> wrote: > I have no idea what I did you warrant you hateful replies towards me, but > they really are not appropriate (in public or private email). I had absolutely no intention of offending you, and am deeply sorry for any offense that I may have caused you. In my reply I merely wanted to flag that I don't have time to go into everything (like most people), that asking questions in a public realm is better because you may then get answers from multiple people, and that people who appear to do some research first tend to get better answers. I also tried to give an example, but that doesn't appear to have been helpful. (I'm fallible like everyone else) My intention there was to be helpful and to explain why I have that view of only replying on list, and it appears to have offended you instead, and I apologise. (one person's direct and helpful speech in one place can be a mortal insult somewhere else) After those couple of paragraphs, I tried to add to your discussion by replying to your specific points which you asked about parallel execution, noting places and examples where it is possible today. (to varying degrees of satisfaction) I then also tried to answer your point of "if something extra could be done, what would probably be generally useful". To that I noted that *my* talk there was cheap, and that execution was hard. Somehow along the way, my intent to try to be helpful to you has resulted in offending and upsetting you, and for that I am truly sorry - life is simply too short for people to upset each other, and in no way was my post intended as "hateful", and once again, my apologies. In future please assume good intentions - I assumed good intentions on your part. I'll bow out at this point. Best Regards, Michael. > >> Date: Sat, 31 Jul 2010 02:08:49 +0100 >> Subject: Re: [pypy-dev] FW: Would the following shared memory model be >> possible? >> From: sparks.m(a)gmail.com >> To: kevinar18(a)hotmail.com >> CC: pypy-dev(a)codespeak.net >> >> On Thu, Jul 29, 2010 at 6:44 PM, Kevin Ar18 <kevinar18(a)hotmail.com> wrote: >> > You brought up a lot of topics. I went ahead and sent you a private >> > email. >> > There's always lots of interesting things I can add to my list of things >> > to >> > learn about. :) >> >> Yes, there are lots of interesting things. I have a limited amount of >> time however (I should be in bed, it's very late here, but I do /try/ >> to reply to on-list mails), so cannot spood feed you. Mailing me >> directly rather than a (relevant) list precludes you getting answers >> from someone other than me. Not being on lists also precludes you >> getting answers to questions by chance. Changing emails and names in >> email headers also makes keeping track of people hard... >> >> (For example you asked off list last year about Kamaelia's license >> from a different email address. Since it wasn't searchable I >> completely forgot. You also asked all sorts of questions but didn't >> want the answers public, so I didn't reply. If instead you'd >> subscribed to the list, and asked there, you'd've found out that >> Kamaelia's license changed - to the Apache Software License v2 ...) >> >> If I mention something you find interesting, please Google first and >> then ask publicly somewhere relevant. (the answer and question are >> then googleable, and you're doing the community a service IMO if you >> ask q's that way - if you're question is somewhere relevant and shows >> you've already googled prior work as far as you can... People are >> time however (I should be in bed, it's very late here, but I do /try/ >> to reply to on-list mails), so cannot spood feed you. Mailing me >> directly rather than a (relevant) list precludes you getting answers >> from someone other than me. Not being on lists also precludes you >> getting answers to questions by chance. Changing emails and names in >> email headers also makes keeping track of people hard... >> >> (For example you asked off list last year about Kamaelia's license >> from a different email address. Since it wasn't searchable I >> completely forgot. You also asked all sorts of questions but didn't >> want the answers public, so I didn't reply. If instead you'd >> subscribed to the list, and asked there, you'd've found out that >> Kamaelia's license changed - to the Apache Software License v2 ...) >> >> always willing to help people who show willing to help themselves in >> my experience.) >> >> >> just looks to me that you're tieing yourself up in knots over things >> >> that aren't problems, when there are some things which could be useful >> >> (in practice) & interesting in this space. >> > The particular issue in this situation is that there is no way to make >> > Kamaelia, FBP, or other concurrency concepts run in parallel (unless you >> > are >> > willing to accept lots of overhead like with the multiprocessing >> > queues). >> > >> > Since you have worked with Kamaelia code a lot... you understand a lot >> > more >> > about implementation details. Do you think the previous shared memory >> > concept or something like it would let you make Kamaelia parallel? >> > If not, can you think of any method that would let you make Kamaelia >> > parallel? >> >> Kamaelia already CAN run components in parallel in different processes >> (has been able to do so for quite some time) or on different >> processors. Indeed, all you do is use a ProcessPipeline or >> ProcessGraphline rather than Pipeline or Graphline, and the components >> in the top level are spread across processes. I still view the code as >> experimental, but it does work, and when needed is very useful. >> >> Kamaelia running on Iron Python can run on seperate processors sharing >> data efficiently (due to lack of GIL there) happily too. Threaded >> components there do that naturally - I don't use IronPython, but it >> does run on Iron Python. On windows this is easiest, though Mono works >> just as well. >> >> I believe Jython also is GIL free, and Kamaelia's Axon runs there >> cleanly too. As a result because Kamaelia is pure python, it runs >> truly in parallel there too (based on hearing from people using >> kamaelia on jython). Cpython is the exception (and a rather big one at >> that). (Pypy has a choice IIUC) >> >> Personally, I think if PyPy worked with generators better (which is >> why I keep an eye on PyPy) and cpyext was improved, it'd provide a >> really compelling platform for me. (I was rather gutted at Europython >> to hear that PyPy's generator support was still ... problematic) >> >> Regarding the *efficiency* and *enforcement* of the approach taken, I >> feel you're chasing the wrong tree, but let's go there. >> >> What approach does baseline (non-Iron Python running) kamaelia take >> for multi-process work? >> >> For historical reasons, it builds on top of pprocess rather than >> multiprocessing module based. This means for interprocess >> communications objects are pickled before being sent over operating >> system pipes. >> >> This provides an obvious communications overhead - and this isn't >> really kamaelia specific at this point. >> >> However, shifting data from one CPU to another is expensive, and only >> worth doing in some circumstances. (Consider a machine with several >> physical CPUs - each has a local CPU cache, and the data needs to be >> transferred from one to another, which is why partly people worry >> about thread/CPU affinity etc) >> >> Basically, if you can manage it, you don't want to shift data between >> CPUs, you want to partition the processing. >> >> ie you may want to start caring about the size of messages and number >> of messages going between processes. Sending small and few between >> processes is going to be preferable to sending large and many for >> throughput purposes. >> >> In the case of small and few, the approach of pickling and sending >> across OS pipes isn't such a bad idea. It works. >> >> If you do want to share data between CPUs, and it sounds like you do, >> then most OSs already provide a means of doing that - threads. The >> conventions people use for using threads are where they become >> unpicked, but as a mechanism, threads do generally work, and work >> well. >> >> As well as channels/boxes, you can use an STM approach, such as than >> in Axon.STM ... >> * http://www.kamaelia.org/STM.html >> * >> http://code.google.com/p/kamaelia/source/browse/trunk/Code/Python/Bindings/… >> >> ...which is logically very similar to version control for variables. A >> downside of STM (at least with this approach) however, is that for it >> to work, you need either copy on write semantics for objects, or full >> copying of objects or similar. Personally I use a biological metaphor >> here, in that channels/boxes and components, and similar perform a >> similar function to axons and neurons in the body, and that STM is >> akin to the hormonal system for maintaining and controlling system >> state. (I modelled biological tree growth many moons ago) >> >> Anyhow, coming back to threads, that brings us back to python, and >> implementations with a GIL, and those without. >> >> For implementations with a GIL, you then have a choice: do I choose to >> try and implement a memory model that _enforces_ data locality? that >> is if a piece of data is in use inside a single "process" or "thread" >> (from hereon I'll use "task" as a generic phrase) that trying to use >> it inside another causes a problem for the task attempting to breach >> the model. >> >> In order to enforce this, I personally believe you'd need to use >> multiple processes, and only share data through dedicated code >> managing shared memory. You could of course do this outside user code. >> To do this you'd need an abstraction that made sense, and something >> like stackless' channels or kamaelia's (in/out) box model makes sense >> there. (The CELL API uses a mailbox metaphor as well for reference) >> >> In that case, you have a choice. You either copy the data into shared >> memory, or you share the data in situ. The former gives you back >> precisely the same overhead previously described, or the latter >> fragments your memory (since you can no longer access it). You could >> also have compaction. >> >> However, personally, I think any possible benefits here are outweighed >> by the costs and complexity. >> >> The alternative is to _encourage_ data locality. That is encourage the >> usage and sharing of data such that whilst you could share data >> between tasks and cause corruption that the common way of using the >> system discourages such actions. In essence that's what I try to do in >> Kamaelia, and it seems to work. Specifically, the model says: >> >> * If I take a piece of data from an inbox, I own it and can do anything >> with it that I like. If you think of a physical piece of paper and >> I take it from an intray, then that really is the case. >> >> * If I put a piece of data in an outbox, I no longer own it and should >> not attempt to do anything more with it. Again, using a physical >> metaphor, and naming scheme helps here. In particular, if I put a >> piece of paper in the post, I can no longer modify it. How it gets >> to its recipient is not my concern either. >> >> In practice this does actually work. If you add in immutable tuples, >> and immutable strings then it becomes a lot clearer how this can work. >> >> Is there a risk here of accidental modification? Yes. However, the >> size and general simplicity of components tends to lead to such >> problems being picked up early. It also enables component level >> acceptance tests. (We tend to build small examples of usage, which in >> turn effectively form acceptance tests) >> >> [ An alternative is to make the "send" primitive make a copy on send. >> That would be quite an overhead, and also limit the types of data you >> can send. ] >> >> In practical terms, it works. (Stackless proves this as well IMO, >> since despite some differences, there's also lots of similarities) >> >> The other question that arises, is "isn't the GIL a problem with >> threads?". Well, the answer to that really depends on what you're >> doing. David Beazely's talk on what happens on mixing different sorts >> of threads shows that it isn't ideal, and if you're hitting that >> behaviour, then actually switching to real processes makes sense. >> However if you're doing CPU intensive work inside a C extension which >> releases the GIL (eg numpy), then it's less of an issue in practice. >> Custom extensions can do the same. >> >> So, for example, picking something which I know colleagues [1] at work >> do, you can use a DVS broadcast capture card to capture video frames, >> pass those between threads which are doing processing on them, and >> inside those threads use c extensions to process the data efficiently >> (since image processing does take time...), and those release the GIL >> boosting throughput. >> >> [1] On this project : >> http://www.bbc.co.uk/rd/projects/2009/10/i3dlive.shtml >> >> So, that makes it all sound great - ie things can, after various >> fashions, run in parallel on various versions of python, to practical >> benefit. But obviously it could be improved. >> >> Personally, I think the project most likely to make a difference here >> is actually pypy. Now, talk is very cheap, and easy, and I'm not >> likely to implement this, so I'll aim to be brief. Execution is hard. >> >> In particular, what I think is most likely to be beneficial is >> something _like_ this: >> >> Assume pypy runs without a GIL. Then allow the creation of a green >> process. A green process is implemented using threads, but with data >> created on the heap such that it defaults to being marked private to >> the thread (ie ala thread local storage, but perhaps implemented >> slightly differently - via references from the thread local storage >> into the heap) rather than shared. Sharing between green processes >> (for channels or boxes) would "simply" be detagged as being owned by >> one thread, and passed to another. >> >> In particular this would mean that you need a mechanism for doing >> this. Simply attempting to call another green process (or thread) from >> another with mutable data types would be sufficient to raise the >> equivalent of a segmentation fault. >> >> Secondly, improve cpyext to the extent that each cpython extension >> gets it's own version of the GIL. (ie each extension runs with its own >> logical runtime, and thinks that it has its own GIL which it can lock >> and release. In practice it's faked by the PyPy runtime. This is >> essentially similar conceptually to creating green processes. >> >> It's worth considering that the Linux kernel went through similar >> changes, in that in the 2.0 days there was a large single big lock, >> which was replaced by ever granular locks. I personally think that >> since there are so many extensions that rely on the existence of the >> GIL simply waving a wand to get rid of it isn't likely. However >> logically providing a GIL per C-Extension may be plausible, and _may_ >> be sufficient. >> >> However, I don't know - it might well not - I've not looked at the >> code, and talk is cheap - execution is hard. >> >> Hopefully the above (cheap :) comments are in some small way useful. >> >> Regards, >> >> >> Michael. >

2 1

Re: [pypy-dev] Would the following shared memory model be possible?
by Andrew Francis 31 Jul '10

31 Jul '10

Hi JP: Message: 1 Date: Thu, 29 Jul 2010 21:24:58 -0000 From: exarkun(a)twistedmatrix.com Subject: Re: [pypy-dev] Would the following shared memory model be possible? To: pypy-dev(a)codespeak.net Message-ID: <20100729212458.2188.24074246.divmod.xquotient.34(a)localhost.localdomain> Content-Type: text/plain; charset="utf-8"; format="flowed" On 08:39 pm, andrewfr_ice(a)yahoo.com wrote: > >Okay. I attended Ray's Hettinger's talk on Monocle. In the past >I have encountered situations where I bumped up with the nesting >problem. >If I recall, the problem involved request handlers that had a RPC style >AND made additional Twisted deferred calls: > >class MyRequestHandler(...): > > @defer.inlineCallbacks > def process(self): > try: > result = yield >client.getPage("http://www.google.com") > except Exception, err: > log.err(err, "process getPage call >failed") > else: > # do some processing with the result > return result > >looks reasonable but Python will balk. JP>Aside from the "return result" (should be defer.returnValue(result), JP>generators can't return with a value), this looks fine to me too. Why JP>do you say Python will balk? Well the return with a value was the deal breaker. I used this example because this is where I came face-to-face with nested generators - and generated a mistrust for them in regard to exotic uses. There was something else about the real example (I am having a hard time finding the posts - somewhere in 2007) - I think it was a very early version of PyAMF and it really wanted a return (HTTP is okay). I believe under the hood, if the protocol returns a deferred or None, the reactor will expect further output in the future. Cheers, Andrew Cheers, Andrew

1 0

S3 2010 deadline extension
by Carl Friedrich Bolz 31 Jul '10

31 Jul '10

The S3 2010 Paper deadline was moved forward by two weeks, and is now August 13, 2010. *** Workshop on Self-sustaining Systems (S3) 2010 *** September 27-28, 2010 The University of Tokyo, Japan http://www.hpi.uni-potsdam.de/swa/s3/s3-10/ In cooperation with ACM SIGPLAN === Call for papers === The Workshop on Self-sustaining Systems (S3) is a forum for discussion of topics relating to computer systems and languages that are able to bootstrap, implement, modify, and maintain themselves. One property of these systems is that their implementation is based on small but powerful abstractions; examples include (amongst others) Squeak/Smalltalk, COLA, Klein/Self, PyPy/Python, Rubinius/Ruby, and Lisp. Such systems are the engines of their own replacement, giving researchers and developers great power to experiment with, and explore future directions from within, their own small language kernels. S3 will be take place September 27-28, 2010 at The University of Tokyo, Japan. It is an exciting opportunity for researchers and practitioners interested in self-sustaining systems to meet and share their knowledge, experience, and ideas for future research and development. --- Submissions and proceedings --- S3 invites submissions of high-quality papers reporting original research, or describing innovative contributions to, or experience with, self-sustaining systems, their implementation, and their application. Papers that depart significantly from established ideas and practices are particularly welcome. Submissions must not have been published previously and must not be under review for any another refereed event or publication. The program committee will evaluate each contributed paper based on its relevance, significance, clarity, and originality. Revised papers will be published as post-proceedings in the ACM Digital Library. Papers should be submitted electronically via EasyChair at http://www.easychair.org/conferences/?conf=s32010 in PDF format. Submissions must be written in English (the official language of the workshop) and must not exceed 10 pages. They should use the ACM SIGPLAN 10 point format, templates for which are available at http://www.acm.org/sigs/sigplan/authorInformation.htm. --- Venue --- The University of Tokyo, Komaba Campus, Japan --- Important dates --- Submission of papers: *EXTENDED* August 13, 2010 Author notification: August 27, 2010 Early registration: September 3, 2010 Revised papers: September 10, 2010 S3 workshop: September 27-28, 2010 Final papers for ACM-DL post-proceedings: October 15, 2010 --- Invited talks --- Yukihiro Matsumoto: "From Lisp to Ruby to Rubinius" Takashi Ikegami: "Sustainable Autonomy and Designing Mind Time" --- Chairs --- Robert Hirschfeld (Hasso-Plattner-Institut Potsdam, Germany) hirschfeld(a)hpi.uni-potsdam.de Hidehiko Masuhara (The University of Tokyo, Japan) masuhara(a)graco.c.u-tokyo.ac.jp Kim Rose (Viewpoints Research Institute, USA) kim.rose(a)vpri.org --- Program committee --- Carl Friedrich Bolz, University of Duesseldorf, Germany Johan Brichau, Universite Catholique de Louvain, Belgium Shigeru Chiba, Tokyo Institute of Technology, Japan Brian Demsky, University of California, Irvine, USA Marcus Denker, INRIA Lille, France Richard P. Gabriel, IBM Research, USA Michael Haupt, Hasso-Plattner-Institut, Germany Robert Hirschfeld, Hasso-Plattner-Institut, Germany (co-chair) Atsushi Igarashi, University of Kyoto, Japan David Lorenz, The Open University, Israel Hidehiko Masuhara, University of Tokyo, Japan (co-chair) Eliot Miranda, Teleplace, USA Ian Piumarta, Viewpoints Research Institute, USA Martin Rinard, MIT, USA Antero Taivalsaari, Nokia, Finland David Ungar, IBM, USA _______________________________________________ fonc mailing list fonc(a)vpri.org http://vpri.org/mailman/listinfo/fonc

1 0

FW: Would the following shared memory model be possible?
by Kevin Ar18 31 Jul '10

31 Jul '10

> Would comments from a project using this approach in real systems be > of interest/use/help? Whilst I didn't know about Morrison's FBP > (Balzer's work predates him btw - don't listen to hype) I had heard of > (and played with) Occam among other more influential things, and > Kamaelia is a real tool. Also there is already a pre-existing FBP tool > for Stackless, and then historically there's also MASCOT & friends. It You brought up a lot of topics. I went ahead and sent you a private email. There's always lots of interesting things I can add to my list of things to learn about. :) > just looks to me that you're tieing yourself up in knots over things > that aren't problems, when there are some things which could be useful > (in practice) & interesting in this space. The particular issue in this situation is that there is no way to make Kamaelia, FBP, or other concurrency concepts run in parallel (unless you are willing to accept lots of overhead like with the multiprocessing queues). Since you have worked with Kamaelia code a lot... you understand a lot more about implementation details. Do you think the previous shared memory concept or something like it would let you make Kamaelia parallel? If not, can you think of any method that would let you make Kamaelia parallel?

2 1

Re: [pypy-dev] Would the following shared memory model be possible?
by Andrew Francis 29 Jul '10

29 Jul '10

Hi Michael: --- On Thu, 7/29/10, Michael Sparks <ms(a)cerenity.org> wrote: > It's a pity we didn't get a chance to chat at the > conference. (I was the one videoing everything for upload after >transcoding :) Yes I noticed. I gave the talk "Prototyping Go's Select with stackless.py for Stackless Python." Much of that talk dealt with rendezvous semantics courtesy via synchronous channels. I will post the Original slides and the Revised version (mistakes corrected ) in a day or two. > > >http://www.kamaelia.org/MiniAxon.html > > > I'm biassed towards Kamaelia (naturally :-), but I agree. > MiniAxon is just a toy/tutorial. Early in kamaelia's history we >considered using stackless, but rejected it simply because we wanted to >work mainly with mainline python, rather than a specialised version. Fair enough. Currently Stackless Python is being integrated with Psyco and will be available as a module. > Other things in Stackless's favour (IIRC) - include the > fact that you can pickle generators, and send them across network >connection, unpickle them and let them continue running. I don't know >if you do the same with tasklets, but I wouldn't be surprised if you do :) As long as you do not have a C Frame involved, you can pickle a tasklet. That was the subject of my "Silly Stackless Python Trick" lighting talk. I was going to demonstrate a version of the Sieve of Eratosthenes that could be pickled and resumed on another machine. However my HP Netbook had a non-standard VGA output connection and I needed to install Stackless on a loaner ThinkPad that died as I hooked it up. However you saw all that :-( > That means you have potential for process migration. Yep. Gabriel Lavoie does a lot of work in that area with PyPy (thanks Gabriel !) > Doing that sensibly though IMO would require better understanding > in the system of what the user is trying to achieve and what they're >sending. (It's easy to think of examples where this causes more pain than >it's worth after all) You have to understand what can be pickled. Occasionally you are in for a surprise (i.e. functools). >You could argue in that case that the biggest _real_ difference is >that we try to use a unified API for different concurrency >methods. Well I would argue that Stackless has a simple elegant model. The addition of select just adds more power.Stackless channels can also serve as generators (they are iterable). I recently took a stab at writing the Sleeping Barber's problem. I think in Stackless, the basic solution was about 30 lines. Very little clutter. > One **highly subjective** other thing in our favour, is > that generators are limited to a single level of control flow >(ie non-nestable without a trampoline). This doesn't sounds like > an advantage, but it tends to lead to simpler components which are >in turn reusable. (and that I view as useful :) Okay. I attended Ray's Hettinger's talk on Monocle. In the past I have encountered situations where I bumped up with the nesting problem. If I recall, the problem involved request handlers that had a RPC style AND made additional Twisted deferred calls: class MyRequestHandler(...): @defer.inlineCallbacks def process(self): try: result = yield client.getPage("http://www.google.com") except Exception, err: log.err(err, "process getPage call failed") else: # do some processing with the result return result looks reasonable but Python will balk. Nested generators. Only way around it is that you had to hope that the Twisted protocol was properly written and chain deferreds. > Have fun, I do :-) Cheers, Andrew

2 1

Re: [pypy-dev] pypy-dev Digest, Vol 360, Issue 13
by Andrew Francis 29 Jul '10

29 Jul '10

Hi Kevin: Message: 1 Date: Tue, 27 Jul 2010 14:20:10 -0400 From: Kevin Ar18 <kevinar18(a)hotmail.com> Subject: Re: [pypy-dev] pre-emptive micro-threads utilizing shared memory message passing? To: <pypy-dev(a)codespeak.net> Message-ID: <SNT110-W23BAAD6B986917DCB074DFAAA70(a)phx.gbl> Content-Type: text/plain; charset="iso-8859-1" >I am attempting to experiment with FBP - Flow Based Programming >(http://www.jpaulmorrison.com/fbp/ and book: http://www.jpaulmorrison.com>/fbp/book.pdf)? There is something very similar in Python: >http://www.kamaelia.org/MiniAxon.html? Also, there are some similarities >to Erlang - the share nothing memory model... and on some very broad >levels, there are similarities that can be found in functional languages. I just came back from EuroPython. A lot of discussion on concurrency.... Well functional languages (like Erlang), variables tend to be immutable. This is a bonus in a concurrent system - makes it easier to reason about the system - and helps to avoid various race conditions. As for the shared memory. I think there is a difference between whether things are shared at the application programmer level, or under the hood controlled by the system. Programmers tend to beare bad at the former. >http://www.kamaelia.org/MiniAxon.html I took a quick look. Maybe I am biased but Stackless Python gives you most of that. Also tasklets and channels can do everything a generator can and more (a generator is more specialised than a coroutine). Also it is easy to mimic asynchrony with a CSP style messaging system where microthreads and channels are cheap. A line from the book "Actors: A Model of Concurrent Computation in Distributed Systems" by Gul A. Agha comes to mind: "synchrony is mere buffered asynchrony." >The process of connecting the boxes together was actually designed to be >programmed visually, as you can see from the examples in the book (I have >no idea if it works well, as I am merely starting to experiment with it). What bought me to Stackless Python and PyPy was work concerning WS-BPEL. Allegedly, WS-BPEL/XLang/WSFL (Web-Services Flow Language) are based on formalisms like pi calculus. Since I don't own a multi-core machine and I am not doing CPU intense stuff, I never really cared. However I have been doing things where I needed to impose logical orderings upon processes (i.e., process C can only run after process A and B are finished). My initial native uses of Stackless (easy to do in anything system based on CSP), resulted in deadlocking the system. So I found understanding deadlock to be very important. >Each box, being a self contained "program," the only data it has access >to is 3 parts: >Implementation of FBP requires a custom scheduler for several reasons: >(1) A box can only run if it has actual data on the "in port(s)"? Thus, >the scheduler would only schedule boxes to run when they can actually >process some data. Stackless Python already works like this. No custom scheduler needed. I would recommend you read Rob Pike's paper "The Implementation of Newsqueak" or some of the Cardilli papers to understand how CSP constructs with channels work. And if you need to customize schedulers - you have two routes 1) Use pre-existing classes and API 2) Experiment with PyPy's stackless.py >(2) In theory, it may be possible to end up with hundreds or thousands of >these light weight boxes.? Using heavy-weight OS threads or processes for every one is out of the question. Stackless Python. >In a perfect world, here's what I might do: * Assume a quad core cpu >(1) Spawn 1 process >(2) Spawn 4 threads & assign each thread to only 1 core -- in other >words, don't let the OS handle moving threads around to different cores >(3) Inside each thread, have a mini scheduler that switches back and >forth between the many micro-threads (or "boxes") -- note that the OS >should not handle any of the switching between micro-threads/boxes as it >does it all wrong (and to heavyweight) for this situation. >(4) Using a shared memory queue, each of the 4 schedulers can get the >next box to run... or add more boxes to the schedule queue. My advice: get stuff properly working under a single threaded model first so you understand the machinery. That said, I think Carlos Eduardo de Paula a few years ago played with adapting Stackless for multi-processing. Second piece of advice: start looking at how Go does things. Stackless Python and Go share a common ancestor. However Go does much more on the multi-core front. Cheers, Andrew

2 1

Re: [pypy-dev] Would the following shared memory model be possible?
by Michael Sparks 29 Jul '10

29 Jul '10

I make it a point these days to only reply on-list. It leads to endless repetition otherwise. If you repost this cc'ing the pypy-dev list I'll reply. If you think it's off topic there, then I see no point. Michael. On Thursday 29 July 2010 18:05:27 you wrote: > Thanks for the reply. > > > Would comments from a project using this approach in real systems be > > of interest/use/help? > > I contacted someone from Kamaelia a while back (probably you). > Yes, use of the dataflow concept would be really useful (no > MIT/BSD/Python/PD license). However, licensing was an issues, so I went > it on my own. I find the concept rather interesting both to maybe learn > from and to actually try and use in an actual application. > > > Whilst I didn't know about Morrison's FBP > > (Balzer's work predates him btw - don't listen to hype) I had heard of > > (and played with) Occam among other more influential things, and > > Kamaelia is a real tool. > > What is this Balzer and Occam? :) Do you have any links I can look at? > > > Also there is already a pre-existing FBP tool > > for Stackless > > The problem is that Stackless is not parallel, which is what I would really > like to do. > > > , and then historically there's also MASCOT & friends. > > Do you have a link about this? -- >>>

1 0

rotting buildbot infrastructure
by Maciej Fijalkowski 28 Jul '10

28 Jul '10

Hello. According to current buildbot status, both osx and win machines are offline. No clue how to get them back. Anyway, our OS X machine is unable to translate pypy, so it's not exactly the best buildbot ever. Can anyone contribute any machine for one of those buildbots? Cheers fijal

3 3

PyPy Proxy
by Hart's Antler 27 Jul '10

27 Jul '10

The code from PyPyGTK has been generalized so that it can work with PyGame, and PyODE. Function calls are improved so that different arg types can be accepted by defining custom wrappers per function. Custom wrappers can also be made for the return values, so different types can be handled as well (it seems that rpython restricts what can be returned from a function to the same types). Proxy objects can move back and forth from CPython to RPython. The wrappers for pygtk, pyode, and pygame are by no means complete, but some basic tests are working. Callbacks are limited to quoted lambdas, but it could be improved. http://pastebin.com/rWEfgMSN I had seen the other proxy method before, but found few examples, how does it work, from Rpython or the PyPy interpreter? http://morepypy.blogspot.com/2009/11/using-cpython-extension-modules-with.h…

2 2

PyPyGTK v0.1
by Hart's Antler 24 Jul '10

24 Jul '10

http://pastebin.com/UhnEurqb The above is a crude way to run pygtk from rpython (through CPython and talking on a pipe), but at least it partially works. Callbacks are limited to quoted lambdas, but some simple return types back to rpython is possible - i'm going to try that next. There is no support for dynamic attribute access, but most of pygtk involves function calls. Where attribute access is required, i guess extra proxy functions could be written. -brett

2 1

rpython questions, **kw, __call__, __getattr__
by Hart's Antler 23 Jul '10

23 Jul '10

Looking through the pypy source code i see **kw, __call__ and __getattr__ are used, but when i try to write my own rpython code that uses these conventions, i get translation errors. Do i need to borrow from "application space" in order to do this or can i just give hints to the annotator? Thanks, -brett #this is allowed def func(*args): print(args) #but this is not? def func(**kw): print(args) #error call pattern too complex #this class fails to translate, are we not allowed to define our own __call__ and __getattr__ in rpython? class A(object): __call__(*args): print(args) __getattr__(self,name): print(name)

3 2

Building a shared library on x86-64 fails due to static linking of libffi
by Jason Creighton 22 Jul '10

22 Jul '10

Hello, While working on asmgcc-64, I ran into this issue. For some reason, PyPy wants to link libffi statically on some platforms, Linux included. But when compiling with the "shared" option (as is done in some asmgcroot tests), you get link errors like: /usr/bin/ld: /usr/lib/libffi.a(ffi64.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC /usr/lib/libffi.a: could not read symbols: Bad value I interpret this to mean that since we building a shared library, the resulting library must be position independent, so we can't link in non-PIC such as is found in the static version of libffi on my system. (Ubuntu 10.04, x86-64). And indeed, if I switch to linking dynamically, the error goes away and things seem to work. However, I don't want to just blindly enable dynamic linking, because there must be a reason it was configured to link statically in the first place. What is that reason? Also, what steps should I take here? I think I need to enable dynamic linking of libffi on x86-64 Linux when building a shared library at the very least, but to reduce the number of code paths, I'm somewhat inclined to link dynamically whether we're building a library or not. What do you guys think? Thanks, Jason

3 2

Re: [pypy-dev] [pypy-svn] r76268 - pypy/branch/micronumpy/pypy/tool
by Maciej Fijalkowski 18 Jul '10

18 Jul '10

Benchmarks generally should go to pypy/benchmarks directory in the main source tree (that is svn+ssh://codespeak.net/svn/pypy/benchmarks) On Sat, Jul 17, 2010 at 12:41 AM, <dan(a)codespeak.net> wrote: > Author: dan > Date: Sat Jul 17 00:41:02 2010 > New Revision: 76268 > > Added: > pypy/branch/micronumpy/pypy/tool/convolve.py > Modified: > pypy/branch/micronumpy/pypy/tool/numpybench.py > Log: > Oops, I forgot the most important part of the benchmark! > > Added: pypy/branch/micronumpy/pypy/tool/convolve.py > ============================================================================== > --- (empty file) > +++ pypy/branch/micronumpy/pypy/tool/convolve.py Sat Jul 17 00:41:02 2010 > @@ -0,0 +1,43 @@ > +from __future__ import division > +from __main__ import numpy as np > + > +def naive_convolve(f, g): > + # f is an image and is indexed by (v, w) > + # g is a filter kernel and is indexed by (s, t), > + # it needs odd dimensions > + # h is the output image and is indexed by (x, y), > + # it is not cropped > + if g.shape[0] % 2 != 1 or g.shape[1] % 2 != 1: > + raise ValueError("Only odd dimensions on filter supported") > + # smid and tmid are number of pixels between the center pixel > + # and the edge, ie for a 5x5 filter they will be 2. > + # > + # The output size is calculated by adding smid, tmid to each > + # side of the dimensions of the input image. > + vmax = f.shape[0] > + wmax = f.shape[1] > + smax = g.shape[0] > + tmax = g.shape[1] > + smid = smax // 2 > + tmid = tmax // 2 > + xmax = vmax + 2*smid > + ymax = wmax + 2*tmid > + # Allocate result image. > + h = np.zeros([xmax, ymax], dtype=f.dtype) > + # Do convolution > + for x in range(xmax): > + for y in range(ymax): > + # Calculate pixel value for h at (x,y). Sum one component > + # for each pixel (s, t) of the filter g. > + s_from = max(smid - x, -smid) > + s_to = min((xmax - x) - smid, smid + 1) > + t_from = max(tmid - y, -tmid) > + t_to = min((ymax - y) - tmid, tmid + 1) > + value = 0 > + for s in range(s_from, s_to): > + for t in range(t_from, t_to): > + v = x - smid + s > + w = y - tmid + t > + value += g[smid - s, tmid - t] * f[v, w] > + h[x, y] = value > + return h > > Modified: pypy/branch/micronumpy/pypy/tool/numpybench.py > ============================================================================== > --- pypy/branch/micronumpy/pypy/tool/numpybench.py (original) > +++ pypy/branch/micronumpy/pypy/tool/numpybench.py Sat Jul 17 00:41:02 2010 > @@ -21,13 +21,29 @@ > return numpy.array(kernel) > > if __name__ == '__main__': > - from sys import argv as args > - width, height, kwidth, kheight = [int(x) for x in args[1:]] > + from optparse import OptionParser > + > + option_parser = OptionParser() > + option_parser.add_option('--kernel-size', dest='kernel', default='3x3', > + help="The size of the convolution kernel, given as WxH. ie 3x3" > + "Note that both dimensions must be odd.") > + option_parser.add_option('--image-size', dest='image', default='256x256', > + help="The size of the image, given as WxH. ie. 256x256") > + option_parser.add_option('--runs', '--count', dest='count', default=1000, > + help="The number of times to run the convolution filter") > + > + options, args = option_parser.parse_args() > + > + def parse_dimension(arg): > + return [int(s.strip()) for s in arg.split('x')] > + > + width, height = parse_dimension(options.image) > + kwidth, kheight = parse_dimension(options.kernel) > + count = int(options.count) > > image = generate_image(width, height) > kernel = generate_kernel(kwidth, kheight) > > from timeit import Timer > convolve_timer = Timer('naive_convolve(image, kernel)', 'from convolve import naive_convolve; from __main__ import image, kernel; gc.enable()') > - count = 100 > print "%.5f sec/pass" % (convolve_timer.timeit(number=count)/count) > _______________________________________________ > pypy-svn mailing list > pypy-svn(a)codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn >

1 0

Windows binaries
by Dave Cross 13 Jul '10

13 Jul '10

Hi, Is there a likely delivery date for Windows binaries of PyPy 1.3? Dave. </PRE> ********************************************************************** Please consider the environment - do you really need to print this email? This email is intended only for the person(s) named above and may contain private and confidential information. If it has come to you in error, please destroy and permanently delete any copy in your possession and contact us on +44 (0) 161 480 4420. The information in this email is copyright © CDL Group Holdings Limited. We cannot accept any liability for any loss or damage sustained as a result of software viruses. It is your responsibility to carry out such virus checking as is necessary before opening any attachment. Cheshire Datasystems Limited uses software which automatically screens incoming emails for inappropriate content and attachments. If the software identifies such content or attachment, the email will be forwarded to our Technology Department for checking. You should be aware that any email which you send to Cheshire Datasystems Limited is subject to this procedure. Cheshire Datasystems Limited, Strata House, Kings Reach Road, Stockport SK4 2HD Registered in England and Wales with Company Number 3991057 VAT registration: 727 1188 33 <PRE>

2 1

Interactive Translation and JIT
by Hart's Antler 09 Jul '10

09 Jul '10

I'm using Jason Creighton branch, and i am trying to test the JIT from interactive translation. Is it now allowed? I'm getting this error: NotImplementedError: --gcrootfinder=asmgcc requires standalone Or am i not setting the options correctly on the translator, here is how i'm translating. from pypy.translator.interactive import Translation t = Translation( pypy_entry_point ) t.config.translation.suggest(jit=True, jit_debug='steps', jit_backend='x86', gc='boehm') t.annotate() t.rtype() f = t.compile_c() f() complete code: http://pastebin.com/T42cqSbz demo: http://www.youtube.com/watch?v=HwbDG3Rdi_Q -brett

2 2

Re: [pypy-dev] [pypy-svn] r75824 - in pypy/branch/interplevel-array/pypy/module/array: . test
by Hakan Ardo 05 Jul '10

05 Jul '10

I've checked in a dict-based version. Not sure it became that clean after all. Is the getattr(space, tc.unwrap) construction ok? On Mon, Jul 5, 2010 at 7:26 AM, Hakan Ardo <Hakan(a)ardoe.net> wrote: > On Sun, Jul 4, 2010 at 10:25 PM, Maciej Fijalkowski <fijall(a)gmail.com> wrote: >> >> Hey. This looks a bit ugly, > > It does, doesn't it :) > >> you can definitely do it with some >> constant dict or something > > Yes, there is an overflow check needed on the integer types but not on > the character an float types, but I guess that could be solved with a > flag in the dict. > > I was actually considering to introduce separate subclasses for each > typecode overriding intem_w and descr_getitem. That would get rid of > the typecode attribute lookup all together. > >> (we have special support for iterating over >> constants and unrolling the iteration, look for unrolling_iterable). >> Also, annotator can fold a bunch of ifs into a switch, but not if "in" >> operator is used (or is fine though). > > That's nice features, good to know about. Thanx. > > > > -- > Håkan Ardö > -- Håkan Ardö

1 0

Re: [pypy-dev] [pypy-svn] r75824 - in pypy/branch/interplevel-array/pypy/module/array: . test
by Hakan Ardo 05 Jul '10

05 Jul '10

On Sun, Jul 4, 2010 at 10:25 PM, Maciej Fijalkowski <fijall(a)gmail.com> wrote: > > Hey. This looks a bit ugly, It does, doesn't it :) > you can definitely do it with some > constant dict or something Yes, there is an overflow check needed on the integer types but not on the character an float types, but I guess that could be solved with a flag in the dict. I was actually considering to introduce separate subclasses for each typecode overriding intem_w and descr_getitem. That would get rid of the typecode attribute lookup all together. > (we have special support for iterating over > constants and unrolling the iteration, look for unrolling_iterable). > Also, annotator can fold a bunch of ifs into a switch, but not if "in" > operator is used (or is fine though). That's nice features, good to know about. Thanx. -- Håkan Ardö

1 0

Re: [pypy-dev] [pypy-svn] r75824 - in pypy/branch/interplevel-array/pypy/module/array: . test
by Maciej Fijalkowski 04 Jul '10

04 Jul '10

> + > + def item_w(self, w_item): > + space=self.space > + if self.typecode == 'c': > + return self.space.str_w(w_item) > + elif self.typecode == 'u': > + return self.space.unicode_w(w_item) > + > + elif self.typecode == 'b': > + item=self.space.int_w(w_item) > + if item<-128: > + msg='signed char is less than minimum' > + raise OperationError(space.w_OverflowError, space.wrap(msg)) > + elif item>127: > + msg='signed char is greater than maximum' > + raise OperationError(space.w_OverflowError, space.wrap(msg)) > + return rffi.cast(rffi.SIGNEDCHAR, item) > + elif self.typecode == 'B': > + item=self.space.int_w(w_item) > + if item<0: > + msg='unsigned byte integer is less than minimum' > + raise OperationError(space.w_OverflowError, space.wrap(msg)) > + elif item>255: > + msg='unsigned byte integer is greater than maximum' > + raise OperationError(space.w_OverflowError, space.wrap(msg)) > + return rffi.cast(rffi.UCHAR, item) > + > + elif self.typecode == 'h': > + item=self.space.int_w(w_item) > + if item<-32768: > + msg='signed short integer is less than minimum' > + raise OperationError(space.w_OverflowError, space.wrap(msg)) > + elif item>32767: > + msg='signed short integer is greater than maximum' > + raise OperationError(space.w_OverflowError, space.wrap(msg)) > + return rffi.cast(rffi.SHORT, item) > + elif self.typecode == 'H': > + item=self.space.int_w(w_item) > + if item<0: > + msg='unsigned short integer is less than minimum' > + raise OperationError(space.w_OverflowError, space.wrap(msg)) > + elif item>65535: > + msg='unsigned short integer is greater than maximum' > + raise OperationError(space.w_OverflowError, space.wrap(msg)) > + return rffi.cast(rffi.USHORT, item) > + > + elif self.typecode in ('i', 'l'): > + item=self.space.int_w(w_item) > + if item<-2147483648: > + msg='signed integer is less than minimum' > + raise OperationError(space.w_OverflowError, space.wrap(msg)) > + elif item>2147483647: > + msg='signed integer is greater than maximum' > + raise OperationError(space.w_OverflowError, space.wrap(msg)) > + return rffi.cast(lltype.Signed, item) > + elif self.typecode in ('I', 'L'): > + item=self.space.int_w(w_item) > + if item<0: > + msg='unsigned integer is less than minimum' > + raise OperationError(space.w_OverflowError, space.wrap(msg)) > + elif item>4294967295: > + msg='unsigned integer is greater than maximum' > + raise OperationError(space.w_OverflowError, space.wrap(msg)) > + return rffi.cast(lltype.Unsigned, item) > + > + elif self.typecode == 'f': > + item=self.space.float_w(w_item) > + return rffi.cast(lltype.SingleFloat, item) > + elif self.typecode == 'd': > + return self.space.float_w(w_item) > + Hey. This looks a bit ugly, you can definitely do it with some constant dict or something (we have special support for iterating over constants and unrolling the iteration, look for unrolling_iterable). Also, annotator can fold a bunch of ifs into a switch, but not if "in" operator is used (or is fine though).

1 0

Interpreter level array implementation
by Hakan Ardo 04 Jul '10

04 Jul '10

Hi, we got the simplest possible interpreter level implementation of an array-like object running (in the interplevel-array branch) and it executes my previous example about 2 times slower than optimized C. Attached is the trace generated by the following example: img=array(640*480); l=0; i=0; while i<640*480: l+=img[i] i+=1 a simplified version of that trace is: 1. [p0, p1, p2, p3, i4, p5, p6, p7, p8, p9, p10, f11, i] 2. i14 = int_lt(i, 307200) 3. guard_true(i14, descr=<Guard1>) 4. guard_nonnull_class(p10, 145745952, descr=<Guard2>) 5. img = getfield_gc(p10, descr=<GcPtrFieldDescr 8>) 6. f17 = getarrayitem_gc(img, i, descr=<FloatArrayDescr>) 7. f18 = float_add(f11, f17) 8. i20 = int_add_ovf(i, 1) 9. guard_no_overflow(, descr=<Guard3>) # 10. i23 = getfield_raw(149604768, descr=<SignedFieldDescr 0>) 11. i25 = int_add(i23, 1) 12. setfield_raw(149604768, i25, descr=<SignedFieldDescr 0>) 13. i28 = int_and(i25, -2131755008) 14. i29 = int_is_true(i28) 15. guard_false(i29, descr=<Guard4>) 16. jump(p0, p1, p2, p3, 27, ConstPtr(ptr31), ConstPtr(ptr32), ConstPtr(ptr33), p8, p9, p10, f18, i20) Does these operation more or less correspond to assembler instructions? I guess that the extra overhead here as compared to the the C version would be line 4, 5, 9 and 10-15. What's 10-15 all about? I guess that most of these additional operation would not affect the performance of more complicated loops as they will only occur once per loop (although combining the guard on line 9 with line 3 might be a possible optimization)? Line 4 will appear once for each array used in the loop and line 5 once for every array access, right? Can the array implementation be designed in someway that would not generate line 5 above? Or would it be possible to get rid of it by some optimization? -- Håkan Ardö

8 13

Re: [pypy-dev] array performace?
by Paolo Giarrusso 03 Jul '10

03 Jul '10

On Fri, Jul 2, 2010 at 10:55, <Ben.Young(a)sungard.com> wrote: >> On Fri, Jul 2, 2010 at 2:26 AM, <Ben.Young(a)sungard.com> wrote: >> >> On Fri, Jul 2, 2010 at 1:47 AM, Paolo Giarrusso >> > <p.giarrusso(a)gmail.com> >> >> wrote: >> >> > On Fri, Jul 2, 2010 at 08:04, Maciej Fijalkowski <fijall(a)gmail.com> >> > wrote: >> >> >> On Thu, Jul 1, 2010 at 1:18 PM, Hakan Ardo <Hakan(a)ardoe.net> wrote: >> >> >>> OK, so making an interpreter level implementation of array.array >> > seams >> >> >>> like a good idea. Would it be possible to get the jit to remove >> > the >> >> >>> wrapping/unwrapping in that case to get better performance than >> >> >>> _rawffi.Array('d'), which is already an interpreter level >> >> >>> implementation? >> >> >> >> >> >> it should work mostly out of the box (you can also try this for >> >> >> _rawffi.array part of module, if you want to). It's probably enough >> > to >> >> >> enable module in pypy/module/pypyjit/policy.py so JIT can have a >> > look >> >> >> there. In case of _rawffi, probably a couple of hints for the jit >> > to >> >> >> not look inside some functions (which do external calls for >> > example) >> >> >> should also be needed, since for example JIT as of now does not >> >> >> support raw mallocs (using C malloc and not our GC). >> >> > >> >> >> Still, making an >> >> >> array module interp-level is probably the sanest approach. >> >> > >> >> > That might be a bad sign. >> >> > For CPython, people recommend to write extensions in C for >> >> > performance, i.e. to make them less maintainable and understandable >> >> > for performance. >> >> > A good JIT should make this unnecessary in as many cases as >> > possible. >> >> > Of course, the array module might be an exception, if it's a single >> >> > case. >> >> > But performance 20x slower than C, with a JIT, is a big warning, >> > since >> >> > fast interpreters are documented to be (in general) just 10x slower >> >> > than C. >> >> >> >> There is a lot of unsupported claims in your sentences, however, >> >> that's not my point. >> >> >> > >> > That's a little harsh. When the JIT was originally developed it was >> > envisaged that it would be faster to re-write code to app level to give >> > speed-ups. If that's changed that's fine, but it's not an "unsupported >> > claim" >> > >> > Ben >> > >> >> Unsupported claim is for example that fast interpreters are 10x slower >> than C. That's the only unsupported claim, but it comes from "The Structure and Performance of Eﬃcient Interpreters". I studied that as a student on VM, you are writing one, so I (unconsciously) guessed that everybody knows that paper - I know that's a completely broken way of writing, but I didn't spot it. >>On what exactly? Did he write this particular benchmark in C >> and in fast interpreter to compare? Another unsupported claim is that >> JIT is 20x slower than C here. I did not claim that - I am aware that it is not even JITted. I complain against the lack of JITting. >> Array module is not even JITted, >> because it's based on _rawffi which itself operates on low-level >> pointers which JIT does not want to deal with. I would say that instead of doing manual annotations or rewriting at the interp-level (which doesn't scale), it would be overall simpler to make the JIT learn itself how to deal with those calls (i.e. inline everything around, leave the external call as a call), once and for all. What you suggest below might be a way to do it. >> That's exactly the >> reason why JIT doesn't look into _rawffi module and making it look >> there doesn't sound like a good idea (instead, we're trying to replace >> it with something JIT-friendly that knows how to do FFI calls into C, >> there is a summer of code project). Well, at the abstraction level I'm speaking, it sounds like there in the end, the JIT will be able to do what is needed. I am not aware of the details. But then, at the end of that project, it seems to me that it should be possible to write the array module in pure Python using this new FFI interface and have the JIT look at it, shouldn't it? I do not concentrate on array specifically - rewriting a few modules at interpreter level is fine. But as a Python developer I should have no need for that. >> All I'm trying to say is that there are valid reasons that array >> module should be on interpreter level and none of this has anything to >> do with incapabilities of the JIT. > Fair enough, and I do see your point, but I think Paolo comment was not aimed at array, just the implication (in this case) that to get performance you need to re-write in rpython. I think his point in general is correct, even if he picked the wrong example to mention it :) (and his 20x claim comes from the original email, so I don't think it's entirely unsupported) Thanks for understanding my point. I'm unsure whether an ideal JIT could allow leaving array at the app-level (and I noted also in the original mail that I was unsure on this). > Of course in this case I'm sure there are good reasons, but it is certainly interesting to see the push towards more rpython code than app-level. I guess that's because the JIT can "see" and accelerate rpython code too I believe, so it’s win-win (because of the code size issues and things like that) > Incidentally, is there a reason that geninterped code is so bloated compared to rpython code that looks like it could have been generated from the app-level equivalent? Would there be a way of annotating the app-level code so that when it's geninterped it's as tight as the equivalent rpython? -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/

7 11

PyPy Speed
by Ben.Young＠sungard.com 02 Jul '10

02 Jul '10

http://speed.pypy.org/overview/ seems to have been unavailable for the last couple of days. It gives a 500 whenever I visit it Ben Young - Senior Software Engineer SunGard - Enterprise House, Vision Park, Histon, Cambridge, CB24 9ZR Tel +44 1223 266042 - Main +44 1223 266100 - http://www.sungard.com/ CONFIDENTIALITY: This email (including any attachments) may contain confidential, proprietary and privileged information, and unauthorized disclosure or use is prohibited. If you received this email in error, please notify the sender and delete this email from your system. Thank you.

3 6

Re: [pypy-dev] array performace?
by Maciej Fijalkowski 02 Jul '10

02 Jul '10

On Fri, Jul 2, 2010 at 2:26 AM, <Ben.Young(a)sungard.com> wrote: >> On Fri, Jul 2, 2010 at 1:47 AM, Paolo Giarrusso > <p.giarrusso(a)gmail.com> >> wrote: >> > On Fri, Jul 2, 2010 at 08:04, Maciej Fijalkowski <fijall(a)gmail.com> > wrote: >> >> On Thu, Jul 1, 2010 at 1:18 PM, Hakan Ardo <Hakan(a)ardoe.net> wrote: >> >>> OK, so making an interpreter level implementation of array.array > seams >> >>> like a good idea. Would it be possible to get the jit to remove > the >> >>> wrapping/unwrapping in that case to get better performance than >> >>> _rawffi.Array('d'), which is already an interpreter level >> >>> implementation? >> >> >> >> it should work mostly out of the box (you can also try this for >> >> _rawffi.array part of module, if you want to). It's probably enough > to >> >> enable module in pypy/module/pypyjit/policy.py so JIT can have a > look >> >> there. In case of _rawffi, probably a couple of hints for the jit > to >> >> not look inside some functions (which do external calls for > example) >> >> should also be needed, since for example JIT as of now does not >> >> support raw mallocs (using C malloc and not our GC). >> > >> >> Still, making an >> >> array module interp-level is probably the sanest approach. >> > >> > That might be a bad sign. >> > For CPython, people recommend to write extensions in C for >> > performance, i.e. to make them less maintainable and understandable >> > for performance. >> > A good JIT should make this unnecessary in as many cases as > possible. >> > Of course, the array module might be an exception, if it's a single >> > case. >> > But performance 20x slower than C, with a JIT, is a big warning, > since >> > fast interpreters are documented to be (in general) just 10x slower >> > than C. >> >> There is a lot of unsupported claims in your sentences, however, >> that's not my point. >> > > That's a little harsh. When the JIT was originally developed it was > envisaged that it would be faster to re-write code to app level to give > speed-ups. If that's changed that's fine, but it's not an "unsupported > claim" > > Ben > Unsupported claim is for example that fast interpreters are 10x slower than C. On what exactly? Did he write this particular benchmark in C and in fast interpreter to compare? Another unsupported claim is that JIT is 20x slower than C here. Array module is not even JITted, because it's based on _rawffi which itself operates on low-level pointers which JIT does not want to deal with. That's exactly the reason why JIT doesn't look into _rawffi module and making it look there doesn't sound like a good idea (instead, we're trying to replace it with something JIT-friendly that knows how to do FFI calls into C, there is a summer of code project). All I'm trying to say is that there are valid reasons that array module should be on interpreter level and none of this has anything to do with incapabilities of the JIT. Cheers, fijal

1 0

array performace?
by Hakan Ardo 02 Jul '10

02 Jul '10

Hi, are there any python construct that the jit will be able to compile into c-type array accesses? Consider the following test: l=0.0 for i in xrange(640,640*480): l+=img[i] intimg[i]=intimg[i-640]+l With the 1.3 release of the jit it executes about 20 times slower than a similar construction in C if I create the arrays using: import _rawffi RAWARRAY = _rawffi.Array('d') img=RAWARRAY(640*480, autofree=True) intimg=RAWARRAY(640*480, autofree=True) Using a list is about 40 times slower and using array.array is about 400 times slower. Any suggestion on how to improve the performance of these kind of constructions? Thanx. -- Håkan Ardö

4 10