Mailman 3 July 2010 - pypy-dev

Building pypy on freebsd 64bit
by Shadrin Eugene 18 Nov '11

18 Nov '11

Hi! I tryed to build pypy on the freebsd, 64bit platform, but failed. First of all I downloaded a 32bit-Python to run pypy. And then: [user@server_name ~/pypy-trunk/pypy/translator/goal]$ ~/py32/bin/python translate.py -Ojit [translation:info] Translating target as defined by targetpypystandalone [platform:execute] gcc -m32 -c -O3 -pthread -fomit-frame-pointer /var/tmp/usession-trunk-22/gcctest.c -o /var/tmp/usession-trunk-22/gcctest.o [platform:WARNING] /var/tmp/usession-trunk-22/gcctest.c:28:2: warning: no newline at end of file [platform:execute] gcc -m32 /var/tmp/usession-trunk-22/gcctest.o -L/lib32 -L/usr/lib32 -L`pwd`/lib32 -Wl,-rpath,/lib32 -Wl,-rpath,/usr/lib32 -pthread -o /var/tmp/usession-trunk-22/gcctest printf("sizeof short=%ld\n", (long)sizeof(short)); printf("sizeof unsigned short=%ld\n", (long)sizeof(unsigned short)); printf("sizeof int=%ld\n", (long)sizeof(int)); printf("sizeof unsigned int=%ld\n", (long)sizeof(unsigned int)); printf("sizeof long=%ld\n", (long)sizeof(long)); printf("sizeof unsigned long=%ld\n", (long)sizeof(unsigned long)); printf("sizeof signed char=%ld\n", (long)sizeof(signed char)); printf("sizeof unsigned char=%ld\n", (long)sizeof(unsigned char)); printf("sizeof long long=%ld\n", (long)sizeof(long long)); printf("sizeof unsigned long long=%ld\n", (long)sizeof(unsigned long long)); printf("sizeof size_t=%ld\n", (long)sizeof(size_t)); printf("sizeof time_t=%ld\n", (long)sizeof(time_t)); printf("sizeof wchar_t=%ld\n", (long)sizeof(wchar_t)); printf("sizeof mode_t=%ld\n", (long)sizeof(mode_t)); printf("sizeof pid_t=%ld\n", (long)sizeof(pid_t)); sizeof short=2 sizeof unsigned short=2 sizeof int=4 sizeof unsigned int=4 sizeof long=8 sizeof unsigned long=8 sizeof signed char=1 sizeof unsigned char=1 sizeof long long=8 sizeof unsigned long long=8 sizeof size_t=8 sizeof time_t=8 sizeof wchar_t=4 sizeof mode_t=2 sizeof pid_t=4 Traceback (most recent call last): File "translate.py", line 300, in <module> main() File "translate.py", line 205, in main targetspec_dic, translateconfig, config, args = parse_options_and_load_target() File "translate.py", line 153, in parse_options_and_load_target targetspec_dic = load_target(targetspec) File "translate.py", line 105, in load_target mod = __import__(specname) File "targetpypystandalone.py", line 5, in <module> from pypy.objspace.std.objspace import StdObjSpace File "/place/home/hitrobot/pypy-trunk/pypy/objspace/std/__init__.py", line 1, in <module> from pypy.objspace.std.objspace import StdObjSpace File "/place/home/hitrobot/pypy-trunk/pypy/objspace/std/objspace.py", line 3, in <module> from pypy.interpreter import pyframe, function, special File "/place/home/hitrobot/pypy-trunk/pypy/interpreter/pyframe.py", line 13, in <module> from pypy.rlib import jit, rstack File "/place/home/hitrobot/pypy-trunk/pypy/rlib/rstack.py", line 10, in <module> from pypy.rpython.lltypesystem import rffi, lltype File "/place/home/hitrobot/pypy-trunk/pypy/rpython/lltypesystem/rffi.py", line 824, in <module> sys.maxint, sizeof(lltype.Signed))) AssertionError: Mixed configuration of the word size of the machine: the underlying Python was compiled with maxint=2147483647, but the C compiler says that 'long' is 8 bytes I got this error, then I added "-m32" flag to gcc argument list (in the posix.py source file) , but error occured again :( Does anybody tryed to build pypy-c on the 64bit FreeBsd? Thank you for your answer! -- Shadrin Eugene

5 7

pre-emptive micro-threads utilizing shared memory message passing?
by Kevin Ar18 12 Aug '10

12 Aug '10

Might as well warn you: This is going to be a rather long post. I'm not sure if this is appropriate to post here or if would fit right in with the mailing list. Sorry, if it is the wrong place to post about this. I've looked through the documenation (http://codespeak.net/pypy/dist/pypy/doc/stackless.html) and didn't really see what I was looking for. I've also investigated several options in the default CPython. What I'm trying to accomplish: I am trying to write a particular threading scenario that follows these rules. It is partly an experiment and partly for actual production code. 1. Hundreds or thousands of micro-threads that are essentially small self-contained programs (not really, but you can think of them that way). 2. No shared state - data is passed around from one micro-thread to another; only one micro-thread has access to the data at a time. (although the programmer gets the impression there is no shared state, in reality, the underlying implementation uses shared memory / shared state for speed; the data does not move; you just pass around a reference/pointer to some shared memory) 3. The micro-threads can run in parallel on different cpu cores, get moved to a different core, etc.... 4. The micro-threads are truly pre-emptive (uses hardware interrupt pre-emption). 5. It is my intention to write my own scheduler that will suspend the micro-threads, start them, control the sharing of data, assign them to different CPU cores etc.... In fact, for my purposes, I MUST write my own scheduler as I have very specific requirements on when they should and should not run. Now, I have spent some time trying to find a way to achieve this ... and I can implement a rather poor version using default Python. However, I don't see any way to implement my ideal version. Maybe someone here might have some pointers for me. Shared Memory between parallel processes ---------------------------------------- Quick Question: Do queues from the multiprocessing module use shared memory? If the answer is YES, you can just skip this section, because that would solve this particular problem. (For simplicity, let's assume a quad core CPU) It is my intent to create 4 threads/processs (one per core) and use the scheduler to assign a micro-thread (of which there may be hundreds) to one of the 4 threads/processes. However, the micro-threads need to exchange data quickly; to do that I need shared memory -- and that is where I'm having some trouble. Normally, 4 threads would be the ideal solution -- as they can run in parallel and use shared memory. However, because of the Python GIL, I can't use threads in this way; thus, I have to use 4 processes, which are not setup to share memory. Question: How can I share Python Objects between processes USING SHARED MEMORY? I do not want to have to copy or "pass" data back and forth between processes or have to use a proxy "server" process. These are both too much of a performance hit for my needs; shared memory is what I need. The multiprocessing module offers me 4 options: "queues", "pipes", "shared memory map", and a "server process". "Shared memory map" won't work as it only handles C values and arrays (not Python objects or variables). "Server Process" sounds like a bad idea. Am I correct in that this option requires extra processing power and does not even use shared memory? If so, that would be a very bad choice for me. The big question then... do "queues" and "pipes" used shared memory or do they pass data back and forth between processes? (if they used shared memory, then that would be perfect) Does PyPy have any other options for me? True Pre-emptive scheduling? ---------------------------- Any way to get pre-emptive micro-threads? Stackless (the real Stackless, not the one in PyPy) has the ability to suspend them after a certain number of interpreter instructions; however, this is prone to problems because it can run much longer than expected. Ideally, I would like to have true pre-emptive scheduling using hardware interrupts based on timing or CPU cycles (like the OS does for real threads). I am currently not aware of any way to achieve this in CPython, PyPy, Unladen Swallow, Stackless, etc.... Are there detailed docs on why the Python GIL exists? ----------------------------------------------------- I don't mean trivial statements like "because of C extensions" or "because the interpreter can't handle it". It may be possible that my particular usage would not require the GIL. However, I won't know this until I can understand what threading problems the Python interpreter has that the GIL was meant to protect against. Is there detailed documentation about this anywhere that covers all the threading issues that the GIL was meant to solve? Thanks, Kevin _________________________________________________________________ Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox. http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:…

10 39

Re: [pypy-dev] FW: Would the following shared memory model be possible?
by Michael Sparks 31 Jul '10

31 Jul '10

[ cc'ing the list in case anyone else took my words the same way as Kevin :-( ] On Sat, Jul 31, 2010 at 5:26 PM, Kevin Ar18 <kevinar18(a)hotmail.com> wrote: > I have no idea what I did you warrant you hateful replies towards me, but > they really are not appropriate (in public or private email). I had absolutely no intention of offending you, and am deeply sorry for any offense that I may have caused you. In my reply I merely wanted to flag that I don't have time to go into everything (like most people), that asking questions in a public realm is better because you may then get answers from multiple people, and that people who appear to do some research first tend to get better answers. I also tried to give an example, but that doesn't appear to have been helpful. (I'm fallible like everyone else) My intention there was to be helpful and to explain why I have that view of only replying on list, and it appears to have offended you instead, and I apologise. (one person's direct and helpful speech in one place can be a mortal insult somewhere else) After those couple of paragraphs, I tried to add to your discussion by replying to your specific points which you asked about parallel execution, noting places and examples where it is possible today. (to varying degrees of satisfaction) I then also tried to answer your point of "if something extra could be done, what would probably be generally useful". To that I noted that *my* talk there was cheap, and that execution was hard. Somehow along the way, my intent to try to be helpful to you has resulted in offending and upsetting you, and for that I am truly sorry - life is simply too short for people to upset each other, and in no way was my post intended as "hateful", and once again, my apologies. In future please assume good intentions - I assumed good intentions on your part. I'll bow out at this point. Best Regards, Michael. > >> Date: Sat, 31 Jul 2010 02:08:49 +0100 >> Subject: Re: [pypy-dev] FW: Would the following shared memory model be >> possible? >> From: sparks.m(a)gmail.com >> To: kevinar18(a)hotmail.com >> CC: pypy-dev(a)codespeak.net >> >> On Thu, Jul 29, 2010 at 6:44 PM, Kevin Ar18 <kevinar18(a)hotmail.com> wrote: >> > You brought up a lot of topics. I went ahead and sent you a private >> > email. >> > There's always lots of interesting things I can add to my list of things >> > to >> > learn about. :) >> >> Yes, there are lots of interesting things. I have a limited amount of >> time however (I should be in bed, it's very late here, but I do /try/ >> to reply to on-list mails), so cannot spood feed you. Mailing me >> directly rather than a (relevant) list precludes you getting answers >> from someone other than me. Not being on lists also precludes you >> getting answers to questions by chance. Changing emails and names in >> email headers also makes keeping track of people hard... >> >> (For example you asked off list last year about Kamaelia's license >> from a different email address. Since it wasn't searchable I >> completely forgot. You also asked all sorts of questions but didn't >> want the answers public, so I didn't reply. If instead you'd >> subscribed to the list, and asked there, you'd've found out that >> Kamaelia's license changed - to the Apache Software License v2 ...) >> >> If I mention something you find interesting, please Google first and >> then ask publicly somewhere relevant. (the answer and question are >> then googleable, and you're doing the community a service IMO if you >> ask q's that way - if you're question is somewhere relevant and shows >> you've already googled prior work as far as you can... People are >> time however (I should be in bed, it's very late here, but I do /try/ >> to reply to on-list mails), so cannot spood feed you. Mailing me >> directly rather than a (relevant) list precludes you getting answers >> from someone other than me. Not being on lists also precludes you >> getting answers to questions by chance. Changing emails and names in >> email headers also makes keeping track of people hard... >> >> (For example you asked off list last year about Kamaelia's license >> from a different email address. Since it wasn't searchable I >> completely forgot. You also asked all sorts of questions but didn't >> want the answers public, so I didn't reply. If instead you'd >> subscribed to the list, and asked there, you'd've found out that >> Kamaelia's license changed - to the Apache Software License v2 ...) >> >> always willing to help people who show willing to help themselves in >> my experience.) >> >> >> just looks to me that you're tieing yourself up in knots over things >> >> that aren't problems, when there are some things which could be useful >> >> (in practice) & interesting in this space. >> > The particular issue in this situation is that there is no way to make >> > Kamaelia, FBP, or other concurrency concepts run in parallel (unless you >> > are >> > willing to accept lots of overhead like with the multiprocessing >> > queues). >> > >> > Since you have worked with Kamaelia code a lot... you understand a lot >> > more >> > about implementation details. Do you think the previous shared memory >> > concept or something like it would let you make Kamaelia parallel? >> > If not, can you think of any method that would let you make Kamaelia >> > parallel? >> >> Kamaelia already CAN run components in parallel in different processes >> (has been able to do so for quite some time) or on different >> processors. Indeed, all you do is use a ProcessPipeline or >> ProcessGraphline rather than Pipeline or Graphline, and the components >> in the top level are spread across processes. I still view the code as >> experimental, but it does work, and when needed is very useful. >> >> Kamaelia running on Iron Python can run on seperate processors sharing >> data efficiently (due to lack of GIL there) happily too. Threaded >> components there do that naturally - I don't use IronPython, but it >> does run on Iron Python. On windows this is easiest, though Mono works >> just as well. >> >> I believe Jython also is GIL free, and Kamaelia's Axon runs there >> cleanly too. As a result because Kamaelia is pure python, it runs >> truly in parallel there too (based on hearing from people using >> kamaelia on jython). Cpython is the exception (and a rather big one at >> that). (Pypy has a choice IIUC) >> >> Personally, I think if PyPy worked with generators better (which is >> why I keep an eye on PyPy) and cpyext was improved, it'd provide a >> really compelling platform for me. (I was rather gutted at Europython >> to hear that PyPy's generator support was still ... problematic) >> >> Regarding the *efficiency* and *enforcement* of the approach taken, I >> feel you're chasing the wrong tree, but let's go there. >> >> What approach does baseline (non-Iron Python running) kamaelia take >> for multi-process work? >> >> For historical reasons, it builds on top of pprocess rather than >> multiprocessing module based. This means for interprocess >> communications objects are pickled before being sent over operating >> system pipes. >> >> This provides an obvious communications overhead - and this isn't >> really kamaelia specific at this point. >> >> However, shifting data from one CPU to another is expensive, and only >> worth doing in some circumstances. (Consider a machine with several >> physical CPUs - each has a local CPU cache, and the data needs to be >> transferred from one to another, which is why partly people worry >> about thread/CPU affinity etc) >> >> Basically, if you can manage it, you don't want to shift data between >> CPUs, you want to partition the processing. >> >> ie you may want to start caring about the size of messages and number >> of messages going between processes. Sending small and few between >> processes is going to be preferable to sending large and many for >> throughput purposes. >> >> In the case of small and few, the approach of pickling and sending >> across OS pipes isn't such a bad idea. It works. >> >> If you do want to share data between CPUs, and it sounds like you do, >> then most OSs already provide a means of doing that - threads. The >> conventions people use for using threads are where they become >> unpicked, but as a mechanism, threads do generally work, and work >> well. >> >> As well as channels/boxes, you can use an STM approach, such as than >> in Axon.STM ... >> * http://www.kamaelia.org/STM.html >> * >> http://code.google.com/p/kamaelia/source/browse/trunk/Code/Python/Bindings/… >> >> ...which is logically very similar to version control for variables. A >> downside of STM (at least with this approach) however, is that for it >> to work, you need either copy on write semantics for objects, or full >> copying of objects or similar. Personally I use a biological metaphor >> here, in that channels/boxes and components, and similar perform a >> similar function to axons and neurons in the body, and that STM is >> akin to the hormonal system for maintaining and controlling system >> state. (I modelled biological tree growth many moons ago) >> >> Anyhow, coming back to threads, that brings us back to python, and >> implementations with a GIL, and those without. >> >> For implementations with a GIL, you then have a choice: do I choose to >> try and implement a memory model that _enforces_ data locality? that >> is if a piece of data is in use inside a single "process" or "thread" >> (from hereon I'll use "task" as a generic phrase) that trying to use >> it inside another causes a problem for the task attempting to breach >> the model. >> >> In order to enforce this, I personally believe you'd need to use >> multiple processes, and only share data through dedicated code >> managing shared memory. You could of course do this outside user code. >> To do this you'd need an abstraction that made sense, and something >> like stackless' channels or kamaelia's (in/out) box model makes sense >> there. (The CELL API uses a mailbox metaphor as well for reference) >> >> In that case, you have a choice. You either copy the data into shared >> memory, or you share the data in situ. The former gives you back >> precisely the same overhead previously described, or the latter >> fragments your memory (since you can no longer access it). You could >> also have compaction. >> >> However, personally, I think any possible benefits here are outweighed >> by the costs and complexity. >> >> The alternative is to _encourage_ data locality. That is encourage the >> usage and sharing of data such that whilst you could share data >> between tasks and cause corruption that the common way of using the >> system discourages such actions. In essence that's what I try to do in >> Kamaelia, and it seems to work. Specifically, the model says: >> >> * If I take a piece of data from an inbox, I own it and can do anything >> with it that I like. If you think of a physical piece of paper and >> I take it from an intray, then that really is the case. >> >> * If I put a piece of data in an outbox, I no longer own it and should >> not attempt to do anything more with it. Again, using a physical >> metaphor, and naming scheme helps here. In particular, if I put a >> piece of paper in the post, I can no longer modify it. How it gets >> to its recipient is not my concern either. >> >> In practice this does actually work. If you add in immutable tuples, >> and immutable strings then it becomes a lot clearer how this can work. >> >> Is there a risk here of accidental modification? Yes. However, the >> size and general simplicity of components tends to lead to such >> problems being picked up early. It also enables component level >> acceptance tests. (We tend to build small examples of usage, which in >> turn effectively form acceptance tests) >> >> [ An alternative is to make the "send" primitive make a copy on send. >> That would be quite an overhead, and also limit the types of data you >> can send. ] >> >> In practical terms, it works. (Stackless proves this as well IMO, >> since despite some differences, there's also lots of similarities) >> >> The other question that arises, is "isn't the GIL a problem with >> threads?". Well, the answer to that really depends on what you're >> doing. David Beazely's talk on what happens on mixing different sorts >> of threads shows that it isn't ideal, and if you're hitting that >> behaviour, then actually switching to real processes makes sense. >> However if you're doing CPU intensive work inside a C extension which >> releases the GIL (eg numpy), then it's less of an issue in practice. >> Custom extensions can do the same. >> >> So, for example, picking something which I know colleagues [1] at work >> do, you can use a DVS broadcast capture card to capture video frames, >> pass those between threads which are doing processing on them, and >> inside those threads use c extensions to process the data efficiently >> (since image processing does take time...), and those release the GIL >> boosting throughput. >> >> [1] On this project : >> http://www.bbc.co.uk/rd/projects/2009/10/i3dlive.shtml >> >> So, that makes it all sound great - ie things can, after various >> fashions, run in parallel on various versions of python, to practical >> benefit. But obviously it could be improved. >> >> Personally, I think the project most likely to make a difference here >> is actually pypy. Now, talk is very cheap, and easy, and I'm not >> likely to implement this, so I'll aim to be brief. Execution is hard. >> >> In particular, what I think is most likely to be beneficial is >> something _like_ this: >> >> Assume pypy runs without a GIL. Then allow the creation of a green >> process. A green process is implemented using threads, but with data >> created on the heap such that it defaults to being marked private to >> the thread (ie ala thread local storage, but perhaps implemented >> slightly differently - via references from the thread local storage >> into the heap) rather than shared. Sharing between green processes >> (for channels or boxes) would "simply" be detagged as being owned by >> one thread, and passed to another. >> >> In particular this would mean that you need a mechanism for doing >> this. Simply attempting to call another green process (or thread) from >> another with mutable data types would be sufficient to raise the >> equivalent of a segmentation fault. >> >> Secondly, improve cpyext to the extent that each cpython extension >> gets it's own version of the GIL. (ie each extension runs with its own >> logical runtime, and thinks that it has its own GIL which it can lock >> and release. In practice it's faked by the PyPy runtime. This is >> essentially similar conceptually to creating green processes. >> >> It's worth considering that the Linux kernel went through similar >> changes, in that in the 2.0 days there was a large single big lock, >> which was replaced by ever granular locks. I personally think that >> since there are so many extensions that rely on the existence of the >> GIL simply waving a wand to get rid of it isn't likely. However >> logically providing a GIL per C-Extension may be plausible, and _may_ >> be sufficient. >> >> However, I don't know - it might well not - I've not looked at the >> code, and talk is cheap - execution is hard. >> >> Hopefully the above (cheap :) comments are in some small way useful. >> >> Regards, >> >> >> Michael. >

2 1

Re: [pypy-dev] Would the following shared memory model be possible?
by Andrew Francis 31 Jul '10

31 Jul '10

Hi JP: Message: 1 Date: Thu, 29 Jul 2010 21:24:58 -0000 From: exarkun(a)twistedmatrix.com Subject: Re: [pypy-dev] Would the following shared memory model be possible? To: pypy-dev(a)codespeak.net Message-ID: <20100729212458.2188.24074246.divmod.xquotient.34(a)localhost.localdomain> Content-Type: text/plain; charset="utf-8"; format="flowed" On 08:39 pm, andrewfr_ice(a)yahoo.com wrote: > >Okay. I attended Ray's Hettinger's talk on Monocle. In the past >I have encountered situations where I bumped up with the nesting >problem. >If I recall, the problem involved request handlers that had a RPC style >AND made additional Twisted deferred calls: > >class MyRequestHandler(...): > > @defer.inlineCallbacks > def process(self): > try: > result = yield >client.getPage("http://www.google.com") > except Exception, err: > log.err(err, "process getPage call >failed") > else: > # do some processing with the result > return result > >looks reasonable but Python will balk. JP>Aside from the "return result" (should be defer.returnValue(result), JP>generators can't return with a value), this looks fine to me too. Why JP>do you say Python will balk? Well the return with a value was the deal breaker. I used this example because this is where I came face-to-face with nested generators - and generated a mistrust for them in regard to exotic uses. There was something else about the real example (I am having a hard time finding the posts - somewhere in 2007) - I think it was a very early version of PyAMF and it really wanted a return (HTTP is okay). I believe under the hood, if the protocol returns a deferred or None, the reactor will expect further output in the future. Cheers, Andrew Cheers, Andrew

1 0

S3 2010 deadline extension
by Carl Friedrich Bolz 31 Jul '10

31 Jul '10

The S3 2010 Paper deadline was moved forward by two weeks, and is now August 13, 2010. *** Workshop on Self-sustaining Systems (S3) 2010 *** September 27-28, 2010 The University of Tokyo, Japan http://www.hpi.uni-potsdam.de/swa/s3/s3-10/ In cooperation with ACM SIGPLAN === Call for papers === The Workshop on Self-sustaining Systems (S3) is a forum for discussion of topics relating to computer systems and languages that are able to bootstrap, implement, modify, and maintain themselves. One property of these systems is that their implementation is based on small but powerful abstractions; examples include (amongst others) Squeak/Smalltalk, COLA, Klein/Self, PyPy/Python, Rubinius/Ruby, and Lisp. Such systems are the engines of their own replacement, giving researchers and developers great power to experiment with, and explore future directions from within, their own small language kernels. S3 will be take place September 27-28, 2010 at The University of Tokyo, Japan. It is an exciting opportunity for researchers and practitioners interested in self-sustaining systems to meet and share their knowledge, experience, and ideas for future research and development. --- Submissions and proceedings --- S3 invites submissions of high-quality papers reporting original research, or describing innovative contributions to, or experience with, self-sustaining systems, their implementation, and their application. Papers that depart significantly from established ideas and practices are particularly welcome. Submissions must not have been published previously and must not be under review for any another refereed event or publication. The program committee will evaluate each contributed paper based on its relevance, significance, clarity, and originality. Revised papers will be published as post-proceedings in the ACM Digital Library. Papers should be submitted electronically via EasyChair at http://www.easychair.org/conferences/?conf=s32010 in PDF format. Submissions must be written in English (the official language of the workshop) and must not exceed 10 pages. They should use the ACM SIGPLAN 10 point format, templates for which are available at http://www.acm.org/sigs/sigplan/authorInformation.htm. --- Venue --- The University of Tokyo, Komaba Campus, Japan --- Important dates --- Submission of papers: *EXTENDED* August 13, 2010 Author notification: August 27, 2010 Early registration: September 3, 2010 Revised papers: September 10, 2010 S3 workshop: September 27-28, 2010 Final papers for ACM-DL post-proceedings: October 15, 2010 --- Invited talks --- Yukihiro Matsumoto: "From Lisp to Ruby to Rubinius" Takashi Ikegami: "Sustainable Autonomy and Designing Mind Time" --- Chairs --- Robert Hirschfeld (Hasso-Plattner-Institut Potsdam, Germany) hirschfeld(a)hpi.uni-potsdam.de Hidehiko Masuhara (The University of Tokyo, Japan) masuhara(a)graco.c.u-tokyo.ac.jp Kim Rose (Viewpoints Research Institute, USA) kim.rose(a)vpri.org --- Program committee --- Carl Friedrich Bolz, University of Duesseldorf, Germany Johan Brichau, Universite Catholique de Louvain, Belgium Shigeru Chiba, Tokyo Institute of Technology, Japan Brian Demsky, University of California, Irvine, USA Marcus Denker, INRIA Lille, France Richard P. Gabriel, IBM Research, USA Michael Haupt, Hasso-Plattner-Institut, Germany Robert Hirschfeld, Hasso-Plattner-Institut, Germany (co-chair) Atsushi Igarashi, University of Kyoto, Japan David Lorenz, The Open University, Israel Hidehiko Masuhara, University of Tokyo, Japan (co-chair) Eliot Miranda, Teleplace, USA Ian Piumarta, Viewpoints Research Institute, USA Martin Rinard, MIT, USA Antero Taivalsaari, Nokia, Finland David Ungar, IBM, USA _______________________________________________ fonc mailing list fonc(a)vpri.org http://vpri.org/mailman/listinfo/fonc

1 0

FW: Would the following shared memory model be possible?
by Kevin Ar18 30 Jul '10

30 Jul '10

> Would comments from a project using this approach in real systems be > of interest/use/help? Whilst I didn't know about Morrison's FBP > (Balzer's work predates him btw - don't listen to hype) I had heard of > (and played with) Occam among other more influential things, and > Kamaelia is a real tool. Also there is already a pre-existing FBP tool > for Stackless, and then historically there's also MASCOT & friends. It You brought up a lot of topics. I went ahead and sent you a private email. There's always lots of interesting things I can add to my list of things to learn about. :) > just looks to me that you're tieing yourself up in knots over things > that aren't problems, when there are some things which could be useful > (in practice) & interesting in this space. The particular issue in this situation is that there is no way to make Kamaelia, FBP, or other concurrency concepts run in parallel (unless you are willing to accept lots of overhead like with the multiprocessing queues). Since you have worked with Kamaelia code a lot... you understand a lot more about implementation details. Do you think the previous shared memory concept or something like it would let you make Kamaelia parallel? If not, can you think of any method that would let you make Kamaelia parallel?

2 1

Re: [pypy-dev] Would the following shared memory model be possible?
by Andrew Francis 29 Jul '10

29 Jul '10

Hi Michael: --- On Thu, 7/29/10, Michael Sparks <ms(a)cerenity.org> wrote: > It's a pity we didn't get a chance to chat at the > conference. (I was the one videoing everything for upload after >transcoding :) Yes I noticed. I gave the talk "Prototyping Go's Select with stackless.py for Stackless Python." Much of that talk dealt with rendezvous semantics courtesy via synchronous channels. I will post the Original slides and the Revised version (mistakes corrected ) in a day or two. > > >http://www.kamaelia.org/MiniAxon.html > > > I'm biassed towards Kamaelia (naturally :-), but I agree. > MiniAxon is just a toy/tutorial. Early in kamaelia's history we >considered using stackless, but rejected it simply because we wanted to >work mainly with mainline python, rather than a specialised version. Fair enough. Currently Stackless Python is being integrated with Psyco and will be available as a module. > Other things in Stackless's favour (IIRC) - include the > fact that you can pickle generators, and send them across network >connection, unpickle them and let them continue running. I don't know >if you do the same with tasklets, but I wouldn't be surprised if you do :) As long as you do not have a C Frame involved, you can pickle a tasklet. That was the subject of my "Silly Stackless Python Trick" lighting talk. I was going to demonstrate a version of the Sieve of Eratosthenes that could be pickled and resumed on another machine. However my HP Netbook had a non-standard VGA output connection and I needed to install Stackless on a loaner ThinkPad that died as I hooked it up. However you saw all that :-( > That means you have potential for process migration. Yep. Gabriel Lavoie does a lot of work in that area with PyPy (thanks Gabriel !) > Doing that sensibly though IMO would require better understanding > in the system of what the user is trying to achieve and what they're >sending. (It's easy to think of examples where this causes more pain than >it's worth after all) You have to understand what can be pickled. Occasionally you are in for a surprise (i.e. functools). >You could argue in that case that the biggest _real_ difference is >that we try to use a unified API for different concurrency >methods. Well I would argue that Stackless has a simple elegant model. The addition of select just adds more power.Stackless channels can also serve as generators (they are iterable). I recently took a stab at writing the Sleeping Barber's problem. I think in Stackless, the basic solution was about 30 lines. Very little clutter. > One **highly subjective** other thing in our favour, is > that generators are limited to a single level of control flow >(ie non-nestable without a trampoline). This doesn't sounds like > an advantage, but it tends to lead to simpler components which are >in turn reusable. (and that I view as useful :) Okay. I attended Ray's Hettinger's talk on Monocle. In the past I have encountered situations where I bumped up with the nesting problem. If I recall, the problem involved request handlers that had a RPC style AND made additional Twisted deferred calls: class MyRequestHandler(...): @defer.inlineCallbacks def process(self): try: result = yield client.getPage("http://www.google.com") except Exception, err: log.err(err, "process getPage call failed") else: # do some processing with the result return result looks reasonable but Python will balk. Nested generators. Only way around it is that you had to hope that the Twisted protocol was properly written and chain deferreds. > Have fun, I do :-) Cheers, Andrew

2 1

Re: [pypy-dev] pypy-dev Digest, Vol 360, Issue 13
by Andrew Francis 29 Jul '10

29 Jul '10

Hi Kevin: Message: 1 Date: Tue, 27 Jul 2010 14:20:10 -0400 From: Kevin Ar18 <kevinar18(a)hotmail.com> Subject: Re: [pypy-dev] pre-emptive micro-threads utilizing shared memory message passing? To: <pypy-dev(a)codespeak.net> Message-ID: <SNT110-W23BAAD6B986917DCB074DFAAA70(a)phx.gbl> Content-Type: text/plain; charset="iso-8859-1" >I am attempting to experiment with FBP - Flow Based Programming >(http://www.jpaulmorrison.com/fbp/ and book: http://www.jpaulmorrison.com>/fbp/book.pdf)? There is something very similar in Python: >http://www.kamaelia.org/MiniAxon.html? Also, there are some similarities >to Erlang - the share nothing memory model... and on some very broad >levels, there are similarities that can be found in functional languages. I just came back from EuroPython. A lot of discussion on concurrency.... Well functional languages (like Erlang), variables tend to be immutable. This is a bonus in a concurrent system - makes it easier to reason about the system - and helps to avoid various race conditions. As for the shared memory. I think there is a difference between whether things are shared at the application programmer level, or under the hood controlled by the system. Programmers tend to beare bad at the former. >http://www.kamaelia.org/MiniAxon.html I took a quick look. Maybe I am biased but Stackless Python gives you most of that. Also tasklets and channels can do everything a generator can and more (a generator is more specialised than a coroutine). Also it is easy to mimic asynchrony with a CSP style messaging system where microthreads and channels are cheap. A line from the book "Actors: A Model of Concurrent Computation in Distributed Systems" by Gul A. Agha comes to mind: "synchrony is mere buffered asynchrony." >The process of connecting the boxes together was actually designed to be >programmed visually, as you can see from the examples in the book (I have >no idea if it works well, as I am merely starting to experiment with it). What bought me to Stackless Python and PyPy was work concerning WS-BPEL. Allegedly, WS-BPEL/XLang/WSFL (Web-Services Flow Language) are based on formalisms like pi calculus. Since I don't own a multi-core machine and I am not doing CPU intense stuff, I never really cared. However I have been doing things where I needed to impose logical orderings upon processes (i.e., process C can only run after process A and B are finished). My initial native uses of Stackless (easy to do in anything system based on CSP), resulted in deadlocking the system. So I found understanding deadlock to be very important. >Each box, being a self contained "program," the only data it has access >to is 3 parts: >Implementation of FBP requires a custom scheduler for several reasons: >(1) A box can only run if it has actual data on the "in port(s)"? Thus, >the scheduler would only schedule boxes to run when they can actually >process some data. Stackless Python already works like this. No custom scheduler needed. I would recommend you read Rob Pike's paper "The Implementation of Newsqueak" or some of the Cardilli papers to understand how CSP constructs with channels work. And if you need to customize schedulers - you have two routes 1) Use pre-existing classes and API 2) Experiment with PyPy's stackless.py >(2) In theory, it may be possible to end up with hundreds or thousands of >these light weight boxes.? Using heavy-weight OS threads or processes for every one is out of the question. Stackless Python. >In a perfect world, here's what I might do: * Assume a quad core cpu >(1) Spawn 1 process >(2) Spawn 4 threads & assign each thread to only 1 core -- in other >words, don't let the OS handle moving threads around to different cores >(3) Inside each thread, have a mini scheduler that switches back and >forth between the many micro-threads (or "boxes") -- note that the OS >should not handle any of the switching between micro-threads/boxes as it >does it all wrong (and to heavyweight) for this situation. >(4) Using a shared memory queue, each of the 4 schedulers can get the >next box to run... or add more boxes to the schedule queue. My advice: get stuff properly working under a single threaded model first so you understand the machinery. That said, I think Carlos Eduardo de Paula a few years ago played with adapting Stackless for multi-processing. Second piece of advice: start looking at how Go does things. Stackless Python and Go share a common ancestor. However Go does much more on the multi-core front. Cheers, Andrew

2 1

Re: [pypy-dev] Would the following shared memory model be possible?
by Michael Sparks 29 Jul '10

29 Jul '10

I make it a point these days to only reply on-list. It leads to endless repetition otherwise. If you repost this cc'ing the pypy-dev list I'll reply. If you think it's off topic there, then I see no point. Michael. On Thursday 29 July 2010 18:05:27 you wrote: > Thanks for the reply. > > > Would comments from a project using this approach in real systems be > > of interest/use/help? > > I contacted someone from Kamaelia a while back (probably you). > Yes, use of the dataflow concept would be really useful (no > MIT/BSD/Python/PD license). However, licensing was an issues, so I went > it on my own. I find the concept rather interesting both to maybe learn > from and to actually try and use in an actual application. > > > Whilst I didn't know about Morrison's FBP > > (Balzer's work predates him btw - don't listen to hype) I had heard of > > (and played with) Occam among other more influential things, and > > Kamaelia is a real tool. > > What is this Balzer and Occam? :) Do you have any links I can look at? > > > Also there is already a pre-existing FBP tool > > for Stackless > > The problem is that Stackless is not parallel, which is what I would really > like to do. > > > , and then historically there's also MASCOT & friends. > > Do you have a link about this? -- >>>

1 0

rotting buildbot infrastructure
by Maciej Fijalkowski 28 Jul '10

28 Jul '10

Hello. According to current buildbot status, both osx and win machines are offline. No clue how to get them back. Anyway, our OS X machine is unable to translate pypy, so it's not exactly the best buildbot ever. Can anyone contribute any machine for one of those buildbots? Cheers fijal

3 3

pypy-dev July 2010